We are investigating a major issue on public and internal reverse proxies (not private).
EDIT 17:55 UTC - we identified the issue (DDOS).
EDIT 17:56 UTC - we fixed the issue on internal reverse proxies.
EDIT 19:15 UTC - we are still working to fix the issue.
EDIT 20:30 UTC - fixed and situation is back to normal. We will publish a post mortem.
Post mortem 2020-11-15
16:45 UTC: Our monitoring throws an alert: public and internal reverse proxies traffic is abnormally decreasing. Dedicated reverse proxies for Premium clients are not impacted. The on-call team starts investigations;
16:53 UTC: We see a lot of HTTP requests timing out with PR_END_OF_FILE_ERROR randomly on multiple reverse proxies.
17:00 UTC: We diagnose lots of IPs running an abnormal DDoS shape on our Paris infrastructure on identified domain names which prevents reverse proxies from accepting connections and causes reduced traffic;
17:30 UTC: After banning these addresses, new ones are used for the attack and we start banning IP ranges. During this period, we are applying custom reverse proxies configurations to limit the attack impact on various clients;
17:56 UTC: We are applying these bans on the internal reverse proxies, the internal situation comes back to normal; then we ban these on public reverse proxies;
18:00 UTC: Traffic is back to normal; PR_END_OF_FILE_ERROR disappeared and we are now facing SSL_ERROR_SYSCALL. We start investigating;
18:24 UTC: We determine these errors are due to configuration errors applied during the reverse proxies configuration changes.
20:06 UTC: All configurations are fixed, everything is working as usual. We are improving reverse proxies auto-configuration to avoid error-prone manual actions. We are fixing custom clients' configuration items and are watching monitoring data closely.
20:14 UTC: Reverse proxy improved auto-configuration is deployed.
20:30 UTC: We announce the end of the incident. The attack logs will be used to improve our DDoS detection system.