Tuesday 12th February 2019

Infrastructure Abnormal amount of 503 errors

We have an abnormal amount of 503 errors served by our reverse proxies. We are investigating.

EDIT 11:31 UTC: Cause has been identified, we are currently fixing the issue on our reverse proxies.

EDIT 11:34 UTC: All reverse proxies now have a consistent state. The issue is fixed.

The issue happened after a configuration error made during a manual operation on some of the reverse proxies. Applications that redeployed since 11:08 UTC were impacted by that issue. Other applications were fine. The changes were rollbacked and will again be tested thoroughly on our test infrastructure.