Monday 30th November 2020

Infrastructure Network outage on multiple servers

A network outage is currently affecting multiple servers. We are currently investigating. Multiple services may be in degraded states or unreachable. Customers applications and databases will experience the same issues.

EDIT 17:22 UTC+1: The network have been restored on those servers. We continue investigating which services are currently impacted. Applications that lost network connectivity to our monitoring are restarting. Applications that crashed because they lost their database access are also restarting.

EDIT 17:45 UTC+1: Deployments may still take some time to start or for those ongoing, to finish. We are cleaning-up the situation.

EDIT 18:17 UTC+1: Deployments are back to normal since 18:05. We are still cleaning up the rest of the mess and making sure everything is back to normal and working fine.

EDIT 18:25 UTC+1: Incident is over.

The issue came during a maintenance of our infrastructure provider during which multiple electricity cables were disconnected on active switches. Some of our servers were linked to those switches, cutting their network access for 5 minutes. Backup network links of those servers were also affected leading to a total loss of network. We will investigate this incident further with the infrastructure provider.