Friday 28th May 2021

Cellar Cellar-c1: network connection issues

Our cellar-c1 cluster is experiencing connection issues. Some buckets might be unavailable as the network between various nodes of the cluster is currently having issues. We are investigating.

Cellar-c2 cluster isn't impacted.

EDIT 08:23 UTC: Connection seems to be back, we have notified both network providers used for Cellar-c1 and are still awaiting an answer. We are waiting a bit more to see if the links are correctly back or if we should expect another issue.

EDIT 08:47 UTC: The connection is now down again.

EDIT 09:35 UTC: The connection has been back up for 15 minutes and the root cause may have been found. We are waiting for explanations from our network provider. In the meantime, this issue may also have affected applications that are connecting to external services. We've seen loss to Scaleway and Azure, there might have been more.

EDIT 10:25 UTC: The issue now seems to be resolved. The root cause wasn't entirely found, current investigations show that a transit provider had an issue and traffic was redirected elsewhere, maybe leading to some links saturation (which would explain why the loss wasn't 100%, but more like 80%).