Tuesday 20th April 2021

Cellar Cellar slowed down and partially unavailable

We are investigating an issue with the Paris Cellar cluster.

17:33 UTC: The issue has been resolved. It was due to a partial upgrade (in progress) of the cluster. Upgraded nodes have been downgraded.

18:08 UTC: The upgrade was in-progress to fix the security issue labelled as CVE-2021-20288. Due to the large number of machines, some of them were not yet up-to-date, which have led to the issue we were facing. Some of the machines were unable to authenticate correctly, leading to a cascading failure of multiple machines that weren't yet patched. Another strategy will be used to continue the upgrade of the cluster.