Past Incidents

Thursday 26th January 2023

FS Buckets [MTL] Increase size of FSBucket storage backend

One server that host FSBucket need additionnal disk space.

EDIT 10:10 UTC Operation to increase the disk space is done. We are redeploying the associated applications

Wednesday 25th January 2023

Logs System Live logs storage backend issue

Live logs system has an issue with the storage backend that put it to read only mode

EDIT 22:56 UTC : The storage backend has left the read-only mode

Infrastructure [PAR] Two hypervisors have rebooted

Two hypervisors have rebooted in the Paris zone. Deployments have been impacted and some applications and databases may be unreachable. We are investigating the issues.

** EDIT 13:59 UTC ** One hypervisor is up and running

** EDIT 14:52 UTC ** The second hypervisor is down due to hardware issues

** EDIT 15:22 UTC ** Applications and databases may be difficult to reach as a load balancer node is hosted on the down hypervisor

** EDIT 17:00 UTC ** Deployments may have been impacted, we are redeploying the system

** EDIT 17:30 UTC ** Hypervisor is up and running. We are cleaning up the last thing

** EDIT 18:17 UTC ** Hypervisors are up and running. All systems seems working normaly

Tuesday 24th January 2023

No incidents reported

Monday 23rd January 2023

No incidents reported

Sunday 22nd January 2023

No incidents reported

Saturday 21st January 2023

No incidents reported

Friday 20th January 2023

Cellar Cellar read-only

At 02:20 UTC, we started having alerts saying the Ceph pools are full. We are investigating this.

04:40 UTC, we take the decision to lower the replication ratio to let the cluster breathe.

A lot of backups failed, though. We will start them again during the day.