Live logs system has an issue with the storage backend that put it to read only mode
EDIT 22:56 UTC : The storage backend has left the read-only mode
Two hypervisors have rebooted in the Paris zone.
Deployments have been impacted and some applications and databases may be unreachable.
We are investigating the issues.
** EDIT 13:59 UTC ** One hypervisor is up and running
** EDIT 14:52 UTC ** The second hypervisor is down due to hardware issues
** EDIT 15:22 UTC ** Applications and databases may be difficult to reach as a load balancer node is hosted on the down hypervisor
** EDIT 17:00 UTC ** Deployments may have been impacted, we are redeploying the system
** EDIT 17:30 UTC ** Hypervisor is up and running. We are cleaning up the last thing
** EDIT 18:17 UTC ** Hypervisors are up and running. All systems seems working normaly
No incidents reported
At 02:20 UTC, we started having alerts saying the Ceph pools are full. We are investigating this.
04:40 UTC, we take the decision to lower the replication ratio to let the cluster breathe.
A lot of backups failed, though. We will start them again during the day.