Past Incidents

An hypervisor needs to be rebooted on our Paris zone. The reboot will happen at 22:00 UTC on Monday, 5th September. Add-ons that support automatic migration will be migrated automatically starting at 21:00 UTC. You can also perform the migration at a time that suits you more before the given deadline.

This will also impact some FSBuckets add-ons during which reads and writes will be unavailable. Applications will be redeployed automatically once the maintenance is over to make sure they correctly re-connect to the FSBucket server.

The maintenance is expected to last 15 minutes.

Impacted users will shortly receive an email with the impacted add-ons.

EDIT 2022-09-05 21:10 UTC: Add-ons migrations is starting

EDIT 2022-09-05 21:40 UTC: Add-ons have been migrated. The hypervisor reboot will happen in twenty minutes.

EDIT 2022-09-05 22:00 UTC: Hypervisor is rebooting

EDIT 2022-09-05 22:28 UTC: Hypervisor has been rebooted in 4 minutes, fsbucket server went back one minute later with most clients reconnecting. We started all affected applications to make sure everyone properly reconnects.

One hypervisor went down in MTL2. We are trying to reboot it.

It affects: 1 load balancer 1 redis add-on 1 mysql add-on The free postgresql databases on MTL.

Update 16:40 after investigating, we decide to redirect the IP of the load balancer to the second LB. A ticket is open at OVHCloud to investigate what seems to be a hardware issue. Update 17:56 OVHCloud team physically checked the server: the RAID card was broken. They changed it and restarted the server. Update 18:05 All VMs on the hypervisor are up and running again.

An hypervisor on the Paris zone is currently unreachable. We are looking into it.

EDIT 17:38 UTC: Hypervisor has been rebooted. Services are being restarted.

EDIT 18:08 UTC: Services have all been restarted. We continue looking into why the hypervisor went down and continue to monitor the situation.

EDIT 18:27 UTC: Initial investigation shows that a KVM kernel bug was encountered, leading to a kernel crash. We will investigate further to see if this can be mitigated by an update. The incident is now over.

We are seeing a network loss towards the New York zone from multiple places since 06:05 UTC. We are looking into the issue. Applications and add-ons may not be reachable from different places and multiple services on the zone (deployments, logs) will not be available.

EDIT 07:04 UTC: We are seeing network improvements to reach the zone. It is currently operational but we are still waiting on confirmation from our provider. From our point of view as of now, traffic towards the zone was dropped when reaching the Level3 network transit. Our network provider seems to have changed it to another provider, allowing us to reach the zone again.

EDIT 12:18 UTC. The network problem is fully resolved. We are still waiting for an incident report from the network operator of the Datacenter. We will share it once available.

EDIT 2022-08-26 14:27 UTC: Here is the report from our provider: It has been identified that the incident is due to a bug found in our device at DRT1. As an initial resolution, our team rebooted the device. Consequently, all alarms cleared and all services were restored after executing the said activity. As of the moment, we can confirm that the link has remained clean and error-free since the service went up.