Past Incidents

An hypervisor on the Montreal zone is unreachable. One of the FSBucket servers of the zone is hosted on it and is therefore unreachable too. This might impact PHP applications as well as any applications using an FSBucket hosted on this server.

We are awaiting information from our infrastructure provider regarding this incident.

EDIT 19:53 UTC: It seems like multiple servers are impacted at the same time, we believe it to be an issue with a specific OVH rack or room. Multiple services on the zone are thus impacted. We are looking at ways to mitigate the issues.

EDIT 20:05 UTC: The servers are reachable again since a few minutes. We are currently making sure everything is fine. OVH incident can be followed here: https://bare-metal-servers.status-ovhcloud.com/incidents/k664s90jxfj0

EDIT 20:15 UTC: Servers in the impacted rack couldn't reach each other up until now. It could have prevented some services to correctly work. It seems like OVH fixed it before we could report it to them. We continue to making sure everything is working as expected.

EDIT 20:36 UTC: The incident is over. We are redeploying all the applications of the zone to be on the safe side.

We are currently having an ingestion issue on our metrics cluster. The root cause has been identified and we are currently working on a fix. Until this incident is fixed, metrics data points might be missing from your metrics dashboards. Access logs are also impacted but will be re-queued later.

EDIT 14:14 UTC: Metrics ingestion is now back to normal. Access logs are being re-queued and are currently lagging a bit.

EDIT 14:20 UTC: Access logs have been ingested and are now up-to-date. The incident is now over.

EDIT 16:25 UTC: The problem came back, we are working on it.

EDIT 16:56 UTC: The problem is now solved again. Another root cause has been identified and has been fixed.

We are encountering slowness on the Cellar infrastructure. We are investigating why.

EDIT 15:05 UTC: The issue has been found and fixed. Performance went back to normal around 13:45 UTC. Additional measures will be taken to avoid this issue in the future.