Past Incidents

We are currently seeing applications having troubles complete their deployments, especially when using dedicated build VM. They may be stuck or very slow at the cache archives upload. We are investigating.

EDIT 10:55 UTC: The root cause has been found. It was only impacting multipart uploads. For deployments already at the upload phase, you will need to cancel the current deployment and start a new one for the problem to be fixed. Sorry for the inconvenience.

An hypervisor went down, we are investigating.

EDIT 22:24 UTC: The hypervisor is up again since 10 minutes. Add-ons are available again. We make sure all applications were redeployed.

EDIT 00:17 UTC: The incident is over.

At 16:29 UTC, a staff member started investigating an alert on one of our hypervisors. They saw the hypervisor could not be logged into anymore.

All services running on that hypervisor are still up and running, but deployments fail to stop the obsolete VMs and we cannot connect to the host itself. We are considering a "semi" kernel crash on the hypervisor's host. We are investigating and may reboot the hypervisor in the following minutes/hours. (First, we try migrating as much important services as possible to avoid causing too much downtime to our customers.)

EDIT 16:46 UTC: We are starting to migrate add-ons on the impacted hypervisor.

EDIT 18:54 UTC: We rebooted the hypervisor, everything went well, all the remaining services are UP again.