Monday 3rd October 2022

Deployments Deployments are DOWN

(Times are UTC) 04:45 - Deployments are broken because of a pulsar issue. We are investigating.

05:45 - To prevent issues on the infrastructure, we disabled all deployments.

05:55 - We detect that some VMs are DOWN. It seems that the pulsar connection issues have overwhelmed the hypervisor's processes.

06:05 - We shut down the processes that fill up the hypervisors. It seems to fix the issue.

06:20 - The deployments seem to be back on tracks. We continue investigating the pulsar issue before putting it back into the deployment processes.

09:09 - We are still experiencing deployments issues. We are investigating.

12:28 - Deployments have been fixed.