Wednesday 12th May 2021

API Core services are experiencing issues

Core services (console, API, metrics, access logs) are experiencing issues. We identified the problem and are working to resolve it.

EDIT 23:02 UTC: the incident is related to one of our hypervisors.

EDIT 23:03 UTC: we restarted the hypervisor; related databases are down.

EDIT 23:04 UTC: hypervisor is up; VMs are starting.

EDIT 23:13 UTC: metrics are down too.

EDIT 23:25 UTC: databases are up. We are now experiencing issues with our internal reverses proxies and console and API are not available.

EDIT 23:30 UTC: we queued the linked applications for a high-priority redeploy to ensure they reconnect to their databases. Core services are still partially down.

EDIT 0:00 UTC: all applications are redeployed.

EDIT 02:56 UTC: we are still working to fix issues on our internal core services (console, API); users applications/addons are not impacted.

EDIT 03:30 UTC: internal core services are back!