Some systems are experiencing issues

Past Incidents

Thursday 26th May 2022

No incidents reported

Wednesday 25th May 2022

Reverse Proxies Some applications are unavailable

Some applications are experiencing issues. We are investigating it.

UPDATE 14:57 UTC: Some Add-ons are being inaccessible due to a faulty proxy. We're removing it from the pool to mitigate.

UPDATE 14:59 UTC: Services are being reloaded to ensure the faulty proxy is removed from the pool.

UPDATE 15:10 UTC: Services are back online for redeployed apps. A faulty sentry induced an abnormal behaviour in the API.

CALL FOR ACTION 15:23 UTC: Remaining applications are currently redeployed. If you're impacted, we advise you to redeploy your app to accelerate the recovery process

Tuesday 24th May 2022

Deployments Issues with deployment not working correctly

We currently have issues with deployments. Deployments may end up with errors asking you to contact our support alongside a stacktrace. We are currently working on a fix.

EDIT 14:59 UTC - We have identified defaulting component which encounters an issue in the connection pooler.

EDIT 15:09 UTC - deployments queue is being consumed and catching up. Issue it mitigated.

EDIT 15:23 UTC - Incident is fixed.

Root cause: we've found an issue in a messaging driver on a couple of isolated servers. Anyway, we've curated out this specific driver to fall back on an alternative messaging layer. In the coming days, we will dive into this specific bug we've found and will communicate the bug fix upstream.

Monday 23rd May 2022

MySQL shared cluster MySQL c5 is experiencing issues

The MySQL c5 shared cluster is experiencing issues. We are investigating.

EDIT 20:02 UTC: the MySQL shared cluster is back online.

Sunday 22nd May 2022

Logs System Logs are experiencing issues

Logs are currently having some ingestion/query issues. We are working on it.

EDIT 21:39 UTC - querying logs is now available.

MySQL shared cluster MySQL c6 is experiencing issues

The MySQL c6 shared cluster of EU zone is experiencing issues. We are investigating.

EDIT 21:39 UTC - shared cluster is now back online

SSH Gateway SSH Gateway is unavailable because of maintenance

The SSH Gateway will undergo a maintenance which will stop the service. Expected downtime is 30 minutes. During this time, SSH access to instances will be unavailable both from the CLI or from the regular SSH tool. Existing SSH connections will be stopped.

Maintenance is expected to start in a few minutes

EDIT 17:56 UTC: Service is back online, you should now be able to SSH to your instances. Sorry for the inconvenience.

Saturday 21st May 2022

Access logs Metrics and access logs are experiencing issues

Metrics and access logs are currently having some ingestion/query issues. We are working on it.

EDIT 23:06 UTC - Storage cluster is now up. We are now catching up the accumulated ingestion lag. Query components will be restarted in a rolling fashion throughout the next 6 hours.

EDIT Sunday 11:27 UTC - Some query components are still reloading

EDIT Sunday 20:27 UTC - We are still experiencing issues on the query components.

EDIT Monday 07:20 UTC - Query is back online

Friday 20th May 2022

Infrastructure [PAR][RETROACTIVE] High number of Monitoring/Unreachable deployments

A few hypervisors on the Paris zone had a configuration issue between 12:21 UTC and 14:16 UTC leading to instances not being properly monitored. This caused Monitoring/Unreachable deployments for the instances hosted on them.

Because of this, those hypervisors became more empty than the others. More VMs were scheduled on them since they had more resources available, which then lead to more Monitoring/Unreachable events.

Instances weren't, for the most part, unreachable, but were redeployed anyway.

This should now be fixed. Sorry for the inconvenience