Past Incidents

Some FS-Bucket add-ons will need to be migrated to a different server for security reasons. During this migration, the Buckets will be in Read-Only mode. Any attempt to create or update a file on the add-on will fail, including for FTP operations. Errors related to Read-only file system are expected during this migration.

The migration is expected to last at most 1 hour. All impacted applications will be redeployed during the migration. After the deployment, applications will be able to write to the bucket. Read operations will not be impacted.

Users of buckets that need to be migrated have received emails.

EDIT 2022-05-31 10:00 UTC: The migration is starting, buckets will be put into read-only.

EDIT 2022-05-31 10:25 UTC: The migration is over. Applications have started redeploying, it should take around 2 hours. You can redeploy your application earlier to finish the migration.

EDIT 2022-05-31 13:11 UTC: All applications have been redeployed, the migration is now over.

We currently have issues with deployments. Deployments may end up with errors asking you to contact our support alongside a stacktrace. We are currently working on a fix.

EDIT 14:59 UTC - We have identified defaulting component which encounters an issue in the connection pooler.

EDIT 15:09 UTC - deployments queue is being consumed and catching up. Issue it mitigated.

EDIT 15:23 UTC - Incident is fixed.

Root cause: we've found an issue in a messaging driver on a couple of isolated servers. Anyway, we've curated out this specific driver to fall back on an alternative messaging layer. In the coming days, we will dive into this specific bug we've found and will communicate the bug fix upstream.

The MySQL c5 shared cluster is experiencing issues. We are investigating.

EDIT 20:02 UTC: the MySQL shared cluster is back online.

Logs are currently having some ingestion/query issues. We are working on it.

EDIT 21:39 UTC - querying logs is now available.

Metrics and access logs are currently having some ingestion/query issues. We are working on it.

EDIT 23:06 UTC - Storage cluster is now up. We are now catching up the accumulated ingestion lag. Query components will be restarted in a rolling fashion throughout the next 6 hours.

EDIT Sunday 11:27 UTC - Some query components are still reloading

EDIT Sunday 20:27 UTC - We are still experiencing issues on the query components.

EDIT Monday 07:20 UTC - Query is back online

A few hypervisors on the Paris zone had a configuration issue between 12:21 UTC and 14:16 UTC leading to instances not being properly monitored. This caused Monitoring/Unreachable deployments for the instances hosted on them.

Because of this, those hypervisors became more empty than the others. More VMs were scheduled on them since they had more resources available, which then lead to more Monitoring/Unreachable events.

Instances weren't, for the most part, unreachable, but were redeployed anyway.

This should now be fixed. Sorry for the inconvenience

Metrics and access logs are currently having some query issues. We are working on it.

EDIT 07:16 UTC - Indexes have been rebuilt. Query is now available.