Past Incidents

Some FS-Bucket add-ons will need to be migrated to a different server for security reasons. During this migration, the Buckets will be in Read-Only mode. Any attempt to create or update a file on the add-on will fail, including for FTP operations. Errors related to Read-only file system are expected during this migration.

The migration is expected to last at most 1 hour. All impacted applications will be redeployed during the migration. After the deployment, application will be able to write to the bucket. Read operations will not be impacted.

EDIT: This maintenance has been postponed to 15:00 UTC+1

EDIT 15:00 UTC+1: The maintenance is starting

EDIT 15:02 UTC+1: The buckets are now read-only

EDIT 15:14 UTC+1: Starting now, you can redeploy your applications if you want to regain write access early. Otherwise, affected applications will be redeployed automatically in the upcoming hour, starting with applications of Clever Cloud Premium customers

EDIT 17:14 UTC+1: The deployment queue finished one hour ago, everything has been working fine so far. This maintenance is over

The 2021-03-07 at 19:40 UTC websites on the RBX went down. We started investigating the issue at 19:45 and saw the RBX reverse proxies were not accepting new connections. We restarted them and everything went back to normal by 19:54.

The culprit was a badly configured NOFILE limit on the RBX reverse proxies. We updated the setting accordingly.

Afterwards: We investigated all the reverse proxies on all the zones to make sure the NOFILE limit was correctly configured everywhere. We updated the reverse proxy software (sozu) to refuse to start when given too few NOFILE. We updated the sozu package to enforce the right NOFILE value upon installation.

We experienced an unexpected issue with a core component of the Metrics system.

The service is completely unavailable at the moment. We are working on it.

08:50 UTC: The faulty component is working. We are working on bringing everything back up.

08:59 UTC: Everything is back up. The ingestion pipeline is catching up.

09:07 UTC: The incident is over.

We are investigating issues with our core API.

EDIT 21:07 - fixed.