Some systems are experiencing issues

Past Incidents

Tuesday 15th June 2021

Planned maintenance on Metrics storage backend, scheduled 2 years ago

Planned maintenance of the storage backend of Clever Cloud Metrics (used for access logs as well) will occur on 2021-06-15 at 11:30 UTC.

The maintenance itself should take no more than an hour. During this time, writes will be queued and reads will be partially available.

Once the maintenance is over, queued-up writes will start being ingested, reads will be available again (except for recent data until queued-up data points are ingested).

11:36 UTC: Maintenance is starting.

12:04 UTC: Maintenance is over. The ingestion pipeline is running at full speed catching up on the queued-up data.

12:18 UTC: Ingestion is caught up.

Friday 11th June 2021

Reverse Proxies PAR: Reverse proxies instability

Reverse proxies on the Paris zone are experiencing instabilities. We are investigating.

EDIT 18:04 UTC+2: One of the reverse proxy stopped accepting new connections. It has been put out of the pool for further investigation. Stability should have been resumed since 2 minutes.

EDIT 18:18 UTC+2: Performance is back to normal. We are going to investigate further why this reverse proxy went into this state without being noticed.

MongoDB shared cluster MongoDB shared cluster on Paris zone is overloaded

MongoDB shared cluster on Paris zone is overloaded. We are investigating what is most likely due to excessive ressources usage of some users.

As a reminder, this cluster is only used by free plans labeled "DEV". This is meant to be used for development and testing purposes only, not production.

If you are using a free plan in production, we suggest you migrate to a dedicated plan using the migration tool in the Clever Cloud console.

10:43 UTC: The cluster is working fine now although it may be slower than usual for now as a node is out of the cluster and will be re-added later.

12:23 UTC: The node mentioned in the last update has been re-added. The incident is over.

Thursday 10th June 2021

No incidents reported

Wednesday 9th June 2021

Logs System Logs are deactivated while we are investigating an issue

Logs are deactivated while we are investigating an issue.

EDIT 19:14 UTC: Logs should now be back to normal. Sorry for the interruption.

API API, console and other Clever Cloud applications partially unreachable

Dedicated load balancers for Clever Cloud's own applications (APIs, Console, website, ...) are overloaded.

We are in the process of adding capacity to resolve this issue.

14:28 UTC: Performance is back to normal.

Reverse Proxies Some applications are responding slowly or are unreachable

At 11:30 UTC we started getting tickets about customer's applications not responding. We started investigating. It looks like the network or the reverse proxies are responsible for that.

EDIT 12:46 UTC: we are experiencing abnormal new connection rates on public reverse proxies.

EDIT 12:50 UTC: we found the responsible application for this new connection rate and are mitigating it.

EDIT 14:19 UTC: Load balancers have been upscaled so they can handle more traffic. Performance is back to normal since 13:12 UTC.

Tuesday 8th June 2021

Logs System Logs ingestion malfunction

Logs ingestion is malfunctioning. We are investigating.

08:00 UTC: New logs are being ingested. Logs emitted during the incident will not be ingested in the main logs storage system. Log drains may start receiving (part of) the older logs, we are still investigating this part.

08:15 UTC: Looks like everything that could be ingested has been ingested. Ingestion delay may still be a little higher than normal though, it should go back to normal soon.

Monday 7th June 2021

No incidents reported

Sunday 6th June 2021

No incidents reported

Saturday 5th June 2021

No incidents reported