Some systems are experiencing issues

Past Incidents

Sunday 11th October 2020

No incidents reported

Saturday 10th October 2020

No incidents reported

Friday 9th October 2020

No incidents reported

Thursday 8th October 2020

FS Buckets Some FS Buckets are experiencing issues

Some FS Buckets addons are experiencing issues. We identified the issue and are working on its resolution.

EDIT 21:25 UTC: The issue is fixed. The PHP applications may not work correctly. We are redeploying them.

EDIT 22:30 UTC : Applications with FS Buckets have been redeployed. The incident is closed.

Post mortem: An incorrect human action conducted the FS Buckets system to follow the wrong path between different storage nodes. We applied fix to avoid this cause.

Cellar Cellar addons are experiencing issues

We identified issues on Cellar addons availability. We identified them and are working on their resolution.

EDIT 15:25 UTC: fixed. We are investigating the reasons.

EDIT 15:45 UTC: we identified the reasons and applied a fix.

Access logs Metrics / Access logs: requests error

Metrics and access logs requests might experience issues following the maintenance of a core component of those features. Requests can either take a very long time to complete or simply answer an error. We are working toward a fix.

Data won't be lost, the ingestion is simply delayed.

Impacted products:

  • Metrics (in console or using the API)
  • Access logs (charts in the console's overview or using the CLI / API)

EDIT 14:03 UTC: Ingestion is now catching up on the delay, everything looks good. Looks like it may take 30 to 40 minutes to go completely back to normal.

EDIT 14:25 UTC: Ingestion has now caught up, everything should be back to normal.

EDIT 21:26 UTC: New issues are ongoing, we are investigating.

EDIT 22:16 UTC: Ingestion is running. We are consuming queues.

EDIT 23:30 UTC: Ingestion is back to normal. Fixed.

Wednesday 7th October 2020

No incidents reported

Tuesday 6th October 2020

Console Console is currently unavailable

Loading the console might result in various errors preventing users from logging in. We are currently investigating. The CLI shouldn't be impacted. Already loaded console webpages shouldn't be impacted either.

EDIT 13:02 UTC: A change causing this issue has been backed out. We will investigate further why it went wrong despite working correctly on our test infrastructure. Sorry for the disruption.

Redis Redis: Redmin dashboard unavailable

The redsmin dashboard for redis add-ons is currently unavailable. The Redsmin provider has been notified. We will update this post as soon as we have an update.

EDIT 10:54 UTC: Redsmin is currently working on a fix.

EDIT 19:54 UTC: The fix seems to be complete. Redsmin interfaces should now be able to load.

Monday 5th October 2020

No incidents reported