Past Incidents

We are currently having difficulties with the logs pipeline. This impacts live logs in the Console / CLI as well as drain logs. We are working on it.

EDIT 14:25 UTC: The issue has been fixed. A fix has been scheduled for deployment this afternoon which should reduce those delivery issues events. We will monitor the fix closely once it gets deployed.

We are currently having difficulties with the logs pipeline. This impacts live logs in the Console / CLI as well as drain logs. We are working on it.

EDIT 15:07 UTC: Live logs and drains are back. Some drains logs may have been lost during the recovery process. Sorry for the inconvenience.

EDIT 15:52 UTC: Live logs and drains are down again, we are looking into it

We identified issues on our metrics/accesslogs storage. We are working to fix the problem which is currently causing timeouts on queries.

EDIT 15:52 UTC: Queries have returned to normal, Metrics and Access logs should now be reachable.

Metrics and access logs are currently having an ingestion issue. Some metrics points will be lost, access logs will be kept and ingested at a later time.

Ingestion is now starting at full capacity again. There will be some delay before having up-to-date access logs but it should be good in a few hours. Sorry for the inconvenience.

The log drains infrastructure went down last night (2022-02-14 around 5 AM) and some drains were lost / are broken.

We are still identifying which ones are broken to restart them. If you see that your drains are broken, please contact the support so we can restart them!

Edit 15:11 — we restarted all drains to be sure. Edit 16:27 — Most of the drains are still broken. We are trying to fix the issue by deleting and re-creating message queues in the logs infrastructure. Edit 16:37 — Deleting and creating back everything seems to have cleaned up the situation. Drains seem to be working again!