(Times in UTC)
09:15 - The RabbitMQ cluster handling live logs started to fail with the "logs" vhost.
We start creating the vhost again.
2021-10-24 08:00 - We notice that parts of the logs system are still not working. We investigate them.
The Logs API keeps crashing for no apparent reason.
11:45 - The Logs API stopped crashing. We don't know why and continue to investigate the reason to fix this for the long term.
Friday 22nd October 2021
APIWebhook and e-mail notifications delivered with a delay
Webhook and e-mail notifications have not been sent since 22:30 UTC on 2021-10-21. The notification service lost its connection to the message queue service and failed to reconnect automatically. This was due to a short network outage between our two Paris datacenters. This issue has been mixed in with others and left unnoticed.
At 11:12 UTC today, the queue has been emptied so all webhooks matching the events during this period have not and will not be sent out. Events from 11:12 to 12:25 UTC have all been sent at once and everything is back to normal since then.