Saturday 23rd October 2021

Logs System Live logs unavailable

(Times in UTC) 09:15 - The RabbitMQ cluster handling live logs started to fail with the "logs" vhost. We start creating the vhost again. 2021-10-24 08:00 - We notice that parts of the logs system are still not working. We investigate them. The Logs API keeps crashing for no apparent reason.

11:45 - The Logs API stopped crashing. We don't know why and continue to investigate the reason to fix this for the long term.