Some systems are experiencing issues

Past Incidents

Monday 25th October 2021

No incidents reported

Sunday 24th October 2021

No incidents reported

Saturday 23rd October 2021

Logs System Live logs unavailable

(Times in UTC) 09:15 - The RabbitMQ cluster handling live logs started to fail with the "logs" vhost. We start creating the vhost again. 2021-10-24 08:00 - We notice that parts of the logs system are still not working. We investigate them. The Logs API keeps crashing for no apparent reason.

11:45 - The Logs API stopped crashing. We don't know why and continue to investigate the reason to fix this for the long term.

Friday 22nd October 2021

API Webhook and e-mail notifications delivered with a delay

Webhook and e-mail notifications have not been sent since 22:30 UTC on 2021-10-21. The notification service lost its connection to the message queue service and failed to reconnect automatically. This was due to a short network outage between our two Paris datacenters. This issue has been mixed in with others and left unnoticed.

At 11:12 UTC today, the queue has been emptied so all webhooks matching the events during this period have not and will not be sent out. Events from 11:12 to 12:25 UTC have all been sent at once and everything is back to normal since then.

Thursday 21st October 2021

No incidents reported

Wednesday 20th October 2021

No incidents reported

Tuesday 19th October 2021

Infrastructure Retroactive: [PAR] Add-on reverse proxy unreachable

An add-on reverse proxy was unreachable on the PAR zone. Some applications might have had issues connecting to their add-ons or may have unexpectedly lost their connections to them.

The reverse proxy has been rebooted and this incident is now over.