Tuesday 12th September 2023

Access logs Metrics and access logs storage layer issue

The storage layer has lost some nodes. We are investigating the issue.

EDIT 13:45 UTC : We have found that we have a network issue which cause storage nodes to timeout and then crash. Those nodes are now up and running, we are beginning the recovery process

EDIT 15:10 UTC : We have finished the recovery process and we are consuming the lag.

EDIT 18:52 UTC : We have almost consume all the data lag (estimate duration is 30 mins left), but there is still 2h of metadata lag.

EDIT 21:00 UTC: We have catched up the data and metadata lag, the query is now open