Some systems are experiencing issues

Past Incidents

Saturday 7th January 2023

Cellar Clock skew on Cellar C2 (Paris zone)

Cellar C2 is having issue with time sync. It may result with a "ClockTooSkewed" error when you try to list or access files.

We are working on fixing the clocks on the Ceph monitoring servers. (Ceph is the software we use to provide the Cellar service.)

EDIT 12:40 UTC+1: One of the reverse proxies in front of the Cellar system was desynchronized. This proxy is now out of the pool for further investigation and the issue should now be fixed.

Friday 6th January 2023

No incidents reported

Thursday 5th January 2023

Infrastructure [Paris] Network instabilities

We are currently experiencing network instabilities on the Paris zone. Our network provider is aware of the issue and we are currently awaiting for more information. One instability was detected at 9:42 UTC+1. No other since then.

Our metrics and access logs stack is currently unavailable, we are working towards bringing it back up.

Update 9:55 am UTC: Metrics and access log storage is now up. We are catching up the lag

Update 14:33 UTC: The lag of the Metrics and access logs platform is now resolved. Regarding the network instabilities, our network provider identified the issue and is working towards resolving it. It may take a few hours to get back to a nominal situation. We did not see any other instabilities since this morning.

Update 15:59 UTC: Another network issue happened at 15:50 UTC and lasted for ~1 minute, parts of the Paris zone was unreachable during that time.

Update 23:11 UTC: No other incident has been seen, we are still waiting for our network provider to ensure that the issue is resolved on their end.

Update 2023-01-09 14:18 UTC: We've seen two new events, one at 13:23 UTC and another at 14:14 UTC. We notified our network provider. Those may be related to the same problems we've seen last week.

Update 2023-01-09 19:47 UTC: Those two events weren't linked to the ones seen last weeks. The reason has been identified by the network provider and has been fixed. We are still waiting for confirmation of resolve on the original issue.

Wednesday 4th January 2023

API API authorization is failing

Some part of the authentication process in the API is failing. We identified the error and are trying to fix it.

Tuesday 3rd January 2023

No incidents reported

Monday 2nd January 2023

No incidents reported

Sunday 1st January 2023

No incidents reported