Some systems are experiencing issues

Past Incidents

Tuesday 5th July 2022

Access logs Ingestion queue issue

One of the server queue storage reach its disk max storage capacity

One of the partition is corrupted, fixing

EDIT 17:10 UTC: The underlying issue has been fixed. The queue is currently being processed. Some events might have been lost during the cluster rebalance. Data points will take a few more hours to be up-to-date in the various dashboards.

EDIT: Queue is in sync

Monday 4th July 2022

API Main API is currently experiencing timeouts

We are currently looking into it. Console and CLI are not working correctly.

A batch was sent by an employee. The throttle interval was set two small and the batch made a huge amount of queries to the database, making it unresponsive. We stopped the batch and will restart it with a higher throttle interval.

Removal of TLS 1.0 and 1.1 from our load balancers, scheduled 1 year ago

When you access a website or an online application, you most often do so in a “secure” way. This is for example the well-known green padlock that symbolizes HTTPS connections in your browser, which has become a standard these years thanks to initiatives like Let’s Encrypt.

This means that the data transferred to the server is encrypted, and that even if they are intercepted, they cannot be read by a third party. This protection has been provided by the TLS (Transport Layer Security) protocol for almost 20 years, whether it’s a personal site, an online shop or an access to your bank’s services.

Over time, this critical technical brick on the Internet has evolved to strengthen the level of security it offers. In August 2018, its version 1.3 (the latest) was released. Meanwhile, versions 1.0 and 1.1 were considered to no longer offer a sufficient level of protection. They have been deprecated by the IETF (Internet Engineering Task Force) since March 2021 and have therefore been gradually removed from recent browsers such as Firefox, Chrome and its derivatives or Safari.

At Clever Cloud, we have seen our customers adopt TLS 1.2 and 1.3 gradually. On our load balancers, based on our in-house and open source reverse proxy Sōzu, the latest version accounts for over 90% of the requests processed each day. TLS 1.2 for just under 9%. TLS 1.0 and 1.1 for only a few tens of thousands of requests per day, less than 0.1% of our traffic.

While we have maintained these versions for compatibility reasons, this will no longer be the case as of June 30. We will of course inform the customers affected by this choice, and encourage them to switch to more recent versions, which will have advantages for them in terms of security, performance and SEO.

Several reminders will be sent between now and the final shutdown of TLS 1.0 and 1.1. If you have any questions on this subject, please contact our support team through the Console.

EDIT 2:00 PM UTC: every public load balancers has been updated with new configuration

Sunday 3rd July 2022

No incidents reported

Saturday 2nd July 2022

No incidents reported

Friday 1st July 2022

No incidents reported

Thursday 30th June 2022

No incidents reported

Wednesday 29th June 2022

Reverse Proxies Network issue and partial reverse proxies outage on RBX

17:12 UTC, there was a unreported network issue. It caused two of our reverse proxies to fail. 17:13 UTC, two alerts get sent through the on-call system. The on-call person ACK both of them, handles the first one and mistake the second one for a redundant alert of the first one. 18:30 UTC, some customers complain about issues between APIs. We start investigating. 19:45 UTC, the culprit is found: a reverse proxy was down. It is restarted and everything goes back to normal. 19:50 UTC, we find the unattended alert and understand the mistake that was made. (reading the two alerts as one issue.)

Storage issue on Warp10

A component stop consuming this queue