Past Incidents

We are experiencing an issue with logs ingestion pipeline. We are looking into it.

EDIT 9h15 UTC : The ingestion pipeline is back to normal. No abnormal delay.

Metrics and Access logs queries are currently unavailable. Data is still ingested, only queries are impacted. ETA for resolution is 15:00 UTC.

This impacts:

• Metrics (Grafana, in the console or using or API)
• Access logs: (requests tiles for an organization / application, CLI access or our API)

EDIT 14:52 UTC: The queries are available again since 14:20 UTC. This incident is over.

Elastic released a security bulletin regarding the impact CVE-2021-44228 has on Elasticsearch. Elastic recommends users to apply the -Dlog4j2.formatMsgNoLookups=true JVM option and restart Elasticsearch. More information in Elastic security bulletin: https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

We will apply this option on all add-ons and restart them as an emergency maintenance. For single node add-ons, this will trigger a short downtime of minimum 1 minute (the approximate time it takes Elasticsearch to boot). For clustered add-ons, no downtime is to be expected as it will be a rolling restart.

Newly created add-ons are already patched.

The restart of all add-ons will start at 15:00 UTC. Sorry for the short notice. Feel free to contact our support if you have any questions.

EDIT 15:05 UTC: Add-ons restart is starting

EDIT 16:10 UTC: Add-ons have been restarted. The maintenance is over.

An add-on reverse proxy crashed at 10:21 UTC and got restarted at 10:24 UTC. During that time, some services connecting to their add-ons might have experienced unexpected connection errors (connection lost, connection refused, ...).

The issue is now fixed.