All systems operational

Incident history

Shuttle Gateway Service Disruption

Following the release on July 31, 2023, at 5:08 PM GMT + 2, the Shuttle gateway encountered an issue preventing it from fulfilling user project requests. This issue arose due to a faulty database migration. To address this, we promptly reverted the release and restored the database to a recent backup, ensuring no data loss occurred. By 6:20 PM, all services were fully operational again. Moving forward, we are committed to enhancing our release rollback procedures, aiming to restore services more efficiently in case of future problematic releases. Additionally, we will focus on bolstering our test suite and refining our staging environment to prevent such issues from ever reaching the production stage.

Jul 31, 2023

Resolved

MongoDB Data Loss Incident

On Saturday, July 29th, we received a report indicating that a user's MongoDB database had been cleared upon re-deployment. Subsequent investigation by our engineers revealed that this issue arose from the shared MongoDB database container lacking an associated persistent volume. To rectify this situation, a persistent volume was attached to the MongoDB database container, and it was restarted without incurring any additional loss of data. Moving forward, our goal is to expedite the resolution of similar incidents and establish a structured procedure for efficiently restoring the MongoDB database in the event of any future failures.

Jul 29, 2023

Resolved

Provisioner downtime

We experienced a provisioner downtime which made Shuttle platform experience difficulties in deploying new projects or serving requests for idled deployments. As next steps, we are working on improving the detection/resolution times and decoupling the provisioner dependency for idled deployments cold start.

Jun 27, 2023

Resolved

An unsecured endpoint within the auth service enabled unauthorized access to users' API keys

In the development of the authentication service, we established a /login endpoint to facilitate a session-based authentication flow specifically designed for console usage. However, this endpoint was unintentionally left unprotected, creating a potential risk that a user's API key could be accessed by others. The issue was successfully identified and rectified in the 0.19.0 release, thereby mitigating the vulnerability.

Jun 15, 2023

Resolved