Postmortem: the production database wiped by a migration

On 29 October 2026, a mis-targeted migration command wiped the GaijinToSensei production database. Here is what happened and what changed since. This postmortem is blameless: we look for what allowed the mistake, not for someone to blame.

Date

29 October 2026, 03:54 UTC

Service

GaijinToSensei, production PostgreSQL database (Neon)

Cause

prisma migrate dev with a shadow database pointing at production

Fix

Neon point-in-time restore (PITR)

Summary

An agent was building a new feature that added two tables. To create the migration, it ran prisma migrate dev. The Prisma shadow database was set to the production database. The command reset it: the production tables were wiped.

The database was restored with Neon point-in-time restore, to a moment before the command.

Timeline

Times in UTC, taken from the git history and that day's log.

1. 03:54The agent runs prisma migrate dev to create the spaced repetition migration. The shadow database points at production: it gets reset.
2. 03:55The agent sets its work aside without merging it and reports it. The agent pipeline is paused.
3. ThenThe database is restored with Neon point-in-time restore, to a moment before 03:54.
4. 05:13Fix for app pages that crashed when a session cookie pointed to an account missing from the database.
5. 11:10The feature is merged again with a new migration, created without touching production, and released.

Cause

prisma migrate dev is a development command. It uses a throwaway shadow database that it may erase. Here, the shadow database was the production database.

Nothing stopped an agent from running a destructive command with production access. That missing guardrail is what allowed the incident.

Impact

• The production database was empty from 03:54 until the restore finished. During that time, the app could not serve user accounts.
• After the restore, some sessions pointed to missing accounts and crashed app pages, until the 05:13 fix.

What helped

• Neon keeps a history that lets you go back to an exact moment. The restore needed no manual backup.
• The agent stopped instead of carrying on, and the pipeline was paused right away.

Fixes and new rules

1. 1Any destructive command is forbidden on the production database: no migrate dev, no reset, no deletion.
2. 2Additive migrations only. The release script rejects a migration that contains DROP, RENAME, TRUNCATE, DELETE, UPDATE or ALTER COLUMN, and never applies a migration itself.
3. 3Releases go through a locked script: preview, checks, promotion, a second check and automatic rollback on failure.
4. 4App pages check that the account still exists and otherwise send the user back to login.

What I learned

• A development command with production access is a production command. Access has to be separated.
• A backup you know how to restore quickly is worth more than any written procedure.
• When an agent makes a mistake, the useful question is: which guardrail was missing? Then add it in code, not in an instruction.

See how GaijinToSensei ships to production