Runbooks
Concrete commands for SMTP, verification, config reloads, local resets, and incidents.
Use this page when something is not working and you need the shortest safe path to isolate the layer that failed.
Fresh Local Database
For local testing, the fastest clean reset is to drop only the Compose database volume and re-run migrations. Do this only for disposable development data.
docker compose down
# Remove the local Postgres volume from Docker Desktop or with a project-specific volume command.
docker compose up -d smirkly-postgres
docker compose --profile dev run --rm smirkly-migrate-dev
SMTP Smoke Test
From the same container or host network where the service runs, check the provider greeting.
curl -v --connect-timeout 10 smtps://smtp.gmail.com:465
curl -v --connect-timeout 10 smtp://smtp.gmail.com:587
If port 587 connects but never returns a greeting, switch Gmail local testing to
AUTH_SMTP_PORT: 465 and AUTH_SMTP_TLS_MODE: tls, then restart the service.
Email Not Delivered
- Check that sign-up returned
201and the user email is correct. - Inspect
email_outboxfor status, attempt count, and latest error. - Check SMTP host, port, TLS mode, username, password, and provider account restrictions.
- Use the resend endpoint after fixing SMTP config.
docker compose exec -T smirkly-postgres psql -U smirkly_auth -d smirkly_auth -c "SELECT id, to_email, status, attempts, next_attempt_at, locked_until, left(coalesce(last_error, ''), 300) AS last_error, created_at, updated_at FROM email_outbox ORDER BY created_at DESC LIMIT 10;"
curl -i -X POST http://localhost:8080/auth/v0/verify-email/resend -H 'Content-Type: application/json' -d '{"email":"user@example.com"}'
Verification Code Expired
Expired or missing codes return 410. Request a fresh code and submit the latest value from the
newest email, not an older message.
curl -i -X POST http://localhost:8080/auth/v0/verify-email -H 'Content-Type: application/json' -d '{"email":"user@example.com","code":"123456"}'
Config Changed But Behavior Did Not
- If only
config_vars.yamlchanged, restart the running process or container. - If C++ source, SQL query files, or CMake source lists changed, rebuild the target and restart.
- If migrations changed, run the migration job before the service version that depends on them.
cmake --build build-debug --parallel --target smirkly-auth
./build-debug/smirkly-auth --config ./configs/static_config.yaml
Secret Exposure
- Revoke or rotate the exposed credential at the provider.
- Update the deployment secret source.
- Restart service instances using that secret.
- Search git history, logs, screenshots, docs, and issue comments for the old value.
- Audit provider access logs if available.
Production Incident Signals
| Signal | Likely cause | First check |
|---|---|---|
Growing email_outbox pending backlog |
SMTP outage, credentials rejected, provider throttling, or worker not running. | Outbox status query and service logs around SMTP errors. |
Spike in 403 EmailNotVerified |
Users cannot complete verification or policy was enabled unexpectedly. | SMTP delivery and latest deployment config. |
| Refresh reuse events | Token theft, replay, client retry bug, or shared cookie store issue. | Session family revocation logs and client release timeline. |
| Database cast errors for IP fields | Bad proxy header parsing or untrusted forwarded headers. | Trusted proxy CIDRs and ingress header behavior. |