Runbooks

Concrete commands for SMTP, verification, config reloads, local resets, and incidents.

Use this page when something is not working and you need the shortest safe path to isolate the layer that failed.

Fresh Local Database

For local testing, the fastest clean reset is to drop only the Compose database volume and re-run migrations. Do this only for disposable development data.

docker compose down
# Remove the local Postgres volume from Docker Desktop or with a project-specific volume command.
docker compose up -d smirkly-postgres
docker compose --profile dev run --rm smirkly-migrate-dev

SMTP Smoke Test

From the same container or host network where the service runs, check the provider greeting.

curl -v --connect-timeout 10 smtps://smtp.gmail.com:465
curl -v --connect-timeout 10 smtp://smtp.gmail.com:587

If port 587 connects but never returns a greeting, switch Gmail local testing to AUTH_SMTP_PORT: 465 and AUTH_SMTP_TLS_MODE: tls, then restart the service.

Email Not Delivered

  1. Check that sign-up returned 201 and the user email is correct.
  2. Inspect email_outbox for status, attempt count, and latest error.
  3. Check SMTP host, port, TLS mode, username, password, and provider account restrictions.
  4. Use the resend endpoint after fixing SMTP config.
docker compose exec -T smirkly-postgres   psql -U smirkly_auth -d smirkly_auth   -c "SELECT id, to_email, status, attempts, next_attempt_at, locked_until, left(coalesce(last_error, ''), 300) AS last_error, created_at, updated_at FROM email_outbox ORDER BY created_at DESC LIMIT 10;"

curl -i -X POST http://localhost:8080/auth/v0/verify-email/resend   -H 'Content-Type: application/json'   -d '{"email":"user@example.com"}'

Verification Code Expired

Expired or missing codes return 410. Request a fresh code and submit the latest value from the newest email, not an older message.

curl -i -X POST http://localhost:8080/auth/v0/verify-email   -H 'Content-Type: application/json'   -d '{"email":"user@example.com","code":"123456"}'

Config Changed But Behavior Did Not

  • If only config_vars.yaml changed, restart the running process or container.
  • If C++ source, SQL query files, or CMake source lists changed, rebuild the target and restart.
  • If migrations changed, run the migration job before the service version that depends on them.
cmake --build build-debug --parallel --target smirkly-auth
./build-debug/smirkly-auth --config ./configs/static_config.yaml

Secret Exposure

  1. Revoke or rotate the exposed credential at the provider.
  2. Update the deployment secret source.
  3. Restart service instances using that secret.
  4. Search git history, logs, screenshots, docs, and issue comments for the old value.
  5. Audit provider access logs if available.

Production Incident Signals

Signal Likely cause First check
Growing email_outbox pending backlog SMTP outage, credentials rejected, provider throttling, or worker not running. Outbox status query and service logs around SMTP errors.
Spike in 403 EmailNotVerified Users cannot complete verification or policy was enabled unexpectedly. SMTP delivery and latest deployment config.
Refresh reuse events Token theft, replay, client retry bug, or shared cookie store issue. Session family revocation logs and client release timeline.
Database cast errors for IP fields Bad proxy header parsing or untrusted forwarded headers. Trusted proxy CIDRs and ingress header behavior.