cuongthai.com

01Build images off-server; the server only pulls and swaps

Problem

The 6 GB server: building both images in parallel was killed by the OS for running out of memory (exit 137). The build cache grew to 7.6 GB on the same disk as PostgreSQL; one deploy died midway with only 1.8 GB left.

Decision

A dedicated build machine (12 cores, 31 GB RAM) builds both images in parallel and pushes them to a registry; the server only pulls, runs migrations and swaps containers. The server no longer keeps a build cache.

Trade-off

About 3× faster (3–6 minutes versus ~15) but one more machine to depend on. Two fallbacks cover it: a sequential build on the server, or a GitHub Actions build when the build machine is unreachable.

02Deploying is not a side effect of git push

Problem

Two workflows once fired on every push to main and raced each other to swap containers — causing two real outages (the feed returned 500 while the schema lagged the image; a container was killed mid-recreate).

Decision

Both deploy workflows became manual-only. A push only runs type-check + lint CI. The deploy script runs the CI checks itself before it pushes main.

Trade-off

Loses the “push and it’s live” convenience; in exchange every deploy has an owner and no two processes step on each other.

03Choosing AI models by measurement, not price lists

Problem

The third-party LLM gateway does not publish real prices; reasoning models also bill for hidden reasoning tokens (measured at ~21% of output tokens). Listed price ratios do not predict the bill.

Decision

Each model is measured on the same task set (gateway ledger delta, latency, correctness, whether it honours max_tokens), then assigned by TASK TYPE: background and machine-read work gets a cheap model, text people read word by word gets a strong one. Changeable by configuration, no deploy needed.

Trade-off

Measurements must be redone when the gateway changes models. Models that ignore token limits are excluded outright, even when cheap — every cost ceiling in the code would be void against them.

04Two keys per model group; a missing key falls back instead of failing

Problem

Each gateway key belongs to one model group; calling another group’s model returns 503 “no channel” — sounds transient, is permanent. The model map ships with the code, keys are added by hand — one mismatch and many features fail together.

Decision

The key is chosen by model-name prefix; if it is missing, the call switches to an equivalent model in a group that has a key and logs a warning.

Trade-off

Users see no error, but features run more expensively (measured: background work 5.1× the cost) until someone reads the warning — hence a checker that really calls every assigned model.

05Three layers of AI cost ceilings

Problem

Previously, an empty setting meant unlimited — and that was the only safeguard of the whole system.

Decision

Background AI work is OFF by default; a per-user daily token cap applies even with no configuration; a two-level daily spend ceiling — the soft $15 level stops background jobs, the hard $40 level stops everything.

Trade-off

Costs inside the system are estimates (the gateway publishes no prices), so the spend ceiling catches anomalies; it is not accounting.

06Hand-written migrations with a drift check before deploy

Problem

A migration already applied in production cannot replay on a scratch database (a duplicated constraint name), which breaks automatic migration generation. Editing deployed migrations is forbidden.

Decision

Migration SQL is written by hand, applied with a deploy command that needs no scratch database, then the schema is diffed against the real database — an empty diff is the pass. A failed production migration stops and is reported, never auto-“resolved”.

Trade-off

Slower and needs discipline; in exchange migration history is never overwritten just to get past an error.

Stale build: a route vanished while the deploy “succeeded”

02/07/2026

What happened

The GIF picker died and messages looked like they had disappeared, even after signing in again.

Root cause

Production ran an old backend bundle that never mounted the new route (it returned 404), after a deploy that synced files without rebuilding the image.

Fix & safeguard

Added a post-deploy smoke test: core routes are called unauthenticated, 401/200 means mounted, 404 fails the whole deploy. The list now holds 70 routes.

Green build, API down with 502 for seven minutes

18/08/2026

What happened

Build, push and swap all reported success; right after, the backend restarted in a loop.

Root cause

The image was built from the wrong Dockerfile: an Alpine (musl) base carrying a glibc Prisma engine — which cannot load.

Fix & safeguard

Always build with the exact file compose uses, and added a gate that runs the image to match libc against the Prisma engine BEFORE pushing. Plus a recovery drill using the previous image still on the host (~40 seconds instead of a 15-minute rebuild).

Config written and reloaded — yet not in effect

25/08/2026

What happened

The nginx sync step reported OK twice in a row (`nginx -t` green, reload green) yet HTTP/2 stayed off and cache headers stayed no-store.

Root cause

The config file was replaced with `mv`. Docker binds a single file by inode at container start, so the container kept reading the old file — the test and the reload both ran against the old config.

Fix & safeguard

Overwrite in place instead of renaming, then compare the sha256 on the host with the file seen from INSIDE the container; on mismatch, restore the old file and stop. Lesson: “written” does not mean “in effect”.

Renaming an enum value broke the seed in production

08/08/2026

What happened

The change passed the entire pre-push checklist, yet the seed step failed in production.

Root cause

The seed script carried its own hand-written copy of the enum type, so it “type-checked against itself”, and the main type check did not cover the seed folder.

Fix & safeguard

Enum types now come straight from the Prisma client; added a separate type-check config for the seed and a real seed run in the checklist whenever the schema changes.

Sessions silently died after 24 hours

02/07/2026

What happened

After a day, every authenticated call returned 401 although the cookie was still present.

Root cause

Tokens lived 24 hours, cookies 7 days, and there was no working token-refresh endpoint.

Fix & safeguard

Added a refresh endpoint (re-checking the account) and a client interceptor: on 401, refresh once and retry. Sessions self-heal without configuration changes.