Skip to content

ADR 001 — The nightly build runs off-box; only artifacts ship

Status: Accepted · Date: 2026-08-08 · Deviates from: spec §3, §5.7, §11.1, §11.2

Context

The spec places the ingest → dbt → embeddings pipeline on the same VPS that serves the app, as a cron-triggered pipeline container (§11.1), with the atomic swap happening locally (§5.7).

That VPS is twakka01 (Hetzner, Helsinki, 46.62.205.109). It is shared with roughly 8–10 other applications across ~19 containers — belayat, kurakani, kathmandu.im, cleverpath, trustkit, dartabase, projecta. Measured 2026-08-08: 5.0 of 7.8 GB RAM in use, 47 of 75 GB disk in use (up from 39 GB in July — it is trending the wrong way). There is a documented incident where a single application left running in dev mode drove load high enough to make SSH and Apache unresponsive for every other tenant.

Decision

The build runs on the developer machine. make all produces dist/groundfact_YYYYMMDD.db, the rendered semantic artifacts, the MkDocs site/, and the favicon cache. ops/publish.sh ships those to /opt/groundfact/, verifies a sha256, performs the ln -sfn atomic swap, and healthchecks. The server runs exactly one container plus a state-backup cron.

Rationale, and a correction

The original argument had two legs — disk pressure and CPU/RAM contention. The disk leg turned out to be wrong and is withdrawn. The S0 feasibility spike measured the full World Bank WDI pull (411 Nepal-covered series × 14 geos × 1960–2025, 198,969 observations) at 1.2 MB parquet / 2.6 MB DuckDB. Extrapolated to global scope across all Phase-1 sources, the database is roughly 200 MB — not the multi-GB artifact the spec's MotherDuck 10 GB threshold implies. Shipping it nightly is trivially cheap.

The decision stands on the remaining leg, which the spike did not undermine:

  1. Ingestion is the heavy step, not storage. FAOSTAT's REST API is down (HTTP 521), so the connector must use bulk ZIPs — 69 datasets, several 50–70 MB each, unzipped and parsed. That is sustained disk and CPU churn on a box with 26 GB free.
  2. Embeddings pull a large toolchain. sentence-transformers plus torch is ~2 GB of dependencies and is CPU-bound at build time. Installing that alongside 19 running containers buys nothing — the output is a FLOAT[384] column that ships inside the .db.
  3. Blast radius. A runaway build on a shared box takes down six other people's projects. Off-box, the worst case is a stale database and a failed publish, and the previous .db keeps serving.

Consequences

  • make all must be run somewhere deliberate. This is a real cost: the pipeline is no longer self-driving, and a forgotten build shows up as a stale freshness badge.
  • /healthz reports db_age_hours and returns 503 beyond 48h, so staleness is detected rather than silently served.
  • Rollback is ln -sfn to the previous .db — cheaper than the spec's in-place design.
  • CI can take over the build later without re-architecting; the seam is ops/publish.sh.