Art-gallery/Documentation/harmonize-dev-prod.md

Harmonize dev and prod (incremental merge)

Bidirectional catalog data and image sync between gallery_dev and gallery_prod, with last-write-wins by timestamp. Use this when both environments may have curator edits since the last release — not when prod should become an exact copy of dev.

For a full prod replace (weekly release), use deploy-dev-to-prod.md (npm run devtoprod:release).

For refreshing dev from prod entirely, use npm run prodto:dev:db (destructive to dev).

---

Rules

Layer	Direction	Conflict resolution
Schema	dev → prod only	Run npm run harmonize:schema (same as dev:migrate on gallery_prod)
Catalog DB	dev ↔ prod	Newer updated_at wins; missing rows copied to the other side (union merge)
Images	dev ↔ prod	Newer file mtime wins; missing files copied to the other side
Users / sessions / audit	not synced	users, session, curator_audit_log stay env-local

Never auto-deletes rows or files that exist on only one side.

---

Prerequisites

1. Both databases on Postgres 192.168.10.122 with updated_at columns applied:

   npm run dev:migrate
   $env:DB_NAME = "gallery_prod"; npm run dev:migrate; Remove-Item Env:\DB_NAME
   

2. Backfill updated_at from image file mtimes (recommended once after migration):

   npm run dev:backfill-updated-at
   npm run harmonize:backfill-updated-at
   

3. infra/docker/.env.prod present with DB_NAME=gallery_prod.

4. SMB share reachable for prod images:

   net use \\192.168.10.122\Gallery /user:YOUR_TRUENAS_USER
   

5. Copy dedicated harmonize config (do not rely on devtoprod.config.json alone — that file is a release profile and its step flags are ignored for harmonize):

   Copy-Item infra/deploy/harmonize.config.example.json infra/deploy/harmonize.config.json
   

   Edit harmonize.config.json (gitignored) — optional smb.user / smb.password, prefer for tie-breaks (dev | prod), schemaChanged: true when new migrations shipped.

6. Prod DB targeting uses infra/docker/.env.prod. scripts/db-env.js loads that file with file values winning over the root .env, so harmonize:db can open gallery_dev and gallery_prod in the same process without the safety check rejecting gallery_dev as a fake prod target.

---

One-command harmonize

npm run harmonize

Dry-run (report only, no writes):

npm run harmonize -- -DryRun

Or:

npm run harmonize:dry-run

Orchestrator steps

Step	npm script	Purpose
backupDev	dev:db:backup	Safety snapshot
backupProd	prod:db:backup	Safety snapshot
schema	harmonize:schema	Apply dev migrations to prod (when schemaChanged: true)
db	harmonize:db	Row-level catalog merge
images	harmonize:images	Bidirectional file merge, then merge artists/paintings (checkup flags + image paths), then regenerate thumbs on dev and prod
verify	curl /api/bounds	Optional smoke check

Reports are written to db/SyncReports/harmonize_db_*.json and harmonize_images_*.json (gitignored).

---

Individual commands

Command	Purpose
npm run harmonize:schema	Migrate prod schema from dev migration files
npm run harmonize:db	Merge catalog rows by updated_at
npm run harmonize:db -- --dry-run	Preview DB changes
npm run harmonize:db -- --prefer=dev	On equal updated_at, dev wins
npm run harmonize:images	Merge image files by mtime, merge artists/paintings checkup flags + image paths, then regenerate thumbs on both sides
npm run harmonize:images -- --skip-thumbnails	File + checkup/path DB merge only (no thumb rebuild)
npm run harmonize:images -- --skip-db	File merge (+ thumbs) without artists/paintings checkup sync
npm run harmonize:images -- --dry-run	Preview file copies and catalog merge (skips thumbnails)
npm run dev:migrate:sync-timestamps	Apply updated_at migration on dev only
npm run dev:backfill-updated-at	Backfill dev updated_at from image mtimes
npm run harmonize:backfill-updated-at	Same backfill on prod

---

Catalog tables synced

Processed in FK order:

historical_eras → art_movements → artists → artist_periods → paintings → painting_influences → painting_influence_sources → painting_annotations → entity_translations → tours → tour_stops

Every synced table needs an updated_at column (including tour_stops — added via migrate-tours.sql / migrate-sync-timestamps.sql).

---

Conflict handling

Harmonize reports conflicts in the JSON report and skips those rows:

Conflict	Cause	Resolution
id_collision	Same id but different natural key (e.g. artist name)	Manual fix in pgAdmin; environments diverged too far
equal_updated_at	Same timestamp, different row content	Re-run with --prefer=dev or --prefer=prod, or edit one side and re-run

Tip: Harmonize regularly from a shared baseline (e.g. after each weekly release) to avoid ID/natural-key collisions from independent inserts.

---

When to use what

Situation	Tool
Weekly release — prod should match dev exactly	npm run devtoprod:release
Mid-week prod curator fix + dev also changed	npm run harmonize
Dev workspace stale — full prod copy	npm run prodto:dev:db
New migration in repo	harmonize:schema or deploy step 4
Only images changed on one side	npm run harmonize:images
Only DB metadata changed	npm run harmonize:db
Pull one artist’s images from prod → dev	Map SMB, then robocopy Duccio* (or the artist prefix) under paintings/ and paintings/thumbs/ — see FAC.md — Dev ↔ prod image sync

---

Example flows

Prod curator uploaded a painting; dev also edited metadata

npm run harmonize

DB rows merge by updated_at; image files merge by mtime. Both sides receive the latest version of each entity.

Schema change + mixed edits

1. Finish and test on dev: npm run dev:migrate
2. Set schemaChanged: true in harmonize.config.json (or enable steps.schema)
3. npm run harmonize

Preview before writing

npm run harmonize:dry-run
# Review db/SyncReports/harmonize_*.json
npm run harmonize

---

Troubleshooting

Symptom	Likely cause	Fix
Refusing prod operation on database "gallery_dev"	Old db-env left DB_NAME from root .env when loading prod	Update to current scripts/db-env.js (.env.prod wins); confirm infra/docker/.env.prod has DB_NAME=gallery_prod
Table tour_stops missing updated_at	Prod/dev schema behind	npm run dev:migrate and migrate prod ($env:DB_NAME="gallery_prod"; npm run dev:migrate; Remove-Item Env:\DB_NAME)
Orchestrator lists release steps (restoreProd, …)	Missing harmonize.config.json; fallback to release config	Copy harmonize.config.example.json → harmonize.config.json (current harmonize.ps1 ignores release-only step keys)

---

Safety

• Pre-flight backups of dev and prod DB (configurable; on by default)
• Prod writes require yes or CONFIRM_PROD=1 (orchestrator sets CONFIRM_PROD=1 when autoConfirm-style run)
• No TRUNCATE — harmonize only inserts/updates changed rows
• Rollback: restore from db/DataBackup/gallery_*_data_*.txt using dev:db:restore or devtoprod:db:restore

See also environments.md and deploy-dev-to-prod.md.