Art-gallery/Documentation/harmonize-dev-prod.md

# Harmonize dev and prod (incremental merge)

Bidirectional **catalog data** and **image** sync between `gallery_dev` and `gallery_prod`, with **last-write-wins** by timestamp. Use this when **both** environments may have curator edits since the last release — not when prod should become an exact copy of dev.

For a full prod replace (weekly release), use [deploy-dev-to-prod.md](deploy-dev-to-prod.md) (`npm run devtoprod:release`).

For refreshing dev from prod entirely, use `npm run prodto:dev:db` (destructive to dev).

---

## Rules

| Layer | Direction | Conflict resolution |
|-------|-----------|---------------------|
| **Schema** | dev → prod only | Run `npm run harmonize:schema` (same as `dev:migrate` on `gallery_prod`) |
| **Catalog DB** | dev ↔ prod | Newer `updated_at` wins; missing rows copied to the other side (union merge) |
| **Images** | dev ↔ prod | Newer file mtime wins; missing files copied to the other side |
| **Users / sessions / audit** | not synced | `users`, `session`, `curator_audit_log` stay env-local |

**Never auto-deletes** rows or files that exist on only one side.

---

## Prerequisites

1. Both databases on Postgres `192.168.10.122` with `updated_at` columns applied:

   ```powershell
   npm run dev:migrate
   $env:DB_NAME = "gallery_prod"; npm run dev:migrate; Remove-Item Env:\DB_NAME
   ```

2. Backfill `updated_at` from image file mtimes (recommended once after migration):

   ```powershell
   npm run dev:backfill-updated-at
   npm run harmonize:backfill-updated-at
   ```

3. [`infra/docker/.env.prod`](../infra/docker/.env.prod) present with `DB_NAME=gallery_prod`.

4. SMB share reachable for prod images:

   ```powershell
   net use \\192.168.10.122\Gallery /user:YOUR_TRUENAS_USER
   ```

5. Copy **dedicated** harmonize config (do not rely on `devtoprod.config.json` alone — that file is a release profile and its step flags are ignored for harmonize):

   ```powershell
   Copy-Item infra/deploy/harmonize.config.example.json infra/deploy/harmonize.config.json
   ```

   Edit `harmonize.config.json` (gitignored) — optional `smb.user` / `smb.password`, `prefer` for tie-breaks (`dev` | `prod`), `schemaChanged: true` when new migrations shipped.

6. Prod DB targeting uses [`infra/docker/.env.prod`](../infra/docker/.env.prod). `scripts/db-env.js` loads that file with **file values winning** over the root `.env`, so `harmonize:db` can open `gallery_dev` and `gallery_prod` in the same process without the safety check rejecting `gallery_dev` as a fake prod target.

---

## One-command harmonize

```powershell
npm run harmonize
```

Dry-run (report only, no writes):

```powershell
npm run harmonize -- -DryRun
```

Or:

```powershell
npm run harmonize:dry-run
```

### Orchestrator steps

| Step | npm script | Purpose |
|------|------------|---------|
| `backupDev` | `dev:db:backup` | Safety snapshot |
| `backupProd` | `prod:db:backup` | Safety snapshot |
| `schema` | `harmonize:schema` | Apply dev migrations to prod (when `schemaChanged: true`) |
| `db` | `harmonize:db` | Row-level catalog merge |
| `images` | `harmonize:images` | Bidirectional file merge, then merge **artists/paintings** (checkup flags + image paths), then regenerate thumbs on **dev and prod** |
| `verify` | curl `/api/bounds` | Optional smoke check |

Reports are written to `db/SyncReports/harmonize_db_*.json` and `harmonize_images_*.json` (gitignored).

---

## Individual commands

| Command | Purpose |
|---------|---------|
| `npm run harmonize:schema` | Migrate prod schema from dev migration files |
| `npm run harmonize:db` | Merge catalog rows by `updated_at` |
| `npm run harmonize:db -- --dry-run` | Preview DB changes |
| `npm run harmonize:db -- --prefer=dev` | On equal `updated_at`, dev wins |
| `npm run harmonize:images` | Merge image files by mtime, merge artists/paintings checkup flags + image paths, then regenerate thumbs on both sides |
| `npm run harmonize:images -- --skip-thumbnails` | File + checkup/path DB merge only (no thumb rebuild) |
| `npm run harmonize:images -- --skip-db` | File merge (+ thumbs) without artists/paintings checkup sync |
| `npm run harmonize:images -- --dry-run` | Preview file copies and catalog merge (skips thumbnails) |
| `npm run dev:migrate:sync-timestamps` | Apply `updated_at` migration on dev only |
| `npm run dev:backfill-updated-at` | Backfill dev `updated_at` from image mtimes |
| `npm run harmonize:backfill-updated-at` | Same backfill on prod |

---

## Catalog tables synced

Processed in FK order:

`historical_eras` → `art_movements` → `artists` → `artist_periods` → `paintings` → `painting_influences` → `painting_influence_sources` → `painting_annotations` → **`entity_translations`** → **`tours`** → **`tour_stops`**

Every synced table needs an `updated_at` column (including `tour_stops` — added via `migrate-tours.sql` / `migrate-sync-timestamps.sql`).

---

## Conflict handling

Harmonize reports conflicts in the JSON report and skips those rows:

| Conflict | Cause | Resolution |
|----------|-------|------------|
| `id_collision` | Same `id` but different natural key (e.g. artist name) | Manual fix in pgAdmin; environments diverged too far |
| `equal_updated_at` | Same timestamp, different row content | Re-run with `--prefer=dev` or `--prefer=prod`, or edit one side and re-run |

**Tip:** Harmonize regularly from a shared baseline (e.g. after each weekly release) to avoid ID/natural-key collisions from independent inserts.

---

## When to use what

| Situation | Tool |
|-----------|------|
| Weekly release — prod should match dev exactly | `npm run devtoprod:release` |
| Mid-week prod curator fix + dev also changed | `npm run harmonize` |
| Dev workspace stale — full prod copy | `npm run prodto:dev:db` |
| New migration in repo | `harmonize:schema` or deploy step 4 |
| Only images changed on one side | `npm run harmonize:images` |
| Only DB metadata changed | `npm run harmonize:db` |
| Pull **one artist’s** images from prod → dev | Map SMB, then robocopy `Duccio*` (or the artist prefix) under `paintings/` and `paintings/thumbs/` — see [FAC.md — Dev ↔ prod image sync](FAC.md#dev--prod-image-sync-smb) |

---

## Example flows

### Prod curator uploaded a painting; dev also edited metadata

```powershell
npm run harmonize
```

DB rows merge by `updated_at`; image files merge by mtime. Both sides receive the latest version of each entity.

### Schema change + mixed edits

1. Finish and test on dev: `npm run dev:migrate`
2. Set `schemaChanged: true` in `harmonize.config.json` (or enable `steps.schema`)
3. `npm run harmonize`

### Preview before writing

```powershell
npm run harmonize:dry-run
# Review db/SyncReports/harmonize_*.json
npm run harmonize
```

---

## Troubleshooting

| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| `Refusing prod operation on database "gallery_dev"` | Old `db-env` left `DB_NAME` from root `.env` when loading prod | Update to current `scripts/db-env.js` (`.env.prod` wins); confirm `infra/docker/.env.prod` has `DB_NAME=gallery_prod` |
| `Table tour_stops missing updated_at` | Prod/dev schema behind | `npm run dev:migrate` and migrate prod (`$env:DB_NAME="gallery_prod"; npm run dev:migrate; Remove-Item Env:\DB_NAME`) |
| Orchestrator lists release steps (`restoreProd`, …) | Missing `harmonize.config.json`; fallback to release config | Copy `harmonize.config.example.json` → `harmonize.config.json` (current `harmonize.ps1` ignores release-only step keys) |

---

## Safety

- Pre-flight backups of dev and prod DB (configurable; on by default)
- Prod writes require `yes` or `CONFIRM_PROD=1` (orchestrator sets `CONFIRM_PROD=1` when `autoConfirm`-style run)
- No TRUNCATE — harmonize only inserts/updates changed rows
- Rollback: restore from `db/DataBackup/gallery_*_data_*.txt` using `dev:db:restore` or `devtoprod:db:restore`

See also [environments.md](environments.md) and [deploy-dev-to-prod.md](deploy-dev-to-prod.md).