Add dev-prod harmonize tool for bidirectional catalog and image sync.
Schema migrates dev to prod only; catalog rows merge by updated_at and images by mtime with union merge and conflict reporting. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Danila Khodjaef
co-authored by
Cursor
Cursor
parent
81ebad2120
commit
f247b418d8
@@ -0,0 +1,173 @@
|
||||
# Harmonize dev and prod (incremental merge)
|
||||
|
||||
Bidirectional **catalog data** and **image** sync between `gallery_dev` and `gallery_prod`, with **last-write-wins** by timestamp. Use this when **both** environments may have curator edits since the last release — not when prod should become an exact copy of dev.
|
||||
|
||||
For a full prod replace (weekly release), use [deploy-dev-to-prod.md](deploy-dev-to-prod.md) (`npm run devtoprod:release`).
|
||||
|
||||
For refreshing dev from prod entirely, use `npm run prodto:dev:db` (destructive to dev).
|
||||
|
||||
---
|
||||
|
||||
## Rules
|
||||
|
||||
| Layer | Direction | Conflict resolution |
|
||||
|-------|-----------|---------------------|
|
||||
| **Schema** | dev → prod only | Run `npm run harmonize:schema` (same as `dev:migrate` on `gallery_prod`) |
|
||||
| **Catalog DB** | dev ↔ prod | Newer `updated_at` wins; missing rows copied to the other side (union merge) |
|
||||
| **Images** | dev ↔ prod | Newer file mtime wins; missing files copied to the other side |
|
||||
| **Users / sessions / audit** | not synced | `users`, `session`, `curator_audit_log` stay env-local |
|
||||
|
||||
**Never auto-deletes** rows or files that exist on only one side.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Both databases on Postgres `192.168.10.122` with `updated_at` columns applied:
|
||||
|
||||
```powershell
|
||||
npm run dev:migrate
|
||||
$env:DB_NAME = "gallery_prod"; npm run dev:migrate; Remove-Item Env:\DB_NAME
|
||||
```
|
||||
|
||||
2. Backfill `updated_at` from image file mtimes (recommended once after migration):
|
||||
|
||||
```powershell
|
||||
npm run dev:backfill-updated-at
|
||||
npm run harmonize:backfill-updated-at
|
||||
```
|
||||
|
||||
3. [`infra/docker/.env.prod`](../infra/docker/.env.prod) present with `DB_NAME=gallery_prod`.
|
||||
|
||||
4. SMB share reachable for prod images:
|
||||
|
||||
```powershell
|
||||
net use \\192.168.10.122\Gallery /user:YOUR_TRUENAS_USER
|
||||
```
|
||||
|
||||
5. Copy harmonize config:
|
||||
|
||||
```powershell
|
||||
Copy-Item infra/deploy/harmonize.config.example.json infra/deploy/harmonize.config.json
|
||||
```
|
||||
|
||||
Edit `harmonize.config.json` (gitignored) — optional `smb.user` / `smb.password`, `prefer` for tie-breaks (`dev` | `prod`), `schemaChanged: true` when new migrations shipped.
|
||||
|
||||
---
|
||||
|
||||
## One-command harmonize
|
||||
|
||||
```powershell
|
||||
npm run harmonize
|
||||
```
|
||||
|
||||
Dry-run (report only, no writes):
|
||||
|
||||
```powershell
|
||||
npm run harmonize -- -DryRun
|
||||
```
|
||||
|
||||
Or:
|
||||
|
||||
```powershell
|
||||
npm run harmonize:dry-run
|
||||
```
|
||||
|
||||
### Orchestrator steps
|
||||
|
||||
| Step | npm script | Purpose |
|
||||
|------|------------|---------|
|
||||
| `backupDev` | `dev:db:backup` | Safety snapshot |
|
||||
| `backupProd` | `prod:db:backup` | Safety snapshot |
|
||||
| `schema` | `harmonize:schema` | Apply dev migrations to prod (when `schemaChanged: true`) |
|
||||
| `db` | `harmonize:db` | Row-level catalog merge |
|
||||
| `images` | `harmonize:images` | Bidirectional file merge |
|
||||
| `verify` | curl `/api/bounds` | Optional smoke check |
|
||||
|
||||
Reports are written to `db/SyncReports/harmonize_db_*.json` and `harmonize_images_*.json` (gitignored).
|
||||
|
||||
---
|
||||
|
||||
## Individual commands
|
||||
|
||||
| Command | Purpose |
|
||||
|---------|---------|
|
||||
| `npm run harmonize:schema` | Migrate prod schema from dev migration files |
|
||||
| `npm run harmonize:db` | Merge catalog rows by `updated_at` |
|
||||
| `npm run harmonize:db -- --dry-run` | Preview DB changes |
|
||||
| `npm run harmonize:db -- --prefer=dev` | On equal `updated_at`, dev wins |
|
||||
| `npm run harmonize:images` | Merge image files by mtime |
|
||||
| `npm run dev:migrate:sync-timestamps` | Apply `updated_at` migration on dev only |
|
||||
| `npm run dev:backfill-updated-at` | Backfill dev `updated_at` from image mtimes |
|
||||
| `npm run harmonize:backfill-updated-at` | Same backfill on prod |
|
||||
|
||||
---
|
||||
|
||||
## Catalog tables synced
|
||||
|
||||
Processed in FK order:
|
||||
|
||||
`historical_eras` → `art_movements` → `artists` → `artist_periods` → `paintings` → `painting_influences` → `painting_influence_sources` → `painting_annotations`
|
||||
|
||||
---
|
||||
|
||||
## Conflict handling
|
||||
|
||||
Harmonize reports conflicts in the JSON report and skips those rows:
|
||||
|
||||
| Conflict | Cause | Resolution |
|
||||
|----------|-------|------------|
|
||||
| `id_collision` | Same `id` but different natural key (e.g. artist name) | Manual fix in pgAdmin; environments diverged too far |
|
||||
| `equal_updated_at` | Same timestamp, different row content | Re-run with `--prefer=dev` or `--prefer=prod`, or edit one side and re-run |
|
||||
|
||||
**Tip:** Harmonize regularly from a shared baseline (e.g. after each weekly release) to avoid ID/natural-key collisions from independent inserts.
|
||||
|
||||
---
|
||||
|
||||
## When to use what
|
||||
|
||||
| Situation | Tool |
|
||||
|-----------|------|
|
||||
| Weekly release — prod should match dev exactly | `npm run devtoprod:release` |
|
||||
| Mid-week prod curator fix + dev also changed | `npm run harmonize` |
|
||||
| Dev workspace stale — full prod copy | `npm run prodto:dev:db` |
|
||||
| New migration in repo | `harmonize:schema` or deploy step 4 |
|
||||
| Only images changed on one side | `npm run harmonize:images` |
|
||||
| Only DB metadata changed | `npm run harmonize:db` |
|
||||
|
||||
---
|
||||
|
||||
## Example flows
|
||||
|
||||
### Prod curator uploaded a painting; dev also edited metadata
|
||||
|
||||
```powershell
|
||||
npm run harmonize
|
||||
```
|
||||
|
||||
DB rows merge by `updated_at`; image files merge by mtime. Both sides receive the latest version of each entity.
|
||||
|
||||
### Schema change + mixed edits
|
||||
|
||||
1. Finish and test on dev: `npm run dev:migrate`
|
||||
2. Set `schemaChanged: true` in `harmonize.config.json` (or enable `steps.schema`)
|
||||
3. `npm run harmonize`
|
||||
|
||||
### Preview before writing
|
||||
|
||||
```powershell
|
||||
npm run harmonize:dry-run
|
||||
# Review db/SyncReports/harmonize_*.json
|
||||
npm run harmonize
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Safety
|
||||
|
||||
- Pre-flight backups of dev and prod DB (configurable; on by default)
|
||||
- Prod writes require `yes` or `CONFIRM_PROD=1` (orchestrator sets `CONFIRM_PROD=1` when `autoConfirm`-style run)
|
||||
- No TRUNCATE — harmonize only inserts/updates changed rows
|
||||
- Rollback: restore from `db/DataBackup/gallery_*_data_*.txt` using `dev:db:restore` or `devtoprod:db:restore`
|
||||
|
||||
See also [environments.md](environments.md) and [deploy-dev-to-prod.md](deploy-dev-to-prod.md).
|
||||
Reference in New Issue
Block a user