Danilka/Art-gallery
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add dev-prod harmonize tool for bidirectional catalog and image sync.

Browse Source 
Schema migrates dev to prod only; catalog rows merge by updated_at and images by mtime with union merge and conflict reporting.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Danila KhodjaefandCursor committed 2026-07-15 11:02:17 +03:00
1 parent 81ebad2120
commit f247b418d8
21 files changed
+1336 -4
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Documentation/FAC.md
+6
View File
@@ -138,6 +138,10 @@ Run `npm run dev:migrate` against prod DB after first deploy with auth vars set
| `npm run prod:db:backup` | Prod backup (reads `infra/docker/.env.prod`) |
| `npm run dev:db:restore -- --file <path>` | Restore backup into **dev** (truncates tables first; prompts `yes`) |
| `npm run devtoprod:db:restore -- --file <path>` | Restore into **prod** (requires confirmation) |
| `npm run harmonize` | Bidirectional catalog DB + image merge by `updated_at` / file mtime — [harmonize-dev-prod.md](harmonize-dev-prod.md) |
| `npm run harmonize:schema` | Apply dev migrations to prod schema only (dev → prod) |
| `npm run harmonize:db` / `harmonize:images` | DB or image merge only |
| `npm run dev:backfill-updated-at` | Backfill catalog `updated_at` from image mtimes (dev) |
**One-time split (recommended):** pgAdmin on dev PC → open [`db/split-dev-prod-pgadmin.sql`](../db/split-dev-prod-pgadmin.sql) → run each STEP on database `postgres`, then verify on `gallery_dev`.
@@ -237,6 +241,8 @@ npm run devtoprod:release # full promote from infra/deploy/devtoprod.conf
npm run devtoprod:thumbnails # rebuild thumb files + DB paths on dev (before backup)
npm run devtoprod:images # dev repo → TrueNAS (promote / first deploy)
npm run prodto:dev:images # TrueNAS → dev repo
npm run harmonize # bidirectional merge (newer wins) — see harmonize-dev-prod.md
npm run harmonize:dry-run # preview DB + image changes only
```
Type `yes` when prompted (or set `autoConfirm: true` in release config). Robocopy exit codes **0–7** = success. Deploy scripts print a final **`===== SUCCESS =====`** or **`===== FAILED =====`** banner.
Documentation/Plans.md
+1 -1
View File
@@ -3,7 +3,7 @@ this file contains draft for future releases and features
1. Multy language support, russian version at least, search for best implementation, preferably story in db and easily expandable. tool to check and correct translation
2. tool to manage links (influence/influenced by ) import csv's ( define format), edit ,add, delete, visualize, map to pictures/ entities
3. tool to monitor/manage (plan actions) of curator actions, markers to check painting/text ?
4. tool to sync prod /env resources (both ways), db structure, db data, images, users etc
4. ~~tool to sync prod /env resources (both ways), db structure, db data, images, users etc~~ — done for catalog DB + images: `npm run harmonize` (schema dev→prod only; users/audit excluded) — [harmonize-dev-prod.md](harmonize-dev-prod.md)
5. curator_audit_log should contain log of actions like fixit, checked, upload etc with details for which entity it was made and details what was the action and outcome
6. ~~create search by entity (painting, artist, movement)~~ — done: timeline header + `GET /api/search`
7. create guided tours (with text/extra infor, set of entities)
Documentation/basics.md
+1
View File
@@ -471,6 +471,7 @@ See [API.md](API.md#authentication) and [data-and-images.md](data-and-images.md#
|----------|----------|
| [setup.md](setup.md) | Install, database, npm scripts |
| [deploy-dev-to-prod.md](deploy-dev-to-prod.md) | Release runbook + one-command `devtoprod:release` |
| [harmonize-dev-prod.md](harmonize-dev-prod.md) | Incremental dev ↔ prod merge (catalog DB + images) |
| [DB_structure.md](DB_structure.md) | Tables and relationships |
| [API.md](API.md) | REST endpoints |
| [data-and-images.md](data-and-images.md) | Image pipeline and seeding |
Documentation/data-and-images.md
+3 -1
View File
@@ -31,7 +31,7 @@ File names are sanitised `{Artist}_{Title}.{ext}`. The image service can redisco
| **Development** | `./data/images/` in repo | Working copy on dev PC |
| **Production** | `/mnt/BasePool/Applications/Gallery/data/images` on TrueNAS | SMB `\\192.168.10.122\Gallery\data\images` |
Promote dev → prod files: `npm run devtoprod:images` (after `net use \\192.168.10.122\Gallery`). Refresh dev from prod: `npm run prodto:dev:images`. See [environments.md](environments.md).
**One-direction promote:** `npm run devtoprod:images` (dev → prod, skip older). **Refresh dev from prod:** `npm run prodto:dev:images`. **Bidirectional merge** (newer file wins): `npm run harmonize:images` or full `npm run harmonize` — see [harmonize-dev-prod.md](harmonize-dev-prod.md). General sync reference: [environments.md](environments.md).
## Scripts overview
@@ -49,6 +49,8 @@ Promote dev → prod files: `npm run devtoprod:images` (after `net use \\192.168
| `fetch-missing-images.js` | `npm run dev:fetch-images` | Downloads files for paintings missing on disk |
| `image-fetcher.js` | *(library)* | Wikimedia / museum resolution used by fetch scripts and API |
| `sync-images-to-prod.ps1` / `sync-images-from-prod.ps1` | `npm run devtoprod:images` / `npm run prodto:dev:images` | Robocopy via SMB `\\192.168.10.122\Gallery` |
| `harmonize-db.js` / `harmonize-images.js` | `npm run harmonize:db` / `harmonize:images` | Bidirectional merge by `updated_at` / file mtime |
| `harmonize.ps1` | `npm run harmonize` | Orchestrator: backups, optional schema, DB + image merge |
| `regenerate-thumbnails.js` | `npm run dev:regenerate-thumbnails` | Rebuild painting thumbs from full images via `sharp` |
| `regenerate-portrait-thumbs.js` | `npm run dev:regenerate-portrait-thumbs` | Rebuild timeline portrait thumbs (~256px) and set `portrait_thumb_path` |
| `audit-painting-images.js` | `npm run dev:audit-painting-images` | Detect thumb/full aspect-ratio mismatches |
Documentation/deploy-dev-to-prod.md
+5 -1
View File
@@ -2,7 +2,11 @@
Step-by-step guide for promoting the **development** version of Gallery to **production**. Covers code, database schema, database data, thumbnail generation, and image files.
> **Default is dev.** This runbook is for a **scheduled release** (~weekly, or when you explicitly decide to ship). Day-to-day work stays on dev — see [environments.md](environments.md#development-first-workflow-default). If the one-time prod install is not done yet, follow [environments.md → One-time setup](environments.md#one-time-setup-full-walkthrough) and [infra/docker/DEPLOY-truenas.md](../infra/docker/DEPLOY-truenas.md) first.
> **Default is dev.** This runbook is for a **scheduled release** (~weekly, or when you explicitly decide to ship). Day-to-day work stays on dev — see [environments.md](environments.md#development-first-workflow-default).
>
> **Mid-week merge:** If both dev and prod have catalog or image edits and you need **last-write-wins** sync instead of a full prod overwrite, use [harmonize-dev-prod.md](harmonize-dev-prod.md) (`npm run harmonize`). Keep this runbook for releases where prod should exactly match dev.
>
> If the one-time prod install is not done yet, follow [environments.md → One-time setup](environments.md#one-time-setup-full-walkthrough) and [infra/docker/DEPLOY-truenas.md](../infra/docker/DEPLOY-truenas.md) first.
| | Dev (source) | Prod (target) |
|---|---|---|
Documentation/environments.md
+7
View File
@@ -305,6 +305,7 @@ See also [Drunkmeyou gitea-https-keenetic-npm-setup.md](../../Drunkmeyou/Documen
| Fast local HMR (no Keenetic) | `npm run dev:server` + `npm run dev:client` |
| Refresh dev DB from prod | `npm run prodto:dev:db` |
| Pull prod images to dev | `npm run prodto:dev:images` |
| Merge dev ↔ prod catalog + images (incremental) | `npm run harmonize` — see [harmonize-dev-prod.md](harmonize-dev-prod.md) |
---
@@ -312,6 +313,8 @@ See also [Drunkmeyou gitea-https-keenetic-npm-setup.md](../../Drunkmeyou/Documen
Run this when you are ready to ship dev to production — **not** after every small change. Typical cadence: **about once a week**.
For **incremental** dev ↔ prod merge (both sides edited), use [harmonize-dev-prod.md](harmonize-dev-prod.md) instead of full restore.
**One command (recommended):** copy [`infra/deploy/devtoprod.config.example.json`](../infra/deploy/devtoprod.config.example.json) to `infra/deploy/devtoprod.config.json`, edit it, then `npm run devtoprod:release` or `deploy-dev-to-prod.cmd`. See [deploy-dev-to-prod.md → One-command release](deploy-dev-to-prod.md#one-command-release-automated).
**Manual steps:** detailed runbook (per-change decision matrix, schema migration, rollback): [deploy-dev-to-prod.md](deploy-dev-to-prod.md).
@@ -338,11 +341,15 @@ Run this when you are ready to ship dev to production — **not** after every sm
| `npm run prodto:dev:db` | Dev PC PowerShell | Clone prod → dev |
| `npm run dev:db:backup` | Dev PC PowerShell | Dev backup |
| `npm run devtoprod:db:restore` | Dev PC PowerShell | Restore into prod |
| `npm run harmonize` | Dev PC PowerShell | Bidirectional catalog + image merge — [harmonize-dev-prod.md](harmonize-dev-prod.md) |
| `npm run harmonize:db` | Dev PC PowerShell | DB merge only |
| `npm run harmonize:images` | Dev PC PowerShell | Image merge only |
## Image sync
| Command | Where | Direction |
|---------|-------|-----------|
| `npm run harmonize` | Dev PC PowerShell | Bidirectional merge (mtime newer wins) — [harmonize-dev-prod.md](harmonize-dev-prod.md) |
| `npm run devtoprod:release` | Dev PC PowerShell | Full config-driven promote (see [deploy-dev-to-prod.md](deploy-dev-to-prod.md#one-command-release-automated)) |
| `npm run devtoprod:thumbnails` | Dev PC PowerShell | Rebuild painting + portrait thumbs on dev before promote |
| `npm run devtoprod:images` | Dev PC PowerShell | Dev → TrueNAS volume |
Documentation/harmonize-dev-prod.md
+173
View File
@@ -0,0 +1,173 @@
# Harmonize dev and prod (incremental merge)
Bidirectional **catalog data** and **image** sync between `gallery_dev` and `gallery_prod`, with **last-write-wins** by timestamp. Use this when **both** environments may have curator edits since the last release — not when prod should become an exact copy of dev.
For a full prod replace (weekly release), use [deploy-dev-to-prod.md](deploy-dev-to-prod.md) (`npm run devtoprod:release`).
For refreshing dev from prod entirely, use `npm run prodto:dev:db` (destructive to dev).
---
## Rules
| Layer | Direction | Conflict resolution |
|-------|-----------|---------------------|
| **Schema** | dev → prod only | Run `npm run harmonize:schema` (same as `dev:migrate` on `gallery_prod`) |
| **Catalog DB** | dev ↔ prod | Newer `updated_at` wins; missing rows copied to the other side (union merge) |
| **Images** | dev ↔ prod | Newer file mtime wins; missing files copied to the other side |
| **Users / sessions / audit** | not synced | `users`, `session`, `curator_audit_log` stay env-local |
**Never auto-deletes** rows or files that exist on only one side.
---
## Prerequisites
1. Both databases on Postgres `192.168.10.122` with `updated_at` columns applied:
```powershell
npm run dev:migrate
$env:DB_NAME = "gallery_prod"; npm run dev:migrate; Remove-Item Env:\DB_NAME
```
2. Backfill `updated_at` from image file mtimes (recommended once after migration):
```powershell
npm run dev:backfill-updated-at
npm run harmonize:backfill-updated-at
```
3. [`infra/docker/.env.prod`](../infra/docker/.env.prod) present with `DB_NAME=gallery_prod`.
4. SMB share reachable for prod images:
```powershell
net use \\192.168.10.122\Gallery /user:YOUR_TRUENAS_USER
```
5. Copy harmonize config:
```powershell
Copy-Item infra/deploy/harmonize.config.example.json infra/deploy/harmonize.config.json
```
Edit `harmonize.config.json` (gitignored) — optional `smb.user` / `smb.password`, `prefer` for tie-breaks (`dev` | `prod`), `schemaChanged: true` when new migrations shipped.
---
## One-command harmonize
```powershell
npm run harmonize
```
Dry-run (report only, no writes):
```powershell
npm run harmonize -- -DryRun
```
Or:
```powershell
npm run harmonize:dry-run
```
### Orchestrator steps
| Step | npm script | Purpose |
|------|------------|---------|
| `backupDev` | `dev:db:backup` | Safety snapshot |
| `backupProd` | `prod:db:backup` | Safety snapshot |
| `schema` | `harmonize:schema` | Apply dev migrations to prod (when `schemaChanged: true`) |
| `db` | `harmonize:db` | Row-level catalog merge |
| `images` | `harmonize:images` | Bidirectional file merge |
| `verify` | curl `/api/bounds` | Optional smoke check |
Reports are written to `db/SyncReports/harmonize_db_*.json` and `harmonize_images_*.json` (gitignored).
---
## Individual commands
| Command | Purpose |
|---------|---------|
| `npm run harmonize:schema` | Migrate prod schema from dev migration files |
| `npm run harmonize:db` | Merge catalog rows by `updated_at` |
| `npm run harmonize:db -- --dry-run` | Preview DB changes |
| `npm run harmonize:db -- --prefer=dev` | On equal `updated_at`, dev wins |
| `npm run harmonize:images` | Merge image files by mtime |
| `npm run dev:migrate:sync-timestamps` | Apply `updated_at` migration on dev only |
| `npm run dev:backfill-updated-at` | Backfill dev `updated_at` from image mtimes |
| `npm run harmonize:backfill-updated-at` | Same backfill on prod |
---
## Catalog tables synced
Processed in FK order:
`historical_eras` → `art_movements` → `artists` → `artist_periods` → `paintings` → `painting_influences` → `painting_influence_sources` → `painting_annotations`
---
## Conflict handling
Harmonize reports conflicts in the JSON report and skips those rows:
| Conflict | Cause | Resolution |
|----------|-------|------------|
| `id_collision` | Same `id` but different natural key (e.g. artist name) | Manual fix in pgAdmin; environments diverged too far |
| `equal_updated_at` | Same timestamp, different row content | Re-run with `--prefer=dev` or `--prefer=prod`, or edit one side and re-run |
**Tip:** Harmonize regularly from a shared baseline (e.g. after each weekly release) to avoid ID/natural-key collisions from independent inserts.
---
## When to use what
| Situation | Tool |
|-----------|------|
| Weekly release — prod should match dev exactly | `npm run devtoprod:release` |
| Mid-week prod curator fix + dev also changed | `npm run harmonize` |
| Dev workspace stale — full prod copy | `npm run prodto:dev:db` |
| New migration in repo | `harmonize:schema` or deploy step 4 |
| Only images changed on one side | `npm run harmonize:images` |
| Only DB metadata changed | `npm run harmonize:db` |
---
## Example flows
### Prod curator uploaded a painting; dev also edited metadata
```powershell
npm run harmonize
```
DB rows merge by `updated_at`; image files merge by mtime. Both sides receive the latest version of each entity.
### Schema change + mixed edits
1. Finish and test on dev: `npm run dev:migrate`
2. Set `schemaChanged: true` in `harmonize.config.json` (or enable `steps.schema`)
3. `npm run harmonize`
### Preview before writing
```powershell
npm run harmonize:dry-run
# Review db/SyncReports/harmonize_*.json
npm run harmonize
```
---
## Safety
- Pre-flight backups of dev and prod DB (configurable; on by default)
- Prod writes require `yes` or `CONFIRM_PROD=1` (orchestrator sets `CONFIRM_PROD=1` when `autoConfirm`-style run)
- No TRUNCATE — harmonize only inserts/updates changed rows
- Rollback: restore from `db/DataBackup/gallery_*_data_*.txt` using `dev:db:restore` or `devtoprod:db:restore`
See also [environments.md](environments.md) and [deploy-dev-to-prod.md](deploy-dev-to-prod.md).
Documentation/setup.md
+2 -1
View File
@@ -110,13 +110,14 @@ Image fetch can take hours if you run it for the entire catalog. The first line
| `npm run prodto:dev:images` | Copy prod images → dev repo |
| `npm run prodto:dev:db` | Clone `gallery_prod` → `gallery_dev` |
| `npm run dev:db:backup` / `devtoprod:db:restore` | Dev backup / promote DB to prod |
| `npm run harmonize` | Bidirectional catalog + image merge (last-write-wins) — [harmonize-dev-prod.md](harmonize-dev-prod.md) |
| `npm run prod:build` | Build production SPA into `client/dist` |
| `npm run dev:start` | API + static SPA on `HOST`:`PORT` (uses root `.env`) |
| `npm run prod:start` | Build client, then start server |
| `npm run dev:server` | API with nodemon reload (local HMR workflow) |
| `npm run dev:client` | Vite dev server on :5173 |
See [environments.md](environments.md) for dev/prod URLs, database split, Docker deploy, and sync commands. Quick reference: [FAC.md](FAC.md).
See [environments.md](environments.md) for dev/prod URLs, database split, Docker deploy, and sync commands. For incremental dev ↔ prod merge (both sides edited), see [harmonize-dev-prod.md](harmonize-dev-prod.md). Quick reference: [FAC.md](FAC.md).
**Production frontend:** build the client, then start the server: