Art-gallery/Documentation/setup.md

# Art Gallery — setup and operations

## Prerequisites

- **Node.js** 20+
- **PostgreSQL** reachable from the app host
- Network access to Wikipedia / Wikimedia (for biographies, seeding, and on-demand image fetch)

## Environment

**Development-first:** The root `.env` is for **dev only** (`DB_NAME=gallery_dev`). All routine work — code, migrations, seeds, image scripts, curator edits — targets dev. Production uses `infra/docker/.env.prod` and TrueNAS; promote on a scheduled release (~weekly), not after every change. See [environments.md — Development-first workflow](environments.md#development-first-workflow-default).

Copy the example file and fill in credentials:

```bash
cp .env.example .env
```

| Variable | Description |
|----------|-------------|
| `DB_HOST` | PostgreSQL host |
| `DB_PORT` | Port (default `5432`) |
| `DB_USER` | Database user |
| `DB_PASSWORD` | Database password |
| `DB_NAME` | Database name (`gallery_dev` for dev; prod uses `gallery_prod`) |
| `PORT` | API listen port (`3451` for `dev:web`; prod container uses `5173`) |
| `HOST` | Bind address (default `0.0.0.0` — required for LAN access) |
| `PUBLIC_URL` | Public URL (dev: `https://devgallery.mysuperlab.netcraze.pro`; prod: `https://gallery.mysuperlab.netcraze.pro`) |
| `TRUST_PROXY` | Set to `true` when behind nginx/reverse proxy (honours `X-Forwarded-*`) |
| `IMAGE_DIR` | Root for cached images (default `./data/images`) |
| `SESSION_SECRET` | Random string for signed session cookies (required for curator login) |
| `SESSION_COOKIE_SECURE` | `false` for local HTTP dev; `true` in prod behind HTTPS |
| `CURATOR_USERNAME` | Bootstrap only — first curator account name (default `curator`) |
| `CURATOR_PASSWORD` | Bootstrap only — password for first curator when `users` table is empty |

`.env` is git-ignored; never commit passwords.

Optional script tuning:

| Variable | Used by | Description |
|----------|---------|-------------|
| `MIN_PAINTINGS` | `expand-catalog` | Minimum paintings per artist (default `6`) |
| `FETCH_MAX_WAIT_SEC` | `fetch-images` | Max seconds per painting in batch runs (default `10`) |

## Install

```bash
npm install
cd client && npm install && cd ..
```

## Database bootstrap

One-shot setup (migrate + seed):

```bash
npm run dev:setup
```

Or step by step:

```bash
npm run dev:migrate    # db/schema.sql + db/migrate-*.sql via server/migrate.js
npm run dev:seed       # eras, movements, artists, flagship paintings (one per artist)
```

`npm run dev:migrate` is safe to re-run on existing databases (uses `IF NOT EXISTS` / `ADD COLUMN IF NOT EXISTS`).

If migration fails with permission errors, grant schema rights to the app user first (see [DB_structure.md](DB_structure.md)).

### Curator accounts (auth migration)

`npm run dev:migrate` applies `db/migrate-auth.sql` (`users`, `curator_audit_log`, `session` tables). When the `users` table is empty and `CURATOR_USERNAME` / `CURATOR_PASSWORD` are set in `.env`, the first curator account is created automatically.

After migrate, sign in from the site header (**Curator login**). Debug mode, Checkup, and all mutating debug APIs require an active curator session. Anonymous visitors browse the timeline and 3D halls without logging in.

See [API.md — Authentication](API.md#authentication) and [basics.md — Developer tools](basics.md#developer-tools-image-audit).

### Recommended post-seed steps

After a fresh seed, run these to match a fully populated local install:

```bash
npm run dev:sync-image-paths           # import paintings from data/images/paintings/ (clone with image files)
npm run dev:fetch-artist-images        # link local portraits or download from Wikipedia
npm run dev:fetch-artist-bios          # bio_short / bio_full from Wikipedia
npm run dev:expand-catalog             # famous works for artists below MIN_PAINTINGS
npm run dev:update-influences          # painting influence graph for detail view + hall exits
npm run dev:migrate:checkup-flags      # optional: review/fixed flags for Checkup page (paintings)
npm run dev:migrate:artist-checkup-flags  # optional: same flags for artist portraits (bio debug)
npm run dev:migrate:search               # optional on very old DBs — also applied by dev:migrate / prod Step 4
npm run dev:migrate:painting-annotations  # optional: art-history notes table
npm run dev:update-painting-annotations   # optional: load curated notes (+ --wikipedia for Wikipedia intros)
npm run dev:fetch-images -- --limit=50                # random sample; 10s max per painting (default)
npm run dev:fetch-images -- --limit=50 --max-wait=120 # same batch size, longer lookup per work
cd client && npm run build && cd ..
```

**Order matters:** `dev:sync-image-paths` should run when the repo already contains painting files under `data/images/paintings/` but the database only has one flagship work per artist (typical after `npm run dev:setup` on a clone). `dev:update-influences` is required for *Influenced By* / *Influenced* panels and golden lamps in the 3D hall — seed does not insert influence edges.

Image fetch can take hours if you run it for the entire catalog. The first line of each run reports **`Missing local files: N`**. Use **`npm run dev:fetch-images -- --limit=N`** for random batches (10s per painting by default), **`--artist="…"`** for one artist in catalog order, or on-demand resolution when viewing a painting in the detail view.

## Run

| Command | Purpose |
|---------|---------|
| `npm run dev:web` | **Public dev stack** — Vite `:5173`, API `:3451` (Keenetic → devgallery…) |
| `npm run prod:docker:publish` | Build + push prod image to Gitea |
| `npm run devtoprod:images` | Copy `data/images/` → TrueNAS via SMB `Gallery` share |
| `npm run prodto:dev:images` | Copy prod images → dev repo |
| `npm run prodto:dev:db` | Clone `gallery_prod` → `gallery_dev` |
| `npm run dev:db:backup` / `devtoprod:db:restore` | Dev backup / promote DB to prod |
| `npm run prod:build` | Build production SPA into `client/dist` |
| `npm run dev:start` | API + static SPA on `HOST`:`PORT` (uses root `.env`) |
| `npm run prod:start` | Build client, then start server |
| `npm run dev:server` | API with nodemon reload (local HMR workflow) |
| `npm run dev:client` | Vite dev server on :5173 |

See [environments.md](environments.md) for dev/prod URLs, database split, Docker deploy, and sync commands. Quick reference: [FAC.md](FAC.md).

**Production frontend:** build the client, then start the server:

```bash
npm run prod:build
        # or: cd client && npm run build && cd ..
npm run dev:start
```

Open http://localhost:5173 (Vite) or http://localhost:3451 (API only).

## Production deployment

Production runs as **`gallery-web`** on TrueNAS at **https://gallery.mysuperlab.netcraze.pro** (Keenetic → `:5173`). Full guide: [environments.md](environments.md) and [infra/docker/DEPLOY-truenas.md](../infra/docker/DEPLOY-truenas.md).

| Access | URL |
|--------|-----|
| Production (Keenetic) | https://gallery.mysuperlab.netcraze.pro |
| Development (Keenetic) | https://devgallery.mysuperlab.netcraze.pro |
| LAN direct (prod container) | http://192.168.10.122:5173 |
| LAN direct (dev PC) | http://192.168.10.70:5173 |

Quick deploy checklist:

1. One-time DB split in **pgAdmin** on dev PC: [`db/split-dev-prod-pgadmin.sql`](../db/split-dev-prod-pgadmin.sql)
2. Dev `.env` → `DB_NAME=gallery_dev`, `PORT=3451`, `PUBLIC_URL=https://devgallery.mysuperlab.netcraze.pro`, plus auth vars (`SESSION_SECRET`, `CURATOR_*`)
3. `npm run dev:migrate` on dev and prod DBs (includes auth tables + bootstrap curator)
4. `net use \\192.168.10.122\Gallery` → `npm run devtoprod:images`
5. `npm run prod:docker:publish` → TrueNAS Custom App from `infra/docker/compose.truenas.yaml` (set auth env in compose)
6. Keenetic: both domains → `:5173`, protocol to device **`http`**, correct IP per environment

### Legacy deployment (optional)

Node on the dev PC at `:3520` with nginx → Vite `:5173` is superseded by TrueNAS Docker prod and `npm run dev:web` for public dev. See [`deploy/nginx-gallery.conf`](../deploy/nginx-gallery.conf) and [`deploy/gallery.service`](../deploy/gallery.service) only if you need a local nginx/systemd setup.

## Maintenance scripts

| Command | Script | Purpose |
|---------|--------|---------|
| `npm run dev:seed` | `scripts/seed-wikipedia.js` | Reload curated Wikipedia data |
| `npm run dev:fetch-artist-bios` | `scripts/fetch-artist-bios.js` | Wikipedia lead sections → `bio_short` / `bio_full` |
| `npm run dev:fetch-artist-bios -- --force` | ↑ | Refresh bios even when already set |
| `npm run dev:expand-catalog` | `scripts/expand-paintings.js` | Insert famous works from `famous-paintings-data.js` |
| `npm run dev:expand-catalog -- --fetch-images` | ↑ | Also download images for new rows (slow) |
| `npm run dev:fetch-images` | `scripts/fetch-missing-images.js` | Search Wikipedia, Commons, museums for missing files |
| `npm run dev:search-missing-paintings` | ↑ (alias) | Same as `fetch-images` |
| `npm run dev:fetch-images -- --artist="Name"` | ↑ | Limit to one artist (catalog order) |
| `npm run dev:fetch-images -- --limit=50` | ↑ | Random sample of N missing paintings |
| `npm run dev:fetch-images -- --limit=250` | ↑ | Larger random batch (processes min(N, missing count)) |
| `npm run dev:fetch-images -- --limit=50 --max-wait=120` | ↑ | Random batch; 120s cap per painting |
| `npm run dev:fetch-images -- --discover-only --limit=20` | ↑ | Fix `wikipedia_title` via search only |
| `npm run dev:regenerate-thumbnails` | `scripts/regenerate-thumbnails.js` | Rebuild all thumbs from full local files |
| `npm run dev:audit-painting-images` | `scripts/audit-painting-images.js` | List thumb/full aspect-ratio mismatches |
| `npm run dev:migrate:thumbnails` | `db/migrate-thumbnails.sql` | Add thumbnail columns |
| `npm run dev:fetch-artist-images` | `scripts/fetch-artist-images.js` | Download portraits or link existing files under `data/images/portraits/` |
| `npm run dev:fetch-artist-images -- --force` | ↑ | Re-fetch even when `portrait_path` is set |
| `npm run dev:sync-image-paths` | `scripts/sync-image-paths.js` | Link `image_path` / `thumbnail_path` and import missing painting rows from disk |
| `npm run dev:sync-image-paths -- --dry-run` | ↑ | Report only, no DB writes |
| `npm run dev:migrate:influence-sources` | `scripts/migrate-influence-sources.js` | Create `painting_influence_sources` + backfill legacy edges |
| `npm run dev:migrate:checkup-flags` | `scripts/migrate-checkup-flags.js` | Add `checkup_checked` / `checkup_fixed` on `paintings` |
| `npm run dev:migrate:artist-checkup-flags` | `scripts/migrate-artist-checkup-flags.js` | Add `checkup_checked` / `checkup_fixed` on `artists` (bio debug) |
| `npm run dev:migrate:search` | `scripts/migrate-search.js` | Same indexes as `migrate-search.sql` (also run by `dev:migrate`) |
| `npm run dev:migrate:painting-annotations` | `scripts/migrate-painting-annotations.js` | Create `painting_annotations` table |
| `npm run dev:migrate:artist-palette` | `scripts/migrate-artist-palette.js` | Add `palette_metadata` JSONB on `artists` |
| `npm run dev:import-painter-palette` | `scripts/import-painter-palette.js` | Enrich artists + influence links from `Inputs/PainterPalette.csv` |
| `npm run dev:analyze-painter-palette` | `scripts/analyze-painter-palette.js` | Report CSV ↔ gallery artist name matches |
| `npm run dev:export-paintings` | `scripts/export-paintings-csv.js` | Write `Output/paintings.csv` (artist, painting, year) |
| `npm run dev:update-painting-annotations` | `scripts/update-painting-annotations.js` | Load curated notes from `painting-annotations-data.js` |
| `npm run dev:update-painting-annotations -- --wikipedia` | ↑ | Add intro sentences from each work’s Wikipedia page |
| `npm run dev:update-painting-annotations -- --wikipedia --wiki-delay=3000` | ↑ | Slower Wikipedia pass when rate-limited (429) |
| `npm run dev:find-duplicates` | `scripts/find-duplicate-paintings.js` | Report duplicate and near-duplicate painting rows |
| `npm run dev:audit-influence-duplicates` | `scripts/audit-influence-duplicates.js` | Report mirrored legacy/sources edges and duplicate influence rows |
| `npm run dev:update-influences` | `scripts/update-influences.js` | Insert influence links (painting / artist / movement) from `art-influences-data.js` |
| `npm run dev:update-influences -- --fetch-images` | ↑ | Also download images for newly created works |
| `npm run dev:update-influences -- --discover` | ↑ | Curated pass + web discovery (Wikipedia, Wikidata, Met, art-history sites) |
| `npm run dev:discover-influences` | `update-influences.js --discover-only` | Discovery pass only |

### Scripts in the repository

These are checked in and maintained:

- `seed-wikipedia.js`, `seed-catalog-data.js` — initial catalog (`npm run dev:seed`)
- `image-fetcher.js` — Wikimedia / museum image resolution
- `fetch-artist-images.js` — artist portrait download / disk linking
- `sync-image-paths.js` — import paintings and align paths from `data/images/paintings/`
- `fetch-artist-bios.js` — artist biographies
- `expand-paintings.js` + `famous-paintings-data.js` — catalog expansion
- `update-influences.js` + `art-influences-data.js` — influence graph (paintings, artists, movements)
- `influence-discovery.js` + `influence-resolver.js` — web discovery and polymorphic source resolution
- `fetch-missing-images.js` — batch image backfill
- `find-duplicate-paintings.js` — duplicate catalog audit
- `audit-influence-duplicates.js` — influence graph duplicate / mirror audit
- `painter-palette-lib.js`, `import-painter-palette.js`, `migrate-artist-palette.js`, `analyze-painter-palette.js` — PainterPalette CSV integration
- `export-paintings-csv.js` — catalog CSV export to `Output/`
- `migrate-checkup-flags.js` — checkup workflow columns (paintings)
- `migrate-artist-checkup-flags.js` — checkup workflow columns (artist portraits)
- `migrate-painting-annotations.js`, `update-painting-annotations.js`, `painting-annotations-data.js` — art-history notes on painting detail
- `regenerate-thumbnails.js`, `audit-painting-images.js`

See [data-and-images.md](data-and-images.md) for pipeline details and override maps.

## Remote repository

Gitea: [Danilka/Art-gallery](https://gitea.mysuperlab.netcraze.pro/Danilka/Art-gallery)

```bash
git clone https://gitea.mysuperlab.netcraze.pro/Danilka/Art-gallery.git
```

After clone: copy `.env.example` → `.env`, install dependencies, run [one-time DB split](Documentation/environments.md#step-b--split-the-database-pgadmin), then post-seed steps. Deploy: [environments.md](Documentation/environments.md).

## Troubleshooting

| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| Empty timeline | DB not seeded | `npm run dev:seed` |
| 502 / 504 on public URL | Keenetic rule wrong (IP, port, or `https` to device) | Dev → `192.168.10.70:5173`; prod → `192.168.10.122:5173`; protocol **`http`** — see [environments.md](environments.md) |
| 503 on public URL | Dev servers not running | `npm run dev:web` (or `dev:server` + `dev:client` for local HMR) |
| **`manifest unknown`** on TrueNAS deploy | Image not in Gitea | `npm run prod:docker:publish` on dev PC first |
| 500 on all `/api/*` | Wrong `.env` or Postgres down | Check connection, logs |
| “Biographical information not yet available” | Bios not fetched | `npm run dev:fetch-artist-bios` |
| Placeholder portraits on timeline | `portrait_path` not set | `npm run dev:fetch-artist-images` |
| Artist hall has only 1–2 paintings | DB not expanded / disk not imported | `npm run dev:sync-image-paths` then `npm run dev:expand-catalog`; extend `famous-paintings-data.js` |
| Movement flow shows “Loading art history…” repeatedly | Stale client refetching on every pan/zoom | Pull latest client; timeline loads catalog once — hard-refresh |
| Movement flow slow on first load | Large artist payload | Client uses `GET /api/artists?timeline=1` (no full bios); rebuild client |
| Movement flow zoom sluggish / dead over portraits | Stale client build | Rebuild client — wheel uses capture listener + interaction-mode rendering; hard-refresh |
| Timeline year labels overlap when zoomed out | Stale client | Rebuild client — `chooseTimelineTickInterval()` adapts step to span and bar width |
| Black frames / canvas covers in 3D gallery | No local image for painting | `npm run dev:fetch-images -- --limit=50` or `--artist="…"`; then `POST …/preload-images` |
| Many `⏱ timeout` lines in fetch batch | Default 10s cap too short for hard works | `--max-wait=120` or raise `FETCH_MAX_WAIT_SEC` |
| White/grey flicker on frames | Texture loading or z-fighting with wall | Rebuild client (`cd client && npm run build`); ensure latest `VirtualGallery.tsx` |
| Wrong painting in 3D gallery frame | Stale or mismatched thumbnail file | `npm run dev:regenerate-thumbnails`; gallery prefers full `image_path` |
| Wrong painting image (fetch) | Bad museum / search match on first download | Add entry to `DIRECT_IMAGE_OVERRIDES` in `scripts/image-fetcher.js`, re-fetch file |
| Empty exit navigation lists | No influence edges for artist | `npm run dev:update-influences`; check `painting_influence_sources` |
| Empty *Influenced By* / *Influenced* on painting detail | No edges for that work | `npm run dev:migrate:influence-sources` then `npm run dev:update-influences`; extend `art-influences-data.js` or run `--discover` |
| Same work listed twice under *Influenced* | Stale server merging legacy + sources tables | Restart server; API reads `painting_influence_sources` only — run `npm run dev:audit-influence-duplicates` to verify DB |
| No golden lamps above frames in 3D hall | Stale API process or no influence edges | Restart server after API changes; run `npm run dev:update-influences` |
| Default Vite page instead of gallery | `client/dist` missing or stale | `cd client && npm run build` |
| Permission denied creating tables | `gallery` user lacks CREATE | Run admin grants, then migrate |
| Wikipedia API rate limit during fetch | Too many requests in a row | Wait and re-run; scripts retry with backoff |
| Checkup **Reviewed** toggle returns 404 | Stale server process missing new routes | Restart `npm run dev:web` or `npm run dev:server` after pulling API changes |
| Debug **More** / **Clear** / **Upload** / **Remove entry** returns 404 | Stale server process | Restart `npm run dev:start` or `npm run dev:server`; routes in `server/index.js` + `server/image-service.js` |
| Debug **Remove entry** — button stuck or missing on next painting | Stale client build | `cd client && npm run build`; hard-refresh — detail view remounts per painting id |
| Debug **Upload** returns 413 Payload Too Large | Base64 JSON exceeds body limit | Server allows 20 MB JSON / 15 MB decoded image; compress file or resize before upload |
| Debug **Upload** — nothing happens after choosing file | Stale client or broken hidden-file `click()` | Pull latest client (`DebugUploadButton` uses `<label>` + `<input>`); hard-refresh |
| Uploaded image reverts after reload (old picture) | Browser cached `/images/…` at same path | Pull latest server + client — API returns `image_cache_key` and URLs use `?v=`; `/images` no longer uses immutable long-term cache |
| No art-history notes on painting detail | Annotations not migrated or loaded | `npm run dev:migrate:painting-annotations` then `npm run dev:update-painting-annotations` |
| **Fix it** fails with `read ECONNRESET` | Remote host dropped connection | Restart server; client sends `searchUrl` / `source`; retry or use Commons URL in overrides |
| Fixed image not shown in 3D gallery | Stale gallery session or cached texture | Rebuild client; fix/upload updates session + `?v=` from `image_cache_key` — use **Back to Gallery** (not browser back) |
| **Back to Timeline** returns to gallery / previous wing | Stale client build | Pull latest client — `goToTimelineHome()` unmounts the hall and resets timeline zoom |
| Catalog search dropdown hidden under timeline | Stale client CSS | Rebuild client — `.site-header` uses `z-index: 110` above the sticky timeline bar |
| Catalog search returns empty / 500 | Search indexes missing | Run `npm run dev:migrate` (includes `migrate-search.sql`) or `npm run dev:migrate:search`; restart API |
| Frame still black after **Checked** | Gallery session not synced | Re-enter hall or toggle debug **Checked** from detail with gallery open behind overlay |
| Duplicate works in gallery / timeline | Double import or variant Wikipedia titles | `npm run dev:find-duplicates`; merge or delete spare rows manually |
| **Failed to load movement gallery** / `Cannot GET /api/movements/:id/gallery` | Stale server process missing route | Restart `npm run dev:web` or `npm run dev:server` after pulling API changes |
| Movement gallery shows generic cream walls | Stale client build | `cd client && npm run build`; hard-refresh browser |
| Windows overlap paintings in movement wing | Stale client | Rebuild client — windows are placed only on side walls in gaps between frames |
| Influence thumbnails cropped on painting detail | Stale client build | `npm run prod:build` — panels use `object-fit: contain` for full image |
| **Curator login** fails / always guest | Auth tables missing or wrong password | Set `SESSION_SECRET` + `CURATOR_PASSWORD` in `.env`, run `npm run dev:migrate`, restart server |
| Debug / Checkup returns **401** | Not signed in as curator | **Curator login** (top-right); session cookie `gallery.sid` must be sent (`credentials: include`) |
| Debug works in UI but API rejects | Stale server without auth middleware | Restart `npm run dev:web` or `npm run dev:server` after pulling auth changes |
| **Empty screen** entering 3D hall (header missing) | Stale client before gallery-session fix | Hard-refresh; pull latest client — hall renders from `view` state, not only `gallerySession` |
| **Dark center** entering 3D hall (header visible, no spinner) | Stale client before gallery loading overlay fix | Hard-refresh; latest client shows **Loading gallery…** / **Loading paintings…** in the canvas area until ready |
| 3D hall black after returning from painting detail | WebGL context lost while hall was hidden | Hard-refresh; latest client remounts canvas when hall becomes active again |