Art-gallery/Documentation/setup.md

# Art Gallery — setup and operations

## Prerequisites

- **Node.js** 20+
- **PostgreSQL** reachable from the app host
- Network access to Wikipedia / Wikimedia (for biographies, seeding, and on-demand image fetch)

## Environment

**Development-first:** The root `.env` is for **dev only** (`DB_NAME=gallery_dev`). All routine work — code, migrations, seeds, image scripts, curator edits — targets dev. Production uses `infra/docker/.env.prod` and TrueNAS; promote on a scheduled release (~weekly), not after every change. See [environments.md — Development-first workflow](environments.md#development-first-workflow-default).

Copy the example file and fill in credentials:

```bash
cp .env.example .env
```

| Variable | Description |
|----------|-------------|
| `DB_HOST` | PostgreSQL host |
| `DB_PORT` | Port (default `5432`) |
| `DB_USER` | Database user |
| `DB_PASSWORD` | Database password |
| `DB_NAME` | Database name (`gallery_dev` for dev; prod uses `gallery_prod`) |
| `PORT` | API listen port (`3451` for `dev:web`; prod container uses `5173`) |
| `HOST` | Bind address (default `0.0.0.0` — required for LAN access) |
| `PUBLIC_URL` | Public URL (dev: `https://devgallery.mysuperlab.netcraze.pro`; prod: `https://gallery.mysuperlab.netcraze.pro`) |
| `TRUST_PROXY` | Set to `true` when behind nginx/reverse proxy (honours `X-Forwarded-*`) |
| `IMAGE_DIR` | Root for cached images (default `./data/images`) |
| `SESSION_SECRET` | Random string for signed session cookies (required for curator login) |
| `SESSION_COOKIE_SECURE` | Optional — omit for auto (HTTPS via proxy → Secure); set `true`/`false` to force |
| `CURATOR_USERNAME` | Bootstrap / reset — curator account name (default `curator`) |
| `CURATOR_PASSWORD` | Bootstrap when `users` is empty; also used by `npm run dev:reset-curator` |

`.env` is git-ignored; never commit passwords.

Optional script tuning:

| Variable | Used by | Description |
|----------|---------|-------------|
| `MIN_PAINTINGS` | `expand-catalog` | Minimum paintings per artist (default `6`) |
| `FETCH_MAX_WAIT_SEC` | `fetch-images` | Max seconds per painting in batch runs (default `10`) |

## Install

```bash
npm install
cd client && npm install && cd ..
```

## Database bootstrap

One-shot setup (migrate + seed):

```bash
npm run dev:setup
```

Or step by step:

```bash
npm run dev:migrate    # db/schema.sql + db/migrate-*.sql via server/migrate.js
npm run dev:seed       # eras, movements, artists, flagship paintings (one per artist)
```

`npm run dev:migrate` is safe to re-run on existing databases (uses `IF NOT EXISTS` / `ADD COLUMN IF NOT EXISTS`).

If migration fails with permission errors, grant schema rights to the app user first (see [DB_structure.md](DB_structure.md)).

### Curator accounts (auth migration)

`npm run dev:migrate` applies `db/migrate-auth.sql` (`users`, `curator_audit_log`, `session` tables). When the `users` table is empty and `CURATOR_USERNAME` / `CURATOR_PASSWORD` are set in `.env`, the first curator account is created automatically. To sync the password from `.env` later, run `npm run dev:reset-curator`.

After migrate, sign in from the site header (**Curator login**). Debug mode, Checkup, Translations, Influences, Tour editor, and all mutating debug APIs require an active curator session. Anonymous visitors browse the timeline, published Tours, and 3D halls without logging in.

See [API.md — Authentication](API.md#authentication) and [basics.md — Developer tools](basics.md#developer-tools-image-audit).

### Recommended post-seed steps

After a fresh seed, run these to match a fully populated local install:

```bash
npm run dev:sync-image-paths           # import paintings from data/images/paintings/ (clone with image files)
npm run dev:fetch-artist-images        # link local portraits or download from Wikipedia
npm run dev:fetch-artist-bios          # bio_short / bio_full from Wikipedia
npm run dev:expand-catalog             # famous works for artists below MIN_PAINTINGS
npm run dev:update-influences          # painting influence graph for detail view + hall exits
npm run dev:migrate:checkup-flags      # optional: review/fixed flags for Checkup page (paintings)
npm run dev:migrate:artist-checkup-flags  # optional: same flags for artist portraits (bio debug)
npm run dev:migrate:search               # optional on very old DBs — also applied by dev:migrate / prod Step 4
npm run dev:migrate:i18n                 # optional — entity_translations (also in dev:migrate)
npm run dev:fetch-artist-bios-ru         # draft Russian bios + Cyrillic names from ru.wikipedia
npm run dev:import-translations -- --file path/to/file.json  # bulk translation import
npm run dev:migrate:painting-annotations  # optional: art-history notes table
npm run dev:update-painting-annotations   # optional: load curated notes (+ --wikipedia for Wikipedia intros)
npm run dev:fetch-images -- --limit=50                # random sample; 10s max per painting (default)
npm run dev:fetch-images -- --limit=50 --max-wait=120 # same batch size, longer lookup per work
cd client && npm run build && cd ..
```

**Order matters:** `dev:sync-image-paths` should run when the repo already contains painting files under `data/images/paintings/` but the database only has one flagship work per artist (typical after `npm run dev:setup` on a clone). `dev:update-influences` is required for *Influenced By* / *Influenced* panels and golden lamps in the 3D hall — seed does not insert influence edges.

Image fetch can take hours if you run it for the entire catalog. The first line of each run reports **`Missing local files: N`**. Use **`npm run dev:fetch-images -- --limit=N`** for random batches (10s per painting by default), **`--artist="…"`** for one artist in catalog order, or on-demand resolution when viewing a painting in the detail view.

## Run

| Command | Purpose |
|---------|---------|
| `npm run dev:web` | **Public dev stack** — Vite `:5173`, API `:3451` (Keenetic → devgallery…) |
| `npm run prod:docker:publish` | Build + push prod image to Gitea |
| `npm run devtoprod:images` | Copy `data/images/` → TrueNAS via SMB `Gallery` share |
| `npm run prodto:dev:images` | Copy prod images → dev repo |
| `npm run prodto:dev:db` | Clone `gallery_prod` → `gallery_dev` |
| `npm run dev:db:backup` / `devtoprod:db:restore` | Dev backup / promote DB to prod |
| `npm run harmonize` | Bidirectional catalog + image merge (last-write-wins) — [harmonize-dev-prod.md](harmonize-dev-prod.md) |
| `npm run prod:build` | Build production SPA into `client/dist` |
| `npm run dev:start` | API + static SPA on `HOST`:`PORT` (uses root `.env`) |
| `npm run prod:start` | Build client, then start server |
| `npm run dev:server` | API with nodemon reload (local HMR workflow) |
| `npm run dev:client` | Vite dev server on :5173 |

See [environments.md](environments.md) for dev/prod URLs, database split, Docker deploy, and sync commands. For Russian UI + catalog text, see [i18n-russian.md](i18n-russian.md). Influence link import/CRUD: [influence-import.md](influence-import.md). Quick reference: [FAC.md](FAC.md).

**Production frontend:** build the client, then start the server:

```bash
npm run prod:build
        # or: cd client && npm run build && cd ..
npm run dev:start
```

Open http://localhost:5173 (Vite) or http://localhost:3451 (API only).

## Production deployment

Production runs as **`gallery-web`** on TrueNAS at **https://gallery.mysuperlab.netcraze.pro** (Keenetic → `:5173`). Full guide: [environments.md](environments.md) and [infra/docker/DEPLOY-truenas.md](../infra/docker/DEPLOY-truenas.md).

| Access | URL |
|--------|-----|
| Production (Keenetic) | https://gallery.mysuperlab.netcraze.pro |
| Development (Keenetic) | https://devgallery.mysuperlab.netcraze.pro |
| LAN direct (prod container) | http://192.168.10.122:5173 |
| LAN direct (dev PC) | http://192.168.10.70:5173 |

Quick deploy checklist:

1. One-time DB split in **pgAdmin** on dev PC: [`db/split-dev-prod-pgadmin.sql`](../db/split-dev-prod-pgadmin.sql)
2. Dev `.env` → `DB_NAME=gallery_dev`, `PORT=3451`, `PUBLIC_URL=https://devgallery.mysuperlab.netcraze.pro`, plus auth vars (`SESSION_SECRET`, `CURATOR_*`)
3. `npm run dev:migrate` on dev and prod DBs (includes auth tables + bootstrap curator)
4. `net use \\192.168.10.122\Gallery` → `npm run devtoprod:images`
5. `npm run prod:docker:publish` → TrueNAS Custom App from `infra/docker/compose.truenas.yaml` (set auth env in compose)
6. Keenetic: both domains → `:5173`, protocol to device **`http`**, correct IP per environment

### Legacy deployment (optional)

Node on the dev PC at `:3520` with nginx → Vite `:5173` is superseded by TrueNAS Docker prod and `npm run dev:web` for public dev. See [`deploy/nginx-gallery.conf`](../deploy/nginx-gallery.conf) and [`deploy/gallery.service`](../deploy/gallery.service) only if you need a local nginx/systemd setup.

## Maintenance scripts

| Command | Script | Purpose |
|---------|--------|---------|
| `npm run dev:seed` | `scripts/seed-wikipedia.js` | Reload curated Wikipedia data |
| `npm run dev:fetch-artist-bios` | `scripts/fetch-artist-bios.js` | Wikipedia lead sections → `bio_short` / `bio_full` |
| `npm run dev:fetch-artist-bios -- --force` | ↑ | Refresh bios even when already set |
| `npm run dev:expand-catalog` | `scripts/expand-paintings.js` | Insert famous works from `famous-paintings-data.js` |
| `npm run dev:expand-catalog -- --fetch-images` | ↑ | Also download images for new rows (slow) |
| `npm run dev:fetch-images` | `scripts/fetch-missing-images.js` | Search Wikipedia, Commons, museums for missing files |
| `npm run dev:search-missing-paintings` | ↑ (alias) | Same as `fetch-images` |
| `npm run dev:fetch-images -- --artist="Name"` | ↑ | Limit to one artist (catalog order) |
| `npm run dev:fetch-images -- --limit=50` | ↑ | Random sample of N missing paintings |
| `npm run dev:fetch-images -- --limit=250` | ↑ | Larger random batch (processes min(N, missing count)) |
| `npm run dev:fetch-images -- --limit=50 --max-wait=120` | ↑ | Random batch; 120s cap per painting |
| `npm run dev:fetch-images -- --discover-only --limit=20` | ↑ | Fix `wikipedia_title` via search only |
| `npm run dev:regenerate-thumbnails` | `scripts/regenerate-thumbnails.js` | Rebuild all thumbs from full local files |
| `npm run dev:audit-painting-images` | `scripts/audit-painting-images.js` | List thumb/full aspect-ratio mismatches |
| `npm run dev:migrate:thumbnails` | `db/migrate-thumbnails.sql` | Add thumbnail columns |
| `npm run dev:fetch-artist-images` | `scripts/fetch-artist-images.js` | Download portraits or link existing files under `data/images/portraits/` |
| `npm run dev:fetch-artist-images -- --force` | ↑ | Re-fetch even when `portrait_path` is set |
| `npm run dev:sync-image-paths` | `scripts/sync-image-paths.js` | Link `image_path` / `thumbnail_path` and import missing painting rows from disk |
| `npm run dev:sync-image-paths -- --dry-run` | ↑ | Report only, no DB writes |
| `npm run dev:migrate:influence-sources` | `scripts/migrate-influence-sources.js` | Create `painting_influence_sources` + backfill legacy edges |
| `npm run dev:migrate:checkup-flags` | `scripts/migrate-checkup-flags.js` | Add `checkup_checked` / `checkup_fixed` on `paintings` |
| `npm run dev:migrate:artist-checkup-flags` | `scripts/migrate-artist-checkup-flags.js` | Add `checkup_checked` / `checkup_fixed` on `artists` (bio debug) |
| `npm run dev:migrate:search` | `scripts/migrate-search.js` | Same indexes as `migrate-search.sql` (also run by `dev:migrate`) |
| `npm run dev:migrate:painting-annotations` | `scripts/migrate-painting-annotations.js` | Create `painting_annotations` table |
| `npm run dev:migrate:artist-palette` | `scripts/migrate-artist-palette.js` | Add `palette_metadata` JSONB on `artists` |
| `npm run dev:import-painter-palette` | `scripts/import-painter-palette.js` | Enrich artists + influence links from `Inputs/PainterPalette.csv` |
| `npm run dev:analyze-painter-palette` | `scripts/analyze-painter-palette.js` | Report CSV ↔ gallery artist name matches |
| `npm run dev:export-paintings` | `scripts/export-paintings-csv.js` | Write `Output/paintings.csv` (artist, painting, year) |
| `npm run dev:update-painting-annotations` | `scripts/update-painting-annotations.js` | Load curated notes from `painting-annotations-data.js` |
| `npm run dev:update-painting-annotations -- --wikipedia` | ↑ | Add intro sentences from each work’s Wikipedia page |
| `npm run dev:update-painting-annotations -- --wikipedia --wiki-delay=3000` | ↑ | Slower Wikipedia pass when rate-limited (429) |
| `npm run dev:find-duplicates` | `scripts/find-duplicate-paintings.js` | Report duplicate and near-duplicate painting rows |
| `npm run dev:audit-influence-duplicates` | `scripts/audit-influence-duplicates.js` | Report mirrored legacy/sources edges and duplicate influence rows |
| `npm run dev:update-influences` | `scripts/update-influences.js` | Insert influence links (painting / artist / movement) from `art-influences-data.js` |
| `npm run dev:update-influences -- --fetch-images` | ↑ | Also download images for newly created works |
| `npm run dev:update-influences -- --discover` | ↑ | Curated pass + web discovery (Wikipedia, Wikidata, Met, art-history sites) |
| `npm run dev:discover-influences` | `update-influences.js --discover-only` | Discovery pass only |

### Scripts in the repository

These are checked in and maintained:

- `seed-wikipedia.js`, `seed-catalog-data.js` — initial catalog (`npm run dev:seed`)
- `image-fetcher.js` — Wikimedia / museum image resolution
- `fetch-artist-images.js` — artist portrait download / disk linking
- `sync-image-paths.js` — import paintings and align paths from `data/images/paintings/`
- `fetch-artist-bios.js` — artist biographies
- `expand-paintings.js` + `famous-paintings-data.js` — catalog expansion
- `update-influences.js` + `art-influences-data.js` — influence graph (paintings, artists, movements)
- `influence-discovery.js` + `influence-resolver.js` — web discovery and polymorphic source resolution
- `fetch-missing-images.js` — batch image backfill
- `find-duplicate-paintings.js` — duplicate catalog audit
- `audit-influence-duplicates.js` — influence graph duplicate / mirror audit
- `painter-palette-lib.js`, `import-painter-palette.js`, `migrate-artist-palette.js`, `analyze-painter-palette.js` — PainterPalette CSV integration
- `export-paintings-csv.js` — catalog CSV export to `Output/`
- `migrate-checkup-flags.js` — checkup workflow columns (paintings)
- `migrate-artist-checkup-flags.js` — checkup workflow columns (artist portraits)
- `migrate-painting-annotations.js`, `update-painting-annotations.js`, `painting-annotations-data.js` — art-history notes on painting detail
- `regenerate-thumbnails.js`, `audit-painting-images.js`

See [data-and-images.md](data-and-images.md) for pipeline details and override maps.

## Remote repository

Gitea: [Danilka/Art-gallery](https://gitea.mysuperlab.netcraze.pro/Danilka/Art-gallery)

```bash
git clone https://gitea.mysuperlab.netcraze.pro/Danilka/Art-gallery.git
```

After clone: copy `.env.example` → `.env`, install dependencies, run [one-time DB split](Documentation/environments.md#step-b--split-the-database-pgadmin), then post-seed steps. Deploy: [environments.md](Documentation/environments.md).

## Troubleshooting

| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| Empty timeline | DB not seeded | `npm run dev:seed` |
| 502 / 504 on public URL | Keenetic rule wrong (IP, port, or `https` to device) | Dev → `192.168.10.70:5173`; prod → `192.168.10.122:5173`; protocol **`http`** — see [environments.md](environments.md) |
| 503 on public URL | Dev servers not running | `npm run dev:web` (or `dev:server` + `dev:client` for local HMR) |
| **`manifest unknown`** on TrueNAS deploy | Image not in Gitea | `npm run prod:docker:publish` on dev PC first |
| 500 on all `/api/*` | Wrong `.env` or Postgres down | Check connection, logs |
| “Biographical information not yet available” | Bios not fetched | `npm run dev:fetch-artist-bios` |
| Placeholder portraits on timeline | `portrait_path` not set | `npm run dev:fetch-artist-images` |
| Artist hall has only 1–2 paintings | DB not expanded / disk not imported | `npm run dev:sync-image-paths` then `npm run dev:expand-catalog`; extend `famous-paintings-data.js` |
| Movement flow shows “Loading art history…” repeatedly | Stale client refetching on every pan/zoom | Pull latest client; timeline loads catalog once — hard-refresh |
| Movement flow slow on first load | Large artist payload | Client uses `GET /api/artists?timeline=1` (no full bios); rebuild client |
| Movement flow zoom sluggish / dead over portraits | Stale client build | Rebuild client — wheel uses capture listener + interaction-mode rendering; hard-refresh |
| Timeline year labels overlap when zoomed out | Stale client | Rebuild client — `chooseTimelineTickInterval()` adapts step to span and bar width |
| Black frames / canvas covers in 3D gallery | No local image for painting | `npm run dev:fetch-images -- --limit=50` or `--artist="…"`; then `POST …/preload-images` |
| Many `⏱ timeout` lines in fetch batch | Default 10s cap too short for hard works | `--max-wait=120` or raise `FETCH_MAX_WAIT_SEC` |
| White/grey flicker on frames | Texture loading or z-fighting with wall | Rebuild client (`cd client && npm run build`); ensure latest `VirtualGallery.tsx` |
| Wrong painting in 3D gallery frame | Stale or mismatched thumbnail file | `npm run dev:regenerate-thumbnails`; gallery prefers full `image_path` |
| Wrong painting image (fetch) | Bad museum / search match on first download | Add entry to `DIRECT_IMAGE_OVERRIDES` in `scripts/image-fetcher.js`, re-fetch file |
| Empty exit navigation lists | No influence edges for artist | `npm run dev:update-influences`; check `painting_influence_sources` |
| Empty *Influenced By* / *Influenced* on painting detail | No edges for that work | `npm run dev:migrate:influence-sources` then `npm run dev:update-influences`; extend `art-influences-data.js` or run `--discover` |
| Same work listed twice under *Influenced* | Stale server merging legacy + sources tables | Restart server; API reads `painting_influence_sources` only — run `npm run dev:audit-influence-duplicates` to verify DB |
| No golden lamps above frames in 3D hall | Stale API process or no influence edges | Restart server after API changes; run `npm run dev:update-influences` |
| Default Vite page instead of gallery | `client/dist` missing or stale | `cd client && npm run build` |
| Permission denied creating tables | `gallery` user lacks CREATE | Run admin grants, then migrate |
| Wikipedia API rate limit during fetch | Too many requests in a row | Wait and re-run; scripts retry with backoff |
| Checkup **Reviewed** toggle returns 404 | Stale server process missing new routes | Restart `npm run dev:web` or `npm run dev:server` after pulling API changes |
| Debug **More** / **Clear** / **Upload** / **Remove entry** returns 404 | Stale server process | Restart `npm run dev:start` or `npm run dev:server`; routes in `server/index.js` + `server/image-service.js` |
| Debug **Remove entry** — button stuck or missing on next painting | Stale client build | `cd client && npm run build`; hard-refresh — detail view remounts per painting id |
| Debug **Upload** returns 413 Payload Too Large | Base64 JSON exceeds body limit | Server allows 20 MB JSON / 15 MB decoded image; compress file or resize before upload |
| Debug **Upload** — nothing happens after choosing file | Stale client or broken hidden-file `click()` | Pull latest client (`DebugUploadButton` uses `<label>` + `<input>`); hard-refresh |
| Uploaded image reverts after reload (old picture) | Browser cached `/images/…` at same path | Pull latest server + client — API returns `image_cache_key` and URLs use `?v=`; `/images` no longer uses immutable long-term cache |
| No art-history notes on painting detail | Annotations not migrated or loaded | `npm run dev:migrate:painting-annotations` then `npm run dev:update-painting-annotations` |
| **Fix it** fails with `read ECONNRESET` | Remote host dropped connection | Restart server; client sends `searchUrl` / `source`; retry or use Commons URL in overrides |
| Fixed image not shown in 3D gallery | Stale gallery session or cached texture | Rebuild client; fix/upload updates session + `?v=` from `image_cache_key` — use **Back to Gallery** (not browser back) |
| **Back to Timeline** returns to gallery / previous wing | Stale client build | Pull latest client — `goToTimelineHome()` unmounts the hall and resets timeline zoom |
| Catalog search dropdown hidden under timeline | Stale client CSS | Rebuild client — `.site-header` uses `z-index: 110` above the sticky timeline bar |
| Catalog search returns empty / 500 | Search indexes missing | Run `npm run dev:migrate` (includes `migrate-search.sql`) or `npm run dev:migrate:search`; restart API |
| `fetch-artist-bios-ru` fails: `index row size … exceeds btree maximum` | Old `entity_translations` index on all `value` text | `npm run dev:migrate` (partial search index on name/title only), then re-run `npm run dev:fetch-artist-bios-ru` |
| Russian UI shows English catalog names | Translations not published | Curator → Translations → Publish; public API serves only `status = published` |
| Influence import leaves many unresolved tokens | Names not in catalog DB | Curator → Influences → Import warnings; fix spelling or add artists first; free-text traditions stay unresolved |
| Influence import created too many edges | Artist-level rows expand to all paintings | Expected (PainterPalette-style); delete unwanted edges in Influences list |
| Influence import blocked: already imported | Same file bytes or mapped data imported before | Expected; use **Import anyway** only if intentional, or skip |
| Frame still black after **Checked** | Gallery session not synced | Re-enter hall or toggle debug **Checked** from detail with gallery open behind overlay |
| Duplicate works in gallery / timeline | Double import or variant Wikipedia titles | `npm run dev:find-duplicates`; merge or delete spare rows manually |
| **Failed to load movement gallery** / `Cannot GET /api/movements/:id/gallery` | Stale server process missing route | Restart `npm run dev:web` or `npm run dev:server` after pulling API changes |
| Movement gallery shows generic cream walls | Stale client build | `cd client && npm run build`; hard-refresh browser |
| Windows overlap paintings in movement wing | Stale client | Rebuild client — windows are placed only on side walls in gaps between frames |
| Influence thumbnails cropped on painting detail | Stale client build | `npm run prod:build` — panels use `object-fit: contain` for full image |
| **Curator login** fails / always guest | Auth tables missing or wrong password | Set `SESSION_SECRET` + `CURATOR_PASSWORD` in `.env`, run `npm run dev:migrate` (or `npm run dev:reset-curator`), restart server |
| Debug / Checkup returns **401** | Not signed in as curator | **Curator login** (top-right); session cookie `gallery.sid` must be sent (`credentials: include`) |
| Debug works in UI but API rejects | Stale server without auth middleware | Restart `npm run dev:web` or `npm run dev:server` after pulling auth changes |
| **Empty screen** entering 3D hall (header missing) | Stale client before gallery-session fix | Hard-refresh; pull latest client — hall renders from `view` state, not only `gallerySession` |
| **Dark center** entering 3D hall (header visible, no spinner) | Stale client before gallery loading overlay fix | Hard-refresh; latest client shows **Loading gallery…** / **Loading paintings…** in the canvas area until ready |
| 3D hall loading overlay never clears | Stuck texture counter or hung HDR | Hard-refresh; current client times out Environment / shader warm-up so the overlay cannot stay forever |
| 3D hall black after returning from painting detail | WebGL context lost while hall was hidden | Hard-refresh; latest client remounts canvas when hall becomes active again |