Art-gallery/Documentation/data-and-images.md

# Art Gallery — data and images

How catalog content, biographies, and artwork files enter the system.

## Principles

1. **No runtime hot-linking** — the UI reads from `/images/…` (local disk). External URLs are used only during ingest.
2. **No AI-generated art or text** — biographies and descriptions come from Wikipedia; influence notes from curated art-history sources.
3. **Local copies** — every displayed image should exist under `data/images/` after seeding or fetch.

## Directory layout

```text
data/images/
├── portraits/           # Artist headshots
│   └── Claude_Monet.jpg
└── paintings/
    ├── Claude_Monet_Water_Lilies.jpg
    └── thumbs/
        └── Claude_Monet_Water_Lilies_thumb.jpg
```

File names are sanitised `{Artist}_{Title}.{ext}`. The image service can rediscover files on disk even when DB paths are empty (`server/image-service.js` → `syncPaintingFromDisk`).

## Scripts overview

| Script | npm command | Role |
|--------|-------------|------|
| `seed-wikipedia.js` | `npm run seed` | Initial eras, movements, artists, paintings, influences |
| `fetch-artist-bios.js` | `npm run fetch-artist-bios` | Wikipedia intros → `bio_short` / `bio_full` |
| `famous-paintings-data.js` | *(data only)* | Curated list of notable works per artist |
| `expand-paintings.js` | `npm run expand-catalog` | Inserts works from data file for thin catalogs |
| `art-influences-data.js` | *(data only)* | Curated painting-to-painting influence edges |
| `update-influences.js` | `npm run update-influences` | Applies influence graph; creates missing artists/works |
| `fetch-missing-images.js` | `npm run fetch-images` | Downloads files for paintings missing on disk |
| `image-fetcher.js` | *(library)* | Wikimedia / museum resolution used by fetch scripts and API |
| `regenerate-thumbnails.js` | `npm run regenerate-thumbnails` | Rebuild thumbs from full images via `sharp` |
| `audit-painting-images.js` | `npm run audit-painting-images` | Detect thumb/full aspect-ratio mismatches |
| `find-duplicate-paintings.js` | `npm run find-duplicates` | Report exact and near-duplicate catalog rows |
| `migrate-checkup-flags.js` | `npm run migrate:checkup-flags` | Add `checkup_checked` / `checkup_fixed` columns |

## Typical workflow

```text
migrate → seed → fetch-artist-bios → expand-catalog → update-influences → fetch-images (per artist or batch) → build client
```

1. **Seed** creates the base catalog (often one flagship painting per modern artist).
2. **fetch-artist-bios** fills biography fields for every artist with a `wikipedia_title`.
3. **expand-catalog** brings each artist up to at least **6** notable works (configurable via `MIN_PAINTINGS`).
4. **fetch-images** downloads artwork files; the 3D gallery needs local files for reliable textures.

## Seeding pipeline

`npm run seed` runs `scripts/seed-wikipedia.js`, which:

1. Inserts **historical eras** and **art movements** (curated date ranges and colours).
2. For each curated **artist**:
   - Creates **artist periods** and **paintings**.
   - May download portraits and painting images (depending on seed script version).
3. Writes **painting_influences** edges from curated scholarship references.

Those influence edges power **3D hall navigation**: predecessors and successors at each artist’s exit doorway are computed from this table (see `GET /api/artists/:id/navigation` in [API.md](API.md)).

Artists are grouped by movement and century; the seed list targets at most ~100 artists per century.

## Artist biographies

`npm run fetch-artist-bios` reads each artist’s `wikipedia_title`, fetches the English Wikipedia **lead section**, and stores:

| Field | Content |
|-------|---------|
| `bio_short` | First two sentences |
| `bio_full` | Full intro (text before the first section heading) |

Flags:

- `--force` — refresh bios even when `bio_full` is already set.

**Disambiguation and title overrides** live in `ARTIST_WIKI_OVERRIDES` inside `scripts/fetch-artist-bios.js`:

| Artist in DB | Wikipedia article used |
|--------------|------------------------|
| Zeuxis | `Zeuxis (painter)` |
| Ivan Klyun | `Ivan Kliun` |
| Jean-Antoine Watteau | `Antoine Watteau` |

The script detects disambiguation pages (“X may refer to:”) and tries fallbacks such as `{name} (painter)` before giving up. Requests are throttled (~3.5 s apart) with retry on HTTP 429.

The bio page (`ArtistBio.tsx`) shows lifespan, movement, summary, full text, and the source Wikipedia title.

## Expanding thin catalogs

Many seed artists arrive with only one famous painting. `npm run expand-catalog` runs `scripts/expand-paintings.js`, which:

1. Finds artists with fewer than `MIN_PAINTINGS` (default **6**).
2. Inserts rows from `scripts/famous-paintings-data.js` that are not already present (normalized title matching skips duplicates).
3. Sets `wikipedia_title` on each new painting for image resolution.

```bash
npm run expand-catalog                    # DB rows only
npm run expand-catalog -- --fetch-images  # also download images (very slow)
```

To add more works, append entries to `famous-paintings-data.js`:

```javascript
{ artist: 'Gustav Klimt', title: 'Portrait of Adele Bloch-Bauer I', year: 1907 },
{ artist: 'Gustav Klimt', title: 'The Kiss', year: 1908, wikipedia_title: 'The Kiss (Klimt painting)' },
```

`wikipedia_title` is optional; it defaults to `title`. Use it when the Wikipedia article name differs from the display title.

Renaissance and medieval masters with large museum catalog dumps (e.g. Raphael, Dürer) are usually above the minimum already; expansion targets Impressionists, modernists, and other artists who had only a single seed painting.

## Painting influence graph

Directed influence links are stored in **`painting_influence_sources`**. Each row connects a painting to a **source** of type `painting`, `artist`, or `movement`, with optional period context (e.g. influence during the work’s creation year).

The legacy **`painting_influences`** table (painting-to-painting only) is still written for hall navigation compatibility and is backfilled into `painting_influence_sources` on migration.

Influence data drives:

- **Painting detail** — *Influenced By* (left) and *Influenced* (right) panels: painting thumbnails, artist portraits, or movement colour swatches, plus notes, aspects, period labels, and citations
- **3D hall exit** — predecessor and successor artists grouped by movement (painting edges only)
- **3D gallery lamps** — golden picture light above frames with any influence edge (`has_influence_links` on painting API responses)

### One-time migration

```bash
npm run migrate:influence-sources   # create table + backfill legacy painting edges
```

### Curated updates

`npm run update-influences` runs `scripts/update-influences.js` against `scripts/art-influences-data.js`:

```bash
npm run update-influences                         # insert curated edges
npm run update-influences -- --fetch-images       # also download images for newly created works
npm run update-influences -- --discover           # curated + web discovery pass
npm run discover-influences                       # discovery only (no curated file pass)
npm run update-influences -- --discover --limit=20  # cap discovery to N works
```

Each entry defines a later `work` and one or more `influencedBy` sources. Legacy single-object form is still supported:

```javascript
{ work: { artist: '...', title: '...', year: 1907 },
  influencedBy: { artist: 'Giotto', title: 'Lamentation' } }
```

Multi-source form (painting, artist, movement):

```javascript
{
  work: { artist: 'Pablo Picasso', title: "Les Demoiselles d'Avignon", year: 1907 },
  influencedBy: [
    { type: 'painting', artist: 'Paul Cézanne', title: 'The Bathers' },
    { type: 'artist', artist: 'Paul Cézanne', period: { duringCreation: true } },
    { type: 'movement', movement: 'Fauvism', period: { start: 1905, end: 1907, note: '...' } },
  ],
  notes, aspects, source_author, source, source_url,
}
```

When `artistMeta` is included, missing artists are created with movement and lifespan. Missing paintings are inserted with `wikipedia_title` for image fetch.

### Web discovery

`scripts/influence-discovery.js` searches art-history sources when `--discover` or `--discover-only` is passed:

- Wikipedia summaries and Wikidata **P737** (influenced by)
- Met Museum collection API
- DuckDuckGo site-restricted search across TheArtStory, Met, Google Arts & Culture, NGA, Art Institute of Chicago, MoMA, Britannica, JSTOR, Oxford Art Online, WikiArt, and Wikipedia

Discovered rows are stored with `confidence: discovered` and `discovered_via` (e.g. `wikipedia`, `wikidata`, `met`, `web:theartstory.org`). Period hints are inferred when the source text mentions influence during creation or a date range overlapping the work’s year.

Extend `art-influences-data.js` for high-quality curated chains; use discovery to suggest additional artist and movement links for manual review.

## Movement lineage (frontend flow diagram)

Separate from the painting influence graph, `client/src/data/movement-lineage.ts` lists **art-movement** predecessor→successor pairs used only by `MovementBands.tsx` to position streams and draw branch connectors (e.g. Impressionism → Post-Impressionism → Fauvism / Cubism / Expressionism).

| Aspect | Detail |
|--------|--------|
| Storage | TypeScript module in the client — **not** a database table |
| Format | `[parentMovementName, childMovementName]` tuples; names must match `art_movements.name` from the seed |
| Multiple parents | Allowed (e.g. Post-Impressionism feeding several modern paths) |
| Sources | Curator notes in the file reference Met essays, TheArtStory, and similar |

To add or fix a branch, edit `MOVEMENT_LINEAGE` in that file and rebuild the client. No migration or API change is required.

## Historical event markers (frontend timeline)

`client/src/data/historical-events.ts` lists **world-history** markers shown on `Timeline.tsx` (French Revolution, World War I/II, etc.).

| Aspect | Detail |
|--------|--------|
| Storage | TypeScript module in the client — **not** a database table |
| Format | `{ id, name, startYear, endYear?, shortLabel? }` — omit `endYear` for a single-year pin |
| Interaction | Click a marker to zoom the shared timeline/movement view to that period |

Edit `HISTORICAL_EVENTS` and rebuild the client to extend the set.

## Batch image fetch

`npm run fetch-images` (alias: `npm run search-missing-paintings`) runs `scripts/fetch-missing-images.js`. It searches multiple sources for paintings without local files:

| Source | Notes |
|--------|--------|
| Wikidata / Wikipedia | Article image + P18 property |
| Wikimedia Commons | Direct file + search |
| Wikipedia search | Discovers better article title when seed `wikipedia_title` is a museum catalog label |
| Google Arts & Culture | Search + asset pages (`artsandculture.google.com`) |
| Met Museum | Open Access API |
| Art Institute of Chicago | IIIF open access |
| Cleveland Museum of Art | CC0 API |
| Rijksmuseum | Public API |
| Musée du Louvre | Collections search (`collections.louvre.fr`) |
| Europeana | European museum aggregator (optional `EUROPEANA_API_KEY` in `.env`) |
| Smithsonian | Optional (`SMITHSONIAN_API_KEY` in `.env`) |
| Harvard Art Museums | Optional (`HARVARD_ART_API_KEY` in `.env`) |

```bash
npm run fetch-images                              # all missing, catalog order (~hours)
npm run fetch-images -- --limit=50                # random sample of 50; 10s max per painting
npm run fetch-images -- --limit=250               # random sample up to N (caps at current missing count)
npm run fetch-images -- --limit=50 --max-wait=120 # slower, more thorough lookup per painting
npm run fetch-images -- --artist="Albrecht Dürer" # one artist, catalog order
npm run fetch-images -- --discover-only --limit=20   # fix wikipedia_title only
npm run fetch-images -- --web-search-only --limit=50 # DuckDuckGo + Commons + multilingual Wikipedia
```

Each run prints **`Missing local files: N`** at startup — that is the current count of catalogued paintings with no full-size or thumbnail file under `data/images/`. A painting counts as present if **either** file exists on disk.

| Flag | Effect |
|------|--------|
| `--limit=N` | Process at most **N** paintings. Queue is a **random sample** of all works missing local files (not alphabetical). |
| `--max-wait=N` | Stop each painting after **N** seconds (default **10**). Logs `⏱ timeout` and continues. |
| `--artist="Name"` | Only that artist’s missing works, in catalog order (`sort_order`, `year`). |
| `--discover-only` | Update `wikipedia_title` via search; no download. |
| `--web-search-only` | Skip museum APIs; use web search + Commons + multilingual Wikipedia. |

Each limited batch run stops per painting after **`--max-wait` seconds** (default **10**), including source lookup and download. While a batch deadline is active, inter-request throttling is skipped and each HTTP call times out at the **remaining** budget (not the full 15s on-demand limit). Override the default via `FETCH_MAX_WAIT_SEC` in `.env` or `--max-wait=N`. On-demand fetches in the web UI keep their separate 15s API timeout and are unaffected.

Re-run the same command to pick a new random batch until the missing count reaches zero.

## On-demand image resolution

When a painting has no local file, `GET /api/paintings/:id/image` triggers `ensurePaintingImages()`:

1. Check DB paths → verify file on disk.
2. Scan disk by `{artist}_{title}` pattern.
3. If still missing and `wikipedia_title` is set, call `scripts/image-fetcher.js`:
   - Resolve overrides and simplified titles
   - **Wikipedia search** when catalog labels fail
   - Wikidata → Wikimedia Commons → Wikipedia page image
   - Fallbacks: Met Museum, Art Institute of Chicago, Cleveland Museum, Rijksmuseum, Smithsonian*, Harvard*
4. Save full image, **generate thumbnail by resizing the full file** (not a separate Commons thumb URL), update DB, serve file.

Separate Wikipedia/Commons thumbnail URLs often resolve to the **wrong work** (e.g. a different painting with a similar title). Thumbnails are always derived locally from the downloaded full image via `sharp` in `scripts/image-fetcher.js`.

Requests are deduplicated (`inflight` map) and timeout after 15 seconds. On-demand resolution uses a ~2.5 s delay between external requests to reduce rate-limit risk; batch `fetch-images` runs skip that delay while the per-painting deadline is active.

## Preload before 3D gallery

`POST /api/artists/:id/preload-images` runs **local-only** linking — no network. Call this when entering an artist’s 3D hall so textures use files already on disk.

The 3D scene uses `galleryImageUrl()`, which never hits the on-demand API (remote latency breaks WebGL texture loading).

## Placeholders

When no image is available:

- `/placeholder-portrait.svg` — timeline and movement-flow portraits
- `/placeholder-art.svg` — paintings in lists and detail view
- **3D gallery** — draped **canvas cover** inside the frame (`CanvasCover` in `VirtualGallery.tsx`); shown when there is no local file, the fetch failed, or the texture has not loaded yet

These live in `client/public/` (and `client/dist/` after build).

## Adding new artists manually

1. Insert rows into `artists`, `artist_periods`, `paintings` (or extend the seed script).
2. Place image files under `data/images/` using the naming convention.
3. Run `npm run fetch-artist-bios` for the new artist’s biography.
4. Add entries to `famous-paintings-data.js` and run `npm run expand-catalog` if needed.
5. Run `npm run fetch-images -- --artist="…"` or rely on preload / on-demand sync.
6. Add influence rows to `painting_influences` with citation fields where possible.

## Image fetcher overrides

`scripts/image-fetcher.js` includes hand-maintained overrides for ambiguous Wikipedia titles and direct URLs (e.g. works whose Commons name does not match the article title, or when museum search returns the wrong work). Extend these maps when automated resolution fails:

| Map | Use when |
|-----|----------|
| `PAINTING_WIKI_OVERRIDES` | DB / seed title should resolve to a different Wikipedia or Wikidata label |
| `DIRECT_IMAGE_OVERRIDES` | You know the exact Commons URL (bypasses Met / Art Institute false matches) |

Examples already in the repo:

- `Self-Portrait Hesitating` → Kauffman, Wikimedia Commons (National Trust)
- `Cherubs of the Sistine Madonna` → Raphael’s putti detail, Wikimedia Commons
- `Madonna and Child (Madonna della Seggiola)` / `Madonna della seggiola` → Raphael’s tondo, Palazzo Pitti
- `Madonna and Child` (Raphael) → *Small Cowper Madonna*, National Gallery of Art
- `Job Cigarette Papers` → Mucha poster disambiguation
- `Charing Cross Bridge` → Derain (not Monet)

After adding an override, delete any wrong cached file under `data/images/paintings/` and re-run fetch or call the on-demand image endpoint for that painting.

## Duplicate paintings

The catalog can contain the same work more than once — usually from a **double import** (identical artist + title + year + image) or **Wikipedia scrape variants** (different article titles for one icon, e.g. Andrei Rublev’s Trinity).

### Find duplicates

```bash
npm run find-duplicates
```

Runs `scripts/find-duplicate-paintings.js`, which reports:

| Report | Rule |
|--------|------|
| **Exact duplicates** | Same `artist_id`, `title`, and `year` |
| **Normalized title duplicates** | Same artist, titles differing only in punctuation/spacing |
| **Rublev / Trinity cluster** | Known multi-entry example for manual merge |

As of a recent audit (~1200 paintings): **52 exact duplicate pairs** (52 removable rows), concentrated in **Hieronymus Bosch** (25), **Albrecht Dürer** (14), and **Domenico Ghirlandaio** (13). Duplicate copies typically share the same image file and have **no influence links**, so the higher id in each pair is safe to delete after review.

**Near-duplicates** (different titles, same work) need curator judgment — e.g. Rublev ids 36, 316, 320, 322, 323 all describe the Trinity icon under different Wikipedia labels; keep id **36** (`Trinity`, wiki `Trinity (Andrei Rublev)`).

`expand-paintings.js` skips inserts when normalized titles match, but duplicates can still appear if seed and expansion use different title strings or if influence discovery creates works independently.

## Debug and checkup image fix

When **Debug mode** is on (home header) or from the **Checkup** page:

1. **Search** — `GET /api/paintings/:id/debug-image-search` tries Google Custom Search (if `GOOGLE_CSE_API_KEY` + `GOOGLE_CSE_CX` are set in `.env`), Google Arts & Culture, Google Images scrape, then DuckDuckGo (`searchGoogleImagesFirst` in `scripts/image-fetcher.js`).
2. **Fix** — `POST /api/paintings/:id/fix-image` downloads the chosen URL via `downloadImageForFix` → `replacePaintingImageFromUrl` in `server/image-service.js`, regenerates the thumbnail with `sharp`, and sets `checkup_fixed` + `checkup_checked`.

### Painting detail debug panel

With debug mode on, `PaintingDetail.tsx` shows a bottom-left panel with search preview and two buttons:

| Button | API | Effect |
|--------|-----|--------|
| **Checked** | `PATCH …/checkup-flags` `{ "checked": true }` | Marks reviewed; 3D frame turns gold |
| **Fix it** | `POST …/fix-image` | Saves image to disk, sets both flags, refreshes detail + gallery |

The client passes `searchUrl`, `source`, and `thumbUrl` from the search result to improve download reliability. After a fix, `HomePage` updates the gallery session and appends a revision query on 3D texture URLs so replaced files reload even when the path is unchanged.

Checkup **Search visible** queues search for filtered rows only (3 concurrent); it does not search the full catalog on load.

See [API.md](API.md#developer-image-audit) and [basics.md](basics.md#developer-tools-image-audit).