Art-gallery/Documentation/data-and-images.md

Art Gallery — data and images

How catalog content, biographies, and artwork files enter the system.

Principles

1. No runtime hot-linking — the UI reads from /images/… (local disk). External URLs are used only during ingest.
2. No AI-generated art or text — biographies and descriptions come from Wikipedia; influence notes from curated art-history sources.
3. Local copies — every displayed image should exist under data/images/ after seeding or fetch.
4. Painting thumbnails must stay in sync with the full image — see Painting thumbnail invariant below. Treat this as a hard requirement for any new feature or script that writes or replaces painting picture files.

Painting thumbnail invariant

Requirement (do not regress): any action that creates, replaces, or clears a painting’s full picture must also regenerate or remove that painting’s dedicated thumbnail under paintings/thumbs/. Never point thumbnail_path at the full-size file as a stand-in, and never reuse a search-result or Commons thumb URL.

Change type	Expected thumb behavior	Primary code
Fix / upload / buffer or URL replace	Write full file, then writePaintingThumb / generateThumbnailFromFull	server/image-service.js
Fetch / seed / expand / influence ingest	Same: thumb from saved full via savePaintingImages	scripts/image-fetcher.js
Disk sync / preload / ensure when full exists but thumb missing	Generate real *_thumb.jpg and update thumbnail_path	ensurePaintingThumbFromFull, syncPaintingFromDisk, ensurePaintingImages, preloadArtistImagesLocal, preloadMovementImagesLocal
Clear image / delete painting	Delete thumb file(s) and null paths (or drop row)	clearPaintingImage, deletePainting
Bulk rebuild after sync/harmonize	regenerate-thumbnails.js / harmonize:images	scripts + npm scripts

Path-only maintenance (sync-image-paths.js) may link existing thumb files without rewriting pixels; if full images changed without a matching thumb step, run npm run dev:regenerate-thumbnails (or harmonize:images).

Directory layout

data/images/
├── portraits/           # Artist headshots (display ~900px wide)
│   ├── Claude_Monet.jpg
│   └── thumbs/          # Timeline thumbnails (~256px)
│       └── Claude_Monet_thumb.jpg
└── paintings/
    ├── Claude_Monet_Water_Lilies.jpg
    └── thumbs/
        └── Claude_Monet_Water_Lilies_thumb.jpg

File names are sanitised {Artist}_{Title}.{ext}. The image service can rediscover files on disk even when DB paths are empty (server/image-service.js → syncPaintingFromDisk).

Dev vs production image storage

Environment	Path on disk	Sync
Development	./data/images/ in repo	Working copy on dev PC
Production	/mnt/BasePool/Applications/Gallery/data/images on TrueNAS	SMB \\192.168.10.122\Gallery\data\images

One-direction promote: npm run devtoprod:images (dev → prod, skip older). Refresh all images on dev from prod: npm run prodto:dev:images. One artist only: map the SMB share, then robocopy that artist’s ArtistName* files under paintings/ and paintings/thumbs/ (see FAC.md). Bidirectional merge (newer file wins, then merge artists/paintings checkup flags + image paths, then rebuild painting + portrait thumbs on both sides): npm run harmonize:images or full npm run harmonize — see harmonize-dev-prod.md. General sync reference: environments.md.

Scripts overview

Script	npm command	Role
seed-wikipedia.js	npm run dev:seed	Initial eras, movements, artists, one flagship painting per artist
seed-catalog-data.js	(data only)	Eras, movements, artist metadata consumed by seed
sync-image-paths.js	npm run dev:sync-image-paths	Import painting rows from disk; set image_path / thumbnail_path
fetch-artist-images.js	npm run dev:fetch-artist-images	Download or link artist portraits under data/images/portraits/
fetch-artist-bios.js	npm run dev:fetch-artist-bios	Wikipedia intros → bio_short / bio_full
famous-paintings-data.js	(data only)	Curated list of notable works per artist
expand-paintings.js	npm run dev:expand-catalog	Inserts works from data file for thin catalogs
art-influences-data.js	(data only)	Curated influence edges (painting / artist / movement)
update-influences.js	npm run dev:update-influences	Applies influence graph; creates missing artists/works
fetch-missing-images.js	npm run dev:fetch-images	Downloads files for paintings missing on disk
image-fetcher.js	(library)	Wikimedia / museum resolution used by fetch scripts and API
sync-images-to-prod.ps1 / sync-images-from-prod.ps1	npm run devtoprod:images / npm run prodto:dev:images	Robocopy via SMB \\192.168.10.122\Gallery
harmonize-db.js / harmonize-images.js	npm run harmonize:db / harmonize:images	Bidirectional merge by updated_at / file mtime; images step also merges artists/paintings checkup flags + paths, then rebuilds thumbs on both sides
harmonize.ps1	npm run harmonize	Orchestrator: backups, optional schema, DB + image merge
regenerate-thumbnails.js	npm run dev:regenerate-thumbnails	Rebuild painting thumbs from full images via sharp
regenerate-portrait-thumbs.js	npm run dev:regenerate-portrait-thumbs	Rebuild timeline portrait thumbs (~256px) and set portrait_thumb_path
audit-painting-images.js	npm run dev:audit-painting-images	Detect thumb/full aspect-ratio mismatches
find-duplicate-paintings.js	npm run dev:find-duplicates	Report exact and near-duplicate catalog rows
migrate-checkup-flags.js	npm run dev:migrate:checkup-flags	Add checkup_checked / checkup_fixed columns

Typical workflow

migrate → seed → sync-image-paths → fetch-artist-images → fetch-artist-bios → expand-catalog → update-influences → fetch-images (per artist or batch) → build client

1. Seed creates the base catalog (one flagship painting per artist; ~100 artists).
2. sync-image-paths imports additional paintings when data/images/paintings/ already contains files from a full clone (filename pattern {Artist}_{Title}.jpg).
3. fetch-artist-images sets portrait_path from local files or Wikipedia.
4. fetch-artist-bios fills biography fields for every artist with a wikipedia_title.
5. expand-catalog brings each artist up to at least 6 notable works (configurable via MIN_PAINTINGS).
6. update-influences loads the influence graph (Influenced By / Influenced panels, 3D hall lamps, exit navigation).
7. fetch-images downloads artwork files still missing on disk; the 3D gallery needs local files for reliable textures.

Seeding pipeline

npm run dev:seed runs scripts/seed-wikipedia.js, which:

1. Inserts historical eras and art movements (curated date ranges and colours from seed-catalog-data.js).
2. For each curated artist:
   • Creates artist periods and one flagship painting.
   • May download portraits and painting images when run with --fetch-images.
3. Does not insert influence edges — run npm run dev:update-influences after seed (see Painting influence graph).

Those influence edges power 3D hall navigation and painting detail panels via painting_influence_sources (see GET /api/artists/:id/navigation and GET /api/paintings/:id in API.md).

Artists are grouped by movement and century; the seed list targets at most ~100 artists per century.

Artist biographies

npm run dev:fetch-artist-bios reads each artist’s wikipedia_title, fetches the English Wikipedia lead section, and stores:

Field	Content
bio_short	First two sentences
bio_full	Full intro (text before the first section heading)

Flags:

• --force — refresh bios even when bio_full is already set.

Disambiguation and title overrides live in ARTIST_WIKI_OVERRIDES inside scripts/fetch-artist-bios.js:

Artist in DB	Wikipedia article used
Zeuxis	Zeuxis (painter)
Ivan Klyun	Ivan Kliun
Jean-Antoine Watteau	Antoine Watteau

The script detects disambiguation pages (“X may refer to:”) and tries fallbacks such as {name} (painter) before giving up. Requests are throttled (~3.5 s apart) with retry on HTTP 429.

The bio page (ArtistBio.tsx) shows lifespan, movement, summary, full text, and the source Wikipedia title.

Expanding thin catalogs

Many seed artists arrive with only one famous painting. npm run dev:expand-catalog runs scripts/expand-paintings.js, which:

1. Finds artists with fewer than MIN_PAINTINGS (default 6).
2. Inserts rows from scripts/famous-paintings-data.js that are not already present (normalized title matching skips duplicates).
3. Sets wikipedia_title on each new painting for image resolution.

npm run dev:expand-catalog                    # DB rows only
npm run dev:expand-catalog -- --fetch-images  # also download images (very slow)

To add more works, append entries to famous-paintings-data.js:

{ artist: 'Gustav Klimt', title: 'Portrait of Adele Bloch-Bauer I', year: 1907 },
{ artist: 'Gustav Klimt', title: 'The Kiss', year: 1908, wikipedia_title: 'The Kiss (Klimt painting)' },

wikipedia_title is optional; it defaults to title. Use it when the Wikipedia article name differs from the display title.

Renaissance and medieval masters with large museum catalog dumps (e.g. Raphael, Dürer) are usually above the minimum already when sync-image-paths has imported files from disk; expansion targets Impressionists, modernists, and other artists who had only a single seed painting.

Importing paintings from disk

When the repository includes a full data/images/paintings/ tree but the database was seeded fresh (one row per artist), run:

npm run dev:sync-image-paths

scripts/sync-image-paths.js:

1. Scans data/images/paintings/ for full-size files (not thumbs/).
2. Matches filenames to artists using the same {Artist}_{Title} sanitisation as server/image-service.js.
3. Updates image_path / thumbnail_path on existing rows when files are found.
4. Inserts missing painting rows for files not yet in the catalog.

Flags:

• --dry-run — report counts only, no DB writes.

Safe to re-run; already-imported works are skipped by normalized title matching.

Typical result on a full clone: ~1,000+ paintings linked from ~1,000 on-disk files.

Artist portraits

Timeline movement flow loads portrait_thumb_path (~256px JPEG under portraits/thumbs/{Artist}_thumb.jpg) when available; biography and 3D exit navigation use full portrait_path. After adding portraits, run npm run dev:regenerate-portrait-thumbs to backfill thumbs on dev.

npm run dev:fetch-artist-images runs scripts/fetch-artist-images.js:

1. For each artist, checks data/images/portraits/{Artist}.jpg (or other extensions) and sets portrait_path when a local file exists.
2. Otherwise downloads from Wikipedia / search fallbacks via image-fetcher.js.

Flags:

• --force — re-fetch even when portrait_path is already set.
• --limit=N — process only the first N artists needing portraits.

Run after seed when portrait files exist on disk but the DB still has null portrait_path values.

Painting influence graph

Directed influence links are stored in painting_influence_sources. Each row connects a painting to a source of type painting, artist, or movement, with optional period context (e.g. influence during the work’s creation year).

The legacy painting_influences table (painting-to-painting only) is still written alongside sources when running npm run dev:update-influences — it keeps script compatibility and matches the backfill migration. The API reads only painting_influence_sources, so each edge appears once in the UI.

Influence data drives:

• Painting detail — Influenced By (left) and Influenced (right) panels: painting thumbnails, artist portraits, or movement colour swatches, plus notes, aspects, period labels, and citations
• 3D hall exit — predecessor and successor artists grouped by movement (from painting and artist sources)
• 3D gallery lamps — golden picture light above frames with any influence edge (has_influence_links on painting API responses); rendered upside down in InfluencePictureLamp

Audit duplicate influence links

Painting-to-painting edges exist in both tables by design. To confirm the database has no stray duplicates and that the API model is clean:

npm run dev:audit-influence-duplicates

Reports: edges present in both tables, duplicate rows within either table (should be 0), and legacy-only / sources-only mismatches. If Influenced ever shows the same successor twice, restart the server after pulling API fixes — responses must not union legacy and sources tables.

One-time migration

npm run dev:migrate:influence-sources   # create table + backfill legacy painting edges

Curated updates

npm run dev:update-influences runs scripts/update-influences.js against scripts/art-influences-data.js:

npm run dev:update-influences                         # insert curated edges
npm run dev:update-influences -- --fetch-images       # also download images for newly created works
npm run dev:update-influences -- --discover           # curated + web discovery pass
npm run dev:discover-influences                       # discovery only (no curated file pass)
npm run dev:update-influences -- --discover --limit=20  # cap discovery to N works

Each entry defines a later work and one or more influencedBy sources. Legacy single-object form is still supported:

{ work: { artist: '...', title: '...', year: 1907 },
  influencedBy: { artist: 'Giotto', title: 'Lamentation' } }

Multi-source form (painting, artist, movement):

{
  work: { artist: 'Pablo Picasso', title: "Les Demoiselles d'Avignon", year: 1907 },
  influencedBy: [
    { type: 'painting', artist: 'Paul Cézanne', title: 'The Bathers' },
    { type: 'artist', artist: 'Paul Cézanne', period: { duringCreation: true } },
    { type: 'movement', movement: 'Fauvism', period: { start: 1905, end: 1907, note: '...' } },
  ],
  notes, aspects, source_author, source, source_url,
}

When artistMeta is included, missing artists are created with movement and lifespan. Missing paintings are inserted with wikipedia_title for image fetch.

Web discovery

scripts/influence-discovery.js searches art-history sources when --discover or --discover-only is passed:

• Wikipedia summaries and Wikidata P737 (influenced by)
• Met Museum collection API
• DuckDuckGo site-restricted search across TheArtStory, Met, Google Arts & Culture, NGA, Art Institute of Chicago, MoMA, Britannica, JSTOR, Oxford Art Online, WikiArt, and Wikipedia

Discovered rows are stored with confidence: discovered and discovered_via (e.g. wikipedia, wikidata, met, web:theartstory.org). Period hints are inferred when the source text mentions influence during creation or a date range overlapping the work’s year.

Extend art-influences-data.js for high-quality curated chains; use discovery to suggest additional artist and movement links for manual review.

Movement lineage (frontend flow diagram)

Separate from the painting influence graph, client/src/data/movement-lineage.ts lists art-movement predecessor→successor pairs used only by MovementBands.tsx to position streams and draw branch connectors (e.g. Impressionism → Post-Impressionism → Fauvism / Cubism / Expressionism).

Aspect	Detail
Storage	TypeScript module in the client — not a database table
Format	[parentMovementName, childMovementName] tuples; names must match art_movements.name from the seed
Multiple parents	Allowed (e.g. Post-Impressionism feeding several modern paths)
Sources	Curator notes in the file reference Met essays, TheArtStory, and similar

To add or fix a branch, edit MOVEMENT_LINEAGE in that file and rebuild the client. No migration or API change is required.

Movement gallery interiors (frontend)

Separate from movement lineage layout, client/src/data/movement-interior-styles.ts defines a unique 3D interior for each seeded art movement (26 styles): wall/floor/ceiling textures, trim colours, window style, and architectural details (columns, coffered ceilings, etc.). Textures are generated procedurally in client/src/utils/galleryProceduralTextures.ts.

Wing layout (up to 55 works per wing, U-shaped hang, window gap placement) lives in client/src/utils/movementHallLayout.ts. Visit order fills the left wall (first work at the entrance), then the far/end wall, then the right (last work at the entrance). The same hang applies to artist halls and guided tours. To change a movement’s look, edit its entry in movement-interior-styles.ts and rebuild the client.

Historical event markers (frontend timeline)

client/src/data/historical-events.ts lists world-history markers shown on Timeline.tsx (French Revolution, World War I/II, etc.).

Aspect	Detail
Storage	TypeScript module in the client — not a database table
Format	{ id, name, startYear, endYear?, shortLabel? } — omit endYear for a single-year pin
Interaction	Click a marker to zoom the shared timeline/movement view to that period
Year axis labels	Dynamic density in Timeline.tsx via chooseTimelineTickInterval() — fewer labels when zoomed out
Pan/zoom batching	createViewChangeScheduler() in timelineView.ts — one React update per animation frame
Vertical guides	TimelineEventGuides.tsx draws faint gold lines (or shaded spans) from the marker row down through the movement flow, aligned to the same year scale

Edit HISTORICAL_EVENTS and rebuild the client to extend the set.

Curator notes (painting editorial text)

Public notes authored by curators on a painting — separate from Wikipedia description, tour stop text, and art-history annotations.

Aspect	Detail
Storage	paintings.curator_notes (TEXT NOT NULL DEFAULT '')
Migration	db/migrate-curator-notes.sql via npm run dev:migrate
UI	Panel on painting detail (after tour notes, before description); curators edit inline when signed in
Hall marker	Small brass plate beneath the frame in the 3D hall when notes are non-empty
API	Included on GET /api/paintings/:id (p.*); update via PATCH /api/paintings/:id/curator-notes (curator)
i18n	Canonical English only for now (not in the translations pipeline)

Painting annotations (art-history notes)

Short art-history citations on the painting detail page — separate from the influence graph and from curator_notes.

Aspect	Detail
Storage	PostgreSQL table painting_annotations
UI	PaintingAnnotations.tsx — numbered markers on the image (when pos_x / pos_y set) plus an “Art history notes” list
API	Included as annotations[] on GET /api/paintings/:id
Curated data	scripts/painting-annotations-data.js — artist/title keys matched via influence-resolver.js
Load	npm run dev:update-painting-annotations (replaces existing rows per painting by default)
Wikipedia pass	npm run dev:update-painting-annotations -- --wikipedia — one intro sentence per work from wikipedia_title; use --wiki-delay=3000 if rate-limited; --no-replace to append without clearing curated rows

Categories include subject, technique, context, and symbolism. Sources cite Gombrich, museum catalogs, and Wikipedia as appropriate.

Batch image fetch

npm run dev:fetch-images (alias: npm run dev:search-missing-paintings) runs scripts/fetch-missing-images.js. It searches multiple sources for paintings without local files:

Source	Notes
Wikidata / Wikipedia	Article image + P18 property
Wikimedia Commons	Direct file + search
Wikipedia search	Discovers better article title when seed wikipedia_title is a museum catalog label
Google Arts & Culture	Search + asset pages (artsandculture.google.com)
Met Museum	Open Access API
Art Institute of Chicago	IIIF open access
Cleveland Museum of Art	CC0 API
Rijksmuseum	Public API
Musée du Louvre	Collections search (collections.louvre.fr)
Europeana	European museum aggregator (optional EUROPEANA_API_KEY in .env)
Smithsonian	Optional (SMITHSONIAN_API_KEY in .env)
Harvard Art Museums	Optional (HARVARD_ART_API_KEY in .env)

npm run dev:fetch-images                              # all missing, catalog order (~hours)
npm run dev:fetch-images -- --limit=50                # random sample of 50; 10s max per painting
npm run dev:fetch-images -- --limit=250               # random sample up to N (caps at current missing count)
npm run dev:fetch-images -- --limit=50 --max-wait=120 # slower, more thorough lookup per painting
npm run dev:fetch-images -- --artist="Albrecht Dürer" # one artist, catalog order
npm run dev:fetch-images -- --discover-only --limit=20   # fix wikipedia_title only
npm run dev:fetch-images -- --web-search-only --limit=50 # DuckDuckGo + Commons + multilingual Wikipedia

Each run prints Missing local files: N at startup — that is the current count of catalogued paintings with no full-size or thumbnail file under data/images/. A painting counts as present if either file exists on disk.

Flag	Effect
--limit=N	Process at most N paintings. Queue is a random sample of all works missing local files (not alphabetical).
--max-wait=N	Stop each painting after N seconds (default 10). Logs ⏱ timeout and continues.
--artist="Name"	Only that artist’s missing works, in catalog order (sort_order, year).
--discover-only	Update wikipedia_title via search; no download.
--web-search-only	Skip museum APIs; use web search + Commons + multilingual Wikipedia.

Each limited batch run stops per painting after --max-wait seconds (default 10), including source lookup and download. While a batch deadline is active, inter-request throttling is skipped and each HTTP call times out at the remaining budget (not the full 15s on-demand limit). Override the default via FETCH_MAX_WAIT_SEC in .env or --max-wait=N. On-demand fetches in the web UI keep their separate 15s API timeout and are unaffected.

Re-run the same command to pick a new random batch until the missing count reaches zero.

On-demand image resolution

When a painting has no local file, GET /api/paintings/:id/image triggers ensurePaintingImages():

1. Check DB paths → verify file on disk.
2. Scan disk by {artist}_{title} pattern.
3. If still missing and wikipedia_title is set, call scripts/image-fetcher.js:
   • Resolve overrides and simplified titles
   • Wikipedia search when catalog labels fail
   • Wikidata → Wikimedia Commons → Wikipedia page image
   • Fallbacks: Met Museum, Art Institute of Chicago, Cleveland Museum, Rijksmuseum, Smithsonian*, Harvard*
4. Save full image, generate thumbnail by resizing the full file (not a separate Commons thumb URL), update DB, serve file. If a full file already exists but a dedicated thumb under paintings/thumbs/ is missing, regenerate the thumb (ensurePaintingThumbFromFull) rather than serving the full image as a thumb.

Separate Wikipedia/Commons thumbnail URLs often resolve to the wrong work (e.g. a different painting with a similar title). Thumbnails are always derived locally from the downloaded full image via sharp in scripts/image-fetcher.js. See Painting thumbnail invariant.

Requests are deduplicated (inflight map) and timeout after 15 seconds. On-demand resolution uses a ~2.5 s delay between external requests to reduce rate-limit risk; batch fetch-images runs skip that delay while the per-painting deadline is active.

Preload before 3D gallery

POST /api/artists/:id/preload-images and POST /api/movements/:id/preload-images are public routes (no curator login). They run local-only linking — no network — and regenerate missing *_thumb.jpg files from full images on disk. The React client calls them automatically when entering an artist or movement 3D hall so textures use thumbs already on disk.

Works without files still show the canvas cover in the frame.

The 3D scene uses galleryImageUrl(), which never hits the on-demand API (remote latency breaks WebGL texture loading).

Placeholders

When no image is available:

• /placeholder-portrait.svg — timeline and movement-flow portraits
• /placeholder-art.svg — paintings in lists and detail view
• 3D gallery — draped canvas cover inside the frame (CanvasCover in VirtualGallery.tsx); shown when there is no local file, the fetch failed, or the texture has not loaded yet

These live in client/public/ (and client/dist/ after build).

Adding new artists manually

1. Insert rows into artists, artist_periods, paintings (or extend the seed script).
2. Place image files under data/images/ using the naming convention.
3. Run npm run dev:fetch-artist-bios for the new artist’s biography.
4. Add entries to famous-paintings-data.js and run npm run dev:expand-catalog if needed.
5. Run npm run dev:fetch-images -- --artist="…" or rely on preload / on-demand sync.
6. Add influence rows via npm run dev:update-influences / art-influences-data.js (writes both painting_influence_sources and legacy painting edges).

PainterPalette external dataset

Inputs/PainterPalette.csv is a curated dataset (~10,000 painters) merging WikiArt, Art500k, and Wikidata with cleaned influence fields. The gallery integrates it for existing DB artists only (not a full catalog import).

One-time setup

npm run dev:migrate:artist-palette    # adds artists.palette_metadata JSONB
npm run dev:import-painter-palette    # enrich + influence links

What gets imported

PainterPalette column	Gallery use
Nationality, gender, styles, birth/death places, occupations, …	Stored in artists.palette_metadata
birth_year, death_year	Fills missing DB years only
Influencedby, Teachers	Artist or movement sources on that artist's paintings
Influencedon, Pupils	Reverse artist sources on the pupil/successor's paintings

Influence rows are written to painting_influence_sources with discovered_via = painter-palette and do not duplicate curated Gombrich edges (unique index + ON CONFLICT DO NOTHING).

Name matching uses normalized strings plus aliases in scripts/painter-palette-lib.js (e.g. Bronzino → Agnolo Bronzino, J. M. W. Turner → J.M.W. Turner). Museum names, WikiArt tags, and dimension strings are filtered out.

Commands

npm run dev:analyze-painter-palette              # match report
npm run dev:import-painter-palette -- --dry-run
npm run dev:import-painter-palette -- --metadata-only
npm run dev:import-painter-palette -- --influences-only

Re-run import-painter-palette after adding gallery artists or updating the CSV; existing palette influence rows are skipped if already present.

Catalog export

Export helpers write under Output/ (gitignored — regenerate as needed):

npm run dev:export-paintings

Writes Output/paintings.csv with columns artist, painting, year (sorted by artist, year, title).

Ad-hoc prod dumps (movements / artists / paintings) can also be written to Output/*.csv for offline review. Influence workbooks ready for the curator import wizard belong under Inputs/ (e.g. Gariff, Story of Art) — see influence-import.md.

Image fetcher overrides

scripts/image-fetcher.js includes hand-maintained overrides for ambiguous Wikipedia titles and direct URLs (e.g. works whose Commons name does not match the article title, or when museum search returns the wrong work). Extend these maps when automated resolution fails:

Map	Use when
PAINTING_WIKI_OVERRIDES	DB / seed title should resolve to a different Wikipedia or Wikidata label
DIRECT_IMAGE_OVERRIDES	You know the exact Commons URL (bypasses Met / Art Institute false matches)

Examples already in the repo:

• Self-Portrait Hesitating → Kauffman, Wikimedia Commons (National Trust)
• Cherubs of the Sistine Madonna → Raphael’s putti detail, Wikimedia Commons
• Madonna and Child (Madonna della Seggiola) / Madonna della seggiola → Raphael’s tondo, Palazzo Pitti
• Madonna and Child (Raphael) → Small Cowper Madonna, National Gallery of Art
• Job Cigarette Papers → Mucha poster disambiguation
• Charing Cross Bridge → Derain (not Monet)

After adding an override, delete any wrong cached file under data/images/paintings/ and re-run fetch or call the on-demand image endpoint for that painting.

Duplicate paintings

The catalog can contain the same work more than once — usually from a double import (identical artist + title + year + image) or Wikipedia scrape variants (different article titles for one icon, e.g. Andrei Rublev’s Trinity).

Find duplicates

npm run dev:find-duplicates

Runs scripts/find-duplicate-paintings.js, which reports:

Report	Rule
Exact duplicates	Same artist_id, title, and year
Normalized title duplicates	Same artist, titles differing only in punctuation/spacing
Rublev / Trinity cluster	Known multi-entry example for manual merge

As of a recent audit (~1200 paintings): 52 exact duplicate pairs (52 removable rows), concentrated in Hieronymus Bosch (25), Albrecht Dürer (14), and Domenico Ghirlandaio (13). Duplicate copies typically share the same image file and have no influence links, so the higher id in each pair is safe to delete after review.

Near-duplicates (different titles, same work) need curator judgment — e.g. Rublev ids 36, 316, 320, 322, 323 all describe the Trinity icon under different Wikipedia labels; keep id 36 (Trinity, wiki Trinity (Andrei Rublev)). For confirmed duplicates, debug Remove entry on painting detail is faster than manual SQL; it deletes files and the row in one step.

expand-paintings.js skips inserts when normalized titles match, but duplicates can still appear if seed and expansion use different title strings or if influence discovery creates works independently.

Debug and checkup image fix

When Debug mode is on (home header) or from the Checkup page:

Show more (home header checkbox, client/src/utils/debugMode.ts) — when debug mode is on, automatically opens the More modal on each painting detail or artist bio page load (same as clicking More).

1. Search — GET /api/paintings/:id/debug-image-search (or …/debug-portrait-search for artists) tries Google Custom Search (if GOOGLE_CSE_API_KEY + GOOGLE_CSE_CX are set in .env), Google Arts & Culture, Google Images scrape, then DuckDuckGo (searchGoogleImagesFirst / searchArtistPortraitFirst in scripts/image-fetcher.js).
2. More — GET …/debug-image-search/more or …/debug-portrait-search/more returns up to 20 ranked candidates (searchPaintingImagesMany / searchArtistPortraitMany). The modal shows each thumbnail with resolution when the search API provides dimensions; otherwise the client probes via GET /api/debug/image-proxy.
3. Fix — POST …/fix-image or …/fix-portrait downloads the chosen URL via downloadImageForFix → replacePaintingImageFromUrl / replaceArtistPortraitFromUrl in server/image-service.js. The server always regenerates thumbnails from the saved full image (writePaintingThumb / writePortraitThumb via sharp — not the search-result thumb URL), updates thumbnail_path / portrait_thumb_path, and sets checkup_fixed + checkup_checked.
4. Clear — POST …/clear-image or …/clear-portrait deletes local file(s), nulls DB paths, sets both flags. Cleared slots stay empty in the UI (no placeholder; checkup_fixed prevents on-demand refetch for paintings).
5. Upload — POST …/upload-image or …/upload-portrait accepts a base64-encoded file in JSON (Express body limit 20 MB; decoded image max 15 MB), validates with sharp, writes to the standard filename under data/images/, and regenerates the matching thumbnail the same way as Fix it. The file picker uses a native <label> + hidden <input> (DebugUploadButton.tsx) so the change event is reliable on Windows/Chromium.
6. Remove entry (painting detail only) — DELETE /api/paintings/:id via deletePainting() in server/image-service.js: deletes image files, removes the DB row (cascade on influence/annotation tables), refetches artist/movement gallery data, remounts the 3D hall, and navigates to the next or previous catalog work with no confirmation dialog.

Debug panel (painting detail and artist bio)

With debug mode on, PaintingDetail.tsx and ArtistBio.tsx show a bottom-left panel with search preview and action buttons:

Button	API (paintings / portraits)	Effect
Checked	PATCH …/checkup-flags { "checked": true }	Marks reviewed; gold frame (paintings) or gold portrait border (artists)
Fix it	POST …/fix-image / …/fix-portrait	Saves top search result to disk, regenerates thumb from full image, sets both flags, refreshes detail + gallery / timeline
More	GET …/debug-*-search/more then fix endpoint	Modal with 20 clickable results (resolution label under each thumb); pick one to replace (thumb regenerated from downloaded full)
Clear	POST …/clear-image / …/clear-portrait	Removes file(s), empty frame in UI
Upload	POST …/upload-image / …/upload-portrait	Local file picker (DebugUploadButton) → save full image + auto-generated thumb; centered Loading… overlay; clears preview and blocks search/fix while uploading
Remove entry	DELETE /api/paintings/:id	Paintings only — permanent delete + gallery refresh + catalog navigation

The client passes searchUrl, source, and thumbUrl from search results to improve download reliability. After a fix, clear, upload, or remove, HomePage updates the gallery session. Image URLs include ?v=<mtime_ms> from API image_cache_key / thumbnail_cache_key (file mtime on disk) so replaced files reload after a full page refresh even when the relative path is unchanged. In-session counters still bump immediately after a mutation.

Checkup Search visible queues search for filtered rows only (3 concurrent); it does not search the full catalog on load.

Curator login required for all debug/checkup UI and mutating API routes. Guests can browse and enter 3D halls normally; preload remains public. See API.md — Authentication.

See API.md and basics.md.