Improve batch image fetch speed and document the pipeline.

Add random queue sampling and a 10s per-painting deadline for fetch-images batches, cap HTTP timeouts to the remaining budget, and update docs plus newly fetched artwork files.

Co-authored-by: Cursor <cursoragent@cursor.com>

.env.example

@@ -9,3 +9,4 @@ IMAGE_DIR=./data/images
 # Optional — enable extra museum search in fetch-missing-images / search-missing-paintings
 # SMITHSONIAN_API_KEY=
 # HARVARD_ART_API_KEY=
+# FETCH_MAX_WAIT_SEC=10

Documentation/data-and-images.md

@@ -150,14 +150,26 @@ Extend the data file to add lineage chains (e.g. Giotto → Masaccio → Michela
 | Harvard Art Museums | Optional (`HARVARD_ART_API_KEY` in `.env`) |
 
 ```bash
-npm run fetch-images                              # all missing (~hours; rate-limited)
-npm run fetch-images -- --limit=50                # batch of 50
-npm run fetch-images -- --artist="Albrecht Dürer"
+npm run fetch-images                              # all missing, catalog order (~hours)
+npm run fetch-images -- --limit=50                # random sample of 50; 10s max per painting
+npm run fetch-images -- --limit=920               # random sample up to N missing works
+npm run fetch-images -- --limit=50 --max-wait=120 # slower, more thorough lookup per painting
+npm run fetch-images -- --artist="Albrecht Dürer" # one artist, catalog order
 npm run fetch-images -- --discover-only --limit=20   # fix wikipedia_title only
 npm run fetch-images -- --web-search-only --limit=50 # DuckDuckGo + Commons + multilingual Wikipedia
 ```
 
-When a better Wikipedia article is found, `wikipedia_title` in the DB is updated automatically.
+| Flag | Effect |
+|------|--------|
+| `--limit=N` | Process at most **N** paintings. Queue is a **random sample** of all works missing local files (not alphabetical). |
+| `--max-wait=N` | Stop each painting after **N** seconds (default **10**). Logs `⏱ timeout` and continues. |
+| `--artist="Name"` | Only that artist’s missing works, in catalog order (`sort_order`, `year`). |
+| `--discover-only` | Update `wikipedia_title` via search; no download. |
+| `--web-search-only` | Skip museum APIs; use web search + Commons + multilingual Wikipedia. |
+
+Each limited batch run stops per painting after **`--max-wait` seconds** (default **10**), including source lookup and download. While a batch deadline is active, inter-request throttling is skipped and each HTTP call times out at the **remaining** budget (not the full 15s on-demand limit). Override the default via `FETCH_MAX_WAIT_SEC` in `.env` or `--max-wait=N`. On-demand fetches in the web UI keep their separate 15s API timeout and are unaffected.
+
+Re-run the same command to pick a new random batch until the missing count reaches zero.
 
 ## On-demand image resolution
 
@@ -174,7 +186,7 @@ When a painting has no local file, `GET /api/paintings/:id/image` triggers `ensu
 
 Separate Wikipedia/Commons thumbnail URLs often resolve to the **wrong work** (e.g. a different painting with a similar title). Thumbnails are always derived locally from the downloaded full image via `sharp` in `scripts/image-fetcher.js`.
 
-Requests are deduplicated (`inflight` map) and timeout after 15 seconds. A ~2.5 s delay between external requests reduces rate-limit risk.
+Requests are deduplicated (`inflight` map) and timeout after 15 seconds. On-demand resolution uses a ~2.5 s delay between external requests to reduce rate-limit risk; batch `fetch-images` runs skip that delay while the per-painting deadline is active.
 
 ## Preload before 3D gallery
 

Documentation/setup.md

@@ -29,6 +29,7 @@ Optional script tuning:
 | Variable | Used by | Description |
 |----------|---------|-------------|
 | `MIN_PAINTINGS` | `expand-catalog` | Minimum paintings per artist (default `6`) |
+| `FETCH_MAX_WAIT_SEC` | `fetch-images` | Max seconds per painting in batch runs (default `10`) |
 
 `.env` is git-ignored; never commit passwords.
 
@@ -64,11 +65,12 @@ After a fresh seed, run these to match a fully populated local install:
 npm run fetch-artist-bios          # bio_short / bio_full from Wikipedia
 npm run expand-catalog             # famous works for artists below MIN_PAINTINGS
 npm run update-influences          # painting influence graph for detail view + hall exits
-npm run fetch-images -- --artist="Gustav Klimt"   # repeat per artist or omit --artist for all
+npm run fetch-images -- --limit=50                # random sample; 10s max per painting (default)
+npm run fetch-images -- --limit=50 --max-wait=120 # same batch size, longer lookup per work
 cd client && npm run build && cd ..
 ```
 
-Image fetch is rate-limited (~3.5 s between external requests) and can take hours for the full catalog. Fetch per artist, or rely on on-demand resolution when viewing a painting in the detail view.
+Image fetch can take hours for the full catalog (~900+ missing works after seed). Use **`npm run fetch-images -- --limit=N`** for random batches (10s per painting by default), **`--artist="…"`** for one artist in catalog order, or on-demand resolution when viewing a painting in the detail view.
 
 ## Run
 
@@ -99,8 +101,9 @@ Open http://localhost:3001 (or your configured `PORT`).
 | `npm run expand-catalog -- --fetch-images` | ↑ | Also download images for new rows (slow) |
 | `npm run fetch-images` | `scripts/fetch-missing-images.js` | Search Wikipedia, Commons, museums for missing files |
 | `npm run search-missing-paintings` | ↑ (alias) | Same as `fetch-images` |
-| `npm run fetch-images -- --artist="Name"` | ↑ | Limit to one artist |
-| `npm run fetch-images -- --limit=50` | ↑ | Process at most N paintings |
+| `npm run fetch-images -- --artist="Name"` | ↑ | Limit to one artist (catalog order) |
+| `npm run fetch-images -- --limit=50` | ↑ | Random sample of N missing paintings |
+| `npm run fetch-images -- --limit=50 --max-wait=120` | ↑ | Random batch; 120s cap per painting |
 | `npm run fetch-images -- --discover-only --limit=20` | ↑ | Fix `wikipedia_title` via search only |
 | `npm run regenerate-thumbnails` | `scripts/regenerate-thumbnails.js` | Rebuild all thumbs from full local files |
 | `npm run audit-painting-images` | `scripts/audit-painting-images.js` | List thumb/full aspect-ratio mismatches |
@@ -143,7 +146,8 @@ After clone: copy `.env.example` → `.env`, install dependencies, run `npm run
 | 500 on all `/api/*` | Wrong `.env` or Postgres down | Check connection, logs |
 | “Biographical information not yet available” | Bios not fetched | `npm run fetch-artist-bios` |
 | Artist hall has only 1–2 paintings | Catalog not expanded | `npm run expand-catalog`; extend `famous-paintings-data.js` |
-| Black frames / canvas covers in 3D gallery | No local image for painting | `npm run fetch-images -- --artist="…"` or `POST …/preload-images` after fetch |
+| Black frames / canvas covers in 3D gallery | No local image for painting | `npm run fetch-images -- --limit=50` or `--artist="…"`; then `POST …/preload-images` |
+| Many `⏱ timeout` lines in fetch batch | Default 10s cap too short for hard works | `--max-wait=120` or raise `FETCH_MAX_WAIT_SEC` |
 | White/grey flicker on frames | Texture loading or z-fighting with wall | Rebuild client (`cd client && npm run build`); ensure latest `VirtualGallery.tsx` |
 | Wrong painting in 3D gallery frame | Stale or mismatched thumbnail file | `npm run regenerate-thumbnails`; gallery prefers full `image_path` |
 | Wrong painting image (fetch) | Bad museum / search match on first download | Add entry to `DIRECT_IMAGE_OVERRIDES` in `scripts/image-fetcher.js`, re-fetch file |

README.md

@@ -40,7 +40,8 @@ Interactive virtual art gallery: zoomable historical timeline, art movement band
    npm run fetch-artist-bios    # Wikipedia biographies for all artists
    npm run expand-catalog       # add famous works for artists with thin catalogs
    npm run update-influences    # art-history lineage links between paintings
-   npm run fetch-images -- --artist="Claude Monet"   # images per artist (slow; repeat)
+   npm run fetch-images -- --limit=50         # random batch of missing images (10s per work)
+   npm run fetch-images -- --artist="Claude Monet"   # one artist in catalog order
    ```
 
 5. Build the client and start the server: