Danilka/Art-gallery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Improve batch image fetch speed and document the pipeline.

Browse Source 
Add random queue sampling and a 10s per-painting deadline for fetch-images batches, cap HTTP timeouts to the remaining budget, and update docs plus newly fetched artwork files.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Danila Khodjaef
2026-06-19 17:59:32 +03:00
co-authored by Cursor
parent d8385d83d6
commit 35c337253d
199 changed files with 206 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/data-and-images.md
+17 -5
View File
@@ -150,14 +150,26 @@ Extend the data file to add lineage chains (e.g. Giotto → Masaccio → Michela
| Harvard Art Museums | Optional (`HARVARD_ART_API_KEY` in `.env`) |
```bash
npm run fetch-images # all missing (~hours; rate-limited)
npm run fetch-images -- --limit=50 # batch of 50
npm run fetch-images -- --artist="Albrecht Dürer"
npm run fetch-images # all missing, catalog order (~hours)
npm run fetch-images -- --limit=50 # random sample of 50; 10s max per painting
npm run fetch-images -- --limit=920 # random sample up to N missing works
npm run fetch-images -- --limit=50 --max-wait=120 # slower, more thorough lookup per painting
npm run fetch-images -- --artist="Albrecht Dürer" # one artist, catalog order
npm run fetch-images -- --discover-only --limit=20 # fix wikipedia_title only
npm run fetch-images -- --web-search-only --limit=50 # DuckDuckGo + Commons + multilingual Wikipedia
```
When a better Wikipedia article is found, `wikipedia_title` in the DB is updated automatically.
| Flag | Effect |
|------|--------|
| `--limit=N` | Process at most **N** paintings. Queue is a **random sample** of all works missing local files (not alphabetical). |
| `--max-wait=N` | Stop each painting after **N** seconds (default **10**). Logs `⏱ timeout` and continues. |
| `--artist="Name"` | Only that artist’s missing works, in catalog order (`sort_order`, `year`). |
| `--discover-only` | Update `wikipedia_title` via search; no download. |
| `--web-search-only` | Skip museum APIs; use web search + Commons + multilingual Wikipedia. |
Each limited batch run stops per painting after **`--max-wait` seconds** (default **10**), including source lookup and download. While a batch deadline is active, inter-request throttling is skipped and each HTTP call times out at the **remaining** budget (not the full 15s on-demand limit). Override the default via `FETCH_MAX_WAIT_SEC` in `.env` or `--max-wait=N`. On-demand fetches in the web UI keep their separate 15s API timeout and are unaffected.
Re-run the same command to pick a new random batch until the missing count reaches zero.
## On-demand image resolution
@@ -174,7 +186,7 @@ When a painting has no local file, `GET /api/paintings/:id/image` triggers `ensu
Separate Wikipedia/Commons thumbnail URLs often resolve to the **wrong work** (e.g. a different painting with a similar title). Thumbnails are always derived locally from the downloaded full image via `sharp` in `scripts/image-fetcher.js`.
Requests are deduplicated (`inflight` map) and timeout after 15 seconds. A ~2.5 s delay between external requests reduces rate-limit risk.
Requests are deduplicated (`inflight` map) and timeout after 15 seconds. On-demand resolution uses a ~2.5 s delay between external requests to reduce rate-limit risk; batch `fetch-images` runs skip that delay while the per-painting deadline is active.
## Preload before 3D gallery