diff --git a/Documentation/API.md b/Documentation/API.md index 22bb529..0029b13 100644 --- a/Documentation/API.md +++ b/Documentation/API.md @@ -751,6 +751,8 @@ Google-family image search for debug / checkup (Custom Search → Google Arts & Download a remote URL and replace the painting’s local full image + thumbnail. Sets `checkup_fixed = true` and `checkup_checked = true`. +**Invariant:** every painting picture write (fix, upload, fetch, disk sync) must regenerate a dedicated `paintings/thumbs/*_thumb.jpg` from the full file — never reuse a remote thumb URL or point `thumbnail_path` at the full image. See [data-and-images.md — Painting thumbnail invariant](data-and-images.md#painting-thumbnail-invariant). + Uses `downloadImageForFix` in `scripts/image-fetcher.js` (browser User-Agent, referer fallbacks, Wikimedia URL upgrades) for reliable downloads from Google Arts, Commons, etc. **Body** diff --git a/Documentation/Plans.md b/Documentation/Plans.md index 0e447ac..8397c8c 100644 --- a/Documentation/Plans.md +++ b/Documentation/Plans.md @@ -1,5 +1,11 @@ this file contains draft for future releases and features +## Standing requirements (do not regress) + +- **Painting thumbnails:** any create/replace/clear of a painting’s full picture must regenerate or remove its dedicated `paintings/thumbs/` file — never use the full image or a remote thumb URL as `thumbnail_path`. See [data-and-images.md — Painting thumbnail invariant](data-and-images.md#painting-thumbnail-invariant). + +## Feature backlog + 1. ~~Multi language support, russian version at least~~ — done: UI i18n (EN/RU) + `entity_translations` DB + curator Translations tool — [i18n-russian.md](i18n-russian.md) 2. ~~tool to manage links (influence/influenced by ) import csv's ( define format), edit ,add, delete, visualize, map to pictures/ entities~~ — done: curator Influences page (list CRUD + import wizard CSV/JSON/XLSX + neighborhood graph) — [influence-import.md](influence-import.md) 3. tool to monitor/manage (plan actions) of curator actions, markers to check painting/text ? diff --git a/Documentation/data-and-images.md b/Documentation/data-and-images.md index d83e896..9b368ea 100644 --- a/Documentation/data-and-images.md +++ b/Documentation/data-and-images.md @@ -7,6 +7,21 @@ How catalog content, biographies, and artwork files enter the system. 1. **No runtime hot-linking** — the UI reads from `/images/…` (local disk). External URLs are used only during ingest. 2. **No AI-generated art or text** — biographies and descriptions come from Wikipedia; influence notes from curated art-history sources. 3. **Local copies** — every displayed image should exist under `data/images/` after seeding or fetch. +4. **Painting thumbnails must stay in sync with the full image** — see [Painting thumbnail invariant](#painting-thumbnail-invariant) below. Treat this as a hard requirement for any new feature or script that writes or replaces painting picture files. + +## Painting thumbnail invariant + +**Requirement (do not regress):** any action that **creates, replaces, or clears** a painting’s full picture must also **regenerate or remove** that painting’s dedicated thumbnail under `paintings/thumbs/`. Never point `thumbnail_path` at the full-size file as a stand-in, and never reuse a search-result or Commons thumb URL. + +| Change type | Expected thumb behavior | Primary code | +|-------------|-------------------------|--------------| +| Fix / upload / buffer or URL replace | Write full file, then `writePaintingThumb` / `generateThumbnailFromFull` | `server/image-service.js` | +| Fetch / seed / expand / influence ingest | Same: thumb from saved full via `savePaintingImages` | `scripts/image-fetcher.js` | +| Disk sync / preload / ensure when full exists but thumb missing | Generate real `*_thumb.jpg` and update `thumbnail_path` | `ensurePaintingThumbFromFull`, `syncPaintingFromDisk`, `ensurePaintingImages`, `preloadArtistImagesLocal` | +| Clear image / delete painting | Delete thumb file(s) and null paths (or drop row) | `clearPaintingImage`, `deletePainting` | +| Bulk rebuild after sync/harmonize | `regenerate-thumbnails.js` / `harmonize:images` | scripts + npm scripts | + +Path-only maintenance (`sync-image-paths.js`) may link existing thumb files without rewriting pixels; if full images changed without a matching thumb step, run `npm run dev:regenerate-thumbnails` (or `harmonize:images`). ## Directory layout @@ -364,9 +379,9 @@ When a painting has no local file, `GET /api/paintings/:id/image` triggers `ensu - **Wikipedia search** when catalog labels fail - Wikidata → Wikimedia Commons → Wikipedia page image - Fallbacks: Met Museum, Art Institute of Chicago, Cleveland Museum, Rijksmuseum, Smithsonian*, Harvard* -4. Save full image, **generate thumbnail by resizing the full file** (not a separate Commons thumb URL), update DB, serve file. +4. Save full image, **generate thumbnail by resizing the full file** (not a separate Commons thumb URL), update DB, serve file. If a full file already exists but a dedicated thumb under `paintings/thumbs/` is missing, regenerate the thumb (`ensurePaintingThumbFromFull`) rather than serving the full image as a thumb. -Separate Wikipedia/Commons thumbnail URLs often resolve to the **wrong work** (e.g. a different painting with a similar title). Thumbnails are always derived locally from the downloaded full image via `sharp` in `scripts/image-fetcher.js`. +Separate Wikipedia/Commons thumbnail URLs often resolve to the **wrong work** (e.g. a different painting with a similar title). Thumbnails are always derived locally from the downloaded full image via `sharp` in `scripts/image-fetcher.js`. See [Painting thumbnail invariant](#painting-thumbnail-invariant). Requests are deduplicated (`inflight` map) and timeout after 15 seconds. On-demand resolution uses a ~2.5 s delay between external requests to reduce rate-limit risk; batch `fetch-images` runs skip that delay while the per-painting deadline is active. diff --git a/server/image-service.js b/server/image-service.js index 62302a9..14216d9 100644 --- a/server/image-service.js +++ b/server/image-service.js @@ -96,10 +96,15 @@ function localFileExists(relPath) { return fs.existsSync(path.join(IMAGE_DIR, relPath)); } -function syncPaintingFromDisk(row) { - const safeBase = `${row.artist_name}_${row.title}`.replace(/[^a-zA-Z0-9_-]/g, '_'); - let imagePath = row.image_path; - let thumbPath = row.thumbnail_path; +function isPaintingThumbRel(rel) { + if (!rel) return false; + return rel.replace(/\\/g, '/').startsWith('paintings/thumbs/'); +} + +/** Discover full/thumb files on disk for a painting basename. */ +function discoverPaintingFilesOnDisk(safeBase) { + let imagePath = null; + let thumbPath = null; for (const ext of ['.jpg', '.jpeg', '.png', '.webp', '.JPG']) { const full = path.join(IMAGE_DIR, 'paintings', safeBase + ext); @@ -111,11 +116,50 @@ function syncPaintingFromDisk(row) { thumbPath = `paintings/thumbs/${safeBase}_thumb${ext}`; } } - if (!thumbPath && imagePath) thumbPath = imagePath; + const jpgThumb = path.join(IMAGE_DIR, 'paintings', 'thumbs', `${safeBase}_thumb.jpg`); + if (!thumbPath && fs.existsSync(jpgThumb)) { + thumbPath = `paintings/thumbs/${safeBase}_thumb.jpg`; + } return { imagePath, thumbPath }; } -/** Fast preload: link local files only, no external API calls */ +/** + * Ensure a dedicated thumbs/ file exists for a full painting image. + * Regenerates from the full file when missing. + */ +async function ensurePaintingThumbFromFull(fullRel, safeBase) { + if (!fullRel || !localFileExists(fullRel)) return null; + const expectedThumb = `paintings/thumbs/${safeBase}_thumb.jpg`; + if (localFileExists(expectedThumb)) return expectedThumb; + try { + return await writePaintingThumb(path.join(IMAGE_DIR, fullRel), safeBase); + } catch (err) { + console.warn(`Painting thumb generation failed for ${safeBase}:`, err.message); + return null; + } +} + +/** Sync/link paths from disk; generate thumbnail when full exists without a thumbs/ file. */ +async function syncPaintingFromDisk(row) { + const safeBase = safePaintingBase(row.artist_name, row.title); + let imagePath = localFileExists(row.image_path) ? row.image_path : null; + let thumbPath = + localFileExists(row.thumbnail_path) && isPaintingThumbRel(row.thumbnail_path) + ? row.thumbnail_path + : null; + + const discovered = discoverPaintingFilesOnDisk(safeBase); + if (!imagePath && discovered.imagePath) imagePath = discovered.imagePath; + if (!thumbPath && discovered.thumbPath) thumbPath = discovered.thumbPath; + + if (!thumbPath && imagePath) { + thumbPath = (await ensurePaintingThumbFromFull(imagePath, safeBase)) || null; + } + + return { imagePath, thumbPath }; +} + +/** Fast preload: link local files only, no external API calls; regenerate missing thumbs from full files. */ async function preloadArtistImagesLocal(artistId) { const rows = await pool.query( `SELECT p.id, p.title, p.image_path, p.thumbnail_path, a.name AS artist_name @@ -127,14 +171,14 @@ async function preloadArtistImagesLocal(artistId) { let linked = 0; for (const row of rows.rows) { - const hasLocal = - localFileExists(row.thumbnail_path) || localFileExists(row.image_path); - if (hasLocal) { + const hasFull = localFileExists(row.image_path); + const hasThumb = localFileExists(row.thumbnail_path) && isPaintingThumbRel(row.thumbnail_path); + if (hasFull && hasThumb) { linked++; continue; } - const synced = syncPaintingFromDisk(row); + const synced = await syncPaintingFromDisk(row); if (synced.imagePath || synced.thumbPath) { await pool.query( `UPDATE paintings SET image_path = COALESCE($1, image_path), thumbnail_path = COALESCE($2, thumbnail_path) WHERE id = $3`, @@ -163,12 +207,27 @@ async function ensurePaintingImages(paintingId, size = 'thumb') { const row = result.rows[0]; const wantThumb = size !== 'full'; + const safeBase = safePaintingBase(row.artist_name, row.title); - if (wantThumb && localFileExists(row.thumbnail_path)) return row.thumbnail_path; + if (wantThumb && localFileExists(row.thumbnail_path) && isPaintingThumbRel(row.thumbnail_path)) { + return row.thumbnail_path; + } if (!wantThumb && localFileExists(row.image_path)) return row.image_path; - if (wantThumb && localFileExists(row.image_path)) return row.image_path; - const synced = syncPaintingFromDisk(row); + // Full on disk but no dedicated thumb — regenerate before falling back to full file. + if (wantThumb && localFileExists(row.image_path)) { + const thumbRel = await ensurePaintingThumbFromFull(row.image_path, safeBase); + if (thumbRel) { + await pool.query(`UPDATE paintings SET thumbnail_path = $1 WHERE id = $2`, [ + thumbRel, + paintingId, + ]); + return thumbRel; + } + return row.image_path; + } + + const synced = await syncPaintingFromDisk(row); if (synced.imagePath || synced.thumbPath) { await pool.query( `UPDATE paintings @@ -177,8 +236,20 @@ async function ensurePaintingImages(paintingId, size = 'thumb') { WHERE id = $3`, [synced.imagePath, synced.thumbPath, paintingId] ); - if (wantThumb && localFileExists(synced.thumbPath)) return synced.thumbPath; - if (wantThumb && localFileExists(synced.imagePath)) return synced.imagePath; + if (wantThumb && localFileExists(synced.thumbPath) && isPaintingThumbRel(synced.thumbPath)) { + return synced.thumbPath; + } + if (wantThumb && localFileExists(synced.imagePath)) { + const thumbRel = await ensurePaintingThumbFromFull(synced.imagePath, safeBase); + if (thumbRel) { + await pool.query(`UPDATE paintings SET thumbnail_path = $1 WHERE id = $2`, [ + thumbRel, + paintingId, + ]); + return thumbRel; + } + return synced.imagePath; + } if (!wantThumb && localFileExists(synced.imagePath)) return synced.imagePath; }