pornboxdBETA
← Field Notes
Devlog

Two new studios, a manual backfill tool, and one ugly URL fix

Today was catalogue-expansion day. Two more studios both publish JSON catalogues, so neither needed an Apify scraper, they plugged into the same writer

Today was catalogue-expansion day. Two more studios both publish JSON catalogues, so neither needed an Apify scraper, they plugged into the same writer pipeline the first JSON-feed studio uses, just with a different feed-shape mapper (about forty lines of code each). The feed auto-detects based on the first item's field names, so adding a third feed shape later means writing one more mapper, not redesigning the import path. A new nats_platform column on the studios table lets multiple studios share the same /nats postback URL but still attribute conversions to the right account, different networks get different platforms.

The manual-backfill path for the missing dates came in as a Tampermonkey userscript instead of an automated scraper. You open a catalogue page in your browser, paste the userscript, a floating panel extracts every video's metadata from the already-rendered Vue.js HTML, and clicks "Send to API" to POST the batch to a new POST /admin/videos/bulk-update endpoint. Since the script runs in your browser with your real session cookies and residential IP, there's no Cloudflare challenge to fight, it's the same HTTP requests a normal visitor makes, just capturing the data. Roughly 100 videos per page, a few seconds to extract, idempotent (only fills NULL fields, safe to re-run). The admin panel also got a proper /admin/videos page with filters (missing date, studio, search) and an inline release-date editor, one click edits a date, the row flashes green on save, and auto-removes from "Missing date" view.

The ugly URL fix: two studios each had a video with slug outdoors, and the old routing couldn't tell them apart. Video URLs are now namespaced under the studio, /videos/studio-a/outdoors vs /videos/studio-b/outdoors. A UNIQUE (studio_id, slug) constraint on the videos table enforces it. On the image pipeline side: added fetch timeouts and a per-run timeout with an admin "Terminate" button, because a few stuck image downloads were tying up the image processor indefinitely.