The hard part of AI artwork isn't the artwork

I have a hobby project that paints children's storybooks. Watercolour pages, a family's own characters, the same faces appearing page after page. It is not a commercial enterprise and I'd like to keep it that way — which is lucky, because it has taught me more about the gap between "AI can generate images" and "AI can do a job" than anything I've built professionally.

The generation part took an afternoon. Gemini's image model (the one Google nicknamed Nano Banana) produces genuinely lovely watercolour illustration from a paragraph of prompt. If all you need is a picture, it's solved.

What I actually needed was the same picture-of-a-character twelve times, in twelve different scenes. That took three or four months of evenings, and the system that eventually worked looks less like prompt engineering and more like a manufacturing quality pipeline.

Drift, and the naive era

The failure mode has a name in my commit history: drift. The dog on page 7 has different ears from the dog on page 2. The boy's jumper changes colour. A character described as small and scruffy arrives tall and groomed halfway through a book, and no reader over four years old misses it.

My first instinct was everyone's first instinct: describe the character harder. Longer attribute lists, firmer language, the word "consistent" doing a lot of hopeful work. The commits from that era read like an argument I was losing — lock the cast more explicitly to defend against drift is a real message from early on. Prompt-only identity got me maybe eight pages out of twelve on-model. The remaining four were the product.

The first real improvement was giving the model something to look at instead of something to read. The pipeline settled on a small hierarchy of reference imagery:

• Character masters — one canonical portrait per character, chosen by a human from a batch of generated candidates. This is the identity.
• Cast sheets — a composite of everyone appearing on a page, passed alongside the page prompt.
• The anchor page — the opening spread, which every later page is conditioned on.

Reference images moved the hit rate a long way. They also introduced two problems I didn't have before, and both are more interesting than the one they solved.

The grader problem

Once you're generating at any volume, checking pages by eye stops being viable — and "does the dog match?" is exactly the kind of question a vision model can answer. So the pipeline grew a verifier: every painted page is checked by a vision model against the cast references, and each book carries a set of invariants (which characters appear where, what must be true of them) that get verified before anything is published.

Grading with the same vendor that painted is cheap and mostly works, with one structural flaw: a systematic blind spot passes every check. If Gemini genuinely can't see the difference between two fox renders, then Gemini-the-grader waves through what Gemini-the-painter got wrong. The fix was a second-opinion sampler — a small percentage of grader verdicts get re-judged by a different vendor's vision model, logged purely for calibration, never changing the outcome. It exists to measure how often the primary grader is wrong, which is a number you cannot get from the primary grader.

The economics of checking are worth pausing on, because they're the opposite of what intuition suggests. An image generation costs on the order of 7¢; a vision check costs around 0.005¢. That ratio means you can be extravagant with verification. A failed check is confirmed by majority vote — three graders must agree a page has drifted before anything gets repainted — and the voting is effectively free. One over-strict grader on a bad day can no longer trigger a re-paint on its own.

Then there are the false positives, which taught me more about the problem than the true ones. The grader flagged a satchel set down at bedtime as a missing-prop failure — it wasn't drift, it was the scene. It flagged mud on a character who'd earned that mud three pages earlier. It judged pyjamas at bedtime as a wardrobe violation because the canonical outfit is a jumper. Each of these became a rule: the identity question the grader is asked is now wardrobe-aware and scene-aware, and distinguishes story-earned change from drift. Writing that prompt was indistinguishable from writing acceptance criteria for a very literal-minded QA engineer.

The pipeline that grew

The full loop, as it stands — with the points of interest that took longest to learn marked on it:

Three of those boxes carry scars:

The anchor gate. Because every later page is conditioned on the opening spread, a drifted page 1 doesn't fail one page — it injects the error into the whole book. Page 1 now has to pass a per-character identity check before pages 2 onward are allowed to paint. Catching one bad page early is worth eleven repaints later.

The sweep baselines against the character's own reference — not the first appearance. The early version compared pages against each other, taking a character's first appearance as truth. Then a character entered mid-book off-model, the sweep enshrined the mistake, and every correct later page got "repaired" toward the wrong version. The baseline had to become the canonical master. The system was doing exactly what I'd built instead of what I'd meant, which is the oldest bug there is.

The repair loop is bounded. Sweep, repaint only the confirmed-drifted pages, re-sweep, at most a configured number of rounds. It either converges or surfaces the leftovers for a human — it never repaints unboundedly and never silently ships a page it knows is wrong. Before this existed the sweep produced findings and I ran the repaints by hand, which is fine for a hobby and disqualifying for anything else.

One more lesson deserves its own line, because it cost me a whole book: identity references fix who is in the picture, and they leak how it's painted. Pass a reference rendered in one visual treatment and the new page inherits brushwork you never asked for. The reference prompt now explicitly scopes what the image is for — and the house style is defined entirely in words, by medium and qualities rather than by naming an artist, which I'd flinched away from early on for reasons that were as much about taste as copyright.

The rug-pull, and renting a GPU

Two things pushed me to stop depending on a hosted image API as the only painter. Google retired the model version I'd built against — the vision grader had to be re-pointed at its successor mid-project, a reminder that a hosted model is a dependency with someone else's roadmap. And the money: my cost meter turned out to be under-counting real Gemini spend by roughly half, and the honest number, multiplied by the retry tax of consistency repaints, wasn't hobby money. (The pipeline now has an out-of-credits circuit breaker across every Gemini call, which is the kind of component you only build after the absence of one has cost you.)

So: RunPod, a rented L40S at about $0.86/hour (an H100 at $3.29 when stock was short), ComfyUI, and ostris's ai-toolkit to train a style LoRA on open weights. The app was already painting through a single ImageGenerator interface, so the self-hosted backend is a one-class swap behind an environment flag — the seam was designed in long before the second implementation existed:

interface ImageGenerator
{
    public function generate(
        string $prompt,
        array $referenceImagePaths = [],
        ?string $aspectRatio = null,
        ?string $imageSize = null,
        ?int $seed = null,
    ): GeneratedImage;
}

Licence was the first filter, before any aesthetic judgement: SDXL's OpenRAIL-M is clean; FLUX.1 [dev] needs a paid licence to self-host; SD 3.5 is revenue-gated. Then the empirical part — Qwen-Image was tried first and rejected because its base aesthetic is glossy 3D render, and a LoRA barely moved it after five hundred steps. SDXL stylises hard in the other direction: it read as soft watercolour within 250 training steps.

The interesting failure was the training data. My corpus was the Gemini pages themselves — detailed, saturated, edge-to-edge colour — so the first LoRA faithfully learned to be Gemini, defeating the point. Escaping the source style took a flywheel:

Process the corpus towards the signature you want (paler, softer edges, breathing room), train, generate a broad batch, cull everything that looks like the old style, retrain on the survivors — and after the first loop, drop the original pages from the dataset entirely. Each turn of the loop sheds more of the source model's fingerprint. Four versions in, the style holds; the one stubborn weakness is close-up human faces, which claw back towards SDXL's photorealism prior. That's not a style problem, it's an identity problem — the same lesson as the Gemini era, wearing different weights — and the eventual answer is per-character LoRAs, which is the next chapter.

The operational footnotes cost me an evening each, so have them for free: RunPod pods are ephemeral and anything outside the mounted network volume is wiped; the SSH proxy is shell-only, so file transfer needs a workaround; and the Jupyter kernel is not the trainer's virtualenv, so training with the wrong Python fails in ways that don't say "wrong Python". None of this is on the pricing page.

A LoRA train runs 30–40 minutes on the L40S. A 48-image batch generates in about four. And the research I did on the economics says owning a GPU only beats renting somewhere north of several hundred books a month — far beyond hobby scale, which is the point: rented spot capacity plus scale-to-zero serverless is how a side project affords a pipeline like this at all.

The uncomfortable summary is that the model was never the product. Gemini paints beautifully; SDXL with a good LoRA paints beautifully. What makes either of them usable is everything wrapped around the call: reference hierarchies, a grader with a second opinion, votes before verdicts, baselines that can't enshrine a mistake, bounded repair, and a publish gate that refuses anything unverified. I built all of that for a bedtime-story hobby. Commercial teams shipping AI imagery without an equivalent are shipping the four drifted pages and hoping nobody's four-year-old notices.