In this post I’ll break down a side project I’ve been building at home: an interactive audiobook that plays like a choose-your-own-adventure. You listen to a novel read in cloned character voices, and at set moments the story pauses and hands control to you. You can type what the main character does next to move the story on, or try something the book doesn’t cover and the character answers you in their own voice.
I didn’t want an AI that rewrites the story. The story is the good part. So the book’s own text always plays as written, and the LLM only steps in when the player goes off-script. Even then it’s only allowed to talk about things the player has already heard.
The whole thing runs on one GPU in my homelab. A Python server streams voice audio and text over a WebSocket to an Unreal Engine 5 client. The surprising part wasn’t getting an LLM to talk. It was finding all the places where an LLM shouldn’t be in the loop at all.

Everything in the top half runs offline and builds the data. Everything in the bottom half runs live while you play, and the LLM only gets called when you go off-script.
See it running
Rail and Trail
The narrative runs in two modes, which I call the Rail and the Trail.
The Rail is the book. It sends the text straight to the voice engine, with no LLM anywhere in that path, and moves forward on its own until it reaches a decision point. Then it waits for the player.
The Trail is improv. If the player types something the book doesn’t cover, a local LLM (Mistral Small 3.2, 24B) answers in character, in first person, in three sentences or less.
Each decision point has a short list of player intents, each with a few short aliases. The server checks the player’s input against those first. A match moves the Rail on to the next part of the story. No match sends the input to the Trail, and the character improvises while the book stays paused. The cheap check always runs first, so the LLM is only called when it’s actually needed, and the HUD shows a small loading animation so the pause doesn’t feel like a crash.
The narrative bible
Everything at runtime is driven by a JSON file per chapter that I call the narrative bible. It holds the text broken into parts, and for each part it stores who is speaking, what the scene looks like, and whether the story pauses there for the player.
My first version stored each part as one block of prose, which fell apart quickly: I couldn’t switch voices in the middle of a paragraph. So every paragraph is now split into lines, and every line is tagged with its own speaker.
{ "type": "paragraph", "indent": true, "lines": [
{ "speaker": "hero_narrator", "text": "...", "kind": "prose" },
{ "speaker": "hero", "text": "...", "kind": "dialog" }
]}
The book is written in first person, so the main character is both narrator and speaker. Splitting those into two speakers means narration and dialogue get different text colours and can be routed to different voices.
Each part also stores a scene description for the LLM and an optional decision point. Every decision point carries a one-line reason for why the story pauses there, which sounds like busywork but makes all 600-plus pause points easy to search and review.
Building the bibles
I wasn’t going to hand-write the JSON, so I built an extractor that generates the bibles from the raw chapter text in five passes. The LLM finds the speakers, splits each chapter into scenes, describes them, and decides where the story should pause. The fifth pass sorts every paragraph into narration, dialogue, system messages and phone messages.
That last pass started as an LLM call too, and it was the wrong tool for the job. On long scenes it would quietly drop paragraphs and still return perfectly valid JSON, so nothing flagged it. For an audiobook, losing text is the one thing that can’t happen. I replaced it with plain regex rules. It now matches the source 100%, runs in under a second, and let me delete the cleanup stage I’d built to catch the LLM’s mistakes.
For the passes that still use the LLM, I tuned for consistency over creativity: low temperature, a fixed seed, a “NEVER do this” list in every prompt, and one retry on failed validation before I check by hand. Results are cached, so a cold run is about 1,300 LLM calls, but a rebuild after a small change takes about 10 seconds.
Keeping the character honest
The first version of the Trail made things up. A player asked about the bathroom, and the character described a clown doll in the bathtub. There is no clown doll in the book.
The scene descriptions only covered the big picture, never what was actually in a room. Prompt rules like “don’t invent things” just swapped one problem for another: the character stopped inventing, but started saying “I don’t see that” about things the book does describe. A prompt can’t stick to the book if it doesn’t have the book.
So I added a search over the text itself. When the player goes off-script, the server pulls the three most relevant passages and hands them to the character as memories. The search only covers what the player has already heard, so nothing can be spoiled. Matching scores alone couldn’t separate real questions from nonsense, so there’s a second, simpler gate: if most of the words in a question never appear in the story so far, the character gets no passages at all. Framing the passages as memories, not as what’s in front of the character, stopped earlier scenes bleeding into the current one.
A 16-question test script now checks the behaviour. It caught a bug in the relevance gate on its first run and currently passes 15 of 16.
Cleaning up the output
Whatever the LLM writes goes straight to the voice engine, one sentence at a time. If the model writes a screenplay label like **Name:**, the voice reads out “asterisk asterisk”. That isn’t a cosmetic bug when you can hear it.
The fix is three layers: the prompt lists the exact patterns the model must never write, the temperature stays low because those labels show up when the model gets creative, and a server-side regex strips name labels, bold, italics and stray asterisks from every sentence before it’s spoken.
Voices and sync
Voices use F5-TTS, which clones a voice from a short reference clip and its transcript. The server streams raw audio straight into Unreal, generating speech roughly two and a half times faster than real time on my GPU.
Per-line voices took longer than they should have. The bibles knew who spoke every line, but the narrator code picked one voice per paragraph, and an audit found 409 paragraphs with a side character’s line read in the narrator’s voice. Grouping each paragraph’s lines by speaker fixed it, and each speaker also gets their own text colour.
The hardest part on the Unreal side was sync. For each phrase the server sends the audio duration, then the text, then the audio. The client runs everything off one queue clocked to the audio, so words appear exactly as they’re spoken, and phone-style chat bubbles pop in on their line. If narration is interrupted, the audio stops, the text clears and the audio clock resets so nothing drifts. Space pauses everything together and resumes in sync.
The hardware
Everything runs on one Linux box, with the LLM and the voice engine sharing a single GPU.
| Part | Spec |
|---|---|
| GPU | Intel Arc Pro B70, 32 GB (upgraded from an Arc B580 12 GB) |
| CPU | AMD Ryzen 7 5800G |
| RAM | 64 GB DDR4 |
| OS | Ubuntu 26.04 LTS, native xe driver, Mesa 26.0.3 Vulkan |
| LLM | Vanilla Ollama on Vulkan, mistral-small3.2 24B Q4_K_M |
| TTS | F5-TTS v1 Base on the Arc’s XPU backend |
| Client | Unreal Engine 5 on a Windows dev box |
With both models loaded the card sits at 18 to 22 GB of its 32 GB. Intel’s own LLM stack for Arc looked like the obvious choice, but on my setup plain Ollama on Vulkan beat it, and after a clean OS rebuild the 24B model was fast enough for both building the bibles and live play.
The editor

I don’t trust the LLM’s pause points enough to ship them untouched, and fixing 600-odd parts in raw JSON wasn’t realistic, so I built a small web editor that runs alongside the game server. It lets me move pause points, edit player intents, split or merge scenes, reassign speakers and edit scene descriptions. It also handles voices: upload a reference clip, have it transcribed automatically, and play any line in that character’s voice. One button commits to git and another rebuilds the search index, so the whole loop of edit, listen, commit and rebuild happens in the browser.
Where it goes next
The main lesson: put the LLM only where improvisation is the point. Twice I started with an LLM or a model score and replaced it with something simpler I could trust, and both times the system got better.
Next up: clean reference audio for the characters still borrowing the narrator’s voice, a better system voice, and trying Kokoro as a faster engine for narration and system lines while keeping F5-TTS for the cloned character voices.



