Author: Mark Hollier

  • Under the Hood: Building an Interactive Audiobook

    In this post I’ll break down a side project I’ve been building at home: an interactive audiobook that plays like a choose-your-own-adventure. You listen to a novel read in cloned character voices, and at set moments the story pauses and hands control to you. You can type what the main character does next to move the story on, or try something the book doesn’t cover and the character answers you in their own voice.

    I didn’t want an AI that rewrites the story. The story is the good part. So the book’s own text always plays as written, and the LLM only steps in when the player goes off-script. Even then it’s only allowed to talk about things the player has already heard.

    The whole thing runs on one GPU in my homelab. A Python server streams voice audio and text over a WebSocket to an Unreal Engine 5 client. The surprising part wasn’t getting an LLM to talk. It was finding all the places where an LLM shouldn’t be in the loop at all.

    System architecture: offline authoring and runtime
    System architecture · offline authoring and runtime

    Everything in the top half runs offline and builds the data. Everything in the bottom half runs live while you play, and the LLM only gets called when you go off-script.

    See it running

    Rail and Trail

    The narrative runs in two modes, which I call the Rail and the Trail.

    The Rail is the book. It sends the text straight to the voice engine, with no LLM anywhere in that path, and moves forward on its own until it reaches a decision point. Then it waits for the player.

    The Trail is improv. If the player types something the book doesn’t cover, a local LLM (Mistral Small 3.2, 24B) answers in character, in first person, in three sentences or less.

    Each decision point has a short list of player intents, each with a few short aliases. The server checks the player’s input against those first. A match moves the Rail on to the next part of the story. No match sends the input to the Trail, and the character improvises while the book stays paused. The cheap check always runs first, so the LLM is only called when it’s actually needed, and the HUD shows a small loading animation so the pause doesn’t feel like a crash.

    The narrative bible

    Everything at runtime is driven by a JSON file per chapter that I call the narrative bible. It holds the text broken into parts, and for each part it stores who is speaking, what the scene looks like, and whether the story pauses there for the player.

    My first version stored each part as one block of prose, which fell apart quickly: I couldn’t switch voices in the middle of a paragraph. So every paragraph is now split into lines, and every line is tagged with its own speaker.

    { "type": "paragraph", "indent": true, "lines": [
      { "speaker": "hero_narrator", "text": "...", "kind": "prose" },
      { "speaker": "hero", "text": "...", "kind": "dialog" }
    ]}

    The book is written in first person, so the main character is both narrator and speaker. Splitting those into two speakers means narration and dialogue get different text colours and can be routed to different voices.

    Each part also stores a scene description for the LLM and an optional decision point. Every decision point carries a one-line reason for why the story pauses there, which sounds like busywork but makes all 600-plus pause points easy to search and review.

    Building the bibles

    I wasn’t going to hand-write the JSON, so I built an extractor that generates the bibles from the raw chapter text in five passes. The LLM finds the speakers, splits each chapter into scenes, describes them, and decides where the story should pause. The fifth pass sorts every paragraph into narration, dialogue, system messages and phone messages.

    That last pass started as an LLM call too, and it was the wrong tool for the job. On long scenes it would quietly drop paragraphs and still return perfectly valid JSON, so nothing flagged it. For an audiobook, losing text is the one thing that can’t happen. I replaced it with plain regex rules. It now matches the source 100%, runs in under a second, and let me delete the cleanup stage I’d built to catch the LLM’s mistakes.

    For the passes that still use the LLM, I tuned for consistency over creativity: low temperature, a fixed seed, a “NEVER do this” list in every prompt, and one retry on failed validation before I check by hand. Results are cached, so a cold run is about 1,300 LLM calls, but a rebuild after a small change takes about 10 seconds.

    Keeping the character honest

    The first version of the Trail made things up. A player asked about the bathroom, and the character described a clown doll in the bathtub. There is no clown doll in the book.

    The scene descriptions only covered the big picture, never what was actually in a room. Prompt rules like “don’t invent things” just swapped one problem for another: the character stopped inventing, but started saying “I don’t see that” about things the book does describe. A prompt can’t stick to the book if it doesn’t have the book.

    So I added a search over the text itself. When the player goes off-script, the server pulls the three most relevant passages and hands them to the character as memories. The search only covers what the player has already heard, so nothing can be spoiled. Matching scores alone couldn’t separate real questions from nonsense, so there’s a second, simpler gate: if most of the words in a question never appear in the story so far, the character gets no passages at all. Framing the passages as memories, not as what’s in front of the character, stopped earlier scenes bleeding into the current one.

    A 16-question test script now checks the behaviour. It caught a bug in the relevance gate on its first run and currently passes 15 of 16.

    Cleaning up the output

    Whatever the LLM writes goes straight to the voice engine, one sentence at a time. If the model writes a screenplay label like **Name:**, the voice reads out “asterisk asterisk”. That isn’t a cosmetic bug when you can hear it.

    The fix is three layers: the prompt lists the exact patterns the model must never write, the temperature stays low because those labels show up when the model gets creative, and a server-side regex strips name labels, bold, italics and stray asterisks from every sentence before it’s spoken.

    Voices and sync

    Voices use F5-TTS, which clones a voice from a short reference clip and its transcript. The server streams raw audio straight into Unreal, generating speech roughly two and a half times faster than real time on my GPU.

    Per-line voices took longer than they should have. The bibles knew who spoke every line, but the narrator code picked one voice per paragraph, and an audit found 409 paragraphs with a side character’s line read in the narrator’s voice. Grouping each paragraph’s lines by speaker fixed it, and each speaker also gets their own text colour.

    The hardest part on the Unreal side was sync. For each phrase the server sends the audio duration, then the text, then the audio. The client runs everything off one queue clocked to the audio, so words appear exactly as they’re spoken, and phone-style chat bubbles pop in on their line. If narration is interrupted, the audio stops, the text clears and the audio clock resets so nothing drifts. Space pauses everything together and resumes in sync.

    The hardware

    Everything runs on one Linux box, with the LLM and the voice engine sharing a single GPU.

    PartSpec
    GPUIntel Arc Pro B70, 32 GB (upgraded from an Arc B580 12 GB)
    CPUAMD Ryzen 7 5800G
    RAM64 GB DDR4
    OSUbuntu 26.04 LTS, native xe driver, Mesa 26.0.3 Vulkan
    LLMVanilla Ollama on Vulkan, mistral-small3.2 24B Q4_K_M
    TTSF5-TTS v1 Base on the Arc’s XPU backend
    ClientUnreal Engine 5 on a Windows dev box

    With both models loaded the card sits at 18 to 22 GB of its 32 GB. Intel’s own LLM stack for Arc looked like the obvious choice, but on my setup plain Ollama on Vulkan beat it, and after a clean OS rebuild the 24B model was fast enough for both building the bibles and live play.

    The editor

    The narrative bible editor: scene context fields and a decision point
    The bible editor · scene context and decision point for one part

    I don’t trust the LLM’s pause points enough to ship them untouched, and fixing 600-odd parts in raw JSON wasn’t realistic, so I built a small web editor that runs alongside the game server. It lets me move pause points, edit player intents, split or merge scenes, reassign speakers and edit scene descriptions. It also handles voices: upload a reference clip, have it transcribed automatically, and play any line in that character’s voice. One button commits to git and another rebuilds the search index, so the whole loop of edit, listen, commit and rebuild happens in the browser.

    Where it goes next

    The main lesson: put the LLM only where improvisation is the point. Twice I started with an LLM or a model score and replaced it with something simpler I could trust, and both times the system got better.

    Next up: clean reference audio for the characters still borrowing the narrator’s voice, a better system voice, and trying Kokoro as a faster engine for narration and system lines while keeping F5-TTS for the cloned character voices.

  • Video Game Audio – Under The Hood – Eon Altar Ambience System

    In this installment of Video Game Audio – Under the hood, I will break down the Ambience System I designed for Eon Altar.

    Eon Altar’s Ambience Design:

    Eon Altar proved to be an interesting project in designing how to tackle the ambiences. Players are either ascending (first level) or descending into a temple trying to find the Eon Altar. I didn’t want to have one static ambience bed for the whole level. I wanted to give the player a sense of change when they explore the level. To do this I broke the Ambience Design into two parts: Ambience Beds and Ambience One shot sweeteners.

    Ambience beds

    Eon Altar’s level design for the most part has a linear progression. I divided the level into three Ambience Zones. Each Zone would have its own ambience bed.

    Ambience Zones boxes

    You may notice the above screenshot shows the ambience zones overlap. I used the zones to send different events to the Ambience System. I used the Trigger Volume “On Enter” event to “prime” the next ambience bed (playing but muted, at −∞ dB). When the players exited the original ambience zone I used the Trigger Volume “On Exit” to send an event to crossfade the currently playing ambience bed and the “prime” ambience bed.

    Ambiences Zone Prefab

    Once players were far enough into the ambience zone they would hit another volume trigger telling the Ambience System to switch Ambience Zone. Changing ambience zones would stop the ambiences from the previous zone.

    The video below demonstrates the ambience bed system in action.

    Ambience One Shot Sweetener

    Now that I had the Ambience beds I needed to put life into the ambiences. I needed a way to play a set of sound groups at random locations and random intervals. So I built the Ambience One Shot Sweetener FSM.

    Ambience One Shot

    What does this thing do?

    Basically it plays a random sound at random locations.

    Let me break down this FSM. When it starts I have a random wait variable, between 5 and 10 seconds in this particular demo. From there it goes to the select sound group and select variation stage. These actions pick a random sound group from a list between 5 and 10 different sounds, and then it picks one of the 5 variations of that sound bank. Now that a chosen sound is set to play we needed to find a location to play the sound. To do this we had to find the location of the Audio Listener.

    What is the Audio Listener?

    Unity uses a component called Audio Listener as the location where the game is listening for the sound. Most games put the audio listener on either the main camera or the player itself. Eon is designed as a local co-op multiplayer game so I decided that the main camera would be too far away and that no particular player should have the audio listener attached to them. I decided to build my own placement of the audio listener, positioning it above and in the center of the party.

    Audio Listener

    This gives the game a wider spread of sound across the front speakers: sound effects for players on the far right come from the right speaker, players on the far left come from the left speaker, and players in the center come from the center speaker.

    I now have a location where to play the sound but I didn’t want the sound to always play on the same location of the audio listener each time. To generate a random location, I randomized a set of numbers for X, Y, and Z and added them to the X, Y, and Z coordinates of the audio listener.

    I found that the sounds would play on top of each other. The random wait timer was not long enough or was too long depending on the type of sound. I needed to wait for the sound to play out before I started the process all over again.

    This proved a little tricky when I created this system, as there was no PlayMaker Event to find the currently playing sound group, and the length of the playing variation. I didn’t want to waste what precious little time I had with the programmer so I dove into PlayMaker and found that you can attach scripts to Variables. By selecting the sound group and variation I had everything I needed to get the sound group game object and by using Master Audio Dynamic Group Variation script I was able to get the length of the sound. I stored that as a delay variable and now it waits for the sound to finish playing before starting the process all over again.

    Here is a video of the Ambience One Shot Sound Effect in action.

  • Video Game Audio – Under The Hood: Eon Altar Music System

    Video Game Audio – Under the hood is a series of blog posts that will shed light on the not so glamorous, technical aspect of implementing audio in video games. In this installment I will break down Eon Altar’s Music System, from our initial designs to the first prototype and finally to what actually made it in the end product.

    What is Eon Altar?

    Eon Altar is a mobile enhanced RPG PC game. Eon Altar uses the best of both free-roaming exploration and turn-based combat.

    Eon Altar was built in Unity 4.6 using Master Audio Unity Store Add-on as its audio middleware solution and PlayMaker used for visual scripting. The majority of the music system was done by myself with limited access to a junior programmer. Most of the audio systems were built in PlayMaker FSMs (Finite State Machine). Master Audio came with PlayMaker FSM Actions as part of its package. You can find links to both PlayMaker and Master Audio at the bottom of this blog.

    Eon Altar’s Initial Adaptive Music Design:

    The original music design we came up with on Eon Altar was an adaptive music layer system. The music system would add or subtract the different layers of the music track depending on what state the party was in. Each level was broken into different musically themed “areas” and each level could have up to three or four different tracks.

    The Different Adaptive States:

    • Quest (base track)
    • Danger Zone (Enemies have been detected)
    • Pre-Battle (Enemies are on the screen)
    • Battle (You are in combat)
    • Low Health (Party health is at 25%)

    Adaptive Music Design Breakdown:

    Quest

    We started off the level with the Quest layers. This was the base track we used for player exploration of the level.

    Danger Zone

    The Danger Zone consisted of when enemies were close to the party or if the party was in a dangerous “Area”.

    Technical requirements built by the junior programmer

    We needed a way to detect how far away the enemies were from the party so the junior programmer created a module called the “Enemy Detection Sphere”. This was a sphere attached to the party and once an enemy spawner prefab entered the sphere the party was in danger.

    With the enemy in the detection sphere a PlayMaker Event Play “Danger Zone” was sent to the music master FSM. (cue up the Kenny Loggins track Danger Zone from Top Gun soundtrack). This was an ongoing gag throughout the production of Eon Altar.

    Pre-Battle (Pre-Combat)

    Pre-Battle is the state where a player could go into combat at any time. The idea was to increase tension through the music while enemies were visible on the screen.

    Battle (Combat)

    As stated at the beginning our combat system is turn-based, so the player, or the enemy, could engage combat at any time.

    Low Health

    The last state “Low Health” is when the players are doing poorly in battle and are about to die.

    Technical requirements needed:

    Since Eon Altar is a co-op game we took a percentage of the party’s health; once we passed that threshold we were in the low health state. We once again sped up the tempo of the track and would add low pass filter to all the SFX, letting the music punch through.

    Combat End in Victory or Death

    When battle was finished in a victory, the music system would transition to an appropriate music state, Pre-Battle, Danger or Quest. When battle ended in death the music would fade out and play a death musical stinger on the Game Over screen.

    Eon Altar’s Adaptive Prototype:

    The first question was: is this even possible with the tools I had? The answer was yes!

    The Master Audio Music System was way more robust than my first impression (a jukebox). Master Audio Music Playlist has an option to synchronize all tracks in a single playlist. This synchronized option allowed the playlist to play all at the same time and in sync. This was exactly what I needed to build the adaptive music system we designed. Master Audio PlayMaker bundle also had every FSM action I needed to create a working prototype.

    The first prototype was built as a proof of concept in our test level.

    Here is a video capture of the Adaptive Music prototype.

    The Inevitable Design Changes:

    The one thing I have learned in all my time working on video games is, at any moment the design of the game can change and will change. This affects audio most of all because we are integrated into most, if not all, systems. Be it design changes, cut features, budget or time constraints, technical difficulties, or it simply doesn’t work. As much as you may think your idea is awesome you can’t be locked in to your design. Eon Altar was no different.

    We had to cut the Pre-battle layer due to gameplay improvements with how the camera worked. The design change allowed the player to use the movement mark to pan the camera around to see ahead of the party. We needed to cut this layer as players would be going in and out of this state all the time.

    Low Health state was cut due to technical limitations of Unity 4.6 and Master Audio. There was no way to do dynamic mix states. The only way you could do any type of DSP (Digital Signal Processing) was on a per-audio-source basis. Master Audio had no DSP support on an Audio Bus. (This all has changed now with Unity 5 and its audio mixer.)

    Design changes and technical issues are one thing, but it all comes down to time. There is never enough time when it comes to anything artistic. During production I sat down with the composer and we hashed out what we were able to deliver in the amount of time we had. At the end we decided to combine Quest and Danger into one track.

    Eon Altar’s Music System

    We cut almost 90% of the original music design. We went from an Adaptive Music system to a basic Exploration and Battle track design. I built a hybrid system that could take either two separated tracks or a single track with a layer option. This gave the composer different options for when he was composing for the different levels.

    The New Music System Events:

    • Exploration
    • Battle
    • Change Track “Change exploration and battle track”
    • Cinematic “Music for Cinematics”
    • Pause Menu
    • TPK “Total Party Kill”
    • Session End

    Exploration/Battle/Change tracks/Cinematic:

    These are the basic events in the music system, exploration music when we are not in combat or Cinematics, battle music when we are in combat. Change the exploration and battle music to a different track. Change track from the two track system to single track and layer system. Last but not least play cinematic music. When leaving a cinematic we could go into battle or quest music. Music track changes were done by placing trigger volumes in the level or built off other gameplay FSMs.

    TPK (Total Party Kill):

    During production the design of what happens when the party dies changed from the level ending to being restored back to a checkpoint (Destiny Marker). I had to design the music system to Fade-out the music on the visual fade-out and play a death musical stinger as the players were teleported to the last activated Destiny Marker. On the visual fade-in the music would fade-in as well. The tricky part of this system was that depending on what Destiny Marker the player returned to the music needed to change to that “areas” quest and battle music and re-enable any of the level’s Music Triggers.

    Pause Menu/Session End:

    The Pause Menu and Session End came very close to the end of the project. This might have been the easiest part of the system to build as Master Audio has a “Pause All” PlayMaker action. This action paused all active audio busses and the music playlists as well. The end of game flow was added to this FSM as it was controlling most of the audio anyway.

    Here is a video of the final Music system that was shipped with Eon Altar.

    Master Audio https://www.assetstore.unity3d.com/en/#!/content/5607

    PlayMaker  https://www.assetstore.unity3d.com/en/#!/content/368

    Eon Altar http://store.steampowered.com/app/382050/