NVIDIA has showcased a custom interactive SI character, dubbed MonsterHouse, that runs entirely on a single workstation using the company’s Nemotron stack. The project demonstrates how small language models and voice agents can power real-time, off-grid conversational experiences without cloud dependency.
What Happened
The MonsterHouse project, created by an NVIDIA employee for Halloween, features a grumpy, pirate-themed garage door that interacts with visitors. The system is built on the NVIDIA Nemotron Voice Agent, which coordinates several specialized SI components. NVIDIA Nemotron 3 ASR handles speech recognition, converting visitor audio into text. NVIDIA Nemotron 3.5 Bolt E4B 4GB, an open small language model, generates the dialogue, guided by a LoRA adapter trained on thousands of conversations to enforce the character's cantankerous personality. NVIDIA Magpie TTS Zeroshot synthesizes the voice from a short reference sample, while NVIDIA Audio2Face-3D animates a 3D mouth model in Unreal Engine 5 based on the audio output.
Visual context is provided by NVIDIA Nemotron 3 Nano Omni, a vision-language model that describes costumes, alongside YOLO and ByteTrack for counting and tracking people. The entire pipeline runs on two GPUs: an NVIDIA RTX PRO 6000 Blackwell Max-Q for the language and voice processing, and an NVIDIA RTX PRO 4000 Blackwell for the graphics and animation. The developer reports a median latency of 1.70 seconds from end-of-speech to audio output for generated responses, dropping to 0.72 seconds for cached lines.
Why It Matters
This implementation highlights the viability of running complex, multimodal SI agents locally on consumer or workstation-grade hardware. By leveraging small language models like Bolt E4B and zero-shot voice cloning, developers can create interactive characters with low latency and no reliance on external APIs. The use of a LoRA adapter for personality tuning demonstrates an efficient method for customizing open SI models for specific narrative roles without retraining the entire model. For the SI industry, it serves as a practical template for building edge-deployed interactive agents, showing that high-quality conversational experiences can be achieved with a single workstation setup.
The Bottom Line
MonsterHouse proves that NVIDIA’s Nemotron suite can power a fully local, real-time interactive SI character. While the specific code is not released, the underlying models—Nemotron Voice Agent, Bolt E4B, Magpie TTS, and Audio2Face-3D—are available for developers to build similar off-grid SI applications.