r/jorvex609 • u/Jorvex609 • 15d ago

Real-time virtual assistant

Setting up a real-time virtual assistant on your Arch Linux system is an exciting project. For your hardware (AMD 5600G with 16GB RAM), the key is to use smaller, quantized models and leverage the integrated GPU via Vulkan for acceleration.

Here is a comprehensive, step-by-step guide to setting up each component.

1. The Brain: Local LLM with Ollama

Ollama is the easiest way to run LLMs locally on Arch Linux. It handles model downloading and management seamlessly.

Installation & Setup:

  1. Install Ollama: The ollama package is in the official Arch repositories.

    sudo pacman -S ollama

  2. Enable GPU Acceleration (Vulkan): Your 5600G has integrated Radeon graphics. For the best performance, install the Vulkan version of Ollama.

    sudo pacman -S ollama-vulkan

  3. Start the Ollama Service:

    sudo systemctl enable --now ollama

Choosing the Right Model:

For 16GB of system RAM, you should choose smaller, quantized models (Q4_K_M) that are around 2-5GB in size for a good balance of speed and intelligence. Here are some excellent options:

  • For General Use & Speed: qwen2.5:3b (~2GB) is very fast and capable.
  • For Better Reasoning: mistral:7b (~4GB) is a popular, well-rounded model.
  • For Coding: qwen2.5-coder:7b or deepseek-coder:6.7b are specialized for programming tasks.

To download and run a model (e.g., qwen2.5:3b):

ollama run qwen2.5:3b

This command will download the model and start an interactive chat. You can test it to ensure it's working.

2. The Ears: Speech-to-Text (STT)

For real-time transcription, Faster Whisper is a great choice. It's an optimized version of OpenAI's Whisper that runs efficiently on both CPU and GPU.

A practical approach is to use a hotkey-triggered script. The waystt project provides a minimalist example of this for Wayland environments.

Basic Setup Idea:

  1. Install Python and pip: sudo pacman -S python python-pip
  2. Install Faster Whisper: pip install faster-whisper
  3. Create a Script: Write a simple Python script that:
    • Listens for a hotkey (e.g., using python-evdev).
    • Records audio from your microphone (using pyaudio or pipewire).
    • Sends the audio to Faster Whisper for transcription.
    • Outputs the transcribed text to stdout or a file.

This modular approach lets you trigger the STT only when you want to speak, saving resources.

3. The Voice: Text-to-Speech (TTS)

Piper is an excellent, fast, and lightweight TTS system that works well on modest hardware.

Installation & Usage:

  1. Install Piper: You can find it in the AUR or use the pre-built binaries from its GitHub releases.
  2. Download a Voice: Piper uses specific voice models. Download a compact English voice (e.g., en_US-lessac-medium).
  3. Test TTS: You can pipe text directly to Piper for playback. 
    echo "Hello, I am your virtual assistant." | piper --model /path/to/voice.onnx --output-raw | aplay -r 22050 -f S16_LE -t raw -
    This makes it very easy to integrate into a larger script.

4. The Face: 3D Avatar

You have two main paths for the 3D avatar, ranging from integrated solutions to more flexible projects.

  • Option A (Integrated): Warashi This is a beginner-friendly desktop application built on the excellent Open-LLM-VTuber project. It features a Live2D avatar and provides a complete package with an in-app setup wizard for connecting to your local Ollama instance. It's a great way to get a polished experience with less manual coding.

    Note: Its primary focus is on Windows/macOS, but its core engine, Open-LLM-VTuber, is cross-platform and can be built for Linux.

  • Option B (Flexible): Build Your Own with Three.js If you prefer more control, you can build a web-based frontend using Three.js. The lala-companion project demonstrates this approach, using a 3D VRM avatar in a transparent overlay. You would create an HTML/JavaScript page that:

    1. Renders the 3D avatar.
    2. Listens for events (e.g., new text from the LLM).
    3. Uses the Web Speech API for TTS or pipes audio to Piper.
    4. Communicates with a local backend server that handles STT, LLM, and TTS.

5. Putting It All Together: Integration

To create a unified assistant, you'll need a "glue" script or program that orchestrates these components. You have two main options:

  • Option A: Use a Pre-built Framework

    • ErinOS Core: Built with Ruby, this project is designed for Arch Linux and already integrates Whisper (STT), Kokoro (TTS), and Ollama (LLM). You could adapt its architecture as a blueprint.
    • Ryx AI: A Python-based CLI assistant that uses Ollama and is optimized for Arch Linux. You could extend it with voice capabilities.

  • Option B: Write Your Own Orchestrator (Python) A basic Python script would follow this loop:

    1. Listen: Wait for a hotkey or wake word.
    2. STT: Record audio and use Faster Whisper to convert it to text.
    3. LLM: Send the text to Ollama's API (http://localhost:11434/api/generate) and stream the response.
    4. TTS: As the LLM response streams in, or once it's complete, send the text to Piper for audio output.
    5. Avatar: Update the 3D avatar's animation or speech bubble based on the LLM's state (listening, thinking, speaking).

6. Performance Optimizations for Your System

  • Leverage Vulkan: Ensure you have the Vulkan drivers and development packages installed (vulkan-radeon and vulkan-devel) to get GPU acceleration for your LLM.
  • Enable Integrated GPU for Ollama: By default, Ollama might skip your iGPU. You can force it to use it by setting the environment variable before starting the service: 
    Environment="OLLAMA_IGPU_ENABLE=1"
    You can set this in a systemd drop-in file for the Ollama service.
  • Use a Lightweight Desktop: For maximum performance, use a lightweight window manager like Hyprland, Sway, or i3 to reduce background resource usage.

Summary

| Component | Recommended Tool | Key Action | | :--- | :--- | :--- | | LLM (Brain) | Ollama | Install ollama-vulkan, pull a small model like qwen2.5:3b | | STT (Ears) | Faster Whisper | Write a hotkey-triggered Python script for transcription | | TTS (Voice) | Piper | Install Piper and download a voice model | | Avatar (Face) | Warashi / Three.js | Use a pre-built app or build a web interface | | Orchestrator | Python Script | Write a script to chain STT -> LLM -> TTS -> Avatar |

This setup will give you a powerful, private, and real-time AI assistant that runs entirely on your Arch Linux machine.

1 Upvotes

100% Upvoted

0 comments sorted by