voice-assistant/agent/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

This is a voice assistant agent that integrates with Home Assistant and provides tool-calling capabilities through a plugin architecture. The system uses OpenAI's API (or compatible endpoints) for LLM interactions and supports multiple languages (English and Swedish).

Architecture

Core Components

• src/backend.py: FastAPI server that loads plugins and exposes the /ask endpoint
• src/agent.py: Main Agent class that orchestrates LLM queries with plugin tools
• src/llm.py: LLM abstraction layer supporting OpenAI, LMStudio, and Llama models
  • OpenAIChat: Handles OpenAI API calls with function calling support (parallel execution)
  • LMStudioChat: Local LLM via LMStudio
  • LlamaChat: Direct llama.cpp integration
• src/db.py: PostgreSQL database with pgvector extension for embeddings
• src/frontend.py: Gradio web interface for chat

Plugin System

The plugin architecture is the heart of this system. All plugins inherit from src/plugins/base_plugin.py and must:

1. Define methods starting with tool_ to expose functions to the LLM
2. Use Pydantic models for type-safe function parameters
3. Return JSON-formatted strings from tool functions
4. Optionally provide a prompt() method to inject context into the system prompt

Plugin discovery: The backend automatically loads all folders in src/plugins/ that contain a plugin.py file with a Plugin class.

Base plugin features:

• Automatically converts tool_* methods into OpenAI function calling format
• Introspects Pydantic models to generate JSON schemas
• Provides access to Home Assistant helper class

Home Assistant integration: Plugins can use self.homeassistant (from src/plugins/homeassistant.py) to interact with Home Assistant via REST API and WebSocket.

Template System

System prompts are managed through Jinja2 templates in src/templates/:

• Templates are language-specific (e.g., ask.en.j2, ask.sv.j2)
• Plugins can inject their own prompts via the prompt() method
• Templates receive: current datetime, plugin prompts, and optional knowledge

Development Commands

Database Setup

Start PostgreSQL with pgvector:

docker-compose up -d

Create database and enable pgvector extension:

CREATE DATABASE rag;
\c rag
CREATE EXTENSION vector;

Run schema initialization:

psql -U postgres -h localhost -p 5433 -d rag -f db.sql

Running the Application

Backend (FastAPI server):

cd src
python -m uvicorn backend:app --host 0.0.0.0 --reload --reload-include config.yml

Frontend (Gradio UI):

cd src
python frontend.py

Configuration

Copy src/config.default.yml to src/config.yml and configure:

• OpenAI API settings (or compatible base_url)
• Home Assistant URL and token
• Plugin-specific settings (Spotify, Västtrafik, etc.)

Working with Plugins

Creating a New Plugin

1. Create folder: src/plugins/myplugin/
2. Create plugin.py with a Plugin class inheriting from BasePlugin
3. Define tool functions with tool_ prefix
4. Use Pydantic models for parameters
5. Add plugin config to config.yml under plugins.myplugin

Example structure:

from pydantic import BaseModel, Field
from ..base_plugin import BasePlugin

class MyInput(BaseModel):
    param: str = Field(..., description="Parameter description")

class Plugin(BasePlugin):
    def tool_my_function(self, input: MyInput):
        """Function description for LLM"""
        # Implementation
        return json.dumps({"status": "success"})

    def prompt(self) -> str | None:
        return "Additional context for the LLM"

Testing Function Calling

The LLM class in src/llm.py handles parallel function execution automatically. When the LLM responds with tool calls, they are executed concurrently using ThreadPoolExecutor.

Important Notes

• Working directory: The backend expects to be run from the src/ directory due to relative imports and file paths
• Database connection: Hardcoded to localhost:5433 in src/db.py
• Function calling: Currently only fully supported with OpenAIChat backend
• Response format: Tool functions must return JSON strings for proper LLM interpretation
• Logging: All modules use Python's logging module at INFO level