128 lines
4.5 KiB
Markdown
128 lines
4.5 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Project Overview
|
|
|
|
This is a voice assistant agent that integrates with Home Assistant and provides tool-calling capabilities through a plugin architecture. The system uses OpenAI's API (or compatible endpoints) for LLM interactions and supports multiple languages (English and Swedish).
|
|
|
|
## Architecture
|
|
|
|
### Core Components
|
|
|
|
- **[src/backend.py](src/backend.py)**: FastAPI server that loads plugins and exposes the `/ask` endpoint
|
|
- **[src/agent.py](src/agent.py)**: Main Agent class that orchestrates LLM queries with plugin tools
|
|
- **[src/llm.py](src/llm.py)**: LLM abstraction layer supporting OpenAI, LMStudio, and Llama models
|
|
- `OpenAIChat`: Handles OpenAI API calls with function calling support (parallel execution)
|
|
- `LMStudioChat`: Local LLM via LMStudio
|
|
- `LlamaChat`: Direct llama.cpp integration
|
|
- **[src/db.py](src/db.py)**: PostgreSQL database with pgvector extension for embeddings
|
|
- **[src/frontend.py](src/frontend.py)**: Gradio web interface for chat
|
|
|
|
### Plugin System
|
|
|
|
The plugin architecture is the heart of this system. All plugins inherit from [src/plugins/base_plugin.py](src/plugins/base_plugin.py) and must:
|
|
|
|
1. Define methods starting with `tool_` to expose functions to the LLM
|
|
2. Use Pydantic models for type-safe function parameters
|
|
3. Return JSON-formatted strings from tool functions
|
|
4. Optionally provide a `prompt()` method to inject context into the system prompt
|
|
|
|
**Plugin discovery**: The backend automatically loads all folders in `src/plugins/` that contain a `plugin.py` file with a `Plugin` class.
|
|
|
|
**Base plugin features**:
|
|
- Automatically converts `tool_*` methods into OpenAI function calling format
|
|
- Introspects Pydantic models to generate JSON schemas
|
|
- Provides access to Home Assistant helper class
|
|
|
|
**Home Assistant integration**: Plugins can use `self.homeassistant` (from [src/plugins/homeassistant.py](src/plugins/homeassistant.py)) to interact with Home Assistant via REST API and WebSocket.
|
|
|
|
### Template System
|
|
|
|
System prompts are managed through Jinja2 templates in `src/templates/`:
|
|
- Templates are language-specific (e.g., `ask.en.j2`, `ask.sv.j2`)
|
|
- Plugins can inject their own prompts via the `prompt()` method
|
|
- Templates receive: current datetime, plugin prompts, and optional knowledge
|
|
|
|
## Development Commands
|
|
|
|
### Database Setup
|
|
|
|
Start PostgreSQL with pgvector:
|
|
```bash
|
|
docker-compose up -d
|
|
```
|
|
|
|
Create database and enable pgvector extension:
|
|
```sql
|
|
CREATE DATABASE rag;
|
|
\c rag
|
|
CREATE EXTENSION vector;
|
|
```
|
|
|
|
Run schema initialization:
|
|
```bash
|
|
psql -U postgres -h localhost -p 5433 -d rag -f db.sql
|
|
```
|
|
|
|
### Running the Application
|
|
|
|
Backend (FastAPI server):
|
|
```bash
|
|
cd src
|
|
python -m uvicorn backend:app --host 0.0.0.0 --reload --reload-include config.yml
|
|
```
|
|
|
|
Frontend (Gradio UI):
|
|
```bash
|
|
cd src
|
|
python frontend.py
|
|
```
|
|
|
|
### Configuration
|
|
|
|
Copy `src/config.default.yml` to `src/config.yml` and configure:
|
|
- OpenAI API settings (or compatible base_url)
|
|
- Home Assistant URL and token
|
|
- Plugin-specific settings (Spotify, Västtrafik, etc.)
|
|
|
|
## Working with Plugins
|
|
|
|
### Creating a New Plugin
|
|
|
|
1. Create folder: `src/plugins/myplugin/`
|
|
2. Create `plugin.py` with a `Plugin` class inheriting from `BasePlugin`
|
|
3. Define tool functions with `tool_` prefix
|
|
4. Use Pydantic models for parameters
|
|
5. Add plugin config to `config.yml` under `plugins.myplugin`
|
|
|
|
Example structure:
|
|
```python
|
|
from pydantic import BaseModel, Field
|
|
from ..base_plugin import BasePlugin
|
|
|
|
class MyInput(BaseModel):
|
|
param: str = Field(..., description="Parameter description")
|
|
|
|
class Plugin(BasePlugin):
|
|
def tool_my_function(self, input: MyInput):
|
|
"""Function description for LLM"""
|
|
# Implementation
|
|
return json.dumps({"status": "success"})
|
|
|
|
def prompt(self) -> str | None:
|
|
return "Additional context for the LLM"
|
|
```
|
|
|
|
### Testing Function Calling
|
|
|
|
The LLM class in [src/llm.py](src/llm.py) handles parallel function execution automatically. When the LLM responds with tool calls, they are executed concurrently using `ThreadPoolExecutor`.
|
|
|
|
## Important Notes
|
|
|
|
- **Working directory**: The backend expects to be run from the `src/` directory due to relative imports and file paths
|
|
- **Database connection**: Hardcoded to localhost:5433 in [src/db.py](src/db.py:17-23)
|
|
- **Function calling**: Currently only fully supported with `OpenAIChat` backend
|
|
- **Response format**: Tool functions must return JSON strings for proper LLM interpretation
|
|
- **Logging**: All modules use Python's logging module at INFO level
|