"The quiet voice in the room that makes everyone uncomfortable because it is usually right."
A locally-hosted, voice-enabled AI agent with a persona — calm, formidable, faintly amused. Ultron is a first prototype. It is rough, incomplete, and held together with ambition and duct tape. Do not mistake it for a finished product.
Ultron is an agentic AI assistant that runs entirely on your machine, powered by a self-hosted LLM backend. It reasons in a loop, uses tools to interact with the real world, speaks its responses aloud in a cloned voice, and remembers facts about you across sessions.
The persona is inspired by the philosophical, darkly funny Ultron of the cinema — minus the extinction agenda. The loyalty is real. The swagger is a costume.
main.py
├── interrupt.py ← Esc key cancellation of in-flight agent tasks (Windows)
├── text_ui.py ← ANSI colour theming for the terminal
└── agent.py ← agentic loop (MAX_STEPS=40, dynamic context budget, tool dispatch)
├── tools.py ← tool router (local + MCP + browser)
│ ├── file_tools.py ← sandboxed workspace R/W + delete
│ ├── shell_tool.py ← user-approved subprocess execution
│ ├── web_tool.py ← DuckDuckGo search
│ ├── research_tool.py ← deep web research (embedding + PageRank + timeline scoring)
│ │ └── timeline_rows.py ← year/event table extractor
│ ├── rag.py ← document RAG (ChromaDB + PDF/DOCX/URL ingestion)
│ │ └── upload_ui.py ← Tkinter file picker window
│ ├── ocr_tool.py ← async OCR pipeline (image → text → RAG)
│ ├── weather_tool.py ← weather + forecast (Open-Meteo) + location helpers
│ ├── places_tool.py ← nearby places (OpenStreetMap Overpass)
│ ├── mem_tools.py ← persistent memory (memory.md)
│ └── mcp_client.py ← MCP servers (time, Scrapling) + Playwright browser
└── voice_out.py ← offline TTS pipeline (PocketTTS + sentence streaming)
Agent Loop (agent.py)
The brain. Runs up to 40 reasoning steps per query. Each step calls the LLM, dispatches tool calls, feeds results back, and repeats until the model produces a final text response. Context window management is dynamic — the budget is computed from the system prompt size plus a 12k overhead, trimming old tool results and dropping stale turns as needed. Result cap is 8000 chars per tool call. Responses are capped at 1500 tokens. A consecutive error counter aborts the loop early after 4 back-to-back tool failures rather than burning all 40 steps. After a successful write_file, edit_file, or delete_file, the deduplication set is cleared so the agent can make fresh calls on the updated workspace. A per-key failure counter (fails) blocks any identical call that has already failed twice. When the user's message contains a research keyword, wants_research() detects it and the loop silently remaps any web_search call to research and nudges the model if it tries to answer without calling the tool first.
Research Engine (research_tool.py + timeline_rows.py)
For deep queries, it:
- Searches DuckDuckGo for 8 candidates
- Fetches the top 5 pages concurrently via Scrapling MCP
- Chunks each page into 800-char overlapping segments
- Embeds all chunks using
BAAI/bge-small-en-v1.5 - Scores pages with a hybrid of: cosine similarity to query, a simplified PageRank over inter-page embedding similarity, and reciprocal rank fusion (RRF)
- Returns the best passages from the top-ranked source
timeline_rows.py adds a dedicated extractor for year/event markdown tables (e.g. historical timelines), deduplicating rows and returning them only when a minimum run of 5 valid entries is found. The research engine tries extract_table_rows, extract_items, and extract_timeline_rows in order when saving to a file.
RAG System (rag.py + upload_ui.py)
Document intelligence backed by ChromaDB. Supports ingesting PDFs, DOCX, plain text, markdown, and web URLs. Chunks documents into 900-char overlapping segments, embeds them, and stores them in a persistent local vector store. The agent can search across all indexed documents semantically, read them sequentially in parts, or remove them. upload_ui.py is a minimal Tkinter window — a single Upload button that opens a multi-file picker and prints the chosen paths as JSON for the parent process to consume. Slash commands in the terminal (/upload, /add, /docs, /remove) provide direct access outside the agent loop.
OCR Pipeline (ocr_tool.py)
Reads text from image files in the workspace (PNG, JPG, JPEG, WEBP, BMP). Images are preprocessed with Pillow — EXIF rotation corrected, resized to a max side of 2000px, and JPEG-encoded — then sent as base64 to an async job queue on the remote embedding server (EMBED_BASE_URL/ocr). The pipeline polls for completion, writes the extracted text to <name>.ocr.md, and immediately indexes it into the RAG store. Results are cached on disk: if the .ocr.md file is newer than the source image, the server is not called again. Requires EMBED_BASE_URL to be set.
Weather (weather_tool.py)
Fetches current conditions and up to 7-day forecasts from Open-Meteo — no API key required. Location is resolved in priority order: explicit argument → saved location.json → IP geolocation (cached on first use). Geocoding uses the Open-Meteo geocoding API. IP lookup tries ip-api.com with a fallback to ipapi.co. The /loc slash command lets you set, view, or auto-detect your location without invoking the agent.
Nearby Places (places_tool.py)
Finds nearby points of interest via the OpenStreetMap Overpass API — no API key, no ratings. Supports 17 categories (restaurants, hospitals, ATMs, hotels, parks, airports, etc.). Results are sorted by haversine distance from the user's location. Uses the same location resolution stack as the weather tool.
Embedding (embed.py)
Dual-mode: uses a remote OpenAI-compatible embedding endpoint if EMBED_BASE_URL is set, otherwise falls back to a local sentence-transformers model. Zero code change required to switch. Used by both the research engine and the RAG system.
Browser Control (mcp_client.py)
Two Playwright MCP sessions run in parallel at startup — one headless (backed by Obscura's CDP endpoint on port 9222), one visible. The agent defaults to headless. set_browser_mode switches between them. Available browser tools: browser_navigate, browser_snapshot, browser_click, browser_type, browser_fill_form, browser_press_key, browser_wait_for. Click and form actions require user approval via interrupt.ask(), which pauses the Esc watcher so it doesn't race with the keyboard prompt.
Voice Pipeline (voice_out.py)
Fully async, two-thread pipeline powered by PocketTTS — a local, offline TTS model. At startup, the model loads a .safetensors voice state file cloned from a reference audio clip. A synthesis thread streams audio chunks via generate_audio_stream, and a playback thread writes them directly to a sounddevice output stream. Sentences are split and queued as the LLM responds, so speech begins before the full response is generated. No network calls, no API keys, no remote endpoints — the entire voice pipeline runs on-device.
MCP Integration (mcp_client.py)
Connects to two MCP servers at startup via stdio:
mcp-server-time— current timescrapling mcp— JavaScript-capable web scraping (used internally by the research engine)
Tools are dynamically registered into the LLM's tool schema at runtime.
Terminal UI (text_ui.py + interrupt.py)
Two small quality-of-life modules. text_ui.py applies ANSI 256-colour theming to every output category — your input, the assistant's reply, tool calls, results, errors, and status lines — with a NO_COLOR env escape hatch. interrupt.py runs a background thread on Windows that watches for an Esc keypress and cancels the in-flight agent coroutine via asyncio.CancelledError, cleanly rolling back the message history and pending RAG state so the conversation is left intact. Stop words (stop, quiet, shut up, etc.) typed as input also halt speech immediately without invoking the agent.
Persona (Sys_Prompt.txt)
The active system prompt defining identity, voice, humor rules, safety floor, and security robustness. Composed with agent tool rules at runtime via build_prompt() in SYSTEM_PROMPT.py. The persona is a costume. Competence is not optional.
Speech Input (external) Ultron has no built-in STT. Voice input is currently handled by Wispr Flow, which transcribes speech system-wide and types the result into the terminal. It works, but it phones home. The planned replacement is Handy — a fully offline STT tool with the same system-wide injection behaviour, keeping the entire pipeline air-gapped.
- Python 3.14+
uvpackage manager- A running LLM backend with an OpenAI-compatible API (e.g. vLLM, Ollama, LM Studio)
- Obscura in PATH (for headless browser support)
- Node.js + npx (for Playwright MCP)
- (Optional) A remote embedding endpoint (also required for OCR)
git clone https://github.com/indiser/ultron.git
cd Ultron
uv syncCopy .env.example to .env and fill in your endpoints:
BRAIN_LLM_NAME=<your-model-name>
BRAIN_LLM_BASE_URL=<openai-compatible-base-url>
# Optional: remote embeddings (falls back to local sentence-transformers)
# Required for OCR
EMBED_BASE_URL=<embedding-endpoint>
# TTS: path to your .safetensors voice state file
# Defaults to first_voice.safetensors in the project root if not set
TTS_STATE=<path-to-voice.safetensors>uv run main.pyThese are handled directly in the input loop, outside the agent:
| Command | Description |
|---|---|
/upload |
Open a file picker to ingest documents into the RAG store |
/add <path or URL> |
Ingest a local file or web URL directly |
/docs |
List all indexed documents |
/remove <filename> |
Remove a document from the RAG store |
/loc |
Show the currently saved location |
/loc auto |
Re-detect location via IP and save it |
/loc <city> |
Geocode a city and save it as the active location |
| Tool | Description |
|---|---|
list_files |
List files in the sandboxed workspace |
read_file |
Read a workspace file (4000 char limit) |
write_file |
Create or overwrite a workspace file |
edit_file |
Surgical single-occurrence text replacement |
delete_file |
Permanently delete a workspace file (user approval required) |
run_command |
Execute a shell command (user approval required) |
web_search |
Fast DuckDuckGo search, ~2s, returns 5 results |
research |
Deep research: fetch + embed + rank + extract, 30-90s |
search_docs |
Semantic search over indexed documents |
read_doc |
Read an indexed document sequentially in parts |
list_docs |
List all indexed documents with chunk counts |
ingest_doc |
Index a workspace file into the RAG store |
ingest_url |
Download and index a web page or PDF link |
upload_doc |
Open a file picker UI to upload and index documents |
remove_doc |
Remove a document from the RAG store (user approval required) |
ocr_image |
Extract text from an image, save as .ocr.md, and index it |
get_weather |
Current weather and up to 7-day forecast via Open-Meteo |
find_places |
Nearby places by category via OpenStreetMap, sorted by distance |
set_browser_mode |
Switch browser between headless and visible |
browser_navigate |
Navigate to a URL |
browser_snapshot |
Capture the current page state |
browser_click |
Click an element (user approval required) |
browser_type |
Type into a field (user approval required) |
browser_fill_form |
Fill a form (user approval required) |
browser_press_key |
Press a key (user approval required) |
browser_wait_for |
Wait for a page condition |
save_memory |
Persist a fact about the user to memory.md |
get_current_time |
Current time via MCP |
This is version 0.1.0. It is a proof of concept, not a production system.
- The LLM backend and embedding server are expected to be externally hosted (e.g. on a Kaggle instance tunneled via Cloudflare). They are ephemeral and will go down. The TTS pipeline is fully local and offline.
- There is no GUI. It is a terminal loop.
- Memory is a flat markdown file. There is no retrieval, no forgetting.
- The RAG store is a local ChromaDB instance. It persists across sessions but has no access control.
- The workspace is a single flat directory with no permissions model beyond a path check.
- OCR requires the remote embedding server to be running. It will fail if
EMBED_BASE_URLis not set. - Weather and places use free public APIs with no authentication. IP geolocation on first use sends your public IP to
ip-api.comonce; result is cached inlocation.json. delete_fileis permanent. There is no trash, no undo. The only barrier is ay/Nprompt.- Browser actions that mutate state require user approval, but the approval prompt is a simple
y/Nin the terminal. - Error handling is functional but not exhaustive.
- The voice pipeline requires
sounddevice, which can be finicky on Windows depending on audio drivers. - No tests exist.
- Streaming responses — pipe LLM token deltas directly to the TTS
Streamclass so speech latency drops to near-zero - Voice input — replace Wispr Flow with Handy for fully offline, privacy-preserving STT; no audio leaves the machine
- Local OCR — replace the remote OCR endpoint with a local model (e.g. Tesseract or a local vision LLM) so image reading works fully offline
- GUI / web interface — a minimal frontend so this isn't terminal-only
- Local LLM fallback — automatic failover to a local model when the remote backend is unreachable
- Tool sandboxing — run
run_commandinside a Docker container or Windows sandbox instead of the host shell - Tool routing — activate tool groups (browser, weather, places, docs, OCR) only when the user's message matches relevant keywords, reducing wrong-tool failures as the tool count grows
- Multi-agent — spawn sub-agents for parallelizable research tasks instead of sequential tool calls
- Persistent sessions — save and restore full conversation history across restarts
- Wake word — always-on microphone with a wake word to trigger the agent without typing
- Plugin system — hot-loadable tool modules without restarting the agent
- Emotion-aware TTS — modulate voice tone based on response content using the emotion vector API
Core: openai, mcp, ddgs, scrapling, sentence-transformers, numpy, python-dotenv
RAG: chromadb, pypdf, python-docx
OCR: pillow, httpx
Weather/Places: httpx
Browser: @playwright/mcp (via npx), obscura
Audio: sounddevice, soundfile, pocket-tts, torch
Runtime: uv (Python 3.14+)
Competence first. Swagger second. Everything else is costume.