Andy Xiao

creative technologist

All work

Local agent platform

Personal project, since February 2026.

A local AI agent on a 16 GB M4 Mac mini. It chats in a native Mac app and on Discord, sets reminders, builds a knowledge graph, paints on a schedule, and hands coding tasks to Claude Code.

Everything runs on the Mac mini. Focus: a harness that makes a 9B model fast, accurate and safe.

The Mac app: a greeting, a question about the distance to the Moon answered with Wikipedia sources, and a reminder offer with Set it and No buttons

Conversation

  • Model. Qwen3.5-9B, MLX 4-bit, with vision.
  • Skill pick. A short model call picks a skill before each reply. Here: a Wikipedia lookup, then a reminder.
  • Approval. Changes to reminders, memories and claims require approval from user.

Architecture

App and Discord messages share a SQLite log.
  • Memory. Language model uses about 6 GB. Image models swap in for each render. A watchdog stops renders when memory runs low.
  • Operations. Health service with automatic fixes, security audit every 30 minutes, nightly backup.
  • Self-improvement. "Dream" turns the agent's suggestions into pull requests through Claude Code.
  • Network. Search and weather queries, news feeds, Discord replies, code tasks sent to Claude Code, and off-site backups.

Menu-bar panel

Prompt cache and routing

Prompt cache

A cold read of the 28,500-token prompt takes 146 seconds.

  • Each prompt continues its predecessor byte for byte.
  • A cache slot per conversation, checkpointed at each new message.
  • Slots prewarm after renders and restarts.

Warm turns reuse over 99% of prompt tokens.

Routing

Model skill pick against regular expressions:

MeasureRegexesModel pick
Asks that reached the right skill18 of 41126 of 137
Reminder asks caught14 of 2323 of 23
"Remember this" facts caught8 of 1817 of 18
False "remember this" offers5 of 431 of 43
Time per call0.48 s

Same 90% overall. Regexes miss 23 of 41 real asks.

Terminal output of the routing eval comparing regexes with the model's pick, including per-skill mistakes
Eval harness: live model, real turn code, isolated database copy.

Knowledge graph

The Knowledge page for a fictional user, Riley: stat tiles, an unverified claim from a visitor, a graph of facts, and a list of facts with their sources and controls
Fictional test user.
  • Extraction. Facts extracted from conversations, each with source quote, confidence and privacy level. Private by default.
  • Uncertainty. Under 80% confidence: ask, don't state.
  • Claims. Statements from other people stay unconfirmed until approved.
  • Corrections. Correct, confirm, forget or block any fact. Duplicates merge instead of deleting.

Image Generation

A calm mountain lake reflecting forested ridges and basalt columns lit by low sun
Qwen-Image 2.1
Looking down into a round pit on the Moon, its basalt walls half in sun, opening into a dark lava tube
Qwen-Image 2.1
A silver tabby cat with dark stripes and golden eyes
Qwen-Image 2.1
A cave of pale blue stalactites and calcite columns, lit amber from within, with a sandy path running into the distance
FLUX.1-dev
  • Projects. Six art projects, each with a renderer and schedule. Structured direction: model fills fields from random menus, code assembles the prompt, a lint flags strays.
  • Literal prompts. Nothing impossible, explicit position and scale, observable physical terms only.
  • Renderers. Qwen-Image 2.1 through mflux since September, about 29 minutes per image. FLUX.1-dev through ComfyUI before that.
  • Behaviors. Twelve scheduled behaviors with chainable steps.
  • Output. 2,477 paintings, 10 to 13 a day.

Runtime