AI Controller

Any Controller becomes an AI controller by declaring tools and adding an agent: block to conf/controller.yaml. The operator types a task in the chat ("put all the batteries in the white box"); a planner model plans it and calls the tools, which start Skills on the robot; a monitor model watches each running skill through the cameras and ends it, or wakes the planner, when something happens.

The harness lives in solarian/core/ai/agent/. There is no separate controller class: the agent is a part of Controller, built when agent: is set.

How a run goes

 user: "put all the batteries in the white box"
   │
   ▼
 planner thread ── wakes on: a message · an action failing or stopped · a monitor flag ·
   │               an operator key · a plan step that needs it · the optional heartbeat
   │  sees: the task, what happened, the plan, the newest camera images (with ids), readings
   │  calls: update_plan · observe · <skill tools> · stop_action · ask_user · finish_task
   │  an action that succeeds starts the plan's next complete step without it
   ▼
 control thread ── starts / stops skills between two ticks (call_on_control_thread)
   │
   ▼
 monitor thread ── every monitor.period_s while actions run:
                   timeouts · check_agent_action tripwires · the skill's monitor questions
                   → finish the action, or wake the planner

A skill tool returns as soon as the skill has started, and a reply that starts one ends the planner's turn: it is woken when something changes. A call the harness refuses, or an observe, is answered at once instead, so the planner can retry or look.

The plan is a queue the harness runs. Each update_plan step can carry the tool and args that carry it out. When an action succeeds and nothing else runs, the harness marks its step done and starts the next step whose call is complete, logging a plan_step event, without calling the planner. It wakes the planner only when the next step needs a point in a fresh image (a tool with image_points), has no tool (checking the result is the planner's), cannot start, or when every step is done. A failure, a timeout, a stop, a monitor flag, the user and the operator always wake it, and the queue waits for its reply. An action the planner starts itself takes the first unfinished step naming its tool. So a pick-and-place costs about four planner turns: plan and start the first move, point the pick, point the place, check and finish.

A run is one episode: when either side records (see Recording), starting a run opens an episode on the sides that are switched on and ending it closes it.

The operator always wins. Any key, click, skill start or stop from the operator pauses the run; the next chat message resumes it, and the planner is told what the operator did. Every command the harness sends carries the run's generation, so a decision made before a cancel, reset or takeover is refused instead of executed after it. When a run ends, on_agent_idle() runs so the controller can bring teleop back.

Declaring tools

A tool is a skill class plus the params the controller binds. The planner fills in the rest:

from solarian.core import Controller
from solarian.core.skill import Tool

ARM = {"arm": "right_arm", "gripper": "right_gripper"}

class MyController(Controller):
    tools = {
        "move_to_pose": Tool(MoveSkill, ARM),
        "nudge_gripper": Tool(MoveSkill, {**ARM, "offer": "delta"}),
        "pick_object": Tool(UMISkill, {"config": "pick_object.yaml"}),
        "place_at": Tool(PlaceSkill, {**ARM, "camera": "right_wrist_camera"}),
    }

    def on_agent_idle(self):
        self.start_skill(CartesianTeleopSkill(params=ARM))

The bound params (module names, which policy config) never reach the model. A tool whose skill has no description is left off the planner's list, with a warning.

The skill contract

Everything the planner and monitor know about a tool comes from the skill, through Skill.tool_spec(params). By default that is these class attributes:

A skill whose behaviour comes from config overrides tool_spec: UMISkill reads an agent: block from the policy yaml it is bound to, and MoveSkill describes the input the binding's offer names (pose with the pose names from move.yaml and their descriptions, a delta nudge, or a gripper aperture).

A move ends when the robot gets there, not after a fixed time, since every second a finished move keeps running delays the next step. MoveSkill ends when every targeted field reads within tolerance of its target, or when none has moved for 0.3 s, counted from 0.5 s after the move was sent: a gripper closed on an object, or a pose just out of reach. The tolerances are 2 units for joints, 2 mm for a gripper aperture, 5 mm and 0.05 rad for a Cartesian pose. timeout_s is only the cap. A pose the arm is already at ends on the first check. A skill of your own that sends an exclusive move can end the same way with solarian_move_skill.Arrival(targets, sent_at).check(self.observations, now).

A skill that reads its target from a tracker overrides services_for(params) to return (module, field, service params): the harness starts those services with the planner's point and waits for the tracker's first result before starting the skill. When the action ends, the harness stops them, unless another running action reads the same field: a tracker left running keeps painting its mask over whatever the action left, such as an object now held in the gripper. With max_target_area set, it measures that first mask: on so106 a pick's object covers 0.1-0.7% of the policy's image, while a point beside it selected 22-48%, so pick_object.yaml refuses a mask over 5% of the camera image.

A skill reports a failure the planner can act on with self.fail("reason"). It raises SkillFailed, which ends the skill as FAILED wherever it runs; the reason reaches the planner through the controller (or the robot's status rows).

Monitor questions

A question costs a vision call every tick, 3-6 s with a model that judges a grasp correctly, so its answer can arrive after the moment it is about. so106's pick therefore has none: it runs to its max_duration_s and the planner checks the newest image, one turn instead of several calls. Its place keeps a released question, since a place that has let go is plain in one frame. This is how the pick was watched:

# conf/skills/pick_object.yaml, before it moved to a time limit
agent:
  description: Grasp one object with the pick policy and lift it. Point at the object in the wrist image.
  max_duration_s: 25
  monitor:
    - name: progress
      type: choice
      instructions: Which phase the pick is in. Compare the previous close-up with the current one.
      choices:
        approaching: >-
          The fingers are open, the target lies ahead of them, and it looks larger
          (closer) in the current close-up than in the previous one.
        grasping: >-
          The fingers are down at the target and closing on it: they are closer
          together in the current close-up than in the previous one.
        holding: >-
          The target is pinched between the fingers: it sits between them, both
          fingers touch it, and the fingers are visibly bent by the pressure on it.
        failed: >-
          The fingers are closer together than the target is wide, with nothing
          between them, or they reopened: wider in the current close-up than in the
          previous one.
      when:
        - {do: finish, value: holding, threshold: 0.8, consecutive: 1, then: home_holding}
        - {do: escalate, value: failed, threshold: 0.8, consecutive: 2, after_s: 6}

A predicate is answered with a probability, a choice with a value and its probability. when is one rule or a list of them. A rule fires when its condition holds consecutive ticks in a row: finish stops the action as succeeded, fail stops it as failed, escalate leaves it running and wakes the planner once per question. threshold is a confidence floor: a tick whose answer is less sure does nothing, and the next tick decides. With a floor and consecutive: 1, one confident tick acts, which is faster and cheaper than waiting for several in a row; measure it on labelled ticks before trusting it. after_s keeps a rule from counting ticks until the action has run that long. then on a finish or fail rule names a tool the harness starts, without arguments, as soon as the action has stopped: the step that always follows, such as carrying a picked object home with the grip kept. After a finish, the follow-up carries the action's plan step on, or takes the next step when the plan names its tool there, and the planner is not woken; it hears what happened at its next wake, at the latest when the follow-up ends. After a fail, the planner is woken at once, and not again when the action has actually stopped. If the skill reports its own failure meanwhile, its reason is kept beside the rule's. A then naming no tool fails when the harness starts. A question without when is only shown to the planner. The planner can add its own conditions to any call with watch_for.

Each tick the monitor sees each camera's last monitor.history frames, oldest first, labelled previous, 1.6 s ago and current, and the readings. With history above 1, an action's first frame is only kept for the next tick: with nothing to compare it with, it is not judged, and the episode log records it as a monitor_frame event. A camera listed in monitor.close_ups is seen only as that region, at capture resolution: what decides a grasp is a few pixels of the fingers, which a full frame at image_max_side blurs away. Other cameras are seen whole at image_max_side. Monitor frames never carry the tracker's overlay, since a tint over the fingers hides the very pixels that decide, and a tracker that slips onto another object misleads. With monitor.mark_target, a 1 px cyan box 3 px outside the bounds of the tracker's mask (else around its box) shows which object the action is on and how wide it is, without covering any of it; the monitor is told to trust the frame over the mark. With monitor.reference on (the default), each tick also gets the frames from when the action started, with the planner's point on the target; turn it off when the close-ups are all a question needs, since frames from before the grasp only distract from it.

Write questions the monitor can answer from the previous frame and the current one:

• Prefer one progress choice with a failed value over several failure predicates. A phase is judged from motion across frames, and "is this failing?" asked during the approach comes back yes.
• Define each choice by what changes between the previous and current frame (closer, closing, reopened), or by what shows in the current one (the fingers bent around the target).
• Give failure rules an after_s long enough for the policy to reach the object, and consecutive: 2: half-closed fingers on the way look like a miss in one frame pair, while a real miss lasts. Finish rules usually need neither. Judge wording by the per-pick replay, not per-tick accuracy: on so106, a looser failed was wrong on a quarter of ticks yet escalated every missed pick on the right tick, while a stricter one cut false answers to 3% and missed a whole failed pick.
• Describe success by what shows in the camera, for any object ("pinched between the fingers, which bend under the pressure"), and the nearest miss too ("closer together than the target is wide, with nothing between them"). Robot-specific detail (finger colour and shape, camera angle) goes in monitor.instructions, once for every skill.

Measure the questions on recorded ticks before trusting them. The episode log keeps every monitor frame, the episode the readings. so106 has two scripts for this, run from the project root:

python scripts/label_monitor_ticks.py   # previous | current close-up over the prompt; keys 1-4 label, s unsure, b back, q quit
python scripts/eval_monitor.py --repeats 3 [--model gpt-6-luna] [--reasoning medium] [--no-target]

The first appends labels to evals/pick_progress.jsonl and resumes at the first tick missing one. With mark_target on, it also asks where the target is in each close-up, since recordings made before episode logs kept no tracker output: drag a box around it as tight as the tracker's mask would be, or right-click when it is out of view; boxes go to evals/pick_targets.jsonl. The second sends each labelled tick as the robot would, marked where the boxes say, and reports accuracy, the confusion between choices, how often an answer met a rule's condition on the wrong tick, how often it fell below its floor, how often repeats of one tick disagree, and latency. It also replays each pick through the skill's rules, consecutive included, and prints the first rule to fire and the label of the tick it fired on.

The planner is told that a monitor flag can be wrong and to check it against the newest image before stopping an action. A timeout reaches it as "without the monitor confirming success", or for an unwatched tool as "ran its limit; judge from the newest image whether it succeeded", not as a failure, and it is told to look before undoing anything: a policy that never stops by itself may have finished its job long before.

For a signal no camera shows (a force sensor, a limit switch, a policy that has stopped moving), override check_agent_action(action_id, tool, elapsed_s) on the controller and return ("finish" | "fail" | "escalate", reason), or with a third element, a tool to start once the action has stopped, as a rule's then. It is polled with the monitor; action_id changes with every action. Its flags reach the planner as a controller check, which the planner trusts. so106 fails a pick or place whose arm and gripper have stayed within a few millimetres for 5 s and starts home_holding: a stall is plain in the readings, whatever the object. Do not use it for hand-tuned thresholds on readings that only correlate with success, such as a gripper's aperture and height: soft fingers read the same holding as empty, and such rules break with every new object.

A watch_for condition the planner adds is checked from 6 s into the action, like a failure rule with after_s: 6, and must be false while the action goes as it should.

Configuration

# conf/controller.yaml
agent:
  planner:
    provider: anthropic          # or openai (chat completions; any compatible server with base_url)
    model: claude-opus-5
    api_key_env: ANTHROPIC_API_KEY
    thinking_budget: 4096
  monitor:
    provider: openai_responses   # or openai_decisions (/v1/decisions): faster, typed answers
    model: gpt-6.1-sol
    reasoning_effort: low
    api_key_env: OPENAI_API_KEY
    period_s: 1.5                # a tick waits for the previous call
    history: 2                   # frames per camera each tick: this one and the last
    image_max_side: 448
    close_ups:                   # x0, y0, x1, y1 as fractions of the capture
      right_wrist_camera/image: [0.25, 0.5, 0.75, 0.9]
    reference: false             # no frames from the action's start
    instructions: |
      What is always in view (the gripper's fingers), and what a held object looks like.
  planner_heartbeat_s: null      # no periodic wakes
  budgets: {max_turns: 80, max_minutes: 20, retries_per_step: 3}
  cameras: [right_wrist_camera/image]
  state_fields: {right_arm: [x, y, z], right_gripper: [aperture_mm]}
  dry_run: false
  system_prompt: |
    What this robot is, what its tools are good and bad at.

Install the model clients with pip install 'solarian[agent]'.

Chat

• pygame window (ui: pygame): a chat column appears on the right. Enter (or a click on the input box) opens the input; while typing, keys type text and never drive the robot (held keys are released first). Enter sends, Esc drops the line. Drag over the conversation to select text and Cmd/Ctrl+C to copy it; Cmd/Ctrl+V pastes into the input, and the wheel scrolls back through the conversation. /cancel ends the run; /reset also forgets the conversation.
• Solarian Base (ui: external): the controller page shows an AI controller panel with the plan, running actions, the conversation and an input. Cancel is a safety command: an observer without the control lock can send it.
• Any front end sends UICommandType.AGENT_MESSAGE ({"text": ...}), AGENT_CANCEL and AGENT_RESET, and reads UISnapshot.agent. The image the planner last saw, with its chosen points marked, is drawn as agent/view.

Episode logs

Every episode the controller records has a log, in recordings/controller/<session>/episodes/<episode>/logs/, holding what the operator and the agent did during it:

• events.jsonl: one JSON object per line, each with t (the synced clock, ISO 8601) and kind.
• images/: the images the events and turns name.
• turns/NNNN.json: every planner call's input and output, numbered from 0001 in each episode.
• overlays/<module>.<field>.jsonl: what the trackers marked on each camera frame.

The controller logs episode_started (with the task) and episode_ended, and every command it carries out, with by: operator or by: agent: skill_started (skill, params), skill_stop (skill, reason), service_started (module, field, params: a tracker and the point it was seeded with) and service_stopped. It also logs skill_ended (skill, completed, failed or stopped, reason), which has no by, since a skill ends by itself as often as it is stopped.

An agent run adds:

• run_started: the run id, task, models, system prompt and tools.
• planner_turn for every planner call: its turn file, latency, token usage, text and the tools it called.
• Every action, monitor answer, plan change, step the harness started and operator takeover. action_started says how long the tracker seed and the start took, and with max_target_area the share of the image the mask covered, as target_area. target_rejected is a call refused for its mask.
• run_ended: status, summary and usage.

The run id is the episode it started in, <session>/<episode>, for example 0003_7f3a9c_cup_stacking/000002. A run that goes on after its episode closes, for example after you take over, seal the episode and press the record key, carries on in the next episode's log. Its first write there is a run_continued event, with the run's config, its status and plan so far, and the episode it continues_from. The next planner turn holds only the messages since the previous one, and every image the new log's events and turns name is saved in it. Between episodes, writes go on in the last episode's log, and a run that started with nothing recording logs from the first episode the controller opens. A run only the robot records, or that nothing records, keeps no log.

Overlays are logged for the fields the controller draws on each camera, one line per new frame that has any: {"ts": capture time, "size": [h, w], <field>: {"mask": base64 PNG} | {"point": [x, y, visibility %]} | {"box": [x0, y0, x1, y1]}}, in the pixels of the frame as it arrived.

so106's scripts/episode_viewer.py plays an episode's video with its events beside it, the agent's and the operator's alike, and the overlays drawn as the controller drew them live. o turns the overlays off and on, and clicking a planner turn prints what the planner was sent and replied. Sessions recorded before logs/ have only the agent's events, in agent/; the viewer and monitor_ticks.py read both.

When the robot recorded the same episodes, its session has the same directory name, so after transferring it off the robot you can copy the logs in beside the robot's episodes:

solarian dataset attach-logs recordings/robot/0003_7f3a9c_cup_stacking

Legacy AIController

The earlier dual-process AIController (a ThinkProcess choosing skills from Jinja prompts, configured by system2_*) is in solarian.core.ai.old for projects that still use it. New projects should use agent:.

See also

• Skills — the units of work the agent calls.
• Controllers — the base class the agent is part of.
• Recording — sessions and episodes the episode logs attach to.