Datasets

A dataset is a collection of episodes: demonstrations of a task, each one a synchronised record of everything the robot sensed, everything it was commanded, and what its cameras saw. Solarian stores one as a session: the directory the robot writes while you teleoperate, which is also the format every dataset op reads and writes.

A session is never edited. Curating it (deleting bad takes, splitting a pick from its place, tracking an object, mirroring) makes a new version that stores only what changed and reads the rest through from its parent. A training set is the last step, exported from whichever version you settle on.

What is in an episode

An episode answers three questions for every instant of a take, and each answer lives in its own file:

• What did the robot sense and do? One row in episode.parquet per module update, with every observation and action as a column. Each signal keeps the rate its hardware ran at; nothing is resampled until export.
• What did the cameras see? One mp4 per camera, at archive resolution, before any lens remap or downscale. The table stores a frame index into it, not pixels.
• Is the take usable? episode.json, written when the episode is sealed: its length, status, task, measured rates and running statistics, so you can choose episodes without opening a table.

What ties the episodes of a session together is session.json: the robot's structure, the column schema with units, and the camera geometry. It is written once, when the session's first episode opens, so every episode in it has the same columns and cameras. A session the controller recorded has the same layout, with source: controller and no robot structure; see Recording.

A session on disk

Here is a real one: an SO-106 with a wrist camera, recorded with solarian record.

recordings/robot/0012_4c9e1b_So106Robot/
├── session.json                         robot, schema, cameras, video settings
├── manifest.jsonl                       one line per sealed episode
└── episodes/
    ├── 000000/
    │   ├── episode.json                 8 KB
    │   ├── episode.parquet              92 KB: 1626 rows, 41 columns
    │   └── video/
    │       └── right_wrist_camera.image.mp4   8.9 MB: 974 frames at 1014x760
    ├── 000001/
    └── ...

Episode directories are numbered from 000000. A camera is named <module>.<field>, after the observation field it fills, so this robot's camera is right_wrist_camera.image.

session.json

{
  "solarian_version": "0.1.1",
  "robot": { "modules": { "right_arm": { ... }, "right_gripper": { ... } }, ... },
  "task": "stack the cups",
  "fps": 30.0,
  "cameras": {
    "right_wrist_camera.image": {
      "width": 1014,
      "height": 760,
      "color_mode": "rgb",
      "source_fps": 30.0,
      "sensor_mode": "2028x1520",
      "optical_remap": "conf/calibrations/right_wrist_camera_gopro_remap.json",
      "observation_resize": [1014, 760],
      "remap_size": [960, 540]
    }
  },
  "video": { "codec": "libx264", "crf": 20, "keyint": 120, "preset": "veryfast", "pix_fmt": "yuv420p" },
  "schema": {
    "t": { "data_class": "TimestampData", "dtype": "float64", "unit": "s" },
    "obs.right_arm.x": { "data_class": "PosData", "dtype": "float32", "unit": "m", "mid": 0.35, "high": 0.45 },
    "obs.right_gripper.aperture_mm": { "data_class": "PosData", "dtype": "float32", "unit": "mm" },
    "cam.right_wrist_camera.image.frame_index": { "data_class": "ImageData", "dtype": "int32", "unit": null },
    "act.right_arm.x": { "data_class": "PosData", "dtype": "float32", "unit": "m" },
    ...
  },
  "obs_fields": { "right_arm": ["shoulder_pan", ..., "x", "y", "z", "rx", "ry", "rz"], "right_gripper": [...] },
  "act_fields": { "right_arm": [...], "right_gripper": ["gripper"] }
}

The camera entry is what makes the archive reusable. The video is stored raw, so the lens remap (optical_remap) and the policy's input size are recorded beside it and applied at export. Recalibrate the camera and re-export, and old sessions pick up the new geometry.

The schema is derived from your project's observations and actions, never hand-written: add a joint and the next session has a column for it. Each column carries its data_class, unit, and the warning bands Solarian Base colours plots with.

episode.parquet

Column names say where a value came from:

A row is written whenever any module publishes, and a module that did not update carries its previous value forward. Eight consecutive rows of the episode above:

     t       obs.right_arm.x   obs.right_gripper.aperture_mm   cam.….frame_index   act.right_arm.x
  10.5817        -0.1416                 78.7677                     314               -0.1439
  10.5906        -0.1416                 78.7677                     315               -0.1439
  10.5978        -0.1416                 78.7677                     315               -0.1439
  10.6163        -0.1422                 78.7677                     315               -0.1439
  10.6292        -0.1422                 78.7677                     316               -0.1439
  10.6419        -0.1422                 78.9004                     316               -0.1439
  10.6895        -0.1428                 78.9004                     317               -0.1472
  10.6989        -0.1428                 78.9004                     318               -0.1472

The arm reported at 16.5 Hz in this take, the gripper at 42.7 Hz and the camera at 25 Hz, so each column changes on its own cadence: that is why the episode has 1626 rows but only 974 video frames. A table sampled at one fixed rate would have discarded the faster signals. The per-module ts columns tell you how stale each value is; export resamples everything onto one grid once a target rate is chosen.

episode.json

{
  "index": 0,
  "start_ts": 1790372667.769,
  "end_ts": 1790372700.300,
  "n_rows": 1626,
  "status": "ok",
  "task": "stack the cups",
  "n_frames": { "right_wrist_camera.image": 974 },
  "video_coverage": { "right_wrist_camera.image": 0.9918 },
  "dropped_frames": { "right_wrist_camera.image": 0 },
  "measured_hz": { "right_arm": 16.5, "right_gripper": 42.7, "right_wrist_camera": 25.1 },
  "stats": {
    "obs.right_arm.shoulder_pan": { "min": -18.58, "max": 31.78, "sum": 14272.17, "sumsq": 414617.6, "count": 1626 },
    ...
  }
}

Only ok episodes are read by ops and training; an episode whose video failed to seal is marked video_failed and skipped. stats holds running sums rather than means, so normalisation over a whole dataset is a sum over episodes rather than a pass over every row. video_coverage is the fraction of frame slots backed by a distinct capture; well below 1 means the camera, or its clock, could not keep up.

manifest.jsonl

One line per sealed episode, appended and fsynced as each one finishes:

{"episode":0,"sealed_at":1790372700.48,"status":"ok","bytes":9390832,"n_rows":1626,"duration":32.53,"transfer":{"state":"pending"},"upload":{"state":"pending"}}

It is the ledger that transfer and upload resume from after a crash. A dropped line is a take the operator threw away.

Files ops add

Ops can add members to an episode beside the table and videos. Built-in ops and tracking plugins write two:

Reading a session

Session opens a directory, a .solarian.zip or a version the same way, by member path:

from solarian.core.recording.archive import Session, video_member

session = Session("recordings/robot/0012_4c9e1b_So106Robot")
session.cameras                     # ['right_wrist_camera.image']
session.meta["schema"]              # session.json, parsed

for index in session.episodes():    # the sealed, ok episodes
    meta = session.episode_meta(index)
    table = session.read_table(index).to_pandas()
    video = session.locate(video_member(index, "right_wrist_camera.image"))
    print(index, meta["n_rows"], table["obs.right_arm.x"].iloc[-1], video)

solarian dataset info prints the same summary from the shell: episodes, frames, tracking coverage per camera, and the ops the session was derived through.

solarian dataset info recordings/robot/0012_4c9e1b_So106Robot

.solarian.zip

A session zipped is still a session. Zip the directory to move or upload it:

zip -r cup_stacking.solarian.zip recordings/robot/0003_7f3a9c_cup_stacking

Videos, tables and .npz files are stored uncompressed inside the zip, as they are compressed already, so an mp4 decodes straight out of the archive without extracting it. The wrapping directory an archiver adds is ignored, and a 7z of the same tree is read the same way.

Uploading a session to Solarian Base skips export. The irreversible choices (a target rate, a policy resolution, a warped image) stay undecided until you make them with solarian export-dataset.

Versions

A version is a session directory with one more file, version.json, that names its parent sessions and, for each of its episodes, the parent episode it starts from. Only the members the op wrote are stored in the version; everything else is read through from the parent, recursively.

versions/04-track/
├── version.json                         parents, the op and its params, episode sources
├── manifest.jsonl
└── episodes/000000/
    ├── episode.json                     always stored, with the op's metadata merged in
    └── targets/
        └── right_wrist_camera.image.npz the one file track added

version.json records where each episode came from and which op made it:

{
  "format": 1,
  "parents": ["../03-split-pick-place-pick"],
  "op": {
    "name": "track", "kind": "map", "source": "plugin:solarian-sam2",
    "params": { "camera": "right_wrist_camera.image", "model": "tiny" },
    "host": "turing", "started_at": 1792620560.75, "finished_at": 1792621012.30
  },
  "episodes": [{ "src": [0, 0] }, { "src": [0, 1] }, ...]
}

So deleting episodes costs a few JSON files and tracking an object costs one npz per episode, however large the videos are. Ops that change pixels, like mirror and split-pick-place, store the new videos and nothing else. Session resolves the layers, so code reading a version cannot tell it from a flat session. Flattening one into a standalone session, or a .solarian.zip, copies every member in and writes a lineage.json recording the ops it went through.

Dataset ops

An op takes one or more sessions and writes new ones. It comes in one of four kinds, named after what one input episode becomes:

solarian dataset ops lists what is installed, and solarian dataset describe <op> prints an op's parameters as JSON Schema:

$ solarian dataset ops
delete-episodes      filter   Delete episodes  (builtin)
merge                combine  Merge datasets  (builtin)
mirror               expand   Mirror left-right  (builtin)
split-pick-place     expand   Split into pick and place -> pick, place  (builtin)
track                map      Track an object [gpu]  (plugin:solarian-sam2)

Running one op

solarian dataset run applies an op once. Parameters are --param key=value, with the value read as YAML so numbers, lists and mappings mean what they look like:

solarian dataset run delete-episodes data/cups --param 'episodes=[3, 7]'
solarian dataset run split-pick-place data/cups --param axis=z --param tol=0.005
solarian dataset run merge data/cups-monday data/cups-tuesday -o data/cups-all

The output defaults to <session>-<op> beside the input (data/cups-delete-episodes), with -<output> appended for an op that writes several, and is a version of its input. An output path ending in .zip, or --flat, stores every member instead.

Computed datasets

Curation is rarely one op. A computed dataset is a directory that groups every version built over one source session, and records each op, its parameters and every per-episode redo in computed.json. That file is the dataset's recipe: rebuild replays it from the source.

data/cups-curated/
├── computed.json
└── versions/
    ├── 01-merge/
    ├── 02-delete-episodes/
    ├── 03-split-pick-place-pick/
    ├── 03-split-pick-place-place/
    ├── 04-track/
    └── 05-mirror/

Versions are named <nn>-<op> and addressed as <dir>@<version>; @source is the source session. apply builds on the version you name, or on the only leaf of the tree if you name none.

Here is a full pass over two days of cup stacking, ending in a pick-only training set:

# Start over Monday's session, and merge in Tuesday's.
solarian dataset new datasets/cups_monday data/cups-curated
solarian dataset apply data/cups-curated merge --with datasets/cups_tuesday

# Drop two bad takes found while watching them in Solarian Base.
solarian dataset apply data/cups-curated delete-episodes --param 'episodes=[3, 7]'

# Cut each take into its pick and its place. The tree now has two leaves.
solarian dataset apply data/cups-curated split-pick-place

# Track the cup on the pick half, reviewing every episode.
solarian dataset apply data/cups-curated@03-split-pick-place-pick track --online \
  --param camera=right_wrist_camera.image --param 'prompt={frame: 0, point: [512, 400]}'
solarian dataset redo data/cups-curated@04-track 4 --param 'prompt={frame: -1, point: [730, 711]}'
solarian dataset accept data/cups-curated@04-track 4
solarian dataset close data/cups-curated@04-track

# Double the data by mirroring, then flatten it to train on.
solarian dataset apply data/cups-curated@04-track mirror
solarian dataset export data/cups-curated@05-mirror cups-pick.solarian.zip

log shows the tree, with each version's state, episode count and parameters:

$ solarian dataset log data/cups-curated
cups_monday  (source)
└─ 01-merge  closed, 120 episodes  {}
   └─ 02-delete-episodes  closed, 118 episodes  {"episodes": [3, 7]}
      ├─ 03-split-pick-place-pick  closed, 118 episodes  {}
      │  └─ 04-track  closed, 118 episodes, 118/118 reviewed  {"camera": "right_wrist_camera.image", ...}
      │     └─ 05-mirror  closed, 236 episodes  {}
      └─ 03-split-pick-place-place  closed, 118 episodes  {}

A version you no longer want goes with everything built on it:

solarian dataset drop data/cups-curated@05-mirror
solarian dataset rebuild data/cups-curated data/cups-curated-v2   # replay the recipe from the source

Reviewing a version

Map and filter ops are often wrong on a few episodes: a tracker loses the object, a threshold drops a good take. Apply one with --review and its version stays open: every episode is processed, then you go through them, redo the ones that are wrong with different parameters, and accept the rest. --online processes nothing up front, so each episode is done when you reach it.

solarian dataset redo data/cups-curated@04-track 12 --param 'prompt={frame: 30, bbox: [400, 300, 620, 520]}'
solarian dataset redo data/cups-curated@04-track 17 --drop     # leave this episode out
solarian dataset accept data/cups-curated@04-track 12
solarian dataset close data/cups-curated@04-track

On a filter version, redo --keep and --drop overrule the op's decision for one of the parent's episodes. Ops only build on closed versions, so nothing a version reads through to can change under it. Solarian Base runs the same review from a computed dataset's page, playing each episode forwards or backwards as you decide.

Writing an op

An op is a file in your project, next to modules/:

myrobot/
└── myrobot/
    └── dataset_ops/
        └── drop_short_episodes.py   # the op drop-short-episodes

Every <package>/dataset_ops/<name>.py is an op, named after the file with _ turned into -, unless the class sets name. There is nothing to register, and it can import your project's code. create-solarian writes drop_short_episodes.py as an example, and solarian add-op <name> writes another from the same template:

from pydantic import BaseModel, Field

from solarian.datasets import DatasetOp, Episode, OpKind


class DropShortEpisodesParams(BaseModel):
    min_frames: int = Field(default=30, ge=1, description="Episodes with fewer rows are dropped.")


class DropShortEpisodesOp(DatasetOp):
    title = "Drop short episodes"
    description = """
    Drop the episodes shorter than min_frames: false starts and aborted demos.
    """
    kind = OpKind.FILTER
    Params = DropShortEpisodesParams

    def keep(self, episode: Episode, params: DropShortEpisodesParams) -> bool:
        return episode.n_rows >= params.min_frames

An Episode gives the op its meta (the parsed episode.json), table(), video(camera), targets(camera) and any other member by path. A MAP op implements map(episode, params, ctx) and returns episode.derive(files, **meta) with only the members it adds or replaces; the runner stores those and reads the rest through from the parent:

import pyarrow.compute as pc
from pydantic import BaseModel, Field

from solarian.core.recording.archive import TABLE_FILE, column_stats, table_bytes
from solarian.datasets import DatasetOp, Episode, OpContext, OpKind

COLUMN = "obs.right_gripper.aperture_mm"


class ClipApertureParams(BaseModel):
    high: float = Field(default=80.0, description="Millimetres. Readings above this are clipped to it.")


class ClipApertureOp(DatasetOp):
    title = "Clip gripper aperture"
    kind = OpKind.MAP
    Params = ClipApertureParams

    def map(self, episode: Episode, params: ClipApertureParams, ctx: OpContext):
        table = episode.table()
        clipped = pc.min_element_wise(table[COLUMN], params.high).cast(table[COLUMN].type)
        table = table.set_column(table.column_names.index(COLUMN), COLUMN, clipped)
        return episode.derive({TABLE_FILE: table_bytes(table)},
                              stats=column_stats(table, episode.meta["stats"]))

The version stores one new episode.parquet and episode.json per episode; the videos stay in the parent.

EXPAND implements expand and yields (output, episode) pairs, and COMBINE implements combine over every input in ctx.inputs. The Params model is the op's form: Solarian Base renders it from its JSON Schema, and --param is checked against it.

solarian dataset ops lists every op with where it came from: builtin, plugin:<package> or project:<path>. A project op with the name of a built-in or plugin op takes its place. A file that fails to import is listed with its error, and the other ops still load. To share an op across projects, move the file into a plugin and add one solarian.dataset_ops entry point; solarian create-plugin op <name> scaffolds one.

Other formats

Solarian Base also opens two formats from outside Solarian, to browse and compare, recognised from the archive's contents rather than its name: a LeRobot v2.1 dataset zipped (meta/info.json is its signature, and it is what solarian export-dataset writes for training), and a UMI replay buffer packed as a .zarr.zip. Both are already resampled to one rate and, in UMI's case, stored as frame arrays rather than video, so dataset ops and versions work on Solarian sessions only.