Dataset Preview
Duplicate
The full dataset viewer is not available (click to read why). Only showing a preview of the rows.
Job has been terminated due to a temporary spike in resource usage and may be restarted later.
Error code:   JobManagerCrashedError

Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.

leader
dict
lifter
dict
observations
dict
physical
dict
posture_reference
dict
sim
dict
timestamps
dict
{"eef":[0.04098762944340706,-0.022157905623316765,0.5557513236999512,0.16337113082408905,0.708559989(...TRUNCATED)
{ "action_mm": [ 0, 0 ], "active_target_mm": [ 0, 0 ], "command_delta_mm": [ 0, 0 ] }
{"eef":[0.04098762199282646,-0.022157905623316765,0.5557513236999512,0.16337114572525024,0.708559989(...TRUNCATED)
{"applied_command":{"dof_velocity_targets":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0(...TRUNCATED)
{"qpos":[-0.021196255460381508,-0.7915393710136414,-0.00801072921603918,1.5211988687515259,-0.009913(...TRUNCATED)
{"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,-0.7071068286(...TRUNCATED)
{"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488000000000,1792454488000000000],"l(...TRUNCATED)
{"eef":[0.040987528860569,-0.022159816697239876,0.5557512044906616,0.16337165236473083,0.70856148004(...TRUNCATED)
{ "action_mm": [ 0, 0 ], "active_target_mm": [ 0, 0 ], "command_delta_mm": [ 0, 0 ] }
{"eef":[0.04098762199282646,-0.022157905623316765,0.5557513236999512,0.16337114572525024,0.708559989(...TRUNCATED)
{"applied_command":{"dof_velocity_targets":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0(...TRUNCATED)
{"qpos":[-0.02049681358039379,-0.791543185710907,-0.0078984210267663,1.5211988687515259,-0.010442748(...TRUNCATED)
{"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED)
{"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488033333248,1792454488033333248],"l(...TRUNCATED)
{"eef":[0.04098748043179512,-0.02216172404587269,0.5557510852813721,0.16337208449840546,0.7085630297(...TRUNCATED)
{ "action_mm": [ 0, 0 ], "active_target_mm": [ 0, 0 ], "command_delta_mm": [ 0, 0 ] }
{"eef":[0.04098762199282646,-0.022157905623316765,0.5557513236999512,0.16337114572525024,0.708559989(...TRUNCATED)
{"applied_command":{"dof_velocity_targets":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0(...TRUNCATED)
{"qpos":[-0.019797371700406075,-0.7915470600128174,-0.007786113303154707,1.5211988687515259,-0.01097(...TRUNCATED)
{"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED)
{"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488066666752,1792454488066666752],"l(...TRUNCATED)
{"eef":[0.040987443178892136,-0.02216363325715065,0.5557510256767273,0.16337239742279053,0.708564579(...TRUNCATED)
{ "action_mm": [ 0, 0 ], "active_target_mm": [ 0, 0 ], "command_delta_mm": [ 0, 0 ] }
{"eef":[0.04098768159747124,-0.022157935425639153,0.5557513236999512,0.16336871683597565,0.708560585(...TRUNCATED)
{"applied_command":{"dof_velocity_targets":[[0.005245814099907875,-0.000029802322387695312,0.0008423(...TRUNCATED)
{"qpos":[-0.019097931683063507,-0.7915509343147278,-0.007673805113881826,1.5211988687515259,-0.01150(...TRUNCATED)
{"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED)
{"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488100000000,1792454488100000000],"l(...TRUNCATED)
{"eef":[0.04098735377192497,-0.02216527797281742,0.5557509660720825,0.16337287425994873,0.7085658311(...TRUNCATED)
{ "action_mm": [ 0, 0 ], "active_target_mm": [ 0, 0 ], "command_delta_mm": [ 0, 0 ] }
{"eef":[0.040986981242895126,-0.022158905863761902,0.5557512640953064,0.16336098313331604,0.70856457(...TRUNCATED)
{"applied_command":{"dof_velocity_targets":[[0.0209832563996315,-0.00011622905731201172,0.0033692223(...TRUNCATED)
{"qpos":[-0.018492907285690308,-0.7915540933609009,-0.007576672360301018,1.5211988687515259,-0.01195(...TRUNCATED)
{"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED)
{"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488133333248,1792454488133333248],"l(...TRUNCATED)
{"eef":[0.0409872941672802,-0.022166922688484192,0.555750846862793,0.16337323188781738,0.70856714248(...TRUNCATED)
{ "action_mm": [ 0, 0 ], "active_target_mm": [ 0, 0 ], "command_delta_mm": [ 0, 0 ] }
{"eef":[0.040985602885484695,-0.022160524502396584,0.5557512640953064,0.16335444152355194,0.70856928(...TRUNCATED)
{"applied_command":{"dof_velocity_targets":[[0.0209832563996315,-0.00011771917343139648,0.0033692340(...TRUNCATED)
{"qpos":[-0.017887884750962257,-0.7915573120117188,-0.007479539606720209,1.5211988687515259,-0.01241(...TRUNCATED)
{"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED)
{"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488166666752,1792454488166666752],"l(...TRUNCATED)
{"eef":[0.04098724201321602,-0.022168567404150963,0.5557507872581482,0.16337351500988007,0.708568513(...TRUNCATED)
{ "action_mm": [ 0, 0 ], "active_target_mm": [ 0, 0 ], "command_delta_mm": [ 0, 0 ] }
{"eef":[0.04098433256149292,-0.022162048146128654,0.5557512044906616,0.16334865987300873,0.708573460(...TRUNCATED)
{"applied_command":{"dof_velocity_targets":[[0.02027507871389389,-0.00011175870895385742,0.003255426(...TRUNCATED)
{"qpos":[-0.017282860353589058,-0.7915604710578918,-0.0073824068531394005,1.5211988687515259,-0.0128(...TRUNCATED)
{"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED)
{"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488200000000,1792454488200000000],"l(...TRUNCATED)
{"eef":[0.040987201035022736,-0.022169996052980423,0.5557507276535034,0.16337381303310394,0.70856964(...TRUNCATED)
{ "action_mm": [ 0, 0 ], "active_target_mm": [ 0, 0 ], "command_delta_mm": [ 0, 0 ] }
{"eef":[0.04098319634795189,-0.022163445129990578,0.5557512044906616,0.16334328055381775,0.708577096(...TRUNCATED)
{"applied_command":{"dof_velocity_targets":[[0.018150731921195984,-0.00009685754776000977,0.00291398(...TRUNCATED)
{"qpos":[-0.017178155481815338,-0.7915610074996948,-0.007361347321420908,1.5211988687515259,-0.01295(...TRUNCATED)
{"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED)
{"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488233333248,1792454488233333248],"l(...TRUNCATED)
{"eef":[0.04098718240857124,-0.022171422839164734,0.5557506084442139,0.16337411105632782,0.708570778(...TRUNCATED)
{ "action_mm": [ 0, 0 ], "active_target_mm": [ 0, 0 ], "command_delta_mm": [ 0, 0 ] }
{"eef":[0.04098205268383026,-0.02216489613056183,0.5557512044906616,0.16333752870559692,0.7085807323(...TRUNCATED)
{"applied_command":{"dof_velocity_targets":[[0.018150685355067253,-0.00009685754776000977,0.00291398(...TRUNCATED)
{"qpos":[-0.01707345061004162,-0.7915615439414978,-0.0073402877897024155,1.5211988687515259,-0.01303(...TRUNCATED)
{"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED)
{"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488266666752,1792454488266666752],"l(...TRUNCATED)
{"eef":[0.040987107902765274,-0.022172847762703896,0.5557505488395691,0.16337446868419647,0.70857191(...TRUNCATED)
{ "action_mm": [ 0, 0 ], "active_target_mm": [ 0, 0 ], "command_delta_mm": [ 0, 0 ] }
{"eef":[0.040981024503707886,-0.022166265174746513,0.5557512044906616,0.16333284974098206,0.70858377(...TRUNCATED)
{"applied_command":{"dof_velocity_targets":[[0.01439834013581276,-0.00007599592208862305,0.002343440(...TRUNCATED)
{"qpos":[-0.0169687457382679,-0.791562020778656,-0.00731922872364521,1.5211988687515259,-0.013119695(...TRUNCATED)
{"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED)
{"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488300000000,1792454488300000000],"l(...TRUNCATED)
End of preview.

Tower-SimData

Project page · Code · Simulator assets

Version 2.0.0 replaces the earlier release with 600 selected simulation demonstrations: 12 paper configurations (A–F × 2/3 layers), with 50 episodes each. The old Task3 episodes are not included. Earlier versions remain available in repository history.

This is a 600-episode simulation-only subset, not the complete 12,000-demonstration SIM/REAL collection described in the paper. Real-world demonstrations are not included in this repository.

Each episode is an independent raw HDF5 file at 30 FPS, with robot states and commands, end-effector poses, four JPEG camera streams, and simulation state. Camera images and all dataset values are preserved from the selected source files. Internal filesystem paths in metadata are redacted; source and published file hashes are recorded separately.

Task identities and dataset size

Public task names follow the paper's A–F definitions. Historical numeric task IDs are retained in MANIFEST.jsonl, TASK_MAPPING.json, and internal HDF5 metadata. Original episode filenames are unchanged. L denotes the 2- or 3-layer configuration. Task identity mapping does not alter recorded trajectories or evaluation labels.

Paper configuration Paper task name Historical ID Public folder Episodes Frames GiB
A-2 Cross Transfer task2-2 taskA-2_cross_transfer_2 50 64,995 4.39
A-3 Cross Transfer task2-3 taskA-3_cross_transfer_3 50 141,832 9.61
B-2 Two-Block Transfer task1-2 taskB-2_two_block_transfer_2 50 129,711 8.87
B-3 Two-Block Transfer task1-3 taskB-3_two_block_transfer_3 50 250,051 17.27
C-2 Extract and Stack task6-2 taskC-2_extract_and_stack_2 50 40,016 2.82
C-3 Extract and Stack task6-3 taskC-3_extract_and_stack_3 50 81,546 5.91
D-2 Lateral Insertion task5-2 taskD-2_lateral_insertion_2 50 46,102 3.17
D-3 Lateral Insertion task5-3 taskD-3_lateral_insertion_3 50 92,763 6.61
E-2 Layerwise Pushdown task4-2 taskE-2_layerwise_pushdown_2 50 85,088 6.07
E-3 Layerwise Pushdown task4-3 taskE-3_layerwise_pushdown_3 50 122,459 9.17
F-2 Tool-Assisted Extraction task7-2 taskF-2_tool_assisted_extraction_2 50 218,555 19.34
F-3 Tool-Assisted Extraction task7-3 taskF-3_tool_assisted_extraction_3 50 385,961 36.69
Total 600 1,659,079 129.93

Repository layout

data/<paper-task-folder>/<original-episode-filename>.hdf5
previews/<paper-task-folder>_start_middle_end.jpg
TASK_MAPPING.json
MANIFEST.jsonl
CHECKSUMS.sha256
schema/tower_simdata_hdf5.json
README.md
LICENSE
CITATION.cff

For example, paper_task: A, paper_config: A-2, legacy_task: task2-2, and layers: 2 identify the same configuration. Each manifest row also preserves the original episode identity, source SHA256, published-file SHA256, frame count and byte size. These are metadata aliases; the HDF5 files have not been relabeled internally.

Selection and validation

The 600 episodes were selected deterministically from the 728 source episodes actually used in the November 2026 twelve-task training dataset. The 24 held-out episodes and previously excluded training episodes are not part of this release. D-2 (historical task5-2) includes 25 episodes per active arm; D-3 (historical task5-3) includes 25 episodes per arm order. This release does not define a new public benchmark train/test split.

Checks include source-file SHA256 against the training provenance, HDF5 dataset structure, synchronized camera frame counts, monotonic row timestamps, finite joint/end-effector values, and decoding the start/middle/end frame of each of four cameras (7,200 sampled images). Sampled image checks are not full-video review or an independent semantic success reassessment. Recorded source success labels are retained.

The selection statements above describe the release preparer's provenance. The source-pool manifest, deterministic selection procedure, held-out identity list, and per-episode qualification receipts are not included here yet, so those claims are not independently reproducible from this repository alone. See VALIDATION.md for the checks independently performed and the remaining release-review items.

Format and compatibility

MANIFEST.jsonl lists every published file with its SHA256, original-source SHA256, bytes, frames, and schema variant. CHECKSUMS.sha256 validates published files. schema/tower_simdata_hdf5.json records the actual dataset inventory by task.

The source schema is jenga_nero_hdf5_v9_discrete_lifter_levels. Public folders now use paper A–F identifiers within the data/<task-folder>/*.hdf5 layout. This release is not byte-schema-compatible with the previous v8 schema. In particular, sim/task_phase uses strings, some v8 diagnostics are absent, and raw physical/command fields are retained. Consumers must inspect the new schema rather than require the old 37 fields. No missing diagnostic fields have been synthesized.

Core fields include observations/qpos[T,16], leader/qpos[T,16], observations/eef[T,14], leader/eef[T,14], lifter/action_mm[T,2], and observations/images/*[T]. Arm order is left then back. Raw commands have not been shifted, resampled, normalized, or converted to model-specific actions. Do not apply the previous release's terminal-row action convention without inspecting the current raw recording.

Stored JPEGs are ready to decode without an additional rotation or flip. The four cameras are right_front_camera, left_back_camera, left_wrist_camera, and back_wrist_camera.

Timing, units, and action targets

leader/qpos and lifter/action_mm are raw command/reference fields; observations/qpos is recorded state. A shared row index does not mean the command was already applied when that image was captured. physical/issued_command/* and physical/applied_command/* expose different stages of the command pipeline and must not be substituted for one another or for leader/qpos without a verified conversion. In the independently inspected C-2 episode, these fields are numerically different; its issued/applied ticks differ by 32 at the end, with a 400 Hz physics grid and an extra_command_delay_ms value of 80. This is a sample observation, not a universal shift to apply to every episode.

The sample HDF5 attributes identify joint angles in radians, gripper widths in meters, end-effector positions in meters, quaternion order xyzw, and lifter commands in millimeters. Arm order is left then back. Read each field's attributes and do not normalize mixed units together. Image samples inspected in A-2 and C-2 are 320×240; the inspection example reports the actual dimensions in any selected episode.

The per-task inventory does not list the previous v8 sim/action_valid, sim/terminal, or sim/success arrays. The inspected files instead contain an episode-level task_success attribute. Do not fabricate per-row success/terminal labels or infer that the final command row is invalid.

Training alignment remains a release-review item: the producer must supply the exact observation-to-target pairing, reference-to-issued-command mapping, delay treatment, and final-row handling used by the training pipeline. Until that is verified, the example below is a raw-data inspection loader, not an endorsed imitation-learning target constructor.

Camera previews

Each preview uses the same representative episode reviewed before publication. Columns are start, middle, and end; each column is a 2×2 four-camera view. Image titles include the paper task, historical ID, original episode name, frame indices, and times.

Task A-2 — Cross Transfer

Historical ID: task2-2.

A-2 — Cross Transfer

Task A-3 — Cross Transfer

Historical ID: task2-3.

A-3 — Cross Transfer

Task B-2 — Two-Block Transfer

Historical ID: task1-2.

B-2 — Two-Block Transfer

Task B-3 — Two-Block Transfer

Historical ID: task1-3.

B-3 — Two-Block Transfer

Task C-2 — Extract and Stack

Historical ID: task6-2.

C-2 — Extract and Stack

Task C-3 — Extract and Stack

Historical ID: task6-3.

C-3 — Extract and Stack

Task D-2 — Lateral Insertion

Historical ID: task5-2.

D-2 — Lateral Insertion

Task D-3 — Lateral Insertion

Historical ID: task5-3.

D-3 — Lateral Insertion

Task E-2 — Layerwise Pushdown

Historical ID: task4-2.

E-2 — Layerwise Pushdown

Task E-3 — Layerwise Pushdown

Historical ID: task4-3.

E-3 — Layerwise Pushdown

Task F-2 — Tool-Assisted Extraction

Historical ID: task7-2.

F-2 — Tool-Assisted Extraction

Task F-3 — Tool-Assisted Extraction

Historical ID: task7-3.

F-3 — Tool-Assisted Extraction

Downloading, reading, and verification

Install huggingface_hub, h5py, numpy, and pillow. The pinned revision below contains the reviewed v2 data and inspection example. Version 2.0.0 is also available on main; pin the revision for reproducible downloads.

from huggingface_hub import snapshot_download

root = snapshot_download(
    repo_id="tower-benchmark/TOWER-SimData",
    repo_type="dataset",
    revision="3c38340e3efbf0bc1858731487e99d59301fdcca",
    local_dir="TOWER-SimData-v2",
    allow_patterns=[
        "data/taskC-2_extract_and_stack_2/*.hdf5",
        "MANIFEST.jsonl", "CHECKSUMS.sha256", "TASK_MAPPING.json",
        "schema/*", "examples/*", "VALIDATION.md", "README.md", "LICENSE", "CITATION.cff",
    ],
)

This example downloads only C-2. To download all 600 episodes (about 129.93 GiB of HDF5 payloads), omit allow_patterns. The inspection example is included in this pinned snapshot.

Inspect a local episode with the included example:

python TOWER-SimData-v2/examples/inspect_episode.py TOWER-SimData-v2/data/<folder>/<episode>.hdf5

The example checks core shapes and finite state/command values, reports units, timestamps and episode-level success, and decodes sampled camera frames. It neither shifts actions nor certifies semantic success or posture compliance.

For a complete download, run from its root:

sha256sum --check CHECKSUMS.sha256

For a partial download, validate the entries for the selected files only; absent files from other tasks are expected. Each selected file must match its manifest byte count and published SHA256. source_sha256 describes the pre-redaction source, not the downloadable file.

License and citation

The dataset is distributed under CC BY 4.0. See LICENSE and CITATION.cff.

Downloads last month
2,666

Models trained or fine-tuned on tower-benchmark/TOWER-SimData