The full dataset viewer is not available (click to read why). Only showing a preview of the rows.
Error code: JobManagerCrashedError
Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
leader dict | lifter dict | observations dict | physical dict | posture_reference dict | sim dict | timestamps dict |
|---|---|---|---|---|---|---|
{"eef":[0.04098762944340706,-0.022157905623316765,0.5557513236999512,0.16337113082408905,0.708559989(...TRUNCATED) | {
"action_mm": [
0,
0
],
"active_target_mm": [
0,
0
],
"command_delta_mm": [
0,
0
]
} | {"eef":[0.04098762199282646,-0.022157905623316765,0.5557513236999512,0.16337114572525024,0.708559989(...TRUNCATED) | {"applied_command":{"dof_velocity_targets":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0(...TRUNCATED) | {"qpos":[-0.021196255460381508,-0.7915393710136414,-0.00801072921603918,1.5211988687515259,-0.009913(...TRUNCATED) | {"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,-0.7071068286(...TRUNCATED) | {"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488000000000,1792454488000000000],"l(...TRUNCATED) |
{"eef":[0.040987528860569,-0.022159816697239876,0.5557512044906616,0.16337165236473083,0.70856148004(...TRUNCATED) | {
"action_mm": [
0,
0
],
"active_target_mm": [
0,
0
],
"command_delta_mm": [
0,
0
]
} | {"eef":[0.04098762199282646,-0.022157905623316765,0.5557513236999512,0.16337114572525024,0.708559989(...TRUNCATED) | {"applied_command":{"dof_velocity_targets":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0(...TRUNCATED) | {"qpos":[-0.02049681358039379,-0.791543185710907,-0.0078984210267663,1.5211988687515259,-0.010442748(...TRUNCATED) | {"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED) | {"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488033333248,1792454488033333248],"l(...TRUNCATED) |
{"eef":[0.04098748043179512,-0.02216172404587269,0.5557510852813721,0.16337208449840546,0.7085630297(...TRUNCATED) | {
"action_mm": [
0,
0
],
"active_target_mm": [
0,
0
],
"command_delta_mm": [
0,
0
]
} | {"eef":[0.04098762199282646,-0.022157905623316765,0.5557513236999512,0.16337114572525024,0.708559989(...TRUNCATED) | {"applied_command":{"dof_velocity_targets":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0(...TRUNCATED) | {"qpos":[-0.019797371700406075,-0.7915470600128174,-0.007786113303154707,1.5211988687515259,-0.01097(...TRUNCATED) | {"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED) | {"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488066666752,1792454488066666752],"l(...TRUNCATED) |
{"eef":[0.040987443178892136,-0.02216363325715065,0.5557510256767273,0.16337239742279053,0.708564579(...TRUNCATED) | {
"action_mm": [
0,
0
],
"active_target_mm": [
0,
0
],
"command_delta_mm": [
0,
0
]
} | {"eef":[0.04098768159747124,-0.022157935425639153,0.5557513236999512,0.16336871683597565,0.708560585(...TRUNCATED) | {"applied_command":{"dof_velocity_targets":[[0.005245814099907875,-0.000029802322387695312,0.0008423(...TRUNCATED) | {"qpos":[-0.019097931683063507,-0.7915509343147278,-0.007673805113881826,1.5211988687515259,-0.01150(...TRUNCATED) | {"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED) | {"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488100000000,1792454488100000000],"l(...TRUNCATED) |
{"eef":[0.04098735377192497,-0.02216527797281742,0.5557509660720825,0.16337287425994873,0.7085658311(...TRUNCATED) | {
"action_mm": [
0,
0
],
"active_target_mm": [
0,
0
],
"command_delta_mm": [
0,
0
]
} | {"eef":[0.040986981242895126,-0.022158905863761902,0.5557512640953064,0.16336098313331604,0.70856457(...TRUNCATED) | {"applied_command":{"dof_velocity_targets":[[0.0209832563996315,-0.00011622905731201172,0.0033692223(...TRUNCATED) | {"qpos":[-0.018492907285690308,-0.7915540933609009,-0.007576672360301018,1.5211988687515259,-0.01195(...TRUNCATED) | {"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED) | {"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488133333248,1792454488133333248],"l(...TRUNCATED) |
{"eef":[0.0409872941672802,-0.022166922688484192,0.555750846862793,0.16337323188781738,0.70856714248(...TRUNCATED) | {
"action_mm": [
0,
0
],
"active_target_mm": [
0,
0
],
"command_delta_mm": [
0,
0
]
} | {"eef":[0.040985602885484695,-0.022160524502396584,0.5557512640953064,0.16335444152355194,0.70856928(...TRUNCATED) | {"applied_command":{"dof_velocity_targets":[[0.0209832563996315,-0.00011771917343139648,0.0033692340(...TRUNCATED) | {"qpos":[-0.017887884750962257,-0.7915573120117188,-0.007479539606720209,1.5211988687515259,-0.01241(...TRUNCATED) | {"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED) | {"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488166666752,1792454488166666752],"l(...TRUNCATED) |
{"eef":[0.04098724201321602,-0.022168567404150963,0.5557507872581482,0.16337351500988007,0.708568513(...TRUNCATED) | {
"action_mm": [
0,
0
],
"active_target_mm": [
0,
0
],
"command_delta_mm": [
0,
0
]
} | {"eef":[0.04098433256149292,-0.022162048146128654,0.5557512044906616,0.16334865987300873,0.708573460(...TRUNCATED) | {"applied_command":{"dof_velocity_targets":[[0.02027507871389389,-0.00011175870895385742,0.003255426(...TRUNCATED) | {"qpos":[-0.017282860353589058,-0.7915604710578918,-0.0073824068531394005,1.5211988687515259,-0.0128(...TRUNCATED) | {"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED) | {"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488200000000,1792454488200000000],"l(...TRUNCATED) |
{"eef":[0.040987201035022736,-0.022169996052980423,0.5557507276535034,0.16337381303310394,0.70856964(...TRUNCATED) | {
"action_mm": [
0,
0
],
"active_target_mm": [
0,
0
],
"command_delta_mm": [
0,
0
]
} | {"eef":[0.04098319634795189,-0.022163445129990578,0.5557512044906616,0.16334328055381775,0.708577096(...TRUNCATED) | {"applied_command":{"dof_velocity_targets":[[0.018150731921195984,-0.00009685754776000977,0.00291398(...TRUNCATED) | {"qpos":[-0.017178155481815338,-0.7915610074996948,-0.007361347321420908,1.5211988687515259,-0.01295(...TRUNCATED) | {"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED) | {"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488233333248,1792454488233333248],"l(...TRUNCATED) |
{"eef":[0.04098718240857124,-0.022171422839164734,0.5557506084442139,0.16337411105632782,0.708570778(...TRUNCATED) | {
"action_mm": [
0,
0
],
"active_target_mm": [
0,
0
],
"command_delta_mm": [
0,
0
]
} | {"eef":[0.04098205268383026,-0.02216489613056183,0.5557512044906616,0.16333752870559692,0.7085807323(...TRUNCATED) | {"applied_command":{"dof_velocity_targets":[[0.018150685355067253,-0.00009685754776000977,0.00291398(...TRUNCATED) | {"qpos":[-0.01707345061004162,-0.7915615439414978,-0.0073402877897024155,1.5211988687515259,-0.01303(...TRUNCATED) | {"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED) | {"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488266666752,1792454488266666752],"l(...TRUNCATED) |
{"eef":[0.040987107902765274,-0.022172847762703896,0.5557505488395691,0.16337446868419647,0.70857191(...TRUNCATED) | {
"action_mm": [
0,
0
],
"active_target_mm": [
0,
0
],
"command_delta_mm": [
0,
0
]
} | {"eef":[0.040981024503707886,-0.022166265174746513,0.5557512044906616,0.16333284974098206,0.70858377(...TRUNCATED) | {"applied_command":{"dof_velocity_targets":[[0.01439834013581276,-0.00007599592208862305,0.002343440(...TRUNCATED) | {"qpos":[-0.0169687457382679,-0.791562020778656,-0.00731922872364521,1.5211988687515259,-0.013119695(...TRUNCATED) | {"block_poses":[[0.37455591559410095,-0.27809953689575195,0.014750001020729542,0.0,0.0,0.70710682868(...TRUNCATED) | {"lifter_command_ros_ns":[0,0],"lifter_source_after_ns":[1792454488300000000,1792454488300000000],"l(...TRUNCATED) |
- Task identities and dataset size
- Repository layout
- Selection and validation
- Format and compatibility
- Camera previews
- Task A-2 — Cross Transfer
- Task A-3 — Cross Transfer
- Task B-2 — Two-Block Transfer
- Task B-3 — Two-Block Transfer
- Task C-2 — Extract and Stack
- Task C-3 — Extract and Stack
- Task D-2 — Lateral Insertion
- Task D-3 — Lateral Insertion
- Task E-2 — Layerwise Pushdown
- Task E-3 — Layerwise Pushdown
- Task F-2 — Tool-Assisted Extraction
- Task F-3 — Tool-Assisted Extraction
- Task A-2 — Cross Transfer
- Downloading, reading, and verification
- License and citation
Tower-SimData
Project page · Code · Simulator assets
Version 2.0.0 replaces the earlier release with 600 selected simulation demonstrations: 12 paper configurations (A–F × 2/3 layers), with 50 episodes each. The old Task3 episodes are not included. Earlier versions remain available in repository history.
This is a 600-episode simulation-only subset, not the complete 12,000-demonstration SIM/REAL collection described in the paper. Real-world demonstrations are not included in this repository.
Each episode is an independent raw HDF5 file at 30 FPS, with robot states and commands, end-effector poses, four JPEG camera streams, and simulation state. Camera images and all dataset values are preserved from the selected source files. Internal filesystem paths in metadata are redacted; source and published file hashes are recorded separately.
Task identities and dataset size
Public task names follow the paper's A–F definitions. Historical numeric task IDs are retained in MANIFEST.jsonl, TASK_MAPPING.json, and internal HDF5 metadata. Original episode filenames are unchanged. L denotes the 2- or 3-layer configuration. Task identity mapping does not alter recorded trajectories or evaluation labels.
| Paper configuration | Paper task name | Historical ID | Public folder | Episodes | Frames | GiB |
|---|---|---|---|---|---|---|
| A-2 | Cross Transfer | task2-2 |
taskA-2_cross_transfer_2 |
50 | 64,995 | 4.39 |
| A-3 | Cross Transfer | task2-3 |
taskA-3_cross_transfer_3 |
50 | 141,832 | 9.61 |
| B-2 | Two-Block Transfer | task1-2 |
taskB-2_two_block_transfer_2 |
50 | 129,711 | 8.87 |
| B-3 | Two-Block Transfer | task1-3 |
taskB-3_two_block_transfer_3 |
50 | 250,051 | 17.27 |
| C-2 | Extract and Stack | task6-2 |
taskC-2_extract_and_stack_2 |
50 | 40,016 | 2.82 |
| C-3 | Extract and Stack | task6-3 |
taskC-3_extract_and_stack_3 |
50 | 81,546 | 5.91 |
| D-2 | Lateral Insertion | task5-2 |
taskD-2_lateral_insertion_2 |
50 | 46,102 | 3.17 |
| D-3 | Lateral Insertion | task5-3 |
taskD-3_lateral_insertion_3 |
50 | 92,763 | 6.61 |
| E-2 | Layerwise Pushdown | task4-2 |
taskE-2_layerwise_pushdown_2 |
50 | 85,088 | 6.07 |
| E-3 | Layerwise Pushdown | task4-3 |
taskE-3_layerwise_pushdown_3 |
50 | 122,459 | 9.17 |
| F-2 | Tool-Assisted Extraction | task7-2 |
taskF-2_tool_assisted_extraction_2 |
50 | 218,555 | 19.34 |
| F-3 | Tool-Assisted Extraction | task7-3 |
taskF-3_tool_assisted_extraction_3 |
50 | 385,961 | 36.69 |
| Total | 600 | 1,659,079 | 129.93 |
Repository layout
data/<paper-task-folder>/<original-episode-filename>.hdf5
previews/<paper-task-folder>_start_middle_end.jpg
TASK_MAPPING.json
MANIFEST.jsonl
CHECKSUMS.sha256
schema/tower_simdata_hdf5.json
README.md
LICENSE
CITATION.cff
For example, paper_task: A, paper_config: A-2, legacy_task: task2-2, and layers: 2 identify the same configuration. Each manifest row also preserves the original episode identity, source SHA256, published-file SHA256, frame count and byte size. These are metadata aliases; the HDF5 files have not been relabeled internally.
Selection and validation
The 600 episodes were selected deterministically from the 728 source episodes actually used in the November 2026 twelve-task training dataset. The 24 held-out episodes and previously excluded training episodes are not part of this release. D-2 (historical task5-2) includes 25 episodes per active arm; D-3 (historical task5-3) includes 25 episodes per arm order. This release does not define a new public benchmark train/test split.
Checks include source-file SHA256 against the training provenance, HDF5 dataset structure, synchronized camera frame counts, monotonic row timestamps, finite joint/end-effector values, and decoding the start/middle/end frame of each of four cameras (7,200 sampled images). Sampled image checks are not full-video review or an independent semantic success reassessment. Recorded source success labels are retained.
The selection statements above describe the release preparer's provenance. The source-pool manifest, deterministic selection procedure, held-out identity list, and per-episode qualification receipts are not included here yet, so those claims are not independently reproducible from this repository alone. See VALIDATION.md for the checks independently performed and the remaining release-review items.
Format and compatibility
MANIFEST.jsonl lists every published file with its SHA256, original-source SHA256, bytes, frames, and schema variant. CHECKSUMS.sha256 validates published files. schema/tower_simdata_hdf5.json records the actual dataset inventory by task.
The source schema is jenga_nero_hdf5_v9_discrete_lifter_levels. Public folders now use paper A–F identifiers within the data/<task-folder>/*.hdf5 layout. This release is not byte-schema-compatible with the previous v8 schema. In particular, sim/task_phase uses strings, some v8 diagnostics are absent, and raw physical/command fields are retained. Consumers must inspect the new schema rather than require the old 37 fields. No missing diagnostic fields have been synthesized.
Core fields include observations/qpos[T,16], leader/qpos[T,16], observations/eef[T,14], leader/eef[T,14], lifter/action_mm[T,2], and observations/images/*[T]. Arm order is left then back. Raw commands have not been shifted, resampled, normalized, or converted to model-specific actions. Do not apply the previous release's terminal-row action convention without inspecting the current raw recording.
Stored JPEGs are ready to decode without an additional rotation or flip. The four cameras are right_front_camera, left_back_camera, left_wrist_camera, and back_wrist_camera.
Timing, units, and action targets
leader/qpos and lifter/action_mm are raw command/reference fields; observations/qpos is recorded state. A shared row index does not mean the command was already applied when that image was captured. physical/issued_command/* and physical/applied_command/* expose different stages of the command pipeline and must not be substituted for one another or for leader/qpos without a verified conversion. In the independently inspected C-2 episode, these fields are numerically different; its issued/applied ticks differ by 32 at the end, with a 400 Hz physics grid and an extra_command_delay_ms value of 80. This is a sample observation, not a universal shift to apply to every episode.
The sample HDF5 attributes identify joint angles in radians, gripper widths in meters, end-effector positions in meters, quaternion order xyzw, and lifter commands in millimeters. Arm order is left then back. Read each field's attributes and do not normalize mixed units together. Image samples inspected in A-2 and C-2 are 320×240; the inspection example reports the actual dimensions in any selected episode.
The per-task inventory does not list the previous v8 sim/action_valid, sim/terminal, or sim/success arrays. The inspected files instead contain an episode-level task_success attribute. Do not fabricate per-row success/terminal labels or infer that the final command row is invalid.
Training alignment remains a release-review item: the producer must supply the exact observation-to-target pairing, reference-to-issued-command mapping, delay treatment, and final-row handling used by the training pipeline. Until that is verified, the example below is a raw-data inspection loader, not an endorsed imitation-learning target constructor.
Camera previews
Each preview uses the same representative episode reviewed before publication. Columns are start, middle, and end; each column is a 2×2 four-camera view. Image titles include the paper task, historical ID, original episode name, frame indices, and times.
Task A-2 — Cross Transfer
Historical ID: task2-2.
Task A-3 — Cross Transfer
Historical ID: task2-3.
Task B-2 — Two-Block Transfer
Historical ID: task1-2.
Task B-3 — Two-Block Transfer
Historical ID: task1-3.
Task C-2 — Extract and Stack
Historical ID: task6-2.
Task C-3 — Extract and Stack
Historical ID: task6-3.
Task D-2 — Lateral Insertion
Historical ID: task5-2.
Task D-3 — Lateral Insertion
Historical ID: task5-3.
Task E-2 — Layerwise Pushdown
Historical ID: task4-2.
Task E-3 — Layerwise Pushdown
Historical ID: task4-3.
Task F-2 — Tool-Assisted Extraction
Historical ID: task7-2.
Task F-3 — Tool-Assisted Extraction
Historical ID: task7-3.
Downloading, reading, and verification
Install huggingface_hub, h5py, numpy, and pillow. The pinned revision below contains the reviewed v2 data and inspection example. Version 2.0.0 is also available on main; pin the revision for reproducible downloads.
from huggingface_hub import snapshot_download
root = snapshot_download(
repo_id="tower-benchmark/TOWER-SimData",
repo_type="dataset",
revision="3c38340e3efbf0bc1858731487e99d59301fdcca",
local_dir="TOWER-SimData-v2",
allow_patterns=[
"data/taskC-2_extract_and_stack_2/*.hdf5",
"MANIFEST.jsonl", "CHECKSUMS.sha256", "TASK_MAPPING.json",
"schema/*", "examples/*", "VALIDATION.md", "README.md", "LICENSE", "CITATION.cff",
],
)
This example downloads only C-2. To download all 600 episodes (about 129.93 GiB of HDF5 payloads), omit allow_patterns. The inspection example is included in this pinned snapshot.
Inspect a local episode with the included example:
python TOWER-SimData-v2/examples/inspect_episode.py TOWER-SimData-v2/data/<folder>/<episode>.hdf5
The example checks core shapes and finite state/command values, reports units, timestamps and episode-level success, and decodes sampled camera frames. It neither shifts actions nor certifies semantic success or posture compliance.
For a complete download, run from its root:
sha256sum --check CHECKSUMS.sha256
For a partial download, validate the entries for the selected files only; absent files from other tasks are expected. Each selected file must match its manifest byte count and published SHA256. source_sha256 describes the pre-redaction source, not the downloadable file.
License and citation
The dataset is distributed under CC BY 4.0. See LICENSE and CITATION.cff.
- Downloads last month
- 2,666











