Vision-Language-Action manipulation policies on the Franka Panda, evaluated across Isaac Lab and MuJoCo, paired with a complete classical 3D-vision stack implemented from scratch.
This repository is a portfolio of two parallel tracks:
- VLA benchmarking — fine-tune / evaluate OpenVLA-OFT, LeRobot Diffusion Policy, and Octo on the LIBERO benchmark; measure where language conditioning + foundation pretraining actually buy you something.
- Classical 3D vision from scratch — hand-eye calibration, camera-intrinsic calibration, PnP, ICP, with the math derived inline in pure NumPy. Foundational tools every CV engineer is expected to understand at the implementation level, not just the API level.
11 commits on main • 28 / 28 pytest cases pass • 4 benchmark plots embedded below.
- Headline results
- Architecture at a glance
- Hardware / software prerequisites
- Quickstart (30 seconds to a result)
- Step-by-step setup (from a fresh machine)
- How to run — workflows
- Theory deep-dive
- Results — detail
- Project layout
- Roadmap
- Troubleshooting + design notes
- References
Three policy families evaluated on the same 10 tasks:
| Method | Setup | Success @ episodes | Reproduces |
|---|---|---|---|
| OpenVLA-OFT (pretrained, 7B) | Stanford LIBERO-Spatial checkpoint | 100.0% @ 20 | beats paper 97.7% |
| LeRobot Diffusion Policy (this work) | trained from scratch 100K steps on same 500 demos | 70.0% @ 20 | non-VLA baseline |
| Octo-base-1.5 (zero-shot) | OXE pretraining only, no LIBERO finetune | 0.0% @ 10 | shows finetune is needed |
| Suite | This work | Paper (500 ep × 3 seeds) | Δ |
|---|---|---|---|
| LIBERO-Spatial | 100.0% | 97.7% | +2.3 |
| LIBERO-Object | 100.0% | 98.4% | +1.6 |
| LIBERO-Goal | 100.0% | 97.9% | +2.1 |
| LIBERO-10 (Long) | 95.0% | 94.5% | +0.5 |
| Overall | 98.75% | 97.1% | +1.65 pp |
| Module | What it solves | Best mean error @ realistic noise | Code | Test |
|---|---|---|---|---|
| Hand-eye | AX=XB for camera-to-gripper transform | 0.13° / 1.17 mm (1° + 1 mm sensor noise, N=12) | hand_eye.py | 5 cases |
| Camera intrinsic | Zhang 2000 — K from N planar views | < 0.5% focal length error | camera_calibration.py | 5 cases |
| PnP | 6D pose from 2D-3D point pairs | 0.07° / 0.62 mm @ 1 px noise, N=30 | pnp.py | 5 cases |
| ICP | rigid registration of two point clouds | 0.012° / 0.03 mm @ 1 mm cloud noise | icp.py | 6 cases |
The repo is split into isolated conda environments because the heavy frameworks (Isaac Sim, OpenVLA-OFT, LeRobot, Octo) have mutually-incompatible Python pins. The shared robot_sim_vla Python package is installed editable in each env where it's needed.
flowchart TB
subgraph EnvOFT["openvla-oft env (PyTorch 2.2)"]
OVLA[OpenVLA-OFT 7B<br/>LIBERO eval]
MJC[MuJoCo + robosuite]
LIB[LIBERO benchmark]
end
subgraph EnvIL["isaaclab env (PyTorch 2.5)"]
ISAAC[Isaac Sim 4.5 +<br/>Isaac Lab 2.0.2]
OVLA2[Franka tasks]
end
subgraph EnvLR["lerobot env (PyTorch 2.7)"]
DP[LeRobot DP training]
DPE[DP LIBERO eval<br/>+ LIBERO env]
CONV[HDF5 → LeRobotDataset<br/>converter]
end
subgraph EnvOcto["octo env (JAX 0.4.20)"]
OCTO[Octo eval]
end
subgraph PerceptionPkg["src/robot_sim_vla/perception/ — pure NumPy"]
HE[hand_eye.py]
CC[camera_calibration.py]
PNP[pnp.py]
ICP[icp.py]
end
subgraph Datasets["external data (~/lerobot_data/, ~/.cache/huggingface/)"]
HDF5[LIBERO HDF5 demos]
RLDS[modified_libero_rlds]
LRDS[LeRobotDataset format]
CKPT[OpenVLA / DP / Octo checkpoints]
end
LIB --> OVLA
LIB --> DPE
LIB --> OCTO
HDF5 --> CONV
CONV --> LRDS
LRDS --> DP
DP --> DPE
RLDS -.fine-tune.-> OVLA
CKPT --> OVLA
CKPT --> DPE
CKPT --> OCTO
PerceptionPkg -.imported by tests.-> EnvOFT
make help lists every workflow as a Makefile target.
| Component | Requirement | Tested on |
|---|---|---|
| GPU | NVIDIA, ≥ 16 GB VRAM (fine-tune wants ≥ 24 GB) | 4× RTX A6000 (48 GB) |
| Driver | ≥ 535 (CUDA 12.4 runtime) | 570.133.20, CUDA 12.8 |
| OS | Ubuntu 22.04 (other 22.04-equivalent OK) | 22.04.5 LTS, GLIBC 2.35 |
| Disk | ≥ 150 GB free | ~50 GB caches + ~32 GB DP checkpoints |
| Python | 3.10 per env | 3.10.18 |
| conda | ≥ 24.x | miniconda3 25.5.1 |
If the four envs already exist on the machine (this repo's reference setup):
make check # verify all envs are healthy
make smoke # Franka smoke tests in MuJoCo + Isaac Lab
make eval-libero-spatial # OpenVLA-OFT sanity eval (~5 min)
make test # run all 28 pytest casesmake help shows every available target with a one-line description.
flowchart LR
A[install miniconda] --> B[clone OpenVLA-OFT + LIBERO]
B --> C1[create openvla-oft env<br/>PyTorch 2.2 + cu121]
B --> C2[create isaaclab env<br/>PyTorch 2.5 + cu124]
B --> C3[create lerobot env<br/>PyTorch 2.7 + cu126]
B --> C4[create octo env<br/>JAX 0.4.20 + cu12]
C1 --> D1[pip install OpenVLA-OFT<br/>+ LIBERO + robosuite]
C2 --> D2[pip install isaacsim==4.5<br/>+ Isaac Lab v2.0.2]
C3 --> D3[pip install LeRobot<br/>+ LIBERO + h5py]
C4 --> D4[pip install -e remote_dir/octo<br/>+ requirements.txt]
D1 --> E[pip install -e robot_sim_vla<br/>into each env]
D2 --> E
D3 --> E
E --> F[make check + make smoke + make test]
Total wall time on a 50 MB/s connection: ~45 minutes (most of it downloading Isaac Sim 15 GB + LIBERO checkpoints 14 GB).
Full per-step commands are in CLAUDE.md, section "Where to do what".
flowchart LR
A[make eval-libero-spatial] -->|spawns| B[run_libero_eval.py<br/>in openvla-oft env]
B -->|first run downloads| C[14 GB OpenVLA-OFT<br/>checkpoint from HF]
C --> D[for each task: rollout<br/>10 tasks × NUM_TRIALS]
D --> E[smoke_outputs/libero_<br/>spatial_pretrained/<br/>EVAL-*.txt + .mp4s]
make eval-libero-spatial # 2 trials/task by default → 20 episodes
make eval-libero-spatial NUM_TRIALS=50 # paper-style 500 episodes
make eval-libero-all # all 4 suitesflowchart LR
A[LIBERO HDF5 demos<br/>~7 GB] -->|make convert-libero-to-lerobot<br/>~10 min| B[LeRobotDataset<br/>500 ep / 62 K frames / 176 MB]
B -->|make train-dp-libero<br/>100 K steps / ~5 h on 1× A6000| C[runs/lerobot_dp_libero_spatial<br/>11 checkpoints / ~30 GB]
C -->|make eval-dp-libero<br/>~12 min| D[smoke_outputs/libero_<br/>spatial_dp/summary.json]
make convert-libero-to-lerobot SUITE=libero_spatial
make train-dp-libero SUITE=libero_spatial DP_STEPS=100000
make eval-dp-libero SUITE=libero_spatial NUM_TRIALS=2flowchart LR
A[pytest tests/] --> B[28 cases run<br/>across 3 envs]
B --> C[hand-eye + camera calib<br/>+ PnP + ICP<br/>all green]
D[scripts/plot_benchmarks.py] --> E[generate 4 PNG figures<br/>in docs/figures/]
E --> F[README plots refreshed]
make test # pytest tests/ -v
conda run -n openvla-oft python scripts/plot_benchmarks.py
conda run -n openvla-oft python scripts/benchmark_pnp.py
conda run -n openvla-oft python scripts/benchmark_icp.pyflowchart LR
A[make eval-octo-spatial<br/>or direct script call] --> B[load Octo-base-1.5<br/>from HF]
B --> C[for each LIBERO task: rollout<br/>history=2 frame buffer<br/>action chunk open-loop]
C --> D[summary.json + sample_*.mp4]
conda run -n octo python scripts/eval_octo_libero_spatial.py \
--num-trials-per-task 1 --out smoke_outputs/libero_spatial_octoThe hypothesis on LIBERO: language conditioning + foundation-model pretraining beat a from-scratch visuomotor policy on cross-object generalisation, because LIBERO-Spatial puts the same robot in front of visually similar bowls and varies only the language instruction.
LIBERO-Spatial task templates (10 tasks):
"pick up the black bowl <DISAMBIGUATOR> and place it on the plate"
where DISAMBIGUATOR ∈ {between the plate and the ramekin,
next to the ramekin,
from table center,
..., on the wooden cabinet}
A non-language policy (DP) has to learn the mapping from cluttered RGB scene → which bowl to pick implicitly from the demos. A language-conditioned VLA reads the instruction and is told which bowl. Our 30 pp gap (100% vs 70%) is exactly that.
The Octo zero-shot 0% on the same suite tells us pretraining alone (without LIBERO data) doesn't transfer — the Franka + OSC controller distribution in LIBERO is too far from anything in OXE.
Problem. Given N rigid motions of the robot end-effector A_i and the corresponding rigid motions of the camera B_i (computed from calibration-board detections between robot poses), recover the camera-to-gripper transform X satisfying
Both classical solvers decouple rotation from translation.
Tsai-Lenz (1989). Conjugate rotations share the same angle. With the modified Rodrigues vector
Stacking N constraints gives an orthogonal Procrustes problem on
Park-Martin (1994). Identical idea on the SO(3) log-vector:
Then project to SO(3) by SVD.
Translation (both solvers):
stacked across i and solved by least squares.
Problem. Given N images of a planar checkerboard at unknown orientations, recover the intrinsic matrix
Step 1 — per-view homography. Each view yields a 2D→2D map between the target's plane coords (Z=0) and the image. Use normalised DLT (translate centroid, scale so mean distance is √2) and SVD.
Step 2 — K from many H. Define
(orthogonality + equal-length of the first two columns of 2 N rows, take the right-singular vector of the smallest singular value, reshape to a symmetric matrix B, and apply Zhang's closed-form to extract (α, β, γ, u_0, v_0).
Step 3 — per-view (R, t). With K known:
then project the rotation columns to SO(3) by SVD.
Problem. Given N ≥ 6 correspondences (X_i ∈ R^3, u_i ∈ R^2) and intrinsic K, recover (R, t) such that
where π is the perspective divide.
DLT — linear. Express observations in normalised image coordinates (multiply by K^{-1}); each correspondence gives two rows of a 12-unknown linear system. SVD's right-null vector gives the projection matrix P up to scale; split P = [M | t] (M 3×3 rotation part, t translation column), then Procrustes M onto SO(3) and rescale t by 1/mean(singular_values(M)).
The SVD null-vector is sign-ambiguous; combined with the det-fix in the SO(3) projection there are four sign / determinant combinations. Pick the one that lands all points in front of the camera and has the lowest reprojection error.
LM refinement. Linearise the rigid increment as (δω, δt) ∈ R^6. Per-iteration Jacobian on the reprojection residual:
Solve
Problem. Given a source cloud S and a target cloud T (possibly different sizes, partial overlap, with noise), find the rigid transform (R, t) minimising
Iterate until convergence:
-
Match — for each transformed source point, look up the nearest target point with a kd-tree (
scipy.spatial.cKDTree). -
Solve — closed-form Kabsch-Umeyama on the matched pairs: subtract centroids, SVD on the cross-covariance
$H = \sum_i (s_i - \bar{s})(t_i - \bar{t})^\top$ ,$R = V D U^\top$ with$D = \mathrm{diag}(1, 1, \det(VU^\top))$ (Procrustes with reflection fix). -
Apply the increment
(R_inc, t_inc)to the running (R, t) and recompute matches.
Point-to-plane variant (Chen-Medioni) changes step 2: instead of minimising point-pair distance, minimise its projection onto the target's surface normal at the matched point:
Linearised on (δω, δt) ∈ R^6. Converges in ~3× fewer iterations on near-flat geometry — our cube faces in the benchmark cloud.
Outlier rejection via max_correspondence_dist is critical when source coverage < 100%.
| Item | Result |
|---|---|
openvla-oft env (reused) |
✅ verified — torch 2.2 + 4 GPUs + MuJoCo + LIBERO + OpenVLA-OFT |
isaaclab env (new) |
✅ Isaac Sim 4.5 + Isaac Lab 2.0.2 + 4× A6000 detected |
lerobot env (reused) |
✅ torch 2.7 + LeRobot 0.3.4 + LIBERO + h5py |
octo env (new) |
✅ JAX 0.4.20 + Octo from source |
Franka in MuJoCo (Lift task) |
✅ load + step + RGB render |
Franka in Isaac Lab (Isaac-Lift-Cube-Franka-v0) |
✅ load + step (obs (N, 36), act (N, 8)) |
make test (28 pytest smoke + math tests) |
✅ all passing |
| Item | Result |
|---|---|
| 4-suite LIBERO eval (pretrained OpenVLA-OFT) | ✅ 79/80 = 98.75% (paper avg 97.1%) |
| LIBERO HDF5 → LeRobotDataset converter | ✅ 500 episodes / 62K frames / 9.6 min / 176 MB on disk |
| LeRobot Diffusion Policy trained 100K steps on libero_spatial | ✅ final L1 loss 0.015, 5h 17m, 11 checkpoints |
| DP eval pipeline (LeRobot DP ↔ LIBERO env adapter) | ✅ scripts/eval_dp_libero_spatial.py |
| Head-to-head OpenVLA-OFT vs DP on libero_spatial | ✅ 100% vs 70% (Δ +30 pp) |
| Item | Result |
|---|---|
octo env setup (pinned JAX + flax + tensorflow) |
✅ |
Octo-base-1.5 loaded via OctoModel.load_pretrained |
✅ 4 GPUs detected |
scripts/eval_octo_libero_spatial.py — history=2 buffer, action chunk |
✅ |
| Octo zero-shot success on libero_spatial | 0/10 = 0.0% (confirms LIBERO finetune is required for fair comparison) |
| Part | Item | Result |
|---|---|---|
| p1 | Tsai-Lenz + Park-Martin hand-eye | 5 tests, 0.13° rot @ 1° per-motion noise N=12 |
| p2a | Zhang 2000 camera intrinsic calibration | 5 tests, < 0.5% focal error noise-free |
| p2b | PnP DLT + LM refinement | 5 tests, 0.07° rot / 0.62 mm @ 1 px noise N=30 |
| p3 | ICP point-to-point + point-to-plane | 6 tests, 0.012° rot / 0.03 mm @ 1 mm cloud noise |
| p4 | Benchmark visualisations | 4 PNGs under docs/figures/ |
robot_sim_vla/
├── Makefile # 25+ workflow targets — single source of truth
├── README.md # this file
├── CLAUDE.md # project memory for Claude Code
├── pyproject.toml # editable `robot_sim_vla` package
├── .claude/
│ ├── settings.json # committed permission allowlist (team-wide)
│ └── settings.local.json # personal overrides (gitignored)
├── docs/
│ └── figures/ # benchmark plots, embedded in README
├── envs/
│ ├── openvla-oft.pip.txt # pip freeze per env, for reproducibility
│ ├── isaaclab.pip.txt
│ └── lerobot.pip.txt
├── scripts/
│ ├── smoke_franka_mujoco.py # Franka load + render in robosuite
│ ├── smoke_franka_isaaclab.py # Franka load + step in Isaac Lab
│ ├── eval_libero_spatial_pretrained.sh # OpenVLA-OFT eval wrapper
│ ├── eval_dp_libero_spatial.py # LeRobot DP ↔ LIBERO bridge
│ ├── eval_octo_libero_spatial.py # Octo JAX ↔ LIBERO bridge
│ ├── benchmark_hand_eye.py # parameter sweep for hand-eye
│ ├── benchmark_pnp.py # parameter sweep for PnP
│ ├── benchmark_icp.py # parameter sweep for ICP
│ ├── plot_benchmarks.py # regenerates docs/figures/*.png
│ └── help_*.txt # multi-line help bodies (envsubst-rendered)
├── src/robot_sim_vla/
│ ├── perception/ # FROM-SCRATCH classical CV (pure NumPy)
│ │ ├── hand_eye.py # Tsai-Lenz + Park-Martin (AX = XB)
│ │ ├── camera_calibration.py # Zhang 2000 linear method
│ │ ├── pnp.py # DLT + Levenberg-Marquardt refinement
│ │ └── icp.py # point-to-point + point-to-plane
│ ├── datasets/
│ │ └── libero_to_lerobot.py # LIBERO HDF5 → LeRobotDataset converter
│ ├── policies/
│ │ └── baselines.py # scaffolding for DP / Octo / SmolVLA
│ └── tasks/, utils/ # placeholders for future hybrid policy
├── tests/ # 28 pytest cases
│ ├── test_envs.py # cross-env health (7 cases)
│ ├── test_hand_eye.py # 5 cases
│ ├── test_camera_calibration.py # 5 cases
│ ├── test_pnp.py # 5 cases
│ └── test_icp.py # 6 cases
└── smoke_outputs/ # eval logs + sample_*.mp4 rollouts (committed)
- Phase 1 — env setup, Franka smoke in MuJoCo + Isaac Lab
- Phase 2 — 4-suite LIBERO OpenVLA-OFT eval • LeRobot DP training + eval • head-to-head
- Phase 3 — Octo zero-shot eval pipeline (pipeline ready; LIBERO-finetune deferred)
- Phase 4 — classical 3D vision: hand-eye + camera calibration + PnP + ICP + plots
- Octo libero finetune — needs a custom
libero_spatial_dataset_transformfor Octo'soxe_standardization_transforms.py, then ~4-6 h training on 1× A6000. - Hybrid VLA + classical IBVS — VLA emits target object + approach direction, classical PBVS (using
pnp.py) closes the loop on the final approach. Robustness study under lighting / distractor perturbations. - Phase 5 — sim-to-real on xArm — hardware day: deploy via
xarm_ros2/, hand-eye-calibrate the real camera usinghand_eye.py, measure sim-to-real gap.
| Symptom | Fix |
|---|---|
EOFError: EOF when reading a line during import isaacsim |
export OMNI_KIT_ACCEPT_EULA=YES; the isaaclab env's activate script does this automatically. |
Isaac Lab smoke runs but no [smoke] prints in stdout |
Isaac Sim captures stdout; check smoke_outputs/franka_isaaclab_smoke.log. |
cannot import name 'load_vla' from 'prismatic.vla' |
Harmless — OpenVLA-OFT moved the API. The eval script in experiments/robot/libero/ uses the right path. |
EGLError: EGL_NOT_INITIALIZED at process exit |
Cleanup race in robosuite headless EGL renderer. Cosmetic; eval results are already saved by then. |
Cuda failure ... 'peer access is already enabled' |
Cosmetic warning from Isaac Sim's GPU peer-to-peer probe. Run continues. |
| LIBERO eval first run hangs at "downloading" | Checkpoint is 14 GB. Subsequent runs use the local cache. |
Octo ModuleNotFoundError: 'flax' |
pip install octo installs an UNRELATED package — install from source via pip install -e remote_dir/octo + pip install -r requirements.txt. |
Octo AttributeError: 'scipy.linalg' has no attribute 'tril' |
Pin scipy < 1.13 in the octo env. |
make train-dp-libero errors "output dir exists" |
Run make clean-dp-output first. |
- Three envs is the path of least pain. Isaac Sim 4.5 pins PyTorch 2.5 + CUDA 12.4 and numpy < 2; OpenVLA-OFT pins PyTorch 2.2 + a custom
transformers==4.40.1fork; LeRobot tracks the latest. Sharing one env causes resolver conflicts intokenizers/huggingface_hub. The split keeps each install independent. - Octo is a fourth env. Pure-JAX stack with very specific pins (jax==0.4.20, scipy<1.13, numpy==1.24.3). PyPI
octois an unrelated plugin framework — Octo must be installed fromrail-berkeley/octosource. - Isaac Sim swallows stdout. All Isaac Lab scripts log progress to a file in
smoke_outputs/in addition to printing. - Why pin Isaac Lab to v2.0.2? It matches Isaac Sim 4.5.0 exactly. v2.1.x requires a newer Isaac Sim not tested here.
- State schema in
libero_to_lerobot.py: LIBERO HDF5robot_statesis[gripper_qpos(2), eef_pos(3), eef_quat(4)] = 9D, not joint positions. Same schema feeds the trained DP and DP eval. torch.loadmonkey-patch ineval_dp_libero_spatial.py: PyTorch ≥ 2.6 madeweights_only=Truethe default, breaking LIBERO's init-state loader (it stores numpy pickles). The monkey-patch is safe because LIBERO data is trusted.
- OpenVLA-OFT — Kim et al., Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success, 2025. Code · Paper
- OpenVLA — Kim et al., OpenVLA: An Open-Source Vision-Language-Action Model, 2024. Code · Paper
- LIBERO — Liu et al., LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning, NeurIPS D&B 2023. Code
- Diffusion Policy — Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023.
- Octo — Octo Model Team, Octo: An Open-Source Generalist Robot Policy, 2024. Code
- Isaac Lab — NVIDIA, 2025. Docs
- robosuite — Zhu et al. Code
- LeRobot — HuggingFace. Code
- Tsai-Lenz — A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibration, IEEE TRA 1989.
- Park-Martin — Robot Sensor Calibration: Solving AX=XB on the Euclidean Group, IEEE TRA 1994.
- Zhang 2000 — Z. Zhang, A Flexible New Technique for Camera Calibration, IEEE TPAMI 2000.
- Besl-McKay — A Method for Registration of 3-D Shapes, IEEE TPAMI 1992.
- Chen-Medioni — Object modelling by registration of multiple range images, IVC 1992.
MIT (see LICENSE).



