Skip to content

About

98.75% on LIBERO · OpenVLA-OFT, Franka, Isaac Lab

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

robot_sim_vla

Vision-Language-Action manipulation policies on the Franka Panda, evaluated across Isaac Lab and MuJoCo, paired with a complete classical 3D-vision stack implemented from scratch.

This repository is a portfolio of two parallel tracks:

  1. VLA benchmarking — fine-tune / evaluate OpenVLA-OFT, LeRobot Diffusion Policy, and Octo on the LIBERO benchmark; measure where language conditioning + foundation pretraining actually buy you something.
  2. Classical 3D vision from scratch — hand-eye calibration, camera-intrinsic calibration, PnP, ICP, with the math derived inline in pure NumPy. Foundational tools every CV engineer is expected to understand at the implementation level, not just the API level.

11 commits on main • 28 / 28 pytest cases pass • 4 benchmark plots embedded below.


Table of contents

  1. Headline results
  2. Architecture at a glance
  3. Hardware / software prerequisites
  4. Quickstart (30 seconds to a result)
  5. Step-by-step setup (from a fresh machine)
  6. How to run — workflows
  7. Theory deep-dive
  8. Results — detail
  9. Project layout
  10. Roadmap
  11. Troubleshooting + design notes
  12. References

Headline results

LIBERO-Spatial: VLA vs non-VLA ladder

Three policy families evaluated on the same 10 tasks:

LIBERO methods bar chart

Method Setup Success @ episodes Reproduces
OpenVLA-OFT (pretrained, 7B) Stanford LIBERO-Spatial checkpoint 100.0% @ 20 beats paper 97.7%
LeRobot Diffusion Policy (this work) trained from scratch 100K steps on same 500 demos 70.0% @ 20 non-VLA baseline
Octo-base-1.5 (zero-shot) OXE pretraining only, no LIBERO finetune 0.0% @ 10 shows finetune is needed

Cross-suite reproduction of OpenVLA-OFT (this work, 20 ep / suite)

Suite This work Paper (500 ep × 3 seeds) Δ
LIBERO-Spatial 100.0% 97.7% +2.3
LIBERO-Object 100.0% 98.4% +1.6
LIBERO-Goal 100.0% 97.9% +2.1
LIBERO-10 (Long) 95.0% 94.5% +0.5
Overall 98.75% 97.1% +1.65 pp

Classical 3D-vision benchmarks (28 / 28 tests pass)

Module What it solves Best mean error @ realistic noise Code Test
Hand-eye AX=XB for camera-to-gripper transform 0.13° / 1.17 mm (1° + 1 mm sensor noise, N=12) hand_eye.py 5 cases
Camera intrinsic Zhang 2000 — K from N planar views < 0.5% focal length error camera_calibration.py 5 cases
PnP 6D pose from 2D-3D point pairs 0.07° / 0.62 mm @ 1 px noise, N=30 pnp.py 5 cases
ICP rigid registration of two point clouds 0.012° / 0.03 mm @ 1 mm cloud noise icp.py 6 cases

Architecture at a glance

The repo is split into isolated conda environments because the heavy frameworks (Isaac Sim, OpenVLA-OFT, LeRobot, Octo) have mutually-incompatible Python pins. The shared robot_sim_vla Python package is installed editable in each env where it's needed.

flowchart TB
    subgraph EnvOFT["openvla-oft env (PyTorch 2.2)"]
        OVLA[OpenVLA-OFT 7B<br/>LIBERO eval]
        MJC[MuJoCo + robosuite]
        LIB[LIBERO benchmark]
    end
    subgraph EnvIL["isaaclab env (PyTorch 2.5)"]
        ISAAC[Isaac Sim 4.5 +<br/>Isaac Lab 2.0.2]
        OVLA2[Franka tasks]
    end
    subgraph EnvLR["lerobot env (PyTorch 2.7)"]
        DP[LeRobot DP training]
        DPE[DP LIBERO eval<br/>+ LIBERO env]
        CONV[HDF5 → LeRobotDataset<br/>converter]
    end
    subgraph EnvOcto["octo env (JAX 0.4.20)"]
        OCTO[Octo eval]
    end

    subgraph PerceptionPkg["src/robot_sim_vla/perception/ — pure NumPy"]
        HE[hand_eye.py]
        CC[camera_calibration.py]
        PNP[pnp.py]
        ICP[icp.py]
    end

    subgraph Datasets["external data (~/lerobot_data/, ~/.cache/huggingface/)"]
        HDF5[LIBERO HDF5 demos]
        RLDS[modified_libero_rlds]
        LRDS[LeRobotDataset format]
        CKPT[OpenVLA / DP / Octo checkpoints]
    end

    LIB --> OVLA
    LIB --> DPE
    LIB --> OCTO
    HDF5 --> CONV
    CONV --> LRDS
    LRDS --> DP
    DP --> DPE
    RLDS -.fine-tune.-> OVLA
    CKPT --> OVLA
    CKPT --> DPE
    CKPT --> OCTO

    PerceptionPkg -.imported by tests.-> EnvOFT
Loading

make help lists every workflow as a Makefile target.


Hardware / software prerequisites

Component Requirement Tested on
GPU NVIDIA, ≥ 16 GB VRAM (fine-tune wants ≥ 24 GB) 4× RTX A6000 (48 GB)
Driver ≥ 535 (CUDA 12.4 runtime) 570.133.20, CUDA 12.8
OS Ubuntu 22.04 (other 22.04-equivalent OK) 22.04.5 LTS, GLIBC 2.35
Disk ≥ 150 GB free ~50 GB caches + ~32 GB DP checkpoints
Python 3.10 per env 3.10.18
conda ≥ 24.x miniconda3 25.5.1

Quickstart — 30 seconds to a result

If the four envs already exist on the machine (this repo's reference setup):

make check                 # verify all envs are healthy
make smoke                 # Franka smoke tests in MuJoCo + Isaac Lab
make eval-libero-spatial   # OpenVLA-OFT sanity eval (~5 min)
make test                  # run all 28 pytest cases

make help shows every available target with a one-line description.


Step-by-step setup (from a fresh machine)

flowchart LR
    A[install miniconda] --> B[clone OpenVLA-OFT + LIBERO]
    B --> C1[create openvla-oft env<br/>PyTorch 2.2 + cu121]
    B --> C2[create isaaclab env<br/>PyTorch 2.5 + cu124]
    B --> C3[create lerobot env<br/>PyTorch 2.7 + cu126]
    B --> C4[create octo env<br/>JAX 0.4.20 + cu12]
    C1 --> D1[pip install OpenVLA-OFT<br/>+ LIBERO + robosuite]
    C2 --> D2[pip install isaacsim==4.5<br/>+ Isaac Lab v2.0.2]
    C3 --> D3[pip install LeRobot<br/>+ LIBERO + h5py]
    C4 --> D4[pip install -e remote_dir/octo<br/>+ requirements.txt]
    D1 --> E[pip install -e robot_sim_vla<br/>into each env]
    D2 --> E
    D3 --> E
    E --> F[make check + make smoke + make test]
Loading

Total wall time on a 50 MB/s connection: ~45 minutes (most of it downloading Isaac Sim 15 GB + LIBERO checkpoints 14 GB).

Full per-step commands are in CLAUDE.md, section "Where to do what".


How to run — workflows

Workflow 1: VLA sanity eval (~5 minutes after model is cached)

flowchart LR
    A[make eval-libero-spatial] -->|spawns| B[run_libero_eval.py<br/>in openvla-oft env]
    B -->|first run downloads| C[14 GB OpenVLA-OFT<br/>checkpoint from HF]
    C --> D[for each task: rollout<br/>10 tasks × NUM_TRIALS]
    D --> E[smoke_outputs/libero_<br/>spatial_pretrained/<br/>EVAL-*.txt + .mp4s]
Loading
make eval-libero-spatial                       # 2 trials/task by default → 20 episodes
make eval-libero-spatial NUM_TRIALS=50         # paper-style 500 episodes
make eval-libero-all                           # all 4 suites

Workflow 2: Diffusion Policy from scratch (~6 hours end-to-end)

flowchart LR
    A[LIBERO HDF5 demos<br/>~7 GB] -->|make convert-libero-to-lerobot<br/>~10 min| B[LeRobotDataset<br/>500 ep / 62 K frames / 176 MB]
    B -->|make train-dp-libero<br/>100 K steps / ~5 h on 1× A6000| C[runs/lerobot_dp_libero_spatial<br/>11 checkpoints / ~30 GB]
    C -->|make eval-dp-libero<br/>~12 min| D[smoke_outputs/libero_<br/>spatial_dp/summary.json]
Loading
make convert-libero-to-lerobot SUITE=libero_spatial
make train-dp-libero SUITE=libero_spatial DP_STEPS=100000
make eval-dp-libero  SUITE=libero_spatial NUM_TRIALS=2

Workflow 3: Classical 3D-vision tests + plots (< 30 seconds)

flowchart LR
    A[pytest tests/] --> B[28 cases run<br/>across 3 envs]
    B --> C[hand-eye + camera calib<br/>+ PnP + ICP<br/>all green]
    D[scripts/plot_benchmarks.py] --> E[generate 4 PNG figures<br/>in docs/figures/]
    E --> F[README plots refreshed]
Loading
make test                                          # pytest tests/ -v
conda run -n openvla-oft python scripts/plot_benchmarks.py
conda run -n openvla-oft python scripts/benchmark_pnp.py
conda run -n openvla-oft python scripts/benchmark_icp.py

Workflow 4: Octo zero-shot eval (~10 min including model download)

flowchart LR
    A[make eval-octo-spatial<br/>or direct script call] --> B[load Octo-base-1.5<br/>from HF]
    B --> C[for each LIBERO task: rollout<br/>history=2 frame buffer<br/>action chunk open-loop]
    C --> D[summary.json + sample_*.mp4]
Loading
conda run -n octo python scripts/eval_octo_libero_spatial.py \
    --num-trials-per-task 1 --out smoke_outputs/libero_spatial_octo

Theory deep-dive

1. Why VLA vs non-VLA matters

The hypothesis on LIBERO: language conditioning + foundation-model pretraining beat a from-scratch visuomotor policy on cross-object generalisation, because LIBERO-Spatial puts the same robot in front of visually similar bowls and varies only the language instruction.

LIBERO-Spatial task templates (10 tasks):
    "pick up the black bowl <DISAMBIGUATOR> and place it on the plate"
    where DISAMBIGUATOR ∈ {between the plate and the ramekin,
                            next to the ramekin,
                            from table center,
                            ..., on the wooden cabinet}

A non-language policy (DP) has to learn the mapping from cluttered RGB scene → which bowl to pick implicitly from the demos. A language-conditioned VLA reads the instruction and is told which bowl. Our 30 pp gap (100% vs 70%) is exactly that.

The Octo zero-shot 0% on the same suite tells us pretraining alone (without LIBERO data) doesn't transfer — the Franka + OSC controller distribution in LIBERO is too far from anything in OXE.

2. Hand-eye calibration — solving AX = XB

Problem. Given N rigid motions of the robot end-effector A_i and the corresponding rigid motions of the camera B_i (computed from calibration-board detections between robot poses), recover the camera-to-gripper transform X satisfying

$$A_i \cdot X = X \cdot B_i \quad \forall i$$

Both classical solvers decouple rotation from translation.

Tsai-Lenz (1989). Conjugate rotations share the same angle. With the modified Rodrigues vector $P = 2\sin(\theta/2),\hat{k}$ for each rotation:

$$P_a^{(i)} = R_X \cdot P_b^{(i)}$$

Stacking N constraints gives an orthogonal Procrustes problem on $\sum_i P_a^{(i)} P_b^{(i)\top}$; SVD + det-fix gives $R_X$.

Park-Martin (1994). Identical idea on the SO(3) log-vector:

$$R_X = M_{yx} \cdot M_{xx}^{-1/2}, \quad M_{xx} = \sum_i \log(R_B^{(i)}) \log(R_B^{(i)})^\top, \quad M_{yx} = \sum_i \log(R_A^{(i)}) \log(R_B^{(i)})^\top$$

Then project to SO(3) by SVD.

Translation (both solvers):

$$(R_A^{(i)} - I) \cdot t_X = R_X \cdot t_B^{(i)} - t_A^{(i)}$$

stacked across i and solved by least squares.

hand-eye error vs noise

3. Camera intrinsic calibration — Zhang 2000

Problem. Given N images of a planar checkerboard at unknown orientations, recover the intrinsic matrix

$$K = \begin{pmatrix} \alpha & \gamma & u_0 \ 0 & \beta & v_0 \ 0 & 0 & 1 \end{pmatrix}$$

Step 1 — per-view homography. Each view yields a 2D→2D map between the target's plane coords (Z=0) and the image. Use normalised DLT (translate centroid, scale so mean distance is √2) and SVD.

Step 2 — K from many H. Define $\omega = (KK^\top)^{-1}$ (the image of the absolute conic — a symmetric 3×3, 6 parameters). Each H gives two constraints:

$$h_1^\top \omega h_2 = 0, \quad h_1^\top \omega h_1 - h_2^\top \omega h_2 = 0$$

(orthogonality + equal-length of the first two columns of $K^{-1} H$, which must be unit vectors). Stack 2 N rows, take the right-singular vector of the smallest singular value, reshape to a symmetric matrix B, and apply Zhang's closed-form to extract (α, β, γ, u_0, v_0).

Step 3 — per-view (R, t). With K known:

$$\lambda = 1 / |K^{-1} h_1|, \quad r_1 = \lambda K^{-1} h_1, \quad r_2 = \lambda K^{-1} h_2, \quad r_3 = r_1 \times r_2, \quad t = \lambda K^{-1} h_3$$

then project the rotation columns to SO(3) by SVD.

4. PnP — 6D pose from 2D-3D correspondences

Problem. Given N ≥ 6 correspondences (X_i ∈ R^3, u_i ∈ R^2) and intrinsic K, recover (R, t) such that

$$u_i = \pi\big(K \cdot (R X_i + t)\big)$$

where π is the perspective divide.

DLT — linear. Express observations in normalised image coordinates (multiply by K^{-1}); each correspondence gives two rows of a 12-unknown linear system. SVD's right-null vector gives the projection matrix P up to scale; split P = [M | t] (M 3×3 rotation part, t translation column), then Procrustes M onto SO(3) and rescale t by 1/mean(singular_values(M)).

The SVD null-vector is sign-ambiguous; combined with the det-fix in the SO(3) projection there are four sign / determinant combinations. Pick the one that lands all points in front of the camera and has the lowest reprojection error.

LM refinement. Linearise the rigid increment as (δω, δt) ∈ R^6. Per-iteration Jacobian on the reprojection residual:

$$J_i = \begin{bmatrix} -\frac{\partial \pi}{\partial p_c}\big|_{p_c=p_c^{(i)}} \cdot [p_c^{(i)}]_\times &amp; \frac{\partial \pi}{\partial p_c}\big|_{p_c=p_c^{(i)}} \end{bmatrix}$$

Solve $(J^\top J + \lambda I) \delta = -J^\top r$ each step, accept if cost decreases (LM-style damping). On 1 px noise with N=30 points, refinement reduces rotation error 3× compared to plain DLT.

PnP error vs noise

5. ICP — rigid registration of point clouds

Problem. Given a source cloud S and a target cloud T (possibly different sizes, partial overlap, with noise), find the rigid transform (R, t) minimising

$$\sum_i |R \cdot s_i + t - \text{nearest}_T(R s_i + t)|^2$$

Iterate until convergence:

  1. Match — for each transformed source point, look up the nearest target point with a kd-tree (scipy.spatial.cKDTree).
  2. Solve — closed-form Kabsch-Umeyama on the matched pairs: subtract centroids, SVD on the cross-covariance $H = \sum_i (s_i - \bar{s})(t_i - \bar{t})^\top$, $R = V D U^\top$ with $D = \mathrm{diag}(1, 1, \det(VU^\top))$ (Procrustes with reflection fix).
  3. Apply the increment (R_inc, t_inc) to the running (R, t) and recompute matches.

Point-to-plane variant (Chen-Medioni) changes step 2: instead of minimising point-pair distance, minimise its projection onto the target's surface normal at the matched point:

$$\sum_i \big((R s_i + t - t_i) \cdot n_i\big)^2$$

Linearised on (δω, δt) ∈ R^6. Converges in ~3× fewer iterations on near-flat geometry — our cube faces in the benchmark cloud.

Outlier rejection via max_correspondence_dist is critical when source coverage < 100%.

ICP convergence + accuracy


Results — detail

Phase 1 — environment + smoke

Item Result
openvla-oft env (reused) ✅ verified — torch 2.2 + 4 GPUs + MuJoCo + LIBERO + OpenVLA-OFT
isaaclab env (new) ✅ Isaac Sim 4.5 + Isaac Lab 2.0.2 + 4× A6000 detected
lerobot env (reused) ✅ torch 2.7 + LeRobot 0.3.4 + LIBERO + h5py
octo env (new) ✅ JAX 0.4.20 + Octo from source
Franka in MuJoCo (Lift task) ✅ load + step + RGB render
Franka in Isaac Lab (Isaac-Lift-Cube-Franka-v0) ✅ load + step (obs (N, 36), act (N, 8))
make test (28 pytest smoke + math tests) ✅ all passing

Phase 2 — comparative benchmarking

Item Result
4-suite LIBERO eval (pretrained OpenVLA-OFT) ✅ 79/80 = 98.75% (paper avg 97.1%)
LIBERO HDF5 → LeRobotDataset converter ✅ 500 episodes / 62K frames / 9.6 min / 176 MB on disk
LeRobot Diffusion Policy trained 100K steps on libero_spatial ✅ final L1 loss 0.015, 5h 17m, 11 checkpoints
DP eval pipeline (LeRobot DP ↔ LIBERO env adapter) ✅ scripts/eval_dp_libero_spatial.py
Head-to-head OpenVLA-OFT vs DP on libero_spatial ✅ 100% vs 70% (Δ +30 pp)

Phase 3 — Octo baseline

Item Result
octo env setup (pinned JAX + flax + tensorflow) ✅
Octo-base-1.5 loaded via OctoModel.load_pretrained ✅ 4 GPUs detected
scripts/eval_octo_libero_spatial.py — history=2 buffer, action chunk ✅
Octo zero-shot success on libero_spatial 0/10 = 0.0% (confirms LIBERO finetune is required for fair comparison)

Phase 4 — classical 3D vision from scratch (✅ complete)

Part Item Result
p1 Tsai-Lenz + Park-Martin hand-eye 5 tests, 0.13° rot @ 1° per-motion noise N=12
p2a Zhang 2000 camera intrinsic calibration 5 tests, < 0.5% focal error noise-free
p2b PnP DLT + LM refinement 5 tests, 0.07° rot / 0.62 mm @ 1 px noise N=30
p3 ICP point-to-point + point-to-plane 6 tests, 0.012° rot / 0.03 mm @ 1 mm cloud noise
p4 Benchmark visualisations 4 PNGs under docs/figures/

Project layout

robot_sim_vla/
├── Makefile                         # 25+ workflow targets — single source of truth
├── README.md                        # this file
├── CLAUDE.md                        # project memory for Claude Code
├── pyproject.toml                   # editable `robot_sim_vla` package
├── .claude/
│   ├── settings.json                # committed permission allowlist (team-wide)
│   └── settings.local.json          # personal overrides (gitignored)
├── docs/
│   └── figures/                     # benchmark plots, embedded in README
├── envs/
│   ├── openvla-oft.pip.txt          # pip freeze per env, for reproducibility
│   ├── isaaclab.pip.txt
│   └── lerobot.pip.txt
├── scripts/
│   ├── smoke_franka_mujoco.py       # Franka load + render in robosuite
│   ├── smoke_franka_isaaclab.py     # Franka load + step in Isaac Lab
│   ├── eval_libero_spatial_pretrained.sh  # OpenVLA-OFT eval wrapper
│   ├── eval_dp_libero_spatial.py    # LeRobot DP ↔ LIBERO bridge
│   ├── eval_octo_libero_spatial.py  # Octo JAX ↔ LIBERO bridge
│   ├── benchmark_hand_eye.py        # parameter sweep for hand-eye
│   ├── benchmark_pnp.py             # parameter sweep for PnP
│   ├── benchmark_icp.py             # parameter sweep for ICP
│   ├── plot_benchmarks.py           # regenerates docs/figures/*.png
│   └── help_*.txt                   # multi-line help bodies (envsubst-rendered)
├── src/robot_sim_vla/
│   ├── perception/                  # FROM-SCRATCH classical CV (pure NumPy)
│   │   ├── hand_eye.py              # Tsai-Lenz + Park-Martin (AX = XB)
│   │   ├── camera_calibration.py    # Zhang 2000 linear method
│   │   ├── pnp.py                   # DLT + Levenberg-Marquardt refinement
│   │   └── icp.py                   # point-to-point + point-to-plane
│   ├── datasets/
│   │   └── libero_to_lerobot.py     # LIBERO HDF5 → LeRobotDataset converter
│   ├── policies/
│   │   └── baselines.py             # scaffolding for DP / Octo / SmolVLA
│   └── tasks/, utils/               # placeholders for future hybrid policy
├── tests/                           # 28 pytest cases
│   ├── test_envs.py                 # cross-env health (7 cases)
│   ├── test_hand_eye.py             # 5 cases
│   ├── test_camera_calibration.py   # 5 cases
│   ├── test_pnp.py                  # 5 cases
│   └── test_icp.py                  # 6 cases
└── smoke_outputs/                   # eval logs + sample_*.mp4 rollouts (committed)

Roadmap

Done (✅)

  • Phase 1 — env setup, Franka smoke in MuJoCo + Isaac Lab
  • Phase 2 — 4-suite LIBERO OpenVLA-OFT eval • LeRobot DP training + eval • head-to-head
  • Phase 3 — Octo zero-shot eval pipeline (pipeline ready; LIBERO-finetune deferred)
  • Phase 4 — classical 3D vision: hand-eye + camera calibration + PnP + ICP + plots

Deferred (✋ — pipeline ready, work day required)

  • Octo libero finetune — needs a custom libero_spatial_dataset_transform for Octo's oxe_standardization_transforms.py, then ~4-6 h training on 1× A6000.
  • Hybrid VLA + classical IBVS — VLA emits target object + approach direction, classical PBVS (using pnp.py) closes the loop on the final approach. Robustness study under lighting / distractor perturbations.
  • Phase 5 — sim-to-real on xArm — hardware day: deploy via xarm_ros2/, hand-eye-calibrate the real camera using hand_eye.py, measure sim-to-real gap.

Troubleshooting + design notes

Common pitfalls (each one cost me time once)

Symptom Fix
EOFError: EOF when reading a line during import isaacsim export OMNI_KIT_ACCEPT_EULA=YES; the isaaclab env's activate script does this automatically.
Isaac Lab smoke runs but no [smoke] prints in stdout Isaac Sim captures stdout; check smoke_outputs/franka_isaaclab_smoke.log.
cannot import name 'load_vla' from 'prismatic.vla' Harmless — OpenVLA-OFT moved the API. The eval script in experiments/robot/libero/ uses the right path.
EGLError: EGL_NOT_INITIALIZED at process exit Cleanup race in robosuite headless EGL renderer. Cosmetic; eval results are already saved by then.
Cuda failure ... 'peer access is already enabled' Cosmetic warning from Isaac Sim's GPU peer-to-peer probe. Run continues.
LIBERO eval first run hangs at "downloading" Checkpoint is 14 GB. Subsequent runs use the local cache.
Octo ModuleNotFoundError: 'flax' pip install octo installs an UNRELATED package — install from source via pip install -e remote_dir/octo + pip install -r requirements.txt.
Octo AttributeError: 'scipy.linalg' has no attribute 'tril' Pin scipy < 1.13 in the octo env.
make train-dp-libero errors "output dir exists" Run make clean-dp-output first.

Design notes

  • Three envs is the path of least pain. Isaac Sim 4.5 pins PyTorch 2.5 + CUDA 12.4 and numpy < 2; OpenVLA-OFT pins PyTorch 2.2 + a custom transformers==4.40.1 fork; LeRobot tracks the latest. Sharing one env causes resolver conflicts in tokenizers/huggingface_hub. The split keeps each install independent.
  • Octo is a fourth env. Pure-JAX stack with very specific pins (jax==0.4.20, scipy<1.13, numpy==1.24.3). PyPI octo is an unrelated plugin framework — Octo must be installed from rail-berkeley/octo source.
  • Isaac Sim swallows stdout. All Isaac Lab scripts log progress to a file in smoke_outputs/ in addition to printing.
  • Why pin Isaac Lab to v2.0.2? It matches Isaac Sim 4.5.0 exactly. v2.1.x requires a newer Isaac Sim not tested here.
  • State schema in libero_to_lerobot.py: LIBERO HDF5 robot_states is [gripper_qpos(2), eef_pos(3), eef_quat(4)] = 9D, not joint positions. Same schema feeds the trained DP and DP eval.
  • torch.load monkey-patch in eval_dp_libero_spatial.py: PyTorch ≥ 2.6 made weights_only=True the default, breaking LIBERO's init-state loader (it stores numpy pickles). The monkey-patch is safe because LIBERO data is trusted.

References

  • OpenVLA-OFT — Kim et al., Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success, 2025. Code · Paper
  • OpenVLA — Kim et al., OpenVLA: An Open-Source Vision-Language-Action Model, 2024. Code · Paper
  • LIBERO — Liu et al., LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning, NeurIPS D&B 2023. Code
  • Diffusion Policy — Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023.
  • Octo — Octo Model Team, Octo: An Open-Source Generalist Robot Policy, 2024. Code
  • Isaac Lab — NVIDIA, 2025. Docs
  • robosuite — Zhu et al. Code
  • LeRobot — HuggingFace. Code
  • Tsai-Lenz — A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibration, IEEE TRA 1989.
  • Park-Martin — Robot Sensor Calibration: Solving AX=XB on the Euclidean Group, IEEE TRA 1994.
  • Zhang 2000 — Z. Zhang, A Flexible New Technique for Camera Calibration, IEEE TPAMI 2000.
  • Besl-McKay — A Method for Registration of 3-D Shapes, IEEE TPAMI 1992.
  • Chen-Medioni — Object modelling by registration of multiple range images, IVC 1992.

License

MIT (see LICENSE).

About

98.75% on LIBERO · OpenVLA-OFT, Franka, Isaac Lab

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages