Hugo Hadfield, Alperen Degirmenci, Jonathan Tremblay
🌐 Website (coming soon) · 📄 Paper (coming soon)
DOPER trains and evaluates keypoint-based 6-DoF object pose models from synthetic data: it
renders COCO-style training splits from a CAD model with Blender, trains a DINOv3 ConvNeXt
detector with a heatmap keypoint head, recovers poses from the keypoints, and reproduces the paper's
BOP evaluations (HANDAL, HOPE v2, HOT3D, T-LESS). It also prepares SPEED+ data, evaluates a
SPEED+ checkpoint on SHIRT, and adapts a model to a real object from NuRec renders. The doper
command drives the packaged recipes; the rendering and training entry points also work on your
own CAD files.
Python 3.11 or newer and uv are required; rendering needs Blender
4.2 LTS or newer (blender on PATH, or export DOPER_BLENDER=/path/to/blender).
From a checkout of this repository:
uv sync
source .venv/bin/activateOptional features are extras (uv sync --extra bop; several at once with repeated --extra):
| Extra | Adds |
|---|---|
bop |
Hugging Face downloads and CAD I/O for the BOP recipes |
isaac |
no packages: Isaac Sim 6.0.1 is a container image (nvcr.io/nvidia/isaac-sim:6.0.1), not a wheel |
usd |
USD scene handling (usd-core) |
keypoints |
numba-accelerated keypoint selection |
refine |
numba L-BFGS-B heatmap refinement and parallel preparation |
tensorrt |
ONNX export for TensorRT (TensorRT itself comes from the platform) |
realsense |
no packages: pyrealsense2 comes from the platform |
filter |
the Kalman filters of the SHIRT evaluation (bayesfilter) |
hdf5 |
HDF5 episode reading for pseudo-labels (h5py) |
speedplus |
the SPEED+ tracked-landmark extractor |
nurec-sam |
SAM2 masks of the NuRec labels stage |
This release contains code only. Trained DOPER checkpoints are not included; they are planned for a separate release.
Building a model with pretrained: true, the default, downloads the pretrained backbone on
your machine:
- The default
convnext_small.dinov3_lvd1689mand the other DINOv3 ConvNeXt backbones come from Hugging Face throughtimm. They are Meta's DINOv3 weights, under the DINOv3 License, which has its own use restrictions. - The optional ResNet backbones use the default torchvision weights.
DOPER does not redistribute either. Downloading and using them is subject to their own terms, and models you train from them may be too.
NOTICE AND DISCLAIMER: This software automatically retrieves, accesses or interacts with external materials. Those retrieved materials are not distributed with this software and are governed solely by separate terms, conditions and licenses. You are solely responsible for finding, reviewing and complying with all applicable terms, conditions, and licenses, and for verifying the security, integrity and suitability of any retrieved materials for your specific use case. This software is provided "AS IS", without warranty of any kind. The author makes no representations or warranties regarding any retrieved materials, and assumes no liability for any losses, damages, liabilities or legal consequences from your use or inability to use this software or any retrieved materials. Use this software and the retrieved materials at your own risk.
examples/minimal/README.md takes a bundled CAD model through
keypoint selection, rendering a train and a validation split, loading them with
doper.dataset.DoperDataset, a training dry run and a short CPU smoke train. Use it first to
check that Blender and the install work, then follow Rendering synthetic data
for your own CAD file.
The BOP, SPEED+ and SHIRT recipes follow the paper. The NuRec workflow takes a recipe for your own capture instead (see NuRec adaptation).
Every doper verb takes --dry-run, which prints the ordered steps with their inputs and
outputs and touches no data: run it first. --root is the dataset root and --runs the output
directory; both are always explicit.
| Recipe | doper verbs |
Not scripted by doper |
|---|---|---|
handal, hopev2, hot3d, tless |
prepare, train, review, evaluate, report |
none |
speedplus |
prepare (--stage fetch, convert, extract-landmarks, layout) |
training a SPEED+ model, and evaluating it on SPEED+ itself |
shirt |
evaluate (--method framewise, candidates, pnp-ukf, pnp-ukf-rts, keypoint-ukf) |
obtaining the SHIRT data, and training the SPEED+ checkpoint it evaluates |
doper prepare handal --root data/bop --dry-run
doper train handal --stage base --root data/bop --runs runs --obj-id 1 --dry-run
doper evaluate handal --split val --root data/bop --runs runs --dry-run
doper report handal --runs runs --dry-run
doper prepare speedplus --stage fetch --root data/speedplus --accept-license --dry-run
doper evaluate shirt --method framewise --root data/shirt --runs runs \
--checkpoint speedplus.pth --keypoints-3d keypoints_3d.json --dry-runThe BOP recipes are described in BOP base recipes. SPEED+ is licensed
CC BY-NC-SA 4.0, which doper prepare speedplus --stage fetch asks you to accept; the SHIRT evaluation takes a
SHIRT tree you already have and a SPEED+ checkpoint (--checkpoint, --keypoints-3d).
Training a model of your own on a rendered or prepared COCO split is python -m doper.train --config FILE, starting from doper/recipes/train_example.yaml.
Rendering code lives in rendering/ (a checkout, not part of the wheel) and needs Blender 4.2 LTS
or newer: put blender on PATH, set DOPER_BLENDER, or pass --blender-bin. The five steps
below take a CAD file to a trainable COCO split; examples/minimal/README.md runs them end to
end on the public-domain 3DBenchy model, which it downloads.
blender --background --python rendering/mesh_setup.py -- --input CAD --output draft.blend --source-units m --center-geometry --centered-mesh-output centered.stlpython -m doper.keypoint_selection --mesh centered.stl --source-units m --object-frame aabb_centered --out-json keypoints.json --object-name object --keypoint-prefix KP_object_blender --background --python rendering/mesh_setup.py -- --input CAD --output object.blend --source-units m --center-geometry --keypoints-input keypoints.json --usd-output object.usda --keypoints-output keypoints_3d.jsonpython rendering/synth_parallel.py --blend-file object.blend --usd object.usda --config configs/synth_example.yaml --outdir data/train --num-images 200 --num-workers 2- Point
DataConfig.rootatdata/and load thetrainsplit withdoper.dataset.DoperDataset.
Step 4 writes images/, masks/, annotations.json, keypoints_3d.json and a per-image scene
record under --outdir. configs/synth_example.yaml is the starter render recipe.
For pose-labeled data, add --rotation-aux to ordinary training. This trains a
coarse orientation classifier alongside the detector, mask and heatmap keypoint heads:
python -m doper.train --config my_training.yaml --rotation-aux --dry-run
python -m doper.train --config my_training.yaml --rotation-auxThe head pools the crop keypoint FPN features to 4 x 4 and classifies them with a cosine
classifier. Its loss updates the shared backbone and keypoint FPN as well as the auxiliary
head. It requires keypoint_head: heatmap and is disabled by default. Pose estimation at
inference still uses keypoints and PnP; this auxiliary classifier does not replace that solver.
COCO annotations must contain a 4 x 4 object_pose: object-to-OpenCV-camera when
camera_extrinsics is absent, or object-to-world with Blender camera-to-world
camera_extrinsics. The bundled renderer and BOP conversion provide these fields. SPEED+
speedplus_q_vbs2tango_true scalar-first quaternions are also supported. Crop rotation
augmentation rotates the target accordingly. Samples without a pose, including background
negatives and box/mask-only labels, receive zero rotation loss; a run with no pose-labeled
training crops is rejected. The target uses the labeled orientation without symmetry reduction.
The defaults are 256 rotation bins, loss weight 1.0, soft-target width 15 degrees, and cosine
logit scale 30. Set rotation_num_bins, rotation_loss_weight, rotation_sigma_deg,
rotation_pool_size, rotation_logit_scale, and rotation_bin_seed in the training YAML
(or use their dashed CLI equivalents). Bins are selected by farthest-point sampling from a
seeded uniform SO(3) candidate pool, independently of the dataset; their exact order and the
learned head are saved in the model checkpoint.
--init-from can add the head to a compatible checkpoint that lacks it. When resuming, keep
the rotation architecture settings unchanged. Validation adds rotation_top1_deg over
pose-labeled crops; best.pth is still selected by keypoint error.
The doper command runs one packaged recipe per BOP dataset (handal, hopev2, hot3d,
tless; or a recipe YAML path). Datasets come only from the public bop-benchmark Hugging Face
repositories at pinned revisions, and every file is SHA-256 checked; install doper[bop] for
the downloads and CAD I/O. --root (dataset) and --runs (outputs) are always explicit, and
--dry-run prints the ordered steps with their inputs and outputs without touching data.
doper prepare handal --root Rfetches and verifies the models, PBR and validation parts, extracts them, then per object selects 64 keypoints on the CAD, converts PBR to COCO, audits the geometry and projections and splits by whole scenes (--obj-id Nfor some objects). Add--parts test --allow-test-inputsto fetch the unlabeled test images for the export. For HOT3D,--parts val test --allow-test-inputsalso stages the native-camera validation panel and test images.doper train handal --stage base --root R --runs O --obj-id Ntrains one object's base model intoO/base/obj_XXXXXXand refuses an existing run unless--force.--override KEY=VALUEchanges a training config value; dataset paths and the base launch keys (init_from,resume,require_*) are refused.doper evaluate handal --split val --root R --runs Oscores validation AR per object (single pass, box jitter and GT-box arms); a re-run keeps results of the same checkpoint and scores only the missing objects, and--forcerescores.--split test [--policy base|selected]writes the BOP CSV toO/test/<policy>/; an interrupted export resumes, and--forcestarts it over (for example after retraining an object). A bank with an object that has no prepared data or no trained model is refused unless--allow-partial-bank.doper report handal --runs OwritesO/report_handal.{json,md}: the per-object validation AR, the appendix A adaptation tables (every method and AR arm per object with its label count, update count and status; skipped or missing students are N/A, never zero; paired macro means and deltas), the follow-up rounds, the fixed-policy rows (base fallbacks, whether the CSV is timed, local AP) and the Table 3 variant rows (with deployment timing).
HANDAL object 40 and HOT3D object 33 have no official PBR. Their bases train only on a
BlenderProc campaign (python -m rendering.blenderproc.campaign stage|run|status, from a
checkout: 1000 layouts of 25 views). doper prepare handal --root R --obj-id 40 --keypoints-only fetches only the models part, selects the object's keypoints on its official
CAD, checks them against the recipe's pinned SHA-256 and writes
R/objects/obj_000040/geometry/keypoints_3d.json: stage the campaign with that file. Then
doper prepare handal --root R --obj-id 40 --generated-root DIR verifies every layout receipt
(a missing or recorded-failed layout is refused by number), requires the campaign's keypoints
to be that selection, converts the renders (HANDAL keeps annotations with an empty visible
mask, HOT3D excludes them) and holds out the layouts with
layout % 10 == 9 for validation. The object then trains, adapts and joins every bank like the
others, so doper evaluate --split test needs no --allow-partial-bank.
T-LESS also has real TRAIN captures from three sensors. After an object's base preparation and
doper prepare tless --root R --parts test --allow-test-inputs (the reserved PrimeSense test
split), doper prepare tless --root R --multisensor --obj-id N fetches the PrimeSense TRAIN
part and the object's original Kinect and Canon TRAIN archives. These come from the T-LESS
authors' site over plain HTTP, not from Hugging Face. Upstream publishes no checksum: the
profile pins the SHA-256 observed for object 1, any other archive is recorded on first download
and must keep that SHA-256. The command writes R/multisensor/obj_XXXXXX/{primesense,kinect,canon}:
keypoints projected with the released poses, PrimeSense's released masks and CAD-projected
masks for Kinect and Canon. Any image whose bytes equal a reserved test image is refused.
A person must inspect its nine overlays and record the review with
doper review tless --root R --runs O --obj-id N --reviewer NAME before the multisensor
adaptation can use it.
doper train handal --stage adapt --method M --root R --runs O --obj-id Nruns one adaptation method of the recipe (static_pseudo,dynamic_pseudo,combined_pseudo,gt_box_mask,gt_box_mask_keypoints; HANDAL alsotest_tta, which needs--allow-test-inputs) from the object's epoch-25 base intoO/M/obj_XXXXXX. A pool below the method's minimum label count writesadaptation_skipped.jsonand trains no student;--labels-onlystops after the verified label pool.doper train handal --stage followup --name obj8_loop --root R --runs O --obj-id 8runs a self-training follow-up round by round (O/obj8_loop/roundNN/obj_XXXXXX,O/obj8_loop/progress.json). A review-gated follow-up (obj7_bootstrap) needsdoper review handal --root R --runs O --obj-id 7 --reviewer NAMEafter its overlays are inspected, or--accept-unreviewed, which is recorded.doper train tless --stage multisensor --root R --runs O --obj-id Nadapts a T-LESS base on the reviewed PrimeSense, Kinect and Canon roots.doper evaluate RECIPE --split val --arm Mscores an adapted arm, a selection alias (static_iterative_round8) or a follow-up round (obj8_loop/round08);--split test --policy Pexports one of the recipe's fixed policies (withMODEL_MATRIX.csvandSKIPS.json) and--policy selectedthe validation-selected bank (selection.jsonlists every candidate without a validation score asunscored; such a candidate is not ranked).--variant {singleview, singleheatmap, multiview, multiheatmap}exports a test variant of a bank (heatmap refinement:--optimizer {lbfgsb,powell},--preparation {numpy,parallel}; L-BFGS-B needsdoper[refine]; multi-view variants are refused for HANDAL). Run a heatmap variant of a bank first: its heatmap rescan must reproduce the cached detection journal, and on a GPU it usually does not bit for bit, so it replaces the journal (a new forward pass). Exports made before it then hold the old pass; each summary records the journal generation per object,doper reportwarns when one policy's CSV and variants differ, and a resumed variant export recomputes the objects whose journal changed.doper report tless --runs O --root R --apcomputes the local BOP24 AP of each test CSV (public T-LESS test GT;--variant V, repeatable, scores the Table 3 variant CSVs instead);doper report RECIPE --runs O --root R --timing [--protocol direct|deployment]times each policy's bank on a CUDA device;doper report handal --runs O --root R --mask-auditwrites the post-training GT-mask IoU of every follow-up round's pool (diagnostic only;--onboarding-zipnames the pinned static onboarding archive when it is not underR/source).
Prepared data links its images with relative symlinks and records root-relative paths, so a
dataset root can be moved as a whole. The source views under R/views (HANDAL without its
defective frames; HOT3D in canonical millimeters with its depth and mask filters) record the
version of the code that built them (CANONICALIZER_VERSION, DEFECT_FILTER_VERSION and
MASK_FILTER_VERSION in doper.bop.hot3d_source), not its bytes; a release that changes their
output bumps the version. A view whose identity no longer matches is refused with its
directory named: delete that directory and rerun doper prepare --force for the affected
objects to rebuild the view and the prepared data on top of it.
DOPER also includes experimental tooling for adapting a synthetic-trained model to a real object from NuRec renders of a reconstruction of its captures. It is not part of the paper. You copy the template recipe examples/nurec/recipe.yaml for your capture; docs/nurec.md lists the requirements and stages.
uv sync --all-extras --group dev creates one environment with the package, its optional
extras and every tool make format check test coverage needs. TensorRT itself comes from the
platform and is not installed by uv sync. The nurec-sam extra builds SAM2 from source; set
SAM2_BUILD_CUDA=0 to skip its optional CUDA extension, which DOPER does not use.
AGENTS.md tells a coding agent how this repository is laid out and how to run it,
and skills/doper-cad-workflow/ is a skill that walks an agent through taking a CAD file to a
trained model with the commands above.
A BibTeX entry will be added here when the paper is available.
DOPER is released under the Apache License 2.0. Vendored third-party code under
doper/_vendor/ keeps its own license; see THIRD_PARTY_NOTICES.md.
This project is currently not accepting contributions. Bug reports and questions are welcome as
issues; see CONTRIBUTING.md. Security issues: see SECURITY.md.
- THIRD_PARTY_NOTICES.md lists the vendored and ported third-party code with its licenses.
- Pretrained backbone weights are downloaded from their publishers under their own licenses; see Pretrained weights.
