Zhaoning Wang1,* Β Β·Β Yi Zhong1,* Β Β·Β Jiawei Fu2 Β Β·Β Henrik I. Christensen2 Β Β·Β Jun Gao1,3
1University of Michigan Β Β 2UC San Diego Β Β 3NVIDIA Β Β Β *Equal contribution
TL;DR
RGB + text query + depth βββΊ AFUN model βββΊ mask + 3D motion curve
From a single RGB observation and a natural-language task description, AFUN predicts a task-conditional functional mask (where to interact) and a 3D post-contact motion curve (how to interact). It pairs frozen pretrained vision-language features with lightweight trainable modules for joint mask + motion prediction.
This repository provides inference code (load a trained AFUN checkpoint and run it on your own images) and the training code for all three stages. Depth is handled automatically β Depth-Anything-3 generates metric depth from a single RGB image, or lingbot-depth refines a noisy sensor (e.g. RealSense) depth map if you provide one.
Contents
- Inference code & demo
- Training code (Stages 1β3)
- Dataset, Evaluation, and Benchmark β see π¦ Dataset & Benchmark and π Evaluation
- Robot Demonstration Code
- Claude Agent md Files (
CLAUDE.md)
git clone --recursive https://github.com/EricWang12/AFUN.git
cd AFUN
# (already cloned non-recursively? run: git submodule update --init --recursive)conda create -n afun python=3.11 -y
conda activate afun
bash install.shNOTE:
install.shinstalls into the active Python 3.11 environment and targets CUDA 13 (cu130). For a different CUDA build, edit the file directly.It downloads ~10 GB of wheels through
$TMPDIRand the finished env is ~9 GB, so pointTMPDIRat a volume with room if/tmpis small (OSError: [Errno 28] No space left on device). On a node without outbound network, setHF_HUB_OFFLINE=1β model construction contacts huggingface.co to revalidate even when every weight is already cached.
SAM3 is gated on Hugging Face: request access at facebook/sam3
and log in once with hf auth login.
The AFUN checkpoint afun.pt is hosted on Google Drive. Fetch it automatically β
this also pre-caches the Hugging Face backbone weights (Qwen3-VL-8B, SAM3, Sonata, and with
--include-depth Depth-Anything-3 and lingbot):
python scripts/prefetch_weights.py --include-depthOr download it by hand from the Google Drive link and place it at:
checkpoints/afun.pt
Backbone weights that are not cached download from Hugging Face on first use.
Run the demo with an RGB image, a text prompt, and (optionally) a sensor-depth folder:
python demo.py --rgb <path_to_rgb_image> --query <text_prompt> --depth-dir <path_to_depth_map>--depth-dir would be a folder containing two files:
<depth-dir>/
βββ depth.npy # raw sensor depth β float32, millimeters, shape (H, W) matching the RGB
βββ cam_K.txt # 3Γ3 camera-intrinsics matrix
By default lingbot refines this depth; pass --no-refine to skip refinement and use the raw sensor depth as-is. Omit --depth-dir
entirely and Depth-Anything-3 estimates depth from the RGB instead.
NOTE: Raw sensor depth without refinement generally does not work well with our model, as it is not trained on such data.
Example 1 β sensor depth (lingbot refines the bundled depth via --depth-dir):
python demo.py --rgb examples/robot_toaster_oven/rgb.png \
--query "Open the toaster oven door" \
--depth-dir examples/robot_toaster_ovenExample 2 β Depth-Anything-3 (no --depth-dir; DA3 estimates depth from the RGB):
python demo.py --rgb examples/kitchen_fridge/rgb.png --query "Open the refrigerator door"Output (per run, under outputs/):
| File | Contents |
|---|---|
pred_seg.png |
RGB + predicted affordance mask (red) + motion trajectory (yellowβblue, yellow = contact) |
pred.npz |
mask, motion parameters, intrinsics, confidence |
pred_3d.html |
interactive 3D point cloud + motion curve (omit with --no-3d) |
meta.json / summary.csv / montage.png |
run metadata (single / batch) |
Run all bundled examples (model loads once, then loops over examples/):
python run_examples.py # add --no-3d to skip the interactive 3D HTMLAll data is released on Hugging Face under the AFUN-dataset organization (CC BY-NC-SA 4.0). Each data point is one folder: an RGB frame, its metric depth map, the ground-truth affordance mask, the ground-truth 3D motion curve, and a language instruction.
| Dataset | What it is | Size |
|---|---|---|
| AFUN | Training set: 44,749 data points from SceneFun3D, RoboMIND, DROID, AgiBot World, RH20T, CALVIN, RLBench and VITRA human videos | 231 GiB |
| AFUN_eval | The three evaluation sets of Table 3: afun_test (121), scenefun3d_test (721), robomind2_test (156). The ADE / FDE / CIM protocol is described in its README |
9.9 GiB |
| AFUN_pool | The full 183,657-point pool that AFUN was curated from. Its 106,225 Ego4D-derived points ship as annotations only, with a script that rebuilds the frames from your own Ego4D download | 363 GiB |
pip install -U huggingface_hub
hf download AFUN-dataset/AFUN_eval --repo-type dataset --local-dir afun_eval # 9.9 GiB β start here
hf download AFUN-dataset/AFUN --repo-type dataset --local-dir afun_train # 231 GiBEach dataset's README documents the folder layout, the trajectory.json schema, and a loading snippet.
AFUN ships as .tar.zst archives; for training, scripts/download_datasets.py afun-train downloads
and extracts them into the layout the code expects (see Training).
python -m src.eval runs the paper's benchmarks and writes one JSONL row per sample plus summary.json
to outputs/eval/<date>_<time>/ (or --output). --limit N evaluates the first N samples of each set.
python scripts/download_datasets.py afun-eval --root $AFUN_DATA_ROOT # afun_test (121), scenefun3d_test (721), robomind2_test (156)
python -m src.eval --task motion --checkpoint checkpoints/afun.pt # all three; or --dataset afun_testSets are read from <--data-root>/afun_eval/<set>/ (default $AFUN_DATA_ROOT). The predicted curve starts at
the median 3D point of the predicted mask (back-projected with the depth map); prediction and ground truth
are linearly resampled to 50 points.
| Metric | Definition |
|---|---|
| ADE / FDE (m) | mean / final-point L2 distance; abs in the camera frame, rel with both curves shifted to start at the origin |
| CIM (%) | the curve start, projected into the image, lies inside the GT mask; failures count as misses |
#fail (num_failures) |
samples without a prediction; left out of ADE / FDE |
The released afun.pt is the Table 3 checkpoint; on afun_test it reproduces Table 3 to within 2 mm,
with the same CIM. For self-trained Stage-3 checkpoints add --config inference_stage3.
--dataset |
Data under $AFUN_DATA_ROOT |
Samples | Query |
|---|---|---|---|
handal-mini, 3doi |
RAGNet handal_mini_val.pkl, 3doi_val.pkl |
1003, 1013 | "Please segment the affordance map of {class} in this image" |
handal-easy, handal-hard, 3doi-easy |
RAGNet *_reasoning_val.pkl |
1003, 1003, 1013 | question + answer |
hova500k |
HOVA-500K/mask_annotation/dataset_index.json, test split |
419 | task instruction |
reasonaff |
ReasonAFF test_new |
600 | problem |
instruct-part |
instruct_part/InstructPart/test |
600 | first instruction |
python -m src.eval --task segmentation --checkpoint checkpoints/afun.pt # all eight setsThe prediction is the top-scoring SAM3 detection above confidence 0.5, compared with the GT mask at the
image's resolution. gIoU averages per-sample IoU; cIoU divides summed intersections by summed unions.
As in the evaluation behind the paper's numbers, samples without a detection above the threshold are
left out of both and counted in num_failures. Table 1 was computed with an earlier checkpoint; the
released afun.pt is close on average but differs per set, by up to 1.5 points gIoU and 7 points cIoU
(HOVA-500K). HOVA-500K masks regenerated with prepare_hova.py also differ from ours, so
hova500k numbers will differ from Table 1.
AFUN is trained in three stages (Sec. 4.2 of the paper). Each stage starts from the weights of the previous one; Qwen3-VL, SAM3, and Sonata stay frozen throughout.
| Stage | Config | What it learns | Data | Batch | Steps |
|---|---|---|---|---|---|
| 1 | train_stage1 |
MetaQuery β SAM3 alignment: SAM3 driven by the projected MetaQueries is regressed (MSE) onto SAM3 driven by its own text encoder | Visual Genome | 196 | 10k |
| 2 | train_stage2 |
Task-conditional segmentation (SAM3 grounding loss) | HOVA-500K, RAGNet, InstructPart, ReasonAFF | 128 | 40k |
| 3 | train_stage3 |
Functional mask + 3D motion curve, jointly | AFUN robot dataset + the Stage-2 data | 96 | 20k |
Batch sizes are totals over 4 GPUs, i.e. 49 / 32 / 24 per GPU; Stage 1 reaches its 49 as 7 samples per pass with 7 gradient-accumulation steps. Every stage uses Adam with a constant learning rate of 2e-4, bf16 autocast, and gradient checkpointing on Qwen3-VL. The paper's runs took about eight days on 4Γ GH200.
All datasets live under one root, $AFUN_DATA_ROOT (default ./data); configs/paths.yaml maps
each dataset to its folder.
export AFUN_DATA_ROOT=/path/to/afun_datasets
export AFUN_OUTPUT_DIR=/path/to/run_outputs # optional, default: ./outputs$AFUN_DATA_ROOT/
βββ visual_genome/ # Stage 1
βββ HOVA-500K/ # Stages 2β3
βββ RAGNet/data/data/ # Stages 2β3
βββ instruct_part/ # Stages 2β3
βββ reasonaff/ # Stages 2β3
βββ afun_train/ # Stage 3
| Dataset | Stages | Training samples | Source |
|---|---|---|---|
| Visual Genome | 1 | 5.4M region captions on 108k images (~15 GB) | jn12/VisualGenome |
| HOVA-500K | 2, 3 | 87,227 | JiaaZ/HOVA-500K + masks from prepare_hova.py |
| RAGNet | 2, 3 | 37,538 | wudongming/RAGNet + HANDAL |
| ReasonAFF | 2, 3 | 2,400 | hqking/affordance-R1 (gated) |
| InstructPart | 2, 3 | 1,800 | Authors' Google Drive |
| AFUN robot dataset | 3 | 44,749 (~231 GB) | AFUN-dataset/AFUN |
scripts/download_datasets.py fetches everything except InstructPart into the expected folders:
python scripts/download_datasets.py visual-genome --root $AFUN_DATA_ROOT
python scripts/download_datasets.py hova500k --root $AFUN_DATA_ROOT # 3doi, Ego4D, HANDAL (~233 GB)
python scripts/download_datasets.py ragnet --root $AFUN_DATA_ROOT
python scripts/download_datasets.py reasonaff --root $AFUN_DATA_ROOT
python scripts/download_datasets.py afun-train --root $AFUN_DATA_ROOTThe downloaded archives (each dataset's .downloads/ folder and RAGNet/data.zip) can be deleted
after extraction.
- ReasonAFF is gated: accept its terms on the dataset page and run
hf auth loginfirst. - RAGNet does not ship the HANDAL images and part masks. The HOVA-500K download includes them;
link them in (or point
--handal-rootat an official HANDAL release):Only records with a task instruction are used (EgoObjects and HANDAL reasoning, OpenX); the annotations that only name the object class are skipped.python scripts/prepare_ragnet.py $AFUN_DATA_ROOT/RAGNet/data/data \ --handal-root $AFUN_DATA_ROOT/HOVA-500K/HANDAL/images
- HOVA-500K provides contact points, not masks. We converted them into masks and task
instructions with Qwen3-VL + SAM3 (Appendix B.4). Our converted set is not public yet;
prepare_hova.pyreruns the conversion on a GPU, so its masks and instructions will differ from ours:It runs Qwen3-VL twice and SAM3 once on each of the ~470k contact annotations, which takes days on one GPU; rerunning it resumes where it stopped. To train without HOVA-500K, addpython scripts/prepare_hova.py $AFUN_DATA_ROOT/HOVA-500K # writes HOVA-500K/mask_annotation/
'~data.datasets.hova500k'(Stage 2) or'~data.segmentation.datasets.hova500k'(Stage 3) to the launch command. - InstructPart: the Hugging Face mirror lacks the annotation files, and Google Drive throttles
scripted downloads, so download it by hand from the authors' Google Drive folder. Place
InstructPart/train/train1800(andInstructPart/testfor evaluation) so that$AFUN_DATA_ROOT/instruct_part/InstructPart/train/train1800/containsimages/,masks/, anddata_train.json. - AFUN robot dataset:
--source <name>downloads a single source (SceneFun3D is 89% of the samples). Each sample is a folder withobs_frame.png,obs_frame_depth.npy(float32, mm),sam_mask.png, andtrajectory.json; the instructions are inafun_train/manifest.json.python scripts/prepare_robot_data.py $AFUN_DATA_ROOT/afun_trainchecks the extraction.
Check that a dataset loads, without loading any model:
python scripts/validate_data.py --dataset ragnet --root $AFUN_DATA_ROOT/RAGNet/data/dataMixtures. Stage 2 draws one dataset per step with probability proportional to its size, except HOVA-500K, which counts as 50,000 samples (β55% HOVA-500K, 41% RAGNet, 3% ReasonAFF, 2% InstructPart). Stage 3 picks robot or segmentation batches at random (1:1), and segmentation batches follow dataset size (β68% HOVA-500K). Without HOVA-500K, RAGNet supplies β90% of the segmentation batches.
torchrun --standalone --nproc_per_node=4 -m src.main --config-name train_stage1
torchrun --standalone --nproc_per_node=4 -m src.main --config-name train_stage2 \
train.ckpt_path=$AFUN_OUTPUT_DIR/afun-stage1/<run>/model_step_10000.pt
torchrun --standalone --nproc_per_node=4 -m src.main --config-name train_stage3 \
train.ckpt_path=$AFUN_OUTPUT_DIR/afun-stage2/<run>/model_step_40000.pttrain.ckpt_path loads the previous stage's weights and starts at step 0 with a fresh optimizer
(without the override, Stages 2 and 3 look for checkpoints/afun_stage1.pt and
checkpoints/afun_stage2.pt). Each run writes model_step_<N>.pt every 1,000 steps to
$AFUN_OUTPUT_DIR/afun-stage<k>/<date>_<time>/.
Useful overrides:
# smaller GPUs: effective batch = batch_size Γ GPUs Γ grad_accumulation_steps
torchrun ... train.batch_size=12 train.grad_accumulation_steps=2
# resume an interrupted run: restores weights, optimizer state, and step
torchrun ... train.resume_path=$AFUN_OUTPUT_DIR/afun-stage3/<run>/model_step_12000.pt
# log to Weights & Biases
torchrun ... train.use_wandb=true
# keep only the latest three checkpoints (each is ~4 GB)
torchrun ... train.training_cfg.rolling_ckpt=trueGPU memory. The defaults fit on 96 GB GPUs: on one RTX PRO 6000 (96 GB) Stages 1β3 peaked at about 39 / 78 / 81 GB. On smaller GPUs, divide
train.batch_sizeand multiplytrain.grad_accumulation_stepsby the same factor.
Tested configuration. Every stage was run on a single GPU, including resuming Stage 3 from a checkpoint. The multi-GPU path is exercised only by a two-process CPU test (
tests/test_distributed.py).
python demo.py --rgb examples/kitchen_fridge/rgb.png --query "Open the refrigerator door" \
--ckpt $AFUN_OUTPUT_DIR/afun-stage3/<run>/model_step_20000.pt --config inference_stage3inference_stage3 builds the 6-layer motion head that train_stage3 trains. The released afun.pt
has a 4-layer motion head and uses the default inference config.
- the 64 MetaQuery token embeddings (32 semantic, 32 motion), two boundary tokens, and their RMSNorm
- the QwenβSAM3 projector
- Stage 3 only: the spline motion head and the projection on top of the frozen Sonata encoder
Checkpoints store only these tensors plus the Adam state (~4 GB), not the frozen backbones.
This project builds on Qwen3-VL, SAM3, Depth-Anything-3, lingbot-depth, and Sonata. We thank the authors for releasing their code and models.
@article{wang2026afun,
title = {{AFUN}: Towards an Affordance Foundation Model for Functionality Understanding},
author = {Wang, Zhaoning and Zhong, Yi and Fu, Jiawei and Christensen, Henrik I. and Gao, Jun},
journal = {arXiv preprint arXiv:2606.02551},
year = {2026},
}