UFACTORY LeRobot

中文版本

UFACTORY robot arm integration with the LeRobot framework for robot learning, data collection, and policy deployment.

Training & Inference Results

Test datasets used during development, for reference only, do NOT reuse. As the robot arm and camera positions during development differ from user setups.

Features

• 🤖 UFACTORY robot control (xArm series)
• 🎮 Multiple teleop modes: GELLO / Pika / UMI / SpaceMouse
• 📷 Multi-camera data collection (RealSense / UMI camera)
• 📊 Dataset recording & management (LeRobot-compatible)
• 🧠 Imitation learning training (ACT / Diffusion Policy / etc.)
• 🚀 Policy evaluation & real-time inference
• 🔧 Mock mode (teleop device only, no physical robot needed)

Requirements

• Ubuntu 22.04 / 24.04
• Python >= 3.10
• CUDA >= 12.0 (recommended for GPU training)
• UFACTORY xArm (optional)

Installation

Base Install

git clone https://github.com/xArm-Developer/lerobot_robot_ufactory.git
cd lerobot_robot_ufactory

# Create a uv virtual environment and sync project dependencies
uv venv --python 3.10
uv sync

Includes: lerobot==0.4.3, xarm-python-sdk, numpy, pyyaml. LeRobot already pulls in torch, opencv, wandb, etc.

Peripheral Modules

Peripheral dependencies are available as optional extras via [module] install.

GELLO Teleop

Dynamixel-based leader arm, joint-space control.

• Once data collection starts, the relative position between the robot arm and camera (D435 / D435i) must remain unchanged.
• The camera position during inference must match the collection setup. If the robot arm or camera changes, previously collected data becomes invalid.

# 1. Install GELLO module
uv sync --extra gello

# 2. Add serial port permissions (re-login required)
sudo usermod -aG dialout $USER

Pika Teleop

Pika Sense handheld + Vive Tracker, Cartesian-space control.

• No requirement for the relative position of the two base stations and the robot arm. Only need to ensure the Pika Sense is within base station range during collection, but base stations must be recalibrated after moving.
• Base station positions for collection and inference do not need to be the same.

# 1. Install peripheral deps (skip transitive deps)
uv pip install pysurvive agx-pypika --no-deps

# 2. Install udev rules (re-plug devices afterwards)
sudo cp src/rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger

Calibrate Vive Tracker before first use: uf-vive-calibrate

UMI Teleop

Universal Manipulation Interface + Vive Tracker, supports dual-arm.

# 1. Install XVSDK (system-level, Ubuntu Focal only)
curl -sL https://raw.githubusercontent.com/xArm-Developer/ufactory_resources/main/fastumi/sdk/XVSDK_focal_amd64.deb -o /tmp/xvsdk.deb && sudo dpkg -i /tmp/xvsdk.deb
sudo apt install -y --fix-broken

# 2. Install peripheral deps
uv pip install pysurvive --no-deps

# 3. Install udev rules (re-plug devices afterwards)
sudo cp src/rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger

Calibrate Vive Tracker before first use: uf-vive-calibrate

Multi-UMI device configuration (two or more devices):

# Increase USB buffer size
sudo sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/s/quiet splash/quiet splash usbcore.usbfs_memory_mb=128/' /etc/default/grub
sync
sudo update-grub
sudo reboot

SpaceMouse Teleop

3Dconnexion SpaceMouse / SpaceNavigator.

# 1. Install SpaceMouse module
uv sync --extra spacemouse

# 2. Install udev rules (re-plug device afterwards)
sudo cp src/rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger

Usage

1. Teleop Testing

Test teleop-to-robot control loop without recording.

# Generic usage
uv run uf-robot-teleop --config_path path/to/config.yaml
uv run uf-robot-teleop --config_path path/to/config.yaml --fps 60  # specify frequency

# Example: xArm6 + UMI teleop
uv run uf-robot-teleop --config_path config/umi/xarm6_umi_record_config.yaml

2. xArm Manual Drag Mode

Use xArm joint teaching mode (mode=2) for manual dragging:

uv run uf-xarm-manual-mode --config_path config/manual_mode/xarm_manual_mode_config.yaml

Set manual_mode to true to enter drag mode. Press Enter to restore normal mode; set it to false to restore normal mode directly. With return_to_initial: true, the script reads the saved initial point through the xArm Studio API and returns to it when drag mode exits.

3. Manual Drag Data Collection

Manual drag recording uses manual_mode: true in the robot configuration and does not configure a teleoperator. During recording, the actual joint state is written as both the observation and action in the LeRobot dataset:

uv run uf-lerobot-record --config_path config/manual_mode/xarm7_manual_record_config.yaml
uv run uf-lerobot-record --config_path config/manual_mode/xarm7_manual_record_config.yaml --resume true

Press Space to start an episode, Right to save it, Left to discard and re-record it, and Esc to stop recording. Reset the arm manually between episodes.

4. Teleop Data Collection

Record datasets via teleop.

# Generic usage
uv run uf-lerobot-record --config_path path/to/record_config.yaml
uv run uf-lerobot-record --config_path path/to/config.yaml --resume true        # resume recording

# Example: xArm6 + UMI data collection
uv run uf-lerobot-record --config_path config/umi/xarm6_umi_record_config.yaml

5. Policy Training

Train imitation learning policies on collected data.

# Generic usage
uv run lerobot-train --policy act --dataset your_dataset_name

# Example: train ACT on xArm6 UMI dataset
uv run lerobot-train --policy act --dataset ufactory/xarm6_umi_datas

Important parameters:

# Note: repo_id is the same as in the record config
# Policy type: ACT, training steps: 800k
# Checkpoints saved every 20k steps, output to lerobot_datas/train (sibling of lerobot directory)
uv run lerobot-train \
  --dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \
  --dataset.repo_id=ufactory/xarm6_umi_datas \
  --policy.type=act \
  --policy.device=cuda \
  --policy.repo_id=ufactory/xarm6_umi_datas \
  --output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \
  --job_name=xarm6_umi_datas \
  --steps=800000 \
  --batch_size=8 \
  --save_freq=20000

6. Inference & Evaluation

Run inference with a trained policy.

# Generic usage
uv run uf-lerobot-eval --config_path path/to/config.yaml --policy.path your_train_path

# Example: run inference with trained ACT policy
uv run uf-lerobot-eval --config_path config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/

Tools

1. Camera Viewer

View and stitch multiple camera feeds.

uv run uf-camera-view -l                           # list all cameras
uv run uf-camera-view -l -T xvisio                 # list XVisio cameras only
uv run uf-camera-view -T xvisio                    # view XVisio cameras (default 1280x1280 YU12)
uv run uf-camera-view -T xvisio -W 640 -H 1920 -F NV12  # specify format
uv run uf-camera-view -T other                     # view other camera types

2. LeRobot Dataset Tools

LeRobot provides dataset utilities for inspecting, editing and managing collected datasets.

View an episode:

e.g. view episode index 17:

uv run lerobot-dataset-viz \
  --root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
  --repo-id ufactory/xarm7_record_datas \
  --display-compressed-images true \
  --episode-index 17

Delete specific episodes:

e.g. delete episodes 18 and 19:

uv run lerobot-edit-dataset \
  --root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
  --repo_id ufactory/xarm7_record_datas \
  --new_repo_id ../xarm7_record_datas_new \
  --operation.type delete_episodes \
  --operation.episode_indices "[18, 19]"

Merge datasets:

uv run lerobot-edit-dataset \
  --root=../../../../lerobot_datas/record \
  --repo_id ufactory/xarm7_record_datas_merge_1_2 \
  --operation.type merge \
  --operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']"

Teleop Comparison

Feature	GELLO	Pika	UMI	SpaceMouse
Control space	Joint space	Cartesian space	Cartesian space	Cartesian space
Tracking	Dynamixel servos	Vive Tracker	UMI SLAM / Vive	3D mouse
Dual-arm	❌	❌	✅	❌
System dep	dialout group	—	XVSDK deb	—

Project Structure

lerobot_robot_ufactory/
├── src/
│   ├── lerobot_robot_ufactory/      # LeRobot plugin package
│   │   ├── robots/                 # Robot control
│   │   │   ├── uf_robot/           #   xArm physical robot
│   │   │   ├── uf_mock_robot/      #   Mock robot simulator
│   │   ├── teleoperators/          # Teleop drivers
│   │   │   ├── base_teleop/        #   Shared base class
│   │   │   ├── gello_teleop/       #   GELLO (Dynamixel leader)
│   │   │   ├── pika_teleop/        #   Pika Sense (handheld + Vive)
│   │   │   ├── umi_teleop/         #   UMI (dual-arm support)
│   │   │   └── space_mouse/        #   SpaceMouse (3D mouse)
│   │   ├── cameras/                # Camera modules
│   │   │   └── umi_camera/         #   UMI camera
│   │   ├── devices/                # External device drivers
│   │   │   ├── pika/               #   Pika serial driver
│   │   │   └── umi/                #   XVLib / Vive Tracker
│   │   ├── scripts/                # Entry-point scripts
│   │   │   ├── uf_robot_teleop.py     # Teleop testing
│   │   │   ├── uf_lerobot_record.py   # Data recording
│   │   │   ├── uf_lerobot_eval.py     # Policy evaluation
│   │   │   ├── uf_camera_view.py      # Camera viewer tool
│   │   │   └── vive_calibrate.py      # Vive Tracker calibration
│   │   ├── context.py              # Teleop context registry
│   │   └── utils/                  # Utilities
├── config/                         # YAML config files
│   ├── gello/
│   ├── pika/
│   ├── umi/
│   └── spacemouse/
├── rules/                          # udev device rules
├── pyproject.toml
└── README.md

Important Notes

Users are expected to thoroughly study the codebase and configuration parameters.
The provided configurations are not guaranteed to work for all scenarios and must be adjusted based on actual hardware setups and task requirements.

In particular, for diffusion policies, the default parameters in LeRobot are primarily designed for simulation and are not optimized for real-world robots.

License

This project is released under the Apache License 2.0. See LICENSE.