wangshuxun/Xarm-DataCollection
1
0
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

main #1

Merged
wangshuxun merged 2 commits from lizhuoyuan/Xarm-DataCollection:main into main 2026-08-09 18:53:51 +08:00
Conversation 0 Commits 2 Files Changed 1 +1983 -512
15 changed files with 1983 additions and 512 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit b37cf9eb9e - Show all commits

427
README.md
View File

@ -1,328 +1,303 @@
# UFACTORY LeRobot # UFACTORY xArm7 · LeRobot (GELLO / Manual Drag)
> [中文版本](README_ZH.md) > [中文版本](README_ZH.md)
UFACTORY robot arm integration with the LeRobot framework for robot learning, data collection, and policy deployment. UFACTORY xArm integration with the [LeRobot](https://github.com/huggingface/lerobot) framework, focused on two data-collection workflows:
## Training & Inference Results - **GELLO** — joint-space teleoperation with a Dynamixel leader arm
- **Manual drag** — record demonstrations by freely moving the arm in xArm teach mode
[Test datasets](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY) used during development, **for reference only, do NOT reuse**. As the robot arm and camera positions during development differ from user setups. Collected data is stored in the standard LeRobot dataset format and can be used for imitation learning (ACT / Diffusion Policy, etc.) and real-time policy inference.
<table>
<tr>
<td width="50%">
<a href="https://www.youtube.com/watch?v=wTiWLiHciT8" target="_blank">
<img src="https://img.youtube.com/vi/wTiWLiHciT8/maxresdefault.jpg" width="100%">
</a>
</td>
<td width="50%">
<a href="https://www.youtube.com/watch?v=IiyvewZh5OY" target="_blank">
<img src="https://img.youtube.com/vi/IiyvewZh5OY/maxresdefault.jpg" width="100%">
</a>
</td>
</tr>
<tr>
<td width="50%">
<a href="https://youtu.be/wBwZH6POk38" target="_blank">
<img src="https://img.youtube.com/vi/wBwZH6POk38/maxresdefault.jpg" width="100%">
</a>
</td>
</tr>
</table>
## Features ## Features
- 🤖 UFACTORY robot control ([xArm series](https://www.ufactory.cc/)) - 🤖 UFACTORY xArm7 control
- 🎮 Multiple teleop modes: GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/) - 🎮 GELLO joint-space teleoperation (Dynamixel leader arm)
- 📷 Multi-camera data collection ([RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI camera) - ✋ Manual drag recording via xArm teach mode
- 📊 Dataset recording & management (LeRobot-compatible) - 📷 Intel RealSense camera observation (D435 / D435i)
- 🧠 Imitation learning training (ACT / Diffusion Policy / etc.) - 📊 LeRobot-compatible dataset recording & management
- 🚀 Policy evaluation & real-time inference - 🧠 Imitation learning training and policy inference
- 🔧 Mock mode (teleop device only, no physical robot needed) - ▶️ Episode replay for recorded manual demonstrations
## Requirements ## Requirements
- Ubuntu 22.04 / 24.04 - Ubuntu 22.04 / 24.04
- Python >= 3.10 - Python >= 3.10
- CUDA >= 12.0 (recommended for GPU training) - CUDA >= 12.0 (recommended for GPU training)
- UFACTORY xArm (optional) - UFACTORY xArm7 and its controller
- GELLO arm (FTDI USB serial)
- Intel RealSense D435 / D435i (for camera observations)
## Installation ## Installation
### Base Install
```bash ```bash
git clone https://github.com/xArm-Developer/lerobot_robot_ufactory.git git clone https://git.weiyantech.cn/wangshuxun/Xarm-DataCollection.git lerobot_xarm7
cd lerobot_robot_ufactory cd lerobot_xarm7
# Create a uv virtual environment and sync project dependencies
uv venv --python 3.10 uv venv --python 3.10
uv sync uv sync --extra gello
``` ```
Includes: `lerobot==0.4.3`, `xarm-python-sdk`, `numpy`, `pyyaml`. LeRobot already pulls in torch, opencv, wandb, etc. The base dependencies include `lerobot==0.4.3` (with Intel RealSense support), `xarm-python-sdk`, `numpy`, `pyyaml`, and `opencv-python`. The `gello` extra adds the GELLO software and Dynamixel SDK.
### Peripheral Modules ### Serial port permission
Peripheral dependencies are available as optional extras via `[module]` install. The GELLO arm connects over a serial port, so add your user to the `dialout` group (re-login afterwards):
#### GELLO Teleop
Dynamixel-based leader arm, joint-space control.
* Once data collection starts, the **relative position** between the robot arm and camera (D435 / D435i) **must remain unchanged**.
* The camera position during inference must match the collection setup. If the robot arm or camera changes, previously collected data becomes invalid.
```bash ```bash
# 1. Install GELLO module
uv sync --extra gello
# 2. Add serial port permissions (re-login required)
sudo usermod -aG dialout $USER sudo usermod -aG dialout $USER
``` ```
#### Pika Teleop Find the GELLO serial port path (used as `teleop.port` in the configs):
Pika Sense handheld + Vive Tracker, Cartesian-space control.
* No requirement for the relative position of the two base stations and the robot arm. Only need to ensure the Pika Sense is within base station range during collection, but **base stations must be recalibrated after moving**.
* Base station positions for collection and inference do not need to be the same.
```bash ```bash
# 1. Install peripheral deps (skip transitive deps) ls /dev/serial/by-id/
uv pip install pysurvive agx-pypika --no-deps
# 2. Install udev rules (re-plug devices afterwards)
sudo cp src/rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
``` ```
> Calibrate Vive Tracker before first use: `uf-vive-calibrate` ## Configuration
#### UMI Teleop Predefined configs are provided under `config/`:
Universal Manipulation Interface + Vive Tracker, supports dual-arm. | Workflow | Config |
|---|---|
| GELLO · xArm7 | `config/gello/xarm7_gello_record_config.yaml` |
| Manual drag · xArm7 | `config/manual_mode/xarm7_manual_record_config.yaml` |
### GELLO config
- `robot.robot_ip` — xArm controller IP (e.g. `192.168.1.245`)
- `robot.robot_dof` — `7`
- `robot.gripper_type` — `1` for the xArm gripper
- `teleop.port` — GELLO serial port (`/dev/serial/by-id/...`)
- `teleop.joint_ids` / `teleop.joint_signs` — per-arm servo mapping and direction
- `teleop.start_joints` — GELLO calibration reference, should match the xArm SDK initial point (degrees)
- `teleop.gripper_id` — GELLO gripper servo ID (`8`; `-1` disables it)
- `dataset.root` / `dataset.repo_id` — where the dataset is stored
- `dataset.single_task` — task description saved with each frame
- `dataset.fps` / `episode_time_s` / `reset_time_s` — recording timing
> The xArm7 config already contains the correct joint mapping; only edit the port, IP, and dataset fields for your setup.
### Manual-drag config
- `robot.manual_mode: true` — enable xArm teach mode (joint free-drive)
- `robot.teach_sensitivity` — teaching sensitivity, valid range 1–5
- `robot.manual_gripper_speed` — gripper velocity in normalized position per second (default `0.5`)
- `robot.observe_joint_vel` — record joint velocities in observations (`false` by default)
- `robot.cameras.camera` — Intel RealSense camera (`serial_number_or_name`, resolution, fps)
- `dataset.root` / `dataset.repo_id` / `single_task` / `fps` / `episode_time_s` / `reset_time_s` / `num_episodes` — dataset settings
### Camera configuration
When adding a camera to the robot config, use the template in
`config/manual_mode/xarm7_manual_record_config.yaml`:
```yaml
robot:
cameras:
camera:
type: intelrealsense # RealSense type, NOT opencv
serial_number_or_name: "148522072685"
width: 640
height: 480
fps: 30
```
- `type` must be `intelrealsense` (RealSense), **not** `opencv` (generic USB camera).
- `serial_number_or_name` must be filled with the actual RealSense serial number **obtained beforehand**, otherwise connecting/recording fails. Get it with:
```bash ```bash
# 1. Install XVSDK (system-level, Ubuntu Focal only) uv run uf-camera-view -l -T realsense # prints each camera's serial number
curl -sL https://raw.githubusercontent.com/xArm-Developer/ufactory_resources/main/fastumi/sdk/XVSDK_focal_amd64.deb -o /tmp/xvsdk.deb && sudo dpkg -i /tmp/xvsdk.deb
sudo apt install -y --fix-broken
# 2. Install peripheral deps
uv pip install pysurvive --no-deps
# 3. Install udev rules (re-plug devices afterwards)
sudo cp src/rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
```
> Calibrate Vive Tracker before first use: `uf-vive-calibrate`
**Multi-UMI device configuration** (two or more devices):
```bash
# Increase USB buffer size
sudo sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/s/quiet splash/quiet splash usbcore.usbfs_memory_mb=128/' /etc/default/grub
sync
sudo update-grub
sudo reboot
```
#### SpaceMouse Teleop
3Dconnexion SpaceMouse / SpaceNavigator.
```bash
# 1. Install SpaceMouse module
uv sync --extra spacemouse
# 2. Install udev rules (re-plug device afterwards)
sudo cp src/rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
``` ```
or with the librealsense tool `rs-enumerate-devices`.
## Usage ## Usage
### 1. Teleop Testing ### 1. GELLO teleop test
Test teleop-to-robot control loop without recording. Test the GELLO → robot control loop without recording:
```bash ```bash
# Generic usage uv run uf-robot-teleop --config_path config/gello/xarm7_gello_record_config.yaml
uv run uf-robot-teleop --config_path path/to/config.yaml uv run uf-robot-teleop --config_path config/gello/xarm7_gello_record_config.yaml --fps 60 # optional loop rate
uv run uf-robot-teleop --config_path path/to/config.yaml --fps 60 # specify frequency
# Example: xArm6 + UMI teleop
uv run uf-robot-teleop --config_path config/umi/xarm6_umi_record_config.yaml
``` ```
### 2. Manual Drag Data Collection `Space` reset & start, `←` reset, `Esc` exit.
Manual drag recording uses `manual_mode: true` in the robot configuration and does not configure a teleoperator. During recording, the actual joint state is written as both the observation and action in the LeRobot dataset. When a gripper is configured, hold `C` to close it slowly and `O` to open it slowly. Adjust the speed with `manual_gripper_speed`, which defaults to `0.5`: ### 2. GELLO data collection
```bash
# Record a new dataset
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml
# Continue recording on an existing dataset
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml -r
# Optional: save episodes in the background
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml -a
```
Controls: `Space` start the episode, `→` save it, `←` discard and re-record it, `Esc` stop recording. The arm resets to its initial point between episodes.
> During collection the **relative position between the robot arm and the camera must not change**, and the camera setup at inference time must match the one used during collection. If the arm or camera moves, previously collected data becomes invalid.
> If the dataset root already exists and `-r` is not given, the script asks whether to overwrite it, resume, or cancel.
### 3. Manual drag recording
```bash ```bash
./start_manual_record.sh ./start_manual_record.sh
./start_manual_record.sh -r ./start_manual_record.sh -r # force resume; fails if the dataset directory does not exist
``` ```
The launcher creates a dataset on its first run and automatically resumes an existing valid dataset. If it reports an empty or incomplete dataset directory, choose a new `dataset.root` or remove that directory after confirming it contains no data. Press `Space` to start an episode, `Right` to save it, `Left` to discard and re-record it, and `Esc` to stop recording. Reset the arm manually between episodes. The record script (`uf-lerobot-record`) reads `dataset.root` from the config and checks the path before recording, then:
### 3. Teleop Data Collection - Directory does not exist → records a new dataset.
- Directory already exists (valid LeRobot dataset) and no `-r` was given → asks interactively:
- `o` overwrite: delete the existing dataset and record a new one
- `r` resume: keep existing episodes and continue recording
- `c` cancel
- Directory exists but is incomplete (missing required metadata or data parquet files) → asks to overwrite it or cancel; non-interactive runs error out instead.
Record datasets via teleop. Pass `-r` to resume directly without asking. Note that `./start_manual_record.sh` keeps its original launcher behavior — it automatically resumes an existing valid dataset (equivalent to `-r`), so run `uv run uf-lerobot-record --config_path config/manual_mode/xarm7_manual_record_config.yaml` directly if you want to see the overwrite/resume prompt.
During recording the arm is in teach mode: the actual joint state is written as both the observation and the action. Hold `C` to slowly close the gripper and `O` to slowly open it. Controls: `Space` start, `→` save, `←` discard & re-record, `Esc` stop. Reset the arm manually between episodes.
> **IMPORTANT: At the start of every episode, wait until the arm has finished resetting and then wait another 5 seconds before operating it, or wait until the console prints `Start Recording` before operating it.** This prevents reset commands from conflicting with operation commands and causing errors.
### 4. Policy training
```bash ```bash
# Generic usage uv run lerobot-train --policy act --dataset ufactory/xarm7_gello_datas
uv run uf-lerobot-record --config_path path/to/record_config.yaml
uv run uf-lerobot-record --config_path path/to/config.yaml --resume true # resume recording
# Example: xArm6 + UMI data collection
uv run uf-lerobot-record --config_path config/umi/xarm6_umi_record_config.yaml
``` ```
### 5. Policy Training Example with explicit training parameters (checkpoints are saved every `save_freq` steps into `output_dir`):
Train imitation learning policies on collected data.
```bash ```bash
# Generic usage
uv run lerobot-train --policy act --dataset your_dataset_name
# Example: train ACT on xArm6 UMI dataset
uv run lerobot-train --policy act --dataset ufactory/xarm6_umi_datas
```
Important parameters:
```bash
# Note: repo_id is the same as in the record config
# Policy type: ACT, training steps: 800k
# Checkpoints saved every 20k steps, output to lerobot_datas/train (sibling of lerobot directory)
uv run lerobot-train \ uv run lerobot-train \
--dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \ --dataset.root=/home/<user>/lerobot_datas/record/ufactory/xarm7_gello_datas \
--dataset.repo_id=ufactory/xarm6_umi_datas \ --dataset.repo_id=ufactory/xarm7_gello_datas \
--policy.type=act \ --policy.type=act \
--policy.device=cuda \ --policy.device=cuda \
--policy.repo_id=ufactory/xarm6_umi_datas \ --policy.repo_id=ufactory/xarm7_gello_datas \
--output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \ --output_dir=/home/<user>/lerobot_datas/train/xarm7_gello_datas \
--job_name=xarm6_umi_datas \ --job_name=xarm7_gello_datas \
--steps=800000 \ --steps=800000 \
--batch_size=8 \ --batch_size=8 \
--save_freq=20000 --save_freq=20000
``` ```
### 6. Inference & Evaluation ### 5. Policy inference
Run inference with a trained policy.
```bash ```bash
# Generic usage uv run uf-lerobot-eval \
uv run uf-lerobot-eval --config_path path/to/config.yaml --policy.path your_train_path --config_path config/gello/xarm7_gello_record_config.yaml \
--policy.path /path/to/train/output/checkpoints/last/pretrained_model/
# Example: run inference with trained ACT policy
uv run uf-lerobot-eval --config_path config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
``` ```
`←` / `→` reset, `Esc` stop.
### 6. Replay recorded episodes
Replay the absolute joint states (`observation.state`) of a manual-drag episode on an xArm7. States are sent as absolute targets at the dataset FPS (default 30), so the motion matches the recording:
```bash
uv run uf-lerobot-replay \
--dataset-root /path/to/xarm7_manual_datas \
--robot-ip 192.168.1.245
# Skip the interactive confirmation (non-interactive use)
uv run uf-lerobot-replay --dataset-root /path/to/xarm7_manual_datas --robot-ip 192.168.1.245 --yes
# Replay another episode
uv run uf-lerobot-replay --dataset-root /path/to/xarm7_manual_datas --robot-ip 192.168.1.245 --episode-index 3
```
The robot first moves to the xArm SDK initial point, then replays the episode and stays at the last state. Make sure the workspace is clear and the recorded initial pose matches the current arm setup.
## IMPORTANT: Robot Power-On and Power-Off
### Power-on
1. **Connect the computer to the robot controller with an Ethernet cable.**
2. **Configure the computer's Ethernet interface to the same IP subnet as the robot IP shown on the controller** (for example, `192.168.1.xxx`).
3. **Open `http://192.168.1.245:18333/` in a browser.** The controller console should be displayed.
4. **Release the emergency-stop button before operating the robot.**
### Power-off
1. **Use the web console to return the arm to its initial position.**
2. **Press the emergency-stop button.**
3. **Turn off the controller power.**
## Tools ## Tools
### 1. Camera Viewer ### Camera viewer
View and stitch multiple camera feeds.
```bash ```bash
uv run uf-camera-view -l # list all cameras uv run uf-camera-view -l # list cameras
uv run uf-camera-view -l -T xvisio # list XVisio cameras only uv run uf-camera-view -T realsense # view RealSense cameras
uv run uf-camera-view -T xvisio # view XVisio cameras (default 1280x1280 YU12)
uv run uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # specify format
uv run uf-camera-view -T other # view other camera types
``` ```
### 2. LeRobot Dataset Tools ### LeRobot dataset tools
LeRobot provides dataset utilities for inspecting, editing and managing collected datasets.
#### View an episode:
e.g. view episode index 17:
```bash ```bash
# View episode 17
uv run lerobot-dataset-viz \ uv run lerobot-dataset-viz \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \ --root=/path/to/record/ufactory/xarm7_manual_datas \
--repo-id ufactory/xarm7_record_datas \ --repo-id ufactory/xarm7_manual_datas \
--display-compressed-images true \ --display-compressed-images true \
--episode-index 17 --episode-index 17
```
#### Delete specific episodes: # Delete episodes 18 and 19
e.g. delete episodes 18 and 19:
```bash
uv run lerobot-edit-dataset \ uv run lerobot-edit-dataset \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \ --root=/path/to/record/ufactory/xarm7_manual_datas \
--repo_id ufactory/xarm7_record_datas \ --repo_id ufactory/xarm7_manual_datas \
--new_repo_id ../xarm7_record_datas_new \ --new_repo_id ../xarm7_manual_datas_new \
--operation.type delete_episodes \ --operation.type delete_episodes \
--operation.episode_indices "[18, 19]" --operation.episode_indices "[18, 19]"
```
#### Merge datasets: # Merge datasets
```bash
uv run lerobot-edit-dataset \ uv run lerobot-edit-dataset \
--root=../../../../lerobot_datas/record \ --root=/path/to/record \
--repo_id ufactory/xarm7_record_datas_merge_1_2 \ --repo_id ufactory/xarm7_datas_merge \
--operation.type merge \ --operation.type merge \
--operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']" --operation.repo_ids "['ufactory/xarm7_datas_1', 'ufactory/xarm7_datas_2']"
``` ```
## Teleop Comparison ## Project structure
| Feature | GELLO | Pika | UMI | SpaceMouse |
|---------|-------|------|-----|------------|
| Control space | Joint space | Cartesian space | Cartesian space | Cartesian space |
| Tracking | Dynamixel servos | Vive Tracker | UMI SLAM / Vive | 3D mouse |
| Dual-arm | ❌ | ❌ | ✅ | ❌ |
| System dep | dialout group | — | XVSDK deb | — |
## Project Structure
``` ```
lerobot_robot_ufactory/ lerobot_xarm7/
├── src/ ├── config/
│ ├── lerobot_robot_ufactory/ # LeRobot plugin package │ ├── gello/ # xArm7 GELLO record config
│ │ ├── robots/ # Robot control │ └── manual_mode/ # xArm7 manual-drag record config
│ │ │ ├── uf_robot/ # xArm physical robot ├── src/lerobot_robot_ufactory/
│ │ │ ├── uf_mock_robot/ # Mock robot simulator │ ├── robots/
│ │ ├── teleoperators/ # Teleop drivers │ │ └── uf_robot/ # xArm control (joint/cartesian, teach mode)
│ │ │ ├── base_teleop/ # Shared base class │ ├── teleoperators/
│ │ │ ├── gello_teleop/ # GELLO (Dynamixel leader) │ │ ├── base_teleop/ # shared teleop base class
│ │ │ ├── pika_teleop/ # Pika Sense (handheld + Vive) │ │ └── gello_teleop/ # GELLO (Dynamixel leader arm)
│ │ │ ├── umi_teleop/ # UMI (dual-arm support) │ ├── scripts/
│ │ │ └── space_mouse/ # SpaceMouse (3D mouse) │ │ ├── uf_robot_teleop.py # teleop test loop
│ │ ├── cameras/ # Camera modules │ │ ├── uf_lerobot_record.py # data collection (incl. manual mode)
│ │ │ └── umi_camera/ # UMI camera │ │ ├── uf_lerobot_eval.py # policy inference
│ │ ├── devices/ # External device drivers │ │ ├── uf_lerobot_replay.py # episode replay
│ │ │ ├── pika/ # Pika serial driver │ │ └── uf_camera_view.py # camera viewer
│ │ │ └── umi/ # XVLib / Vive Tracker │ └── configs/parser.py # config loading / CLI overrides
│ │ ├── scripts/ # Entry-point scripts ├── start_manual_record.sh # manual-drag launcher
│ │ │ ├── uf_robot_teleop.py # Teleop testing
│ │ │ ├── uf_lerobot_record.py # Data recording
│ │ │ ├── uf_lerobot_eval.py # Policy evaluation
│ │ │ ├── uf_camera_view.py # Camera viewer tool
│ │ │ └── vive_calibrate.py # Vive Tracker calibration
│ │ ├── context.py # Teleop context registry
│ │ └── utils/ # Utilities
├── config/ # YAML config files
│ ├── gello/
│ ├── pika/
│ ├── umi/
│ └── spacemouse/
├── rules/ # udev device rules
├── pyproject.toml ├── pyproject.toml
└── README.md ├── README.md
└── README_ZH.md
``` ```
## Important Notes ## Important notes
Users are expected to thoroughly study the codebase and configuration parameters. - The provided configs are **examples**: edit IPs, serial ports, camera serials, dataset paths, and task descriptions to match your hardware.
The provided configurations are **not guaranteed to work for all scenarios** and must be adjusted based on actual hardware setups and task requirements. - For GELLO data, keep the robot–camera relative pose identical between collection and inference.
- The LeRobot default parameters for diffusion policies are mostly designed for simulation and are **not optimized for real robots** — tune them for your task.
In particular, for **diffusion policies**, the default parameters in LeRobot are primarily designed for simulation and **are not optimized for real-world robots**. - Check the workspace for obstacles and keep the arm's initial pose consistent with the recorded data before replay or inference.
## License ## License

459
README_ZH.md
View File

@ -1,337 +1,328 @@
# UFACTORY LeRobot # UFACTORY xArm7 · LeRobot（GELLO / 手动拖拽）
> [English Version](README.md) > [English Version](README.md)
UFACTORY(深圳市众为创造科技有限公司) 机械臂与 LeRobot 框架集成项目，支持多种遥操作方式的数据采集、策略训练和部署推理。 UFACTORY xArm 与 [LeRobot](https://github.com/huggingface/lerobot) 框架的集成项目，专注于两种数据采集方式：
## 训练推理效果 - **GELLO** — 使用 Dynamixel 示教臂的关节空间遥操作
- **手动拖拽** — 在 xArm 示教模式下自由拖动机械臂录制演示
点击下载开发时[采集的数据集](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY)，**仅供参考，不可复用**。因为用户机械臂和摄像头位置和开发测试时不一致。 采集的数据以标准 LeRobot 数据集格式保存，可用于模仿学习训练（ACT / Diffusion Policy 等）和实时策略推理。
<table>
<tr>
<td width="50%">
<a href="https://www.bilibili.com/video/BV12xFjzzEaX" target="_blank">
<img src="https://i2.hdslb.com/bfs/archive/7b325df5fb4c16e922b66d27b56d8fb6534f8b46.jpg" width="100%">
</a>
</td>
<td width="50%">
<a href="https://www.bilibili.com/video/BV16ccizHE2P" target="_blank">
<img src="https://i2.hdslb.com/bfs/archive/8c5d5e9370577fa89d06a175aceea282d9b2eb9a.jpg" width="100%">
</a>
</td>
</tr>
<tr>
<td width="50%">
<a href="https://www.bilibili.com/video/BV1xGEy6mE3i" target="_blank">
<img src="https://i2.hdslb.com/bfs/archive/c8bbac04a736b7043a20b753e11afea7627bdae2.jpg" width="100%">
</a>
</td>
</tr>
</table>
## 功能特性 ## 功能特性
- 🤖 UFACTORY 机械臂控制（[xArm 系列](https://www.ufactory.cc/)） - 🤖 UFACTORY xArm7 控制
- 🎮 多种遥操作方式：GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/) - 🎮 GELLO 关节空间遥操作（Dynamixel 示教臂）
- 📷 多摄像头数据采集（[RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI 相机） - ✋ xArm 示教模式手动拖拽采集
- 📊 数据集录制与管理（兼容 LeRobot 格式） - 📷 Intel RealSense 相机观测（D435 / D435i）
- 🧠 模仿学习训练（ACT / Diffusion Policy 等） - 📊 兼容 LeRobot 格式的数据集录制与管理
- 🚀 策略评估与实时推理 - 🧠 模仿学习训练与策略推理
- 🔧 Mock 机器人模拟（只用遥操作设备采集数据） - ▶️ 手动演示数据的 episode 回放
## 环境要求 ## 环境要求
- Ubuntu 22.04 / 24.04 - Ubuntu 22.04 / 24.04
- Python >= 3.10 - Python >= 3.10
- CUDA >= 12.0（GPU 训练推荐） - CUDA >= 12.0（GPU 训练推荐）
- UFACTORY 机械臂（xArm 系列，可选） - UFACTORY xArm7 及控制器
- GELLO 示教臂（FTDI USB 串口）
- Intel RealSense D435 / D435i（需要相机观测时）
## 安装 ## 安装
### 基础项目安装
```bash ```bash
git clone https://github.com/xArm-Developer/lerobot_robot_ufactory.git git clone https://git.weiyantech.cn/wangshuxun/Xarm-DataCollection.git lerobot_xarm7
cd lerobot_robot_ufactory cd lerobot_xarm7
# 创建 uv 虚拟环境并同步项目依赖
uv venv --python 3.10 uv venv --python 3.10
uv sync uv sync --extra gello
``` ```
包含：`lerobot==0.4.3`、`xarm-python-sdk`、`numpy`、`pyyaml`（lerobot 已自动携带 torch、opencv、wandb 等训练相关依赖）。 基础依赖包含 `lerobot==0.4.3`（带 Intel RealSense 支持）、`xarm-python-sdk`、`numpy`、`pyyaml` 和 `opencv-python`。`gello` 可选依赖会额外安装 GELLO 软件和 Dynamixel SDK。
### 外设模块安装 ### 串口权限
外设依赖以可选模块形式提供，通过 `[模块名]` 安装。 GELLO 示教臂通过串口连接，需要将当前用户加入 `dialout` 组（重新登录后生效）：
#### GELLO 遥操作
适用于 GELLO 示教臂（Dynamixel 舵机方案），控制空间为关节空间。
* 一旦开始数据采集，机械臂与摄像头（D435 / D435i）的**相对位置必须保持不变**。
* 推理时的摄像头位置必须与采集时相同。若机械臂或摄像头发生变化，此前采集的数据将无效。
```bash ```bash
# 1. 安装 GELLO 模块
uv sync --extra gello
# 2. 添加串口权限（重新登录后生效）
sudo usermod -aG dialout $USER sudo usermod -aG dialout $USER
``` ```
#### Pika 遥操作 查看 GELLO 串口路径（用于配置文件中的 `teleop.port`）：
适用于 Pika Sense 手持示教器 + Vive Tracker，控制空间为笛卡尔空间。
* 两个基站和机械臂相对位置没有要求，只需要保证采集时pika sense在基站范围内，但**基站移动后需要重新校准**。
* 采集和推理时基站位置可不相同。
```bash ```bash
# 1. 安装外设依赖（不需要它们的间接依赖） ls /dev/serial/by-id/
uv pip install pysurvive agx-pypika --no-deps
# 2. 安装 udev 规则（重新插拔设备后生效）
sudo cp rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
``` ```
> Vive Tracker 首次使用前需校准：`uf-vive-calibrate` ## 配置
#### UMI 遥操作 项目在 `config/` 下提供了预置配置文件：
适用于 UMI（Universal Manipulation Interface）方案，含 Vive Tracker 追踪，支持双机械臂。 | 采集方式 | 配置文件 |
|---|---|
| GELLO · xArm7 | `config/gello/xarm7_gello_record_config.yaml` |
| 手动拖拽 · xArm7 | `config/manual_mode/xarm7_manual_record_config.yaml` |
### GELLO 配置说明
- `robot.robot_ip` — xArm 控制器 IP（如 `192.168.1.245`）
- `robot.robot_dof` — `7`
- `robot.gripper_type` — `1` 表示 xArm 夹爪
- `teleop.port` — GELLO 串口路径（`/dev/serial/by-id/...`）
- `teleop.joint_ids` / `teleop.joint_signs` — 各型号机械臂的舵机映射与方向
- `teleop.start_joints` — GELLO 校准参考值（角度），应与 xArm SDK 初始点一致
- `teleop.gripper_id` — GELLO 夹爪舵机 ID（`8`；`-1` 表示无夹爪）
- `dataset.root` / `dataset.repo_id` — 数据集保存位置
- `dataset.single_task` — 随每一帧保存的任务描述
- `dataset.fps` / `episode_time_s` / `reset_time_s` — 录制时序参数
> xArm7 的配置已包含正确的关节映射，一般只需要修改串口、IP 和数据集路径。
### 手动拖拽配置说明
- `robot.manual_mode: true` — 开启 xArm 示教模式（关节自由拖动）
- `robot.teach_sensitivity` — 示教灵敏度，有效范围 1–5
- `robot.manual_gripper_speed` — 夹爪速度（每秒归一化位置变化，默认 `0.5`）
- `robot.observe_joint_vel` — 是否在观测中记录关节速度（默认 `false`）
- `robot.cameras.camera` — Intel RealSense 相机配置（`serial_number_or_name`、分辨率、fps）
- `dataset.root` / `dataset.repo_id` / `single_task` / `fps` / `episode_time_s` / `reset_time_s` / `num_episodes` — 数据集配置
### 相机配置说明
需要给机器人配置添加相机时，参考 `config/manual_mode/xarm7_manual_record_config.yaml` 中的模板：
```yaml
robot:
cameras:
camera:
type: intelrealsense # RealSense 类型，不是 opencv
serial_number_or_name: "148522072685"
width: 640
height: 480
fps: 30
```
- `type` 必须是 `intelrealsense`（RealSense 类型），**不能**写成 `opencv`（普通 USB 相机类型）。
- `serial_number_or_name` 需要先获取 RealSense 相机序列号再填写，否则连接/录制会报错。获取序列号：
```bash ```bash
# 1. 安装 XVSDK（系统级依赖，仅支持 Ubuntu Focal） uv run uf-camera-view -l -T realsense # 列出每台相机的序列号
curl -sL https://raw.githubusercontent.com/xArm-Developer/ufactory_resources/main/fastumi/sdk/XVSDK_focal_amd64.deb -o /tmp/xvsdk.deb && sudo dpkg -i /tmp/xvsdk.deb
sudo apt install -y --fix-broken
# 2. 安装外设依赖
uv pip install pysurvive --no-deps
# 3. 安装 udev 规则（重新插拔设备后生效）
sudo cp rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
``` ```
> Vive Tracker 首次使用前需校准：`uf-vive-calibrate` 也可以使用 librealsense 自带的 `rs-enumerate-devices`。
**多 UMI 设备配置**（使用两台及以上时）：
```bash
# 增加 USB 缓冲区大小
sudo sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/s/quiet splash/quiet splash usbcore.usbfs_memory_mb=128/' /etc/default/grub
sync
sudo update-grub
sudo reboot
```
#### SpaceMouse 遥操作
适用于 3Dconnexion SpaceMouse / SpaceNavigator。
```bash
# 1. 安装 SpaceMouse 模块
uv sync --extra spacemouse
# 2. 安装 udev 规则（重新插拔设备后生效）
sudo cp rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
```
## 使用 ## 使用
### 1. 遥操作测试 ### 1. GELLO 遥操作测试
测试遥操作设备与机械臂的联动，不录制数据。 不录制数据，仅测试 GELLO 与机械臂的联动：
```bash ```bash
# 通用格式 uv run uf-robot-teleop --config_path config/gello/xarm7_gello_record_config.yaml
uv run uf-robot-teleop --config_path path/to/config.yaml uv run uf-robot-teleop --config_path config/gello/xarm7_gello_record_config.yaml --fps 60 # 可选，指定循环频率
uv run uf-robot-teleop --config_path path/to/config.yaml --fps 60 # 指定频率
# 示例: xArm6 + UMI 遥操作
uv run uf-robot-teleop --config_path config/umi/xarm6_umi_record_config.yaml
``` ```
### 2. 人工拖拽数据采集 `Space` 复位并开始，`←` 复位，`Esc` 退出。
人工拖拽录制使用 robot 配置中的 `manual_mode: true`，不需要配置 teleop。录制过程中，机械臂的实际关节状态会作为 observation 和 action 写入 LeRobot 数据集；如果配置了夹爪，还可以按住 `C` 缓慢闭合、按住 `O` 缓慢张开。夹爪速度通过 `manual_gripper_speed` 配置，默认值为 `0.5`： ### 2. GELLO 数据采集
```bash
# 录制新数据集
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml
# 在已有数据集上续录
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml -r
# 可选：后台异步保存 episode
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml -a
```
按键控制：`Space` 开始当前 episode，`→` 保存，`←` 放弃并重录，`Esc` 停止录制。每个 episode 之间机械臂会自动复位到初始点。
> 采集过程中**机械臂与相机（D435 / D435i）的相对位置必须保持不变**，推理时的相机位置必须与采集时一致。若机械臂或相机发生变化，此前采集的数据将失效。
> 如果数据集目录已存在且未加 `-r`，脚本会询问是覆盖、续录还是取消。
### 3. 手动拖拽数据采集
```bash ```bash
./start_manual_record.sh ./start_manual_record.sh
./start_manual_record.sh -r ./start_manual_record.sh -r # 强制续录；数据集目录不存在时会报错
``` ```
启动脚本首次运行时创建数据集，检测到有效数据集时会自动续录。如果提示数据目录为空或不完整，请修改 `dataset.root`，或者确认没有数据后删除该目录。按 `Space` 开始当前 episode，按 `Right` 保存，按 `Left` 放弃并重录当前 episode，按 `Esc` 停止录制。episode 之间可以手动复位机械臂。 录制脚本（`uf-lerobot-record`）会从配置读取 `dataset.root` 并在录制前检查路径，然后：
### 3. 遥操作数据采集 - 目录不存在：直接录制新数据集。
- 目录已存在（有效 LeRobot 数据集）且未加 `-r`：交互询问：
- `o` 覆盖：删除已有数据集，重新录制
- `r` 续录：保留已有 episode，继续录制
- `c` 取消
- 目录存在但不完整（缺少必要元数据或数据 parquet 文件）：询问覆盖或取消；非交互运行时直接报错。
通过遥操作录制数据集。 加 `-r` 可跳过询问直接续录。注意 `./start_manual_record.sh` 保持原有启动脚本行为——检测到有效数据集会自动续录（相当于 `-r`），如果想看到覆盖/续录的询问，请直接用 `uv run uf-lerobot-record --config_path config/manual_mode/xarm7_manual_record_config.yaml` 运行。
录制时机械臂处于示教模式，实际关节状态会同时作为 observation 和 action 写入数据集。按住 `C` 缓慢闭合夹爪，按住 `O` 缓慢张开。按键控制：`Space` 开始，`→` 保存，`←` 放弃并重录，`Esc` 停止。episode 之间手动复位机械臂。
> **重要：每个 episode 开始时，必须等待机械臂复位完成后再等待 5 秒，然后才能开始操作；或者确认控制台打印 `Start Recording` 后再开始操作。** 这样可以避免机械臂复位控制指令与操作指令冲突导致报错。
### 4. 策略训练
```bash ```bash
# 通用格式 uv run lerobot-train --policy act --dataset ufactory/xarm7_gello_datas
uv run uf-lerobot-record --config_path path/to/record_config.yaml
uv run uf-lerobot-record --config_path path/to/config.yaml --resume true # 续录
# 示例: xArm6 + UMI 数据采集
uv run uf-lerobot-record --config_path config/umi/xarm6_umi_record_config.yaml
``` ```
### 4. 数据重放 带完整训练参数的示例（每 `save_freq` 步保存一次 checkpoint 到 `output_dir`）：
`datasets/xarm7_manual_replay` 是人工拖拽录制的 LeRobot 数据集。回放脚本使用其中的
`observation.state`，将 7 个关节弧度值和归一化夹爪位置作为**绝对目标值**发送给 xArm7，
不会将相邻帧相减，也不会累加成相对动作。脚本默认按数据集的 30 FPS 播放一个 episode。
启动前会要求确认，连接后会先自动移动到 xArm SDK 初始点；播放结束后保持最后一帧姿态并断开连接：
```bash ```bash
uv run uf-lerobot-replay \
--dataset-root /home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay \
--robot-ip 192.168.1.245
```
无人值守运行时可以使用 `--yes` 跳过确认。执行前请确认机械臂工作空间无障碍物，且数据中的初始姿态与当前设备匹配。
### 5. Lerobot训练
采集数据后，使用 LeRobot 训练管道进行模仿学习训练。
```bash
# 通用格式
uv run lerobot-train --policy act --dataset your_dataset_name
```
参数示例：
```bash
# 注意: repo_id就是采集时配置文件里面的repo_id
# 这里训练策略policy.type选用act，训练steps为80w次
# 训练过程每2w次保存一次结果，结果输出到和lerobot同级目录下的lerobot_datas/train里面
uv run lerobot-train \ uv run lerobot-train \
--dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \ --dataset.root=/home/<user>/lerobot_datas/record/ufactory/xarm7_gello_datas \
--dataset.repo_id=ufactory/xarm6_umi_datas \ --dataset.repo_id=ufactory/xarm7_gello_datas \
--policy.type=act \ --policy.type=act \
--policy.device=cuda \ --policy.device=cuda \
--policy.repo_id=ufactory/xarm6_umi_datas \ --policy.repo_id=ufactory/xarm7_gello_datas \
--output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \ --output_dir=/home/<user>/lerobot_datas/train/xarm7_gello_datas \
--job_name=xarm6_umi_datas \ --job_name=xarm7_gello_datas \
--steps=800000 \ --steps=800000 \
--batch_size=8 \ --batch_size=8 \
--save_freq=20000 --save_freq=20000
``` ```
### 6. 推理 ### 5. 策略推理
指定模型进行推理
```bash ```bash
# 通用格式 uv run uf-lerobot-eval \
uv run uf-lerobot-eval --config_path path/to/config.yaml --policy.path your_train_path --config_path config/gello/xarm7_gello_record_config.yaml \
--policy.path /path/to/train/output/checkpoints/last/pretrained_model/
# 示例：使用训练好的 ACT 策略进行推理
uv run uf-lerobot-eval --config_path config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
``` ```
## 工具集 `←` / `→` 复位，`Esc` 停止。
### 1. 摄像头查看器 ### 6. 回放已录制 episode
查看和拼接多路摄像头画面。 将手动拖拽数据集的绝对关节状态（`observation.state`）回放到 xArm7。脚本按数据集 FPS（默认 30）将状态作为**绝对目标值**发送，不做差分或累加，因此运动轨迹与录制时一致：
```bash ```bash
uv run uf-camera-view -l # 列出所有摄像头 # 默认回放第一条
uv run uf-camera-view -l -T xvisio # 仅列出 XVisio 摄像头 uv run uf-lerobot-replay \
uv run uf-camera-view -T xvisio # 查看 XVisio 摄像头（默认 1280x1280 YU12） --dataset-root /path/to/xarm7_manual_datas \
uv run uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # 指定格式 --robot-ip 192.168.1.245
uv run uf-camera-view -T other # 查看其他类型摄像头
# 跳过交互确认（无人值守）
uv run uf-lerobot-replay --dataset-root /path/to/xarm7_manual_datas --robot-ip 192.168.1.245 --yes
# 回放其他 episode
uv run uf-lerobot-replay --dataset-root /path/to/xarm7_manual_datas --robot-ip 192.168.1.245 --episode-index 3
``` ```
### 2. Lerobot数据集工具 回放开始前机械臂会先移动到 xArm SDK 初始点，播放结束后保持最后一帧姿态并断开连接。执行前请确认工作空间无障碍物，且数据中的初始姿态与当前设备一致。
Lerobot提供一些数据集工具，方便对采集的数据集进行增删查操作。
## 重要：机械臂开关机事项
### 开机
1. **使用网线将电脑连接到机械臂控制器。**
2. **参考控制器上标注的机械臂 IP，将电脑以太网接口配置到同一网段**（例如 `192.168.1.xxx`）。
3. **在浏览器中访问 `http://192.168.1.245:18333/`**，应出现控制台界面。
4. **操作机械臂前，抬起急停按钮。**
### 关机
1. **先通过网页控制将机械臂返回到初始位置。**
2. **再按下急停按钮。**
3. **关闭控制器电源。**
## 工具
### 摄像头查看器
### 查看某个索引的episode:
例如查看索引号为17的episode:
```bash ```bash
uv run uf-camera-view -l # 列出所有摄像头
uv run uf-camera-view -T realsense # 查看 RealSense 摄像头
```
网页预览会扫描所有 RealSense 摄像头，默认使用 `640x480`、`30fps`，页面中可勾选设备切换或同时显示多路画面：
```bash
uv run uf-realsense-view
```
脚本使用 LeRobot 的 `RealSenseCamera`。如果当前环境中的 OpenCV 是 LeRobot 默认的
headless 版本，会自动启动网页预览。同一台机器上打开
`http://127.0.0.1:8765/`；从其他机器访问时，请将 `127.0.0.1` 替换为运行脚本机器的实际 IP。
网页服务默认监听 `0.0.0.0`，也可以用 `--host 127.0.0.1` 限制为本机访问；还可以通过
`--backend opencv` 强制使用 OpenCV 窗口。
也可以只打开指定设备；需要多个设备时重复 `--serial`：
```bash
uv run uf-realsense-view \
--serial 148522072685 --width 640 --height 480 --fps 30
uv run uf-realsense-view \
--serial 148522072685 --serial SECOND_CAMERA_SERIAL
```
OpenCV 窗口按 `q` 或 `Esc` 退出；网页模式按 `Ctrl+C` 退出。无桌面环境时可以用
`--no-display` 检查是否能持续取帧。
### LeRobot 数据集工具
```bash
# 查看索引为 17 的 episode
uv run lerobot-dataset-viz \ uv run lerobot-dataset-viz \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \ --root=/path/to/record/ufactory/xarm7_manual_datas \
--repo-id ufactory/xarm7_record_datas \ --repo-id ufactory/xarm7_manual_datas \
--display-compressed-images true \ --display-compressed-images true \
--episode-index 17 --episode-index 17
```
### 删除某些索引的episodes: # 删除索引为 18 和 19 的 episode
例如删除索引号为18和19的episode:
```bash
uv run lerobot-edit-dataset \ uv run lerobot-edit-dataset \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \ --root=/path/to/record/ufactory/xarm7_manual_datas \
--repo_id ufactory/xarm7_record_datas \ --repo_id ufactory/xarm7_manual_datas \
--new_repo_id ../xarm7_record_datas_new \ --new_repo_id ../xarm7_manual_datas_new \
--operation.type delete_episodes \ --operation.type delete_episodes \
--operation.episode_indices "[18, 19]" --operation.episode_indices "[18, 19]"
```
### 合并数据集 # 合并数据集
```bash
uv run lerobot-edit-dataset \ uv run lerobot-edit-dataset \
--root=../../../../lerobot_datas/record \ --root=/path/to/record \
--repo_id ufactory/xarm7_record_datas_merge_1_2 \ --repo_id ufactory/xarm7_datas_merge \
--operation.type merge \ --operation.type merge \
--operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']" --operation.repo_ids "['ufactory/xarm7_datas_1', 'ufactory/xarm7_datas_2']"
``` ```
## 遥操作方式对比
| 特性 | GELLO | Pika | UMI | SpaceMouse |
|------|-------|------|-----|------------|
| 控制空间 | 关节空间 | 笛卡尔空间 | 笛卡尔空间 | 笛卡尔空间 |
| 跟踪方式 | Dynamixel 舵机 | Vive Tracker | UMI SLAM / Vive | 3D 鼠标 |
| 双臂支持 | ❌ | ❌ | ✅ | ❌ |
| 系统依赖 | dialout 组 | — | XVSDK deb | — |
## 项目结构 ## 项目结构
``` ```
lerobot_robot_ufactory/ lerobot_xarm7/
├── src/ ├── config/
│ ├── lerobot_robot_ufactory/ # LeRobot 插件包 │ ├── gello/ # xArm7 GELLO 录制配置
│ │ ├── robots/ # 机器人控制 │ └── manual_mode/ # xArm7 手动拖拽录制配置
│ │ │ ├── uf_robot/ # xArm 实体机器人 ├── src/lerobot_robot_ufactory/
│ │ │ ├── uf_mock_robot/ # 仿真 Mock 机器人 │ ├── robots/
│ │ ├── teleoperators/ # 遥操作器 │ │ └── uf_robot/ # xArm 控制（关节/笛卡尔空间、示教模式）
│ │ │ ├── base_teleop/ # 共享基类 │ ├── teleoperators/
│ │ │ ├── gello_teleop/ # GELLO (Dynamixel 示教臂) │ │ ├── base_teleop/ # 遥操作基类
│ │ │ ├── pika_teleop/ # Pika Sense (手持示教器 + Vive) │ │ └── gello_teleop/ # GELLO（Dynamixel 示教臂）
│ │ │ ├── umi_teleop/ # UMI (含双机械臂) │ ├── scripts/
│ │ │ └── space_mouse/ # SpaceMouse (3D 鼠标) │ │ ├── uf_robot_teleop.py # 遥操作测试
│ │ ├── cameras/ # 摄像头模块 │ │ ├── uf_lerobot_record.py # 数据采集（含手动模式）
│ │ │ └── umi_camera/ # UMI 相机 │ │ ├── uf_lerobot_eval.py # 策略推理
│ │ ├── devices/ # 外部设备驱动 │ │ ├── uf_lerobot_replay.py # episode 回放
│ │ │ ├── pika/ # Pika 串口驱动 │ │ └── uf_camera_view.py # 摄像头查看器
│ │ │ └── umi/ # XVLib / Vive Tracker │ └── configs/parser.py # 配置加载 / CLI 覆盖
│ │ ├── scripts/ # 执行脚本 ├── start_manual_record.sh # 手动拖拽启动脚本
│ │ │ ├── uf_robot_teleop.py # 遥操作测试
│ │ │ ├── uf_lerobot_record.py # 数据采集
│ │ │ ├── uf_lerobot_eval.py # 策略评估
│ │ │ ├── uf_camera_view.py # 摄像头查看工具
│ │ │ └── vive_calibrate.py # Vive Tracker 校准
│ │ ├── context.py # Teleop 上下文注册
│ │ └── utils/ # 工具函数
├── config/ # YAML 配置文件
│ ├── gello/
│ ├── pika/
│ ├── umi/
│ └── spacemouse/
├── rules/ # udev 设备规则
├── pyproject.toml ├── pyproject.toml
└── README.md ├── README.md
└── README_ZH.md
``` ```
## 重要提示 ## 重要提示
用户需要全面研究整个代码库，并了解相关的配置参数，因为代码中所写的配置并非适用于所有使用场景和设置，所以用户需要研究代码或相关理论，以获取相关知识，并自行进行修改和调整。特别是对于扩散策略(diffusion policy)，LeRobot 中的默认参数可能仅用于模拟，并未针对实际机器人场景进行优化。
- 提供的配置都是**示例**：请根据实际硬件修改 IP、串口、相机序列号、数据集路径和任务描述。
- GELLO 数据采集与推理时，机械臂与相机的相对位姿必须保持一致。
- LeRobot 中扩散策略（Diffusion Policy）的默认参数主要面向仿真，**未针对真实机器人优化**，需要根据任务自行调整。
- 回放或推理前，请确认工作空间无障碍物，并保证机械臂初始姿态与录制数据一致。
## 许可证 ## 许可证

16
backup/README.md Normal file
View File

@ -0,0 +1,16 @@
# Backup
Previous full versions of the project READMEs, kept before the documentation was
rewritten (2026-08) to focus only on the currently used workflows
(GELLO teleop and manual drag).
- `README_old.md` — previous English README (Pika / UMI / SpaceMouse / demo videos, etc.)
- `README_ZH_old.md` — previous Chinese README
The backups are verbatim copies of the files as they existed at commit `b5d46c4`
and can be restored with:
```bash
cp backup/README_old.md README.md
cp backup/README_ZH_old.md README_ZH.md
```

338
backup/README_ZH_old.md Normal file
View File

@ -0,0 +1,338 @@
# UFACTORY LeRobot
> [English Version](README.md)
UFACTORY(深圳市众为创造科技有限公司) 机械臂与 LeRobot 框架集成项目，支持多种遥操作方式的数据采集、策略训练和部署推理。
## 训练推理效果
点击下载开发时[采集的数据集](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY)，**仅供参考，不可复用**。因为用户机械臂和摄像头位置和开发测试时不一致。
<table>
<tr>
<td width="50%">
<a href="https://www.bilibili.com/video/BV12xFjzzEaX" target="_blank">
<img src="https://i2.hdslb.com/bfs/archive/7b325df5fb4c16e922b66d27b56d8fb6534f8b46.jpg" width="100%">
</a>
</td>
<td width="50%">
<a href="https://www.bilibili.com/video/BV16ccizHE2P" target="_blank">
<img src="https://i2.hdslb.com/bfs/archive/8c5d5e9370577fa89d06a175aceea282d9b2eb9a.jpg" width="100%">
</a>
</td>
</tr>
<tr>
<td width="50%">
<a href="https://www.bilibili.com/video/BV1xGEy6mE3i" target="_blank">
<img src="https://i2.hdslb.com/bfs/archive/c8bbac04a736b7043a20b753e11afea7627bdae2.jpg" width="100%">
</a>
</td>
</tr>
</table>
## 功能特性
- 🤖 UFACTORY 机械臂控制（[xArm 系列](https://www.ufactory.cc/)）
- 🎮 多种遥操作方式：GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/)
- 📷 多摄像头数据采集（[RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI 相机）
- 📊 数据集录制与管理（兼容 LeRobot 格式）
- 🧠 模仿学习训练（ACT / Diffusion Policy 等）
- 🚀 策略评估与实时推理
- 🔧 Mock 机器人模拟（只用遥操作设备采集数据）
## 环境要求
- Ubuntu 22.04 / 24.04
- Python >= 3.10
- CUDA >= 12.0（GPU 训练推荐）
- UFACTORY 机械臂（xArm 系列，可选）
## 安装
### 基础项目安装
```bash
git clone https://github.com/xArm-Developer/lerobot_robot_ufactory.git
cd lerobot_robot_ufactory
# 创建 uv 虚拟环境并同步项目依赖
uv venv --python 3.10
uv sync
```
包含：`lerobot==0.4.3`、`xarm-python-sdk`、`numpy`、`pyyaml`（lerobot 已自动携带 torch、opencv、wandb 等训练相关依赖）。
### 外设模块安装
外设依赖以可选模块形式提供，通过 `[模块名]` 安装。
#### GELLO 遥操作
适用于 GELLO 示教臂（Dynamixel 舵机方案），控制空间为关节空间。
* 一旦开始数据采集，机械臂与摄像头（D435 / D435i）的**相对位置必须保持不变**。
* 推理时的摄像头位置必须与采集时相同。若机械臂或摄像头发生变化，此前采集的数据将无效。
```bash
# 1. 安装 GELLO 模块
uv sync --extra gello
# 2. 添加串口权限（重新登录后生效）
sudo usermod -aG dialout $USER
```
#### Pika 遥操作
适用于 Pika Sense 手持示教器 + Vive Tracker，控制空间为笛卡尔空间。
* 两个基站和机械臂相对位置没有要求，只需要保证采集时pika sense在基站范围内，但**基站移动后需要重新校准**。
* 采集和推理时基站位置可不相同。
```bash
# 1. 安装外设依赖（不需要它们的间接依赖）
uv pip install pysurvive agx-pypika --no-deps
# 2. 安装 udev 规则（重新插拔设备后生效）
sudo cp rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
```
> Vive Tracker 首次使用前需校准：`uf-vive-calibrate`
#### UMI 遥操作
适用于 UMI（Universal Manipulation Interface）方案，含 Vive Tracker 追踪，支持双机械臂。
```bash
# 1. 安装 XVSDK（系统级依赖，仅支持 Ubuntu Focal）
curl -sL https://raw.githubusercontent.com/xArm-Developer/ufactory_resources/main/fastumi/sdk/XVSDK_focal_amd64.deb -o /tmp/xvsdk.deb && sudo dpkg -i /tmp/xvsdk.deb
sudo apt install -y --fix-broken
# 2. 安装外设依赖
uv pip install pysurvive --no-deps
# 3. 安装 udev 规则（重新插拔设备后生效）
sudo cp rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
```
> Vive Tracker 首次使用前需校准：`uf-vive-calibrate`
**多 UMI 设备配置**（使用两台及以上时）：
```bash
# 增加 USB 缓冲区大小
sudo sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/s/quiet splash/quiet splash usbcore.usbfs_memory_mb=128/' /etc/default/grub
sync
sudo update-grub
sudo reboot
```
#### SpaceMouse 遥操作
适用于 3Dconnexion SpaceMouse / SpaceNavigator。
```bash
# 1. 安装 SpaceMouse 模块
uv sync --extra spacemouse
# 2. 安装 udev 规则（重新插拔设备后生效）
sudo cp rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
```
## 使用
### 1. 遥操作测试
测试遥操作设备与机械臂的联动，不录制数据。
```bash
# 通用格式
uv run uf-robot-teleop --config_path path/to/config.yaml
uv run uf-robot-teleop --config_path path/to/config.yaml --fps 60 # 指定频率
# 示例: xArm6 + UMI 遥操作
uv run uf-robot-teleop --config_path config/umi/xarm6_umi_record_config.yaml
```
### 2. 人工拖拽数据采集
人工拖拽录制使用 robot 配置中的 `manual_mode: true`，不需要配置 teleop。录制过程中，机械臂的实际关节状态会作为 observation 和 action 写入 LeRobot 数据集；如果配置了夹爪，还可以按住 `C` 缓慢闭合、按住 `O` 缓慢张开。夹爪速度通过 `manual_gripper_speed` 配置，默认值为 `0.5`：
```bash
./start_manual_record.sh
./start_manual_record.sh -r
```
启动脚本首次运行时创建数据集，检测到有效数据集时会自动续录。如果提示数据目录为空或不完整，请修改 `dataset.root`，或者确认没有数据后删除该目录。按 `Space` 开始当前 episode，按 `Right` 保存，按 `Left` 放弃并重录当前 episode，按 `Esc` 停止录制。episode 之间可以手动复位机械臂。
### 3. 遥操作数据采集
通过遥操作录制数据集。
```bash
# 通用格式
uv run uf-lerobot-record --config_path path/to/record_config.yaml
uv run uf-lerobot-record --config_path path/to/config.yaml --resume true # 续录
# 示例: xArm6 + UMI 数据采集
uv run uf-lerobot-record --config_path config/umi/xarm6_umi_record_config.yaml
```
### 4. 数据重放
`datasets/xarm7_manual_replay` 是人工拖拽录制的 LeRobot 数据集。回放脚本使用其中的
`observation.state`，将 7 个关节弧度值和归一化夹爪位置作为**绝对目标值**发送给 xArm7，
不会将相邻帧相减，也不会累加成相对动作。脚本默认按数据集的 30 FPS 播放一个 episode。
启动前会要求确认，连接后会先自动移动到 xArm SDK 初始点；播放结束后保持最后一帧姿态并断开连接：
```bash
uv run uf-lerobot-replay \
--dataset-root /home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay \
--robot-ip 192.168.1.245
```
无人值守运行时可以使用 `--yes` 跳过确认。执行前请确认机械臂工作空间无障碍物，且数据中的初始姿态与当前设备匹配。
### 5. Lerobot训练
采集数据后，使用 LeRobot 训练管道进行模仿学习训练。
```bash
# 通用格式
uv run lerobot-train --policy act --dataset your_dataset_name
```
参数示例：
```bash
# 注意: repo_id就是采集时配置文件里面的repo_id
# 这里训练策略policy.type选用act，训练steps为80w次
# 训练过程每2w次保存一次结果，结果输出到和lerobot同级目录下的lerobot_datas/train里面
uv run lerobot-train \
--dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \
--dataset.repo_id=ufactory/xarm6_umi_datas \
--policy.type=act \
--policy.device=cuda \
--policy.repo_id=ufactory/xarm6_umi_datas \
--output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \
--job_name=xarm6_umi_datas \
--steps=800000 \
--batch_size=8 \
--save_freq=20000
```
### 6. 推理
指定模型进行推理
```bash
# 通用格式
uv run uf-lerobot-eval --config_path path/to/config.yaml --policy.path your_train_path
# 示例：使用训练好的 ACT 策略进行推理
uv run uf-lerobot-eval --config_path config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
```
## 工具集
### 1. 摄像头查看器
查看和拼接多路摄像头画面。
```bash
uv run uf-camera-view -l # 列出所有摄像头
uv run uf-camera-view -l -T xvisio # 仅列出 XVisio 摄像头
uv run uf-camera-view -T xvisio # 查看 XVisio 摄像头（默认 1280x1280 YU12）
uv run uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # 指定格式
uv run uf-camera-view -T other # 查看其他类型摄像头
```
### 2. Lerobot数据集工具
Lerobot提供一些数据集工具，方便对采集的数据集进行增删查操作。
### 查看某个索引的episode:
例如查看索引号为17的episode:
```bash
uv run lerobot-dataset-viz \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
--repo-id ufactory/xarm7_record_datas \
--display-compressed-images true \
--episode-index 17
```
### 删除某些索引的episodes:
例如删除索引号为18和19的episode:
```bash
uv run lerobot-edit-dataset \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
--repo_id ufactory/xarm7_record_datas \
--new_repo_id ../xarm7_record_datas_new \
--operation.type delete_episodes \
--operation.episode_indices "[18, 19]"
```
### 合并数据集
```bash
uv run lerobot-edit-dataset \
--root=../../../../lerobot_datas/record \
--repo_id ufactory/xarm7_record_datas_merge_1_2 \
--operation.type merge \
--operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']"
```
## 遥操作方式对比
| 特性 | GELLO | Pika | UMI | SpaceMouse |
|------|-------|------|-----|------------|
| 控制空间 | 关节空间 | 笛卡尔空间 | 笛卡尔空间 | 笛卡尔空间 |
| 跟踪方式 | Dynamixel 舵机 | Vive Tracker | UMI SLAM / Vive | 3D 鼠标 |
| 双臂支持 | ❌ | ❌ | ✅ | ❌ |
| 系统依赖 | dialout 组 | — | XVSDK deb | — |
## 项目结构
```
lerobot_robot_ufactory/
├── src/
│ ├── lerobot_robot_ufactory/ # LeRobot 插件包
│ │ ├── robots/ # 机器人控制
│ │ │ ├── uf_robot/ # xArm 实体机器人
│ │ │ ├── uf_mock_robot/ # 仿真 Mock 机器人
│ │ ├── teleoperators/ # 遥操作器
│ │ │ ├── base_teleop/ # 共享基类
│ │ │ ├── gello_teleop/ # GELLO (Dynamixel 示教臂)
│ │ │ ├── pika_teleop/ # Pika Sense (手持示教器 + Vive)
│ │ │ ├── umi_teleop/ # UMI (含双机械臂)
│ │ │ └── space_mouse/ # SpaceMouse (3D 鼠标)
│ │ ├── cameras/ # 摄像头模块
│ │ │ └── umi_camera/ # UMI 相机
│ │ ├── devices/ # 外部设备驱动
│ │ │ ├── pika/ # Pika 串口驱动
│ │ │ └── umi/ # XVLib / Vive Tracker
│ │ ├── scripts/ # 执行脚本
│ │ │ ├── uf_robot_teleop.py # 遥操作测试
│ │ │ ├── uf_lerobot_record.py # 数据采集
│ │ │ ├── uf_lerobot_eval.py # 策略评估
│ │ │ ├── uf_camera_view.py # 摄像头查看工具
│ │ │ └── vive_calibrate.py # Vive Tracker 校准
│ │ ├── context.py # Teleop 上下文注册
│ │ └── utils/ # 工具函数
├── config/ # YAML 配置文件
│ ├── gello/
│ ├── pika/
│ ├── umi/
│ └── spacemouse/
├── rules/ # udev 设备规则
├── pyproject.toml
└── README.md
```
## 重要提示
用户需要全面研究整个代码库，并了解相关的配置参数，因为代码中所写的配置并非适用于所有使用场景和设置，所以用户需要研究代码或相关理论，以获取相关知识，并自行进行修改和调整。特别是对于扩散策略(diffusion policy)，LeRobot 中的默认参数可能仅用于模拟，并未针对实际机器人场景进行优化。
## 许可证
本项目基于 Apache License 2.0 发布，详见 [LICENSE](LICENSE) 文件。

329
backup/README_old.md Normal file
View File

@ -0,0 +1,329 @@
# UFACTORY LeRobot
> [中文版本](README_ZH.md)
UFACTORY robot arm integration with the LeRobot framework for robot learning, data collection, and policy deployment.
## Training & Inference Results
[Test datasets](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY) used during development, **for reference only, do NOT reuse**. As the robot arm and camera positions during development differ from user setups.
<table>
<tr>
<td width="50%">
<a href="https://www.youtube.com/watch?v=wTiWLiHciT8" target="_blank">
<img src="https://img.youtube.com/vi/wTiWLiHciT8/maxresdefault.jpg" width="100%">
</a>
</td>
<td width="50%">
<a href="https://www.youtube.com/watch?v=IiyvewZh5OY" target="_blank">
<img src="https://img.youtube.com/vi/IiyvewZh5OY/maxresdefault.jpg" width="100%">
</a>
</td>
</tr>
<tr>
<td width="50%">
<a href="https://youtu.be/wBwZH6POk38" target="_blank">
<img src="https://img.youtube.com/vi/wBwZH6POk38/maxresdefault.jpg" width="100%">
</a>
</td>
</tr>
</table>
## Features
- 🤖 UFACTORY robot control ([xArm series](https://www.ufactory.cc/))
- 🎮 Multiple teleop modes: GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/)
- 📷 Multi-camera data collection ([RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI camera)
- 📊 Dataset recording & management (LeRobot-compatible)
- 🧠 Imitation learning training (ACT / Diffusion Policy / etc.)
- 🚀 Policy evaluation & real-time inference
- 🔧 Mock mode (teleop device only, no physical robot needed)
## Requirements
- Ubuntu 22.04 / 24.04
- Python >= 3.10
- CUDA >= 12.0 (recommended for GPU training)
- UFACTORY xArm (optional)
## Installation
### Base Install
```bash
git clone https://github.com/xArm-Developer/lerobot_robot_ufactory.git
cd lerobot_robot_ufactory
# Create a uv virtual environment and sync project dependencies
uv venv --python 3.10
uv sync
```
Includes: `lerobot==0.4.3`, `xarm-python-sdk`, `numpy`, `pyyaml`. LeRobot already pulls in torch, opencv, wandb, etc.
### Peripheral Modules
Peripheral dependencies are available as optional extras via `[module]` install.
#### GELLO Teleop
Dynamixel-based leader arm, joint-space control.
* Once data collection starts, the **relative position** between the robot arm and camera (D435 / D435i) **must remain unchanged**.
* The camera position during inference must match the collection setup. If the robot arm or camera changes, previously collected data becomes invalid.
```bash
# 1. Install GELLO module
uv sync --extra gello
# 2. Add serial port permissions (re-login required)
sudo usermod -aG dialout $USER
```
#### Pika Teleop
Pika Sense handheld + Vive Tracker, Cartesian-space control.
* No requirement for the relative position of the two base stations and the robot arm. Only need to ensure the Pika Sense is within base station range during collection, but **base stations must be recalibrated after moving**.
* Base station positions for collection and inference do not need to be the same.
```bash
# 1. Install peripheral deps (skip transitive deps)
uv pip install pysurvive agx-pypika --no-deps
# 2. Install udev rules (re-plug devices afterwards)
sudo cp src/rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
```
> Calibrate Vive Tracker before first use: `uf-vive-calibrate`
#### UMI Teleop
Universal Manipulation Interface + Vive Tracker, supports dual-arm.
```bash
# 1. Install XVSDK (system-level, Ubuntu Focal only)
curl -sL https://raw.githubusercontent.com/xArm-Developer/ufactory_resources/main/fastumi/sdk/XVSDK_focal_amd64.deb -o /tmp/xvsdk.deb && sudo dpkg -i /tmp/xvsdk.deb
sudo apt install -y --fix-broken
# 2. Install peripheral deps
uv pip install pysurvive --no-deps
# 3. Install udev rules (re-plug devices afterwards)
sudo cp src/rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
```
> Calibrate Vive Tracker before first use: `uf-vive-calibrate`
**Multi-UMI device configuration** (two or more devices):
```bash
# Increase USB buffer size
sudo sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/s/quiet splash/quiet splash usbcore.usbfs_memory_mb=128/' /etc/default/grub
sync
sudo update-grub
sudo reboot
```
#### SpaceMouse Teleop
3Dconnexion SpaceMouse / SpaceNavigator.
```bash
# 1. Install SpaceMouse module
uv sync --extra spacemouse
# 2. Install udev rules (re-plug device afterwards)
sudo cp src/rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
```
## Usage
### 1. Teleop Testing
Test teleop-to-robot control loop without recording.
```bash
# Generic usage
uv run uf-robot-teleop --config_path path/to/config.yaml
uv run uf-robot-teleop --config_path path/to/config.yaml --fps 60 # specify frequency
# Example: xArm6 + UMI teleop
uv run uf-robot-teleop --config_path config/umi/xarm6_umi_record_config.yaml
```
### 2. Manual Drag Data Collection
Manual drag recording uses `manual_mode: true` in the robot configuration and does not configure a teleoperator. During recording, the actual joint state is written as both the observation and action in the LeRobot dataset. When a gripper is configured, hold `C` to close it slowly and `O` to open it slowly. Adjust the speed with `manual_gripper_speed`, which defaults to `0.5`:
```bash
./start_manual_record.sh
./start_manual_record.sh -r
```
The launcher creates a dataset on its first run and automatically resumes an existing valid dataset. If it reports an empty or incomplete dataset directory, choose a new `dataset.root` or remove that directory after confirming it contains no data. Press `Space` to start an episode, `Right` to save it, `Left` to discard and re-record it, and `Esc` to stop recording. Reset the arm manually between episodes.
### 3. Teleop Data Collection
Record datasets via teleop.
```bash
# Generic usage
uv run uf-lerobot-record --config_path path/to/record_config.yaml
uv run uf-lerobot-record --config_path path/to/config.yaml --resume true # resume recording
# Example: xArm6 + UMI data collection
uv run uf-lerobot-record --config_path config/umi/xarm6_umi_record_config.yaml
```
### 5. Policy Training
Train imitation learning policies on collected data.
```bash
# Generic usage
uv run lerobot-train --policy act --dataset your_dataset_name
# Example: train ACT on xArm6 UMI dataset
uv run lerobot-train --policy act --dataset ufactory/xarm6_umi_datas
```
Important parameters:
```bash
# Note: repo_id is the same as in the record config
# Policy type: ACT, training steps: 800k
# Checkpoints saved every 20k steps, output to lerobot_datas/train (sibling of lerobot directory)
uv run lerobot-train \
--dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \
--dataset.repo_id=ufactory/xarm6_umi_datas \
--policy.type=act \
--policy.device=cuda \
--policy.repo_id=ufactory/xarm6_umi_datas \
--output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \
--job_name=xarm6_umi_datas \
--steps=800000 \
--batch_size=8 \
--save_freq=20000
```
### 6. Inference & Evaluation
Run inference with a trained policy.
```bash
# Generic usage
uv run uf-lerobot-eval --config_path path/to/config.yaml --policy.path your_train_path
# Example: run inference with trained ACT policy
uv run uf-lerobot-eval --config_path config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
```
## Tools
### 1. Camera Viewer
View and stitch multiple camera feeds.
```bash
uv run uf-camera-view -l # list all cameras
uv run uf-camera-view -l -T xvisio # list XVisio cameras only
uv run uf-camera-view -T xvisio # view XVisio cameras (default 1280x1280 YU12)
uv run uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # specify format
uv run uf-camera-view -T other # view other camera types
```
### 2. LeRobot Dataset Tools
LeRobot provides dataset utilities for inspecting, editing and managing collected datasets.
#### View an episode:
e.g. view episode index 17:
```bash
uv run lerobot-dataset-viz \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
--repo-id ufactory/xarm7_record_datas \
--display-compressed-images true \
--episode-index 17
```
#### Delete specific episodes:
e.g. delete episodes 18 and 19:
```bash
uv run lerobot-edit-dataset \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
--repo_id ufactory/xarm7_record_datas \
--new_repo_id ../xarm7_record_datas_new \
--operation.type delete_episodes \
--operation.episode_indices "[18, 19]"
```
#### Merge datasets:
```bash
uv run lerobot-edit-dataset \
--root=../../../../lerobot_datas/record \
--repo_id ufactory/xarm7_record_datas_merge_1_2 \
--operation.type merge \
--operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']"
```
## Teleop Comparison
| Feature | GELLO | Pika | UMI | SpaceMouse |
|---------|-------|------|-----|------------|
| Control space | Joint space | Cartesian space | Cartesian space | Cartesian space |
| Tracking | Dynamixel servos | Vive Tracker | UMI SLAM / Vive | 3D mouse |
| Dual-arm | ❌ | ❌ | ✅ | ❌ |
| System dep | dialout group | — | XVSDK deb | — |
## Project Structure
```
lerobot_robot_ufactory/
├── src/
│ ├── lerobot_robot_ufactory/ # LeRobot plugin package
│ │ ├── robots/ # Robot control
│ │ │ ├── uf_robot/ # xArm physical robot
│ │ │ ├── uf_mock_robot/ # Mock robot simulator
│ │ ├── teleoperators/ # Teleop drivers
│ │ │ ├── base_teleop/ # Shared base class
│ │ │ ├── gello_teleop/ # GELLO (Dynamixel leader)
│ │ │ ├── pika_teleop/ # Pika Sense (handheld + Vive)
│ │ │ ├── umi_teleop/ # UMI (dual-arm support)
│ │ │ └── space_mouse/ # SpaceMouse (3D mouse)
│ │ ├── cameras/ # Camera modules
│ │ │ └── umi_camera/ # UMI camera
│ │ ├── devices/ # External device drivers
│ │ │ ├── pika/ # Pika serial driver
│ │ │ └── umi/ # XVLib / Vive Tracker
│ │ ├── scripts/ # Entry-point scripts
│ │ │ ├── uf_robot_teleop.py # Teleop testing
│ │ │ ├── uf_lerobot_record.py # Data recording
│ │ │ ├── uf_lerobot_eval.py # Policy evaluation
│ │ │ ├── uf_camera_view.py # Camera viewer tool
│ │ │ └── vive_calibrate.py # Vive Tracker calibration
│ │ ├── context.py # Teleop context registry
│ │ └── utils/ # Utilities
├── config/ # YAML config files
│ ├── gello/
│ ├── pika/
│ ├── umi/
│ └── spacemouse/
├── rules/ # udev device rules
├── pyproject.toml
└── README.md
```
## Important Notes
Users are expected to thoroughly study the codebase and configuration parameters.
The provided configurations are **not guaranteed to work for all scenarios** and must be adjusted based on actual hardware setups and task requirements.
In particular, for **diffusion policies**, the default parameters in LeRobot are primarily designed for simulation and **are not optimized for real-world robots**.
## License
This project is released under the Apache License 2.0. See [LICENSE](LICENSE).

6
config/manual_mode/xarm7_manual_record_config.yaml
View File

@ -24,10 +24,10 @@ robot:
fps: 30 fps: 30
dataset: dataset:
root: "/home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay_pick_pen" root: "/home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay_pick_bottle"
repo_id: "ufactory/xarm7_manual_datas" repo_id: "zjuscl/xarm7_manual_replay_pick_bottle"
# Task description stored with each recorded frame. # Task description stored with each recorded frame.
single_task: "Describe the task being demonstrated." single_task: "Pick up the black bottle and place it on the blue bag"
fps: 30 fps: 30
episode_time_s: 60 episode_time_s: 60
# Reserved reset duration between episodes in seconds. # Reserved reset duration between episodes in seconds.

1
pyproject.toml
View File

@ -39,6 +39,7 @@ uf-lerobot-eval = "lerobot_robot_ufactory.scripts.uf_lerobot_eval:main"
uf-lerobot-replay = "lerobot_robot_ufactory.scripts.uf_lerobot_replay:main" uf-lerobot-replay = "lerobot_robot_ufactory.scripts.uf_lerobot_replay:main"
uf-vive-calibrate = "lerobot_robot_ufactory.scripts.vive_calibrate:main" uf-vive-calibrate = "lerobot_robot_ufactory.scripts.vive_calibrate:main"
uf-camera-view = "lerobot_robot_ufactory.scripts.uf_camera_view:main" uf-camera-view = "lerobot_robot_ufactory.scripts.uf_camera_view:main"
uf-realsense-view = "lerobot_robot_ufactory.scripts.uf_realsense_view:main"
uf-camera-test = "lerobot_robot_ufactory.scripts.uf_camera_test:main" uf-camera-test = "lerobot_robot_ufactory.scripts.uf_camera_test:main"
[project.optional-dependencies] [project.optional-dependencies]

4
replay_trajectory.sh Executable file
View File

@ -0,0 +1,4 @@
#!/bin/bash
uv run uf-lerobot-replay \
--dataset-root /home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay_pick_bottle \
--robot-ip 192.168.1.245

6
src/lerobot_robot_ufactory/robots/uf_robot/uf_robot.py
View File

@ -449,9 +449,9 @@ class UFRobot(Robot, Thread):
warn_code = getattr(arm, "warn_code", "unknown") warn_code = getattr(arm, "warn_code", "unknown")
return f"mode={mode}, state={state}, error_code={error_code}, warn_code={warn_code}" return f"mode={mode}, state={state}, error_code={error_code}, warn_code={warn_code}"
def _check_motion_code(self, command: str, code: int) -> None: def _check_motion_code(self, command: str, code: int | None) -> None:
"""Fail loudly when the SDK rejects a joint command.""" """Fail loudly when the SDK explicitly rejects a motion command."""
if code != 0: if code is not None and code != 0:
raise RuntimeError(f"{command} failed, code={code}, {self._motion_status()}") raise RuntimeError(f"{command} failed, code={code}, {self._motion_status()}")
def send_action(self, action: dict) -> np.ndarray: def send_action(self, action: dict) -> np.ndarray:

29
src/lerobot_robot_ufactory/scripts/uf_lerobot_eval.py
View File

@ -38,6 +38,7 @@ from lerobot.configs.policies import PreTrainedConfig
from lerobot.scripts.lerobot_record import DatasetRecordConfig from lerobot.scripts.lerobot_record import DatasetRecordConfig
from lerobot.datasets.lerobot_dataset import LeRobotDataset from lerobot.datasets.lerobot_dataset import LeRobotDataset
from lerobot_robot_ufactory.utils.utils import init_keyboard_listener from lerobot_robot_ufactory.utils.utils import init_keyboard_listener
from lerobot_robot_ufactory.utils.action_safety import ActionSafetyConfig, ActionSafetyGuard
from lerobot_robot_ufactory.devices.umi.vive_tracker.transformations import Transformations from lerobot_robot_ufactory.devices.umi.vive_tracker.transformations import Transformations
@ -120,9 +121,11 @@ class EvalConfig:
return ["policy"] return ["policy"]
def eval_loop(cfg: EvalConfig, relative=False, rx_continuous=False): def eval_loop(cfg: EvalConfig, relative=False, rx_continuous=False, safety_guard: ActionSafetyGuard | None = None):
init_logging() init_logging()
logging.info(pformat(asdict(cfg))) logging.info(pformat(asdict(cfg)))
if safety_guard is not None:
safety_guard.log_config()
robot = make_robot_from_config(cfg.robot) robot = make_robot_from_config(cfg.robot)
@ -261,6 +264,8 @@ def eval_loop(cfg: EvalConfig, relative=False, rx_continuous=False):
prev_robot_dict[key] = {'type': 1 if is_tcp else 0, 'pose': np.array(pose)} prev_robot_dict[key] = {'type': 1 if is_tcp else 0, 'pose': np.array(pose)}
prev_action_dict[key] = {'type': 1 if is_tcp else 0, 'pose': np.array(pose)} prev_action_dict[key] = {'type': 1 if is_tcp else 0, 'pose': np.array(pose)}
safety_halted = False
while True: while True:
start_loop_t = time.perf_counter() start_loop_t = time.perf_counter()
@ -269,6 +274,12 @@ def eval_loop(cfg: EvalConfig, relative=False, rx_continuous=False):
print("\n********** Policy Eval Episode (Reset) **********") print("\n********** Policy Eval Episode (Reset) **********")
break break
# Safety halt: stop inference and stop sending actions,
# wait for the operator to press the right arrow key to resume
if safety_halted:
precise_sleep(sleep_time_s)
continue
# Get robot observation # Get robot observation
obs = robot.get_observation() obs = robot.get_observation()
@ -398,6 +409,16 @@ def eval_loop(cfg: EvalConfig, relative=False, rx_continuous=False):
# gripper_raw = (future_gripper_norm + 1) / 2 * (_gripper_max - _gripper_min) + _gripper_min # gripper_raw = (future_gripper_norm + 1) / 2 * (_gripper_max - _gripper_min) + _gripper_min
# robot_action_to_send['right.gripper.pos'] = 1.0 if gripper_raw > 0.4 else 0.0 # robot_action_to_send['right.gripper.pos'] = 1.0 if gripper_raw > 0.4 else 0.0
# Safety check: any violation triggers an e-stop; the action is
# NOT sent and the loop waits for the operator to press right arrow
if safety_guard is not None:
violation = safety_guard.check(robot_action_to_send, curr_robot_dict, keys)
if violation is not None:
safety_halted = True
logging.error(f"*** SAFETY HALT *** {violation}")
print(f"\n*** SAFETY HALT *** {violation}\nAction was NOT sent. Press right arrow (->) to reset and resume, ESC to exit.")
continue
robot.send_action(robot_action_to_send) robot.send_action(robot_action_to_send)
dt_s = time.perf_counter() - start_loop_t dt_s = time.perf_counter() - start_loop_t
@ -418,11 +439,15 @@ def main():
parser = argparse.ArgumentParser(description='configuration args') parser = argparse.ArgumentParser(description='configuration args')
parser.add_argument('--relative', action='store_true', help='is relative motion or not') parser.add_argument('--relative', action='store_true', help='is relative motion or not')
parser.add_argument('--rx_continuous', action='store_true', help='rx continuous or not') parser.add_argument('--rx_continuous', action='store_true', help='rx continuous or not')
parser.add_argument('--enable_safety', action='store_true', help='enable action safety guard (thresholds in ActionSafetyConfig)')
args, unknown = parser.parse_known_args() args, unknown = parser.parse_known_args()
sys.argv = [sys.argv[0]] + unknown sys.argv = [sys.argv[0]] + unknown
register_third_party_plugins() register_third_party_plugins()
cfg = get_cfg() cfg = get_cfg()
eval_loop(cfg, args.relative, args.rx_continuous) # Action safety guard: tune thresholds in ActionSafetyConfig directly.
# Any violation triggers an e-stop; press right arrow to resume.
safety_guard = ActionSafetyGuard(ActionSafetyConfig(enabled=args.enable_safety))
eval_loop(cfg, args.relative, args.rx_continuous, safety_guard)
if __name__ == "__main__": if __name__ == "__main__":

90
src/lerobot_robot_ufactory/scripts/uf_lerobot_record.py
View File

@ -493,12 +493,102 @@ def _print_record_controls(is_recorded, manual_mode):
print(f'⌨ {controls}') print(f'⌨ {controls}')
def _ask_choice(prompt: str, options: dict[str, str]) -> str:
"""Prompt the user to pick one of the given options (lowercase keys)."""
keys = "/".join(options)
while True:
print(f"\n{prompt}")
for key, description in options.items():
print(f" [{key}] {description}")
try:
choice = input(f"Choose [{keys}]: ").strip().lower()
except EOFError:
print("No input available, cancelling.")
raise SystemExit(1)
if choice in options:
return choice
print(f"Invalid choice, please enter {keys}.")
def _missing_dataset_files(root: Path) -> list[str]:
"""Return the local files required before a dataset can be resumed."""
missing = []
for relative_path in ("meta/info.json", "meta/tasks.parquet"):
if not (root / relative_path).is_file():
missing.append(relative_path)
if not any((root / "meta" / "episodes").glob("*/*.parquet")):
missing.append("meta/episodes/*/*.parquet")
if not any((root / "data").glob("*/*.parquet")):
missing.append("data/*/*.parquet")
return missing
def _prepare_dataset_root(cfg: UFRecordConfig) -> None:
"""Prepare an existing dataset root without pre-creating a new one."""
root = Path(cfg.dataset.root)
existed = root.exists()
if not existed:
if cfg.resume:
raise RuntimeError(f"Cannot resume because the dataset directory does not exist: {root}")
return
missing = _missing_dataset_files(root)
if missing:
missing_text = ", ".join(missing)
message = (
f"Dataset directory is incomplete and cannot be resumed: {root}\n"
f"Missing: {missing_text}"
)
if cfg.resume or not sys.stdin.isatty():
raise RuntimeError(message)
choice = _ask_choice(
message,
options={
"o": "Overwrite: remove this directory and record a new dataset",
"c": "Cancel",
},
)
if choice == "o":
shutil.rmtree(root)
else:
raise SystemExit("Recording cancelled.")
return
if cfg.resume:
return
# A valid LeRobot dataset already exists.
if not sys.stdin.isatty():
# Non-interactive run: keep the previous auto-resume behaviour.
cfg.resume = True
print(f"Existing dataset found, resuming recording (non-interactive): {root}")
return
choice = _ask_choice(
f"Dataset directory already exists: {root}",
options={
"o": "Overwrite: delete the existing dataset and record a new one",
"r": "Resume: keep existing episodes and continue recording",
"c": "Cancel",
},
)
if choice == "o":
shutil.rmtree(root)
elif choice == "r":
cfg.resume = True
else:
raise SystemExit("Recording cancelled.")
def record(cfg: UFRecordConfig, async_save: bool = False) -> LeRobotDataset: def record(cfg: UFRecordConfig, async_save: bool = False) -> LeRobotDataset:
init_logging() init_logging()
logging.info(pformat(asdict(cfg))) logging.info(pformat(asdict(cfg)))
if cfg.display_data: if cfg.display_data:
init_rerun(session_name="recording") init_rerun(session_name="recording")
_prepare_dataset_root(cfg)
robot = make_robot_from_config(cfg.robot) robot = make_robot_from_config(cfg.robot)
teleop = make_teleoperator_from_config(cfg.teleop) if cfg.teleop is not None else None teleop = make_teleoperator_from_config(cfg.teleop) if cfg.teleop is not None else None
manual_mode = bool(getattr(cfg.robot, "manual_mode", False)) manual_mode = bool(getattr(cfg.robot, "manual_mode", False))

570
src/lerobot_robot_ufactory/scripts/uf_realsense_view.py Normal file
View File

@ -0,0 +1,570 @@
#!/usr/bin/env python3
"""Web viewer for one or more Intel RealSense color streams."""
from __future__ import annotations
import argparse
import http.server
import json
import threading
import time
from dataclasses import dataclass
from typing import Any
from urllib.parse import parse_qs, urlsplit
import cv2
import pyrealsense2 as rs
from lerobot.cameras import ColorMode
from lerobot.cameras.realsense import RealSenseCamera, RealSenseCameraConfig
DEFAULT_SERIAL = None
DEFAULT_WIDTH = 640
DEFAULT_HEIGHT = 480
DEFAULT_FPS = 30
DEFAULT_HOST = "0.0.0.0"
DEFAULT_PORT = 8765
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Preview one or more Intel RealSense color streams in a browser"
)
parser.add_argument(
"-s",
"--serial",
action="append",
default=DEFAULT_SERIAL,
help="camera serial number; repeat to limit the device list to selected cameras",
)
parser.add_argument("-W", "--width", type=int, default=DEFAULT_WIDTH, help="color width")
parser.add_argument("-H", "--height", type=int, default=DEFAULT_HEIGHT, help="color height")
parser.add_argument(
"-F", "--fps", type=int, default=DEFAULT_FPS, help="capture and preview FPS (default: 30)"
)
parser.add_argument(
"--backend",
choices=("auto", "opencv", "web"),
default="web",
help="preview backend (default: web; web provides device selection)",
)
parser.add_argument(
"--host",
default=DEFAULT_HOST,
help="web server bind address (default: 0.0.0.0; use 127.0.0.1 for local-only access)",
)
parser.add_argument("--port", type=int, default=DEFAULT_PORT, help="web server port")
parser.add_argument(
"--no-display",
action="store_true",
help="capture frames without opening a preview",
)
return parser.parse_args(argv)
def opencv_has_gui() -> bool:
return next(
("NONE" not in line for line in cv2.getBuildInformation().splitlines() if "GUI:" in line),
False,
)
@dataclass(frozen=True)
class RealSenseDevice:
serial: str
name: str
product_line: str
def discover_devices() -> list[RealSenseDevice]:
"""Discover connected RealSense devices without opening their streams."""
devices: list[RealSenseDevice] = []
context = rs.context()
for device in context.query_devices():
devices.append(
RealSenseDevice(
serial=device.get_info(rs.camera_info.serial_number),
name=device.get_info(rs.camera_info.name),
product_line=device.get_info(rs.camera_info.product_line),
)
)
return devices
class _CameraFeed:
"""Own one camera connection and publish its latest JPEG frame."""
def __init__(self, device: RealSenseDevice, width: int, height: int, fps: int) -> None:
self.device = device
self.width = width
self.height = height
self.fps = fps
self.camera = RealSenseCamera(
RealSenseCameraConfig(
serial_number_or_name=device.serial,
width=width,
height=height,
fps=fps,
color_mode=ColorMode.BGR,
)
)
self.condition = threading.Condition()
self.jpeg: bytes | None = None
self.frame: Any | None = None
self.frame_id = 0
self.frame_count = 0
self.error: str | None = None
self.connected = False
self.stopped = False
self._stop_event = threading.Event()
self._thread: threading.Thread | None = None
def connect(self) -> None:
self.camera.connect()
self.connected = True
self._thread = threading.Thread(
target=self._capture_loop,
name=f"realsense-{self.device.serial}",
daemon=True,
)
self._thread.start()
def _set_error(self, error: Exception) -> None:
with self.condition:
self.error = f"{type(error).__name__}: {error}"
self.condition.notify_all()
def _capture_loop(self) -> None:
period = 1.0 / self.fps
next_deadline = time.monotonic()
while not self._stop_event.is_set():
try:
frame = self.camera.read()
encoded, buffer = cv2.imencode(".jpg", frame)
if not encoded:
raise RuntimeError("JPEG encoding failed")
with self.condition:
self.frame = frame
self.jpeg = buffer.tobytes()
self.frame_id += 1
self.frame_count += 1
self.error = None
self.condition.notify_all()
except Exception as exc:
self._set_error(exc)
if self._stop_event.wait(0.1):
break
next_deadline += period
delay = next_deadline - time.monotonic()
if delay > 0:
self._stop_event.wait(delay)
elif -delay > period * 2:
next_deadline = time.monotonic()
def wait_for_frame(self, last_frame_id: int) -> tuple[bytes | None, int, bool]:
with self.condition:
self.condition.wait_for(
lambda: self.frame_id > last_frame_id or self.stopped,
timeout=1.0,
)
return self.jpeg, self.frame_id, self.stopped
def latest_frame(self) -> Any | None:
with self.condition:
return None if self.frame is None else self.frame.copy()
def status(self, selected: bool) -> dict[str, Any]:
with self.condition:
return {
"serial": self.device.serial,
"name": self.device.name,
"product_line": self.device.product_line,
"width": self.width,
"height": self.height,
"fps": self.fps,
"connected": self.connected,
"selected": selected,
"frame_count": self.frame_count,
"error": self.error,
}
def stop(self) -> None:
self._stop_event.set()
with self.condition:
self.stopped = True
self.condition.notify_all()
if self._thread is not None:
self._thread.join(timeout=2.0)
if self.connected:
try:
self.camera.disconnect()
except Exception as exc:
self._set_error(exc)
self.connected = False
class _PreviewState:
def __init__(self, feeds: list[_CameraFeed], selected: list[str], fps: int) -> None:
self.feeds = {feed.device.serial: feed for feed in feeds}
self.selected = set(selected)
self.fps = fps
def get_feed(self, serial: str | None) -> _CameraFeed | None:
return self.feeds.get(serial) if serial else None
def device_status(self) -> list[dict[str, Any]]:
return [
feed.status(feed.device.serial in self.selected)
for feed in self.feeds.values()
]
def stop(self) -> None:
for feed in self.feeds.values():
feed.stop()
WEB_PAGE = """<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>RealSense Viewer</title>
<style>
:root { color-scheme: dark; font-family: system-ui, sans-serif; }
* { box-sizing: border-box; }
body { margin: 0; background: #101214; color: #e8eaed; }
header { position: sticky; top: 0; z-index: 2; padding: 14px 18px; background: #191c20; border-bottom: 1px solid #30343a; }
h1 { margin: 0 0 10px; font-size: 20px; font-weight: 600; }
#status { color: #9da5af; font-size: 13px; }
.toolbar { display: flex; flex-wrap: wrap; gap: 8px; align-items: center; margin-top: 12px; }
button { border: 1px solid #4a515b; border-radius: 5px; padding: 7px 11px; color: #e8eaed; background: #262b31; cursor: pointer; }
button:hover { background: #323941; }
#devices { display: flex; flex-wrap: wrap; gap: 8px; margin-top: 12px; }
.device { display: flex; align-items: center; gap: 7px; padding: 7px 9px; border: 1px solid #3b4149; border-radius: 5px; background: #20242a; font-size: 13px; }
.device input { width: 16px; height: 16px; accent-color: #4da3ff; }
.device small { color: #9da5af; }
.device.error { border-color: #a44d4d; }
#grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(360px, 1fr)); gap: 12px; padding: 14px; }
.tile { min-width: 0; overflow: hidden; background: #191c20; border: 1px solid #30343a; border-radius: 6px; }
.tile h2 { margin: 0; padding: 9px 11px; font-size: 14px; font-weight: 500; }
.tile img { display: block; width: 100%; height: auto; background: #08090a; }
.empty { padding: 36px 18px; color: #9da5af; text-align: center; }
</style>
</head>
<body>
<header>
<h1>RealSense Viewer</h1>
<div id="status">Loading devices...</div>
<div class="toolbar">
<button id="all" type="button">Select all</button>
<button id="none" type="button">Clear</button>
</div>
<div id="devices"></div>
</header>
<main id="grid"><div class="empty">Loading...</div></main>
<script>
const devicesEl = document.getElementById('devices');
const gridEl = document.getElementById('grid');
const statusEl = document.getElementById('status');
let devices = [];
let selected = new Set();
function deviceLabel(device) {
return `${device.name} (${device.serial})`;
}
function streamUrl(serial) {
return `/stream?serial=${encodeURIComponent(serial)}`;
}
function renderGrid() {
gridEl.replaceChildren();
const visible = devices.filter(device => selected.has(device.serial) && device.connected);
if (!visible.length) {
const empty = document.createElement('div');
empty.className = 'empty';
empty.textContent = 'Select a connected device to display.';
gridEl.appendChild(empty);
return;
}
for (const device of visible) {
const tile = document.createElement('section');
tile.className = 'tile';
const title = document.createElement('h2');
title.textContent = deviceLabel(device);
const image = document.createElement('img');
image.alt = deviceLabel(device);
image.src = streamUrl(device.serial);
tile.append(title, image);
gridEl.appendChild(tile);
}
}
function renderDevices() {
devicesEl.replaceChildren();
for (const device of devices) {
const label = document.createElement('label');
label.className = `device${device.error ? ' error' : ''}`;
const checkbox = document.createElement('input');
checkbox.type = 'checkbox';
checkbox.checked = selected.has(device.serial);
checkbox.disabled = !device.connected;
checkbox.addEventListener('change', () => {
if (checkbox.checked) selected.add(device.serial);
else selected.delete(device.serial);
renderGrid();
});
const text = document.createElement('span');
text.textContent = deviceLabel(device);
const details = document.createElement('small');
details.textContent = device.connected ? `${device.width}x${device.height} @ ${device.fps} Hz` : (device.error || 'offline');
label.append(checkbox, text, details);
devicesEl.appendChild(label);
}
}
async function refresh() {
try {
const response = await fetch('/api/devices', { cache: 'no-store' });
const payload = await response.json();
devices = payload.devices;
if (!selected.size) {
for (const device of devices.filter(device => device.selected && device.connected)) selected.add(device.serial);
}
statusEl.textContent = `${devices.filter(device => device.connected).length}/${devices.length} devices connected · ${payload.fps} Hz`;
renderDevices();
renderGrid();
} catch (error) {
statusEl.textContent = `Device service unavailable: ${error}`;
}
}
document.getElementById('all').addEventListener('click', () => {
for (const device of devices) if (device.connected) selected.add(device.serial);
renderDevices(); renderGrid();
});
document.getElementById('none').addEventListener('click', () => {
selected.clear(); renderDevices(); renderGrid();
});
refresh();
</script>
</body>
</html>
""".encode("utf-8")
class _PreviewHandler(http.server.BaseHTTPRequestHandler):
state: _PreviewState
def _send_bytes(self, status: int, content_type: str, body: bytes) -> None:
try:
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
self.wfile.flush()
except (BrokenPipeError, ConnectionResetError):
pass
def do_GET(self) -> None:
parsed = urlsplit(self.path)
path = parsed.path
if path == "/":
self._send_bytes(200, "text/html; charset=utf-8", WEB_PAGE)
return
if path == "/favicon.ico":
self._send_bytes(204, "text/plain; charset=utf-8", b"")
return
if path == "/health":
self._send_bytes(200, "application/json; charset=utf-8", self._json(self.state.device_status()))
return
if path == "/api/devices":
self._send_bytes(
200,
"application/json; charset=utf-8",
self._json(self.state.device_status(), include_fps=True),
)
return
if path == "/stream":
serial = parse_qs(parsed.query).get("serial", [None])[0]
self._stream(serial)
return
try:
self.send_error(404)
except (BrokenPipeError, ConnectionResetError):
pass
def _json(self, devices: list[dict[str, Any]], include_fps: bool = False) -> bytes:
payload: dict[str, Any] = {"ok": True, "devices": devices}
if include_fps:
payload["fps"] = self.state.fps
return json.dumps(payload, ensure_ascii=False).encode("utf-8")
def _stream(self, serial: str | None) -> None:
feed = self.state.get_feed(serial)
if feed is None:
self._send_bytes(404, "text/plain; charset=utf-8", b"Unknown camera serial\n")
return
if not feed.connected:
self._send_bytes(503, "text/plain; charset=utf-8", b"Camera is not connected\n")
return
try:
self.send_response(200)
self.send_header("Cache-Control", "no-cache, private")
self.send_header("Pragma", "no-cache")
self.send_header("Content-Type", "multipart/x-mixed-replace; boundary=frame")
self.end_headers()
last_frame_id = 0
while True:
jpeg, frame_id, stopped = feed.wait_for_frame(last_frame_id)
if stopped:
return
if jpeg is None or frame_id == last_frame_id:
continue
last_frame_id = frame_id
self.wfile.write(
b"--frame\r\nContent-Type: image/jpeg\r\n"
+ f"Content-Length: {len(jpeg)}\r\n\r\n".encode()
+ jpeg
+ b"\r\n"
)
self.wfile.flush()
except (BrokenPipeError, ConnectionResetError):
pass
def log_message(self, format: str, *args: Any) -> None:
pass
def start_web_server(
state: _PreviewState, host: str, port: int
) -> http.server.ThreadingHTTPServer:
handler = type("RealSensePreviewHandler", (_PreviewHandler,), {"state": state})
server = http.server.ThreadingHTTPServer((host, port), handler)
server.daemon_threads = True
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
if host == "0.0.0.0":
print(f"Open http://127.0.0.1:{port}/ on this machine to view the streams.")
print(f"For another machine, use http://<server-ip>:{port}/")
else:
print(f"Open http://{host}:{port}/ in a browser to view the streams.")
print("Press Ctrl+C to exit.")
return server
def _make_mosaic(feeds: list[_CameraFeed]) -> Any | None:
tiles = []
for feed in feeds:
frame = feed.latest_frame()
if frame is None:
continue
cv2.putText(
frame,
f"{feed.device.name} {feed.device.serial}",
(10, 28),
cv2.FONT_HERSHEY_SIMPLEX,
0.65,
(0, 255, 255),
2,
cv2.LINE_AA,
)
tiles.append(frame)
if not tiles:
return None
height = max(tile.shape[0] for tile in tiles)
aligned = [
cv2.resize(tile, (int(tile.shape[1] * height / tile.shape[0]), height))
if tile.shape[0] != height
else tile
for tile in tiles
]
return cv2.hconcat(aligned)
def main() -> None:
args = parse_args()
if args.fps <= 0:
raise SystemExit("FPS must be positive")
if args.no_display:
backend = "none"
elif args.backend == "auto":
backend = "opencv" if opencv_has_gui() else "web"
else:
backend = args.backend
devices = discover_devices()
requested = list(dict.fromkeys(args.serial or []))
if requested:
requested_set = set(requested)
devices = [device for device in devices if device.serial in requested_set]
missing = sorted(requested_set - {device.serial for device in devices})
if missing:
print(f"Requested cameras not found: {', '.join(missing)}")
if not devices:
raise SystemExit("No RealSense cameras found.")
feeds: list[_CameraFeed] = []
for device in devices:
feed = _CameraFeed(device, args.width, args.height, args.fps)
try:
print(f"Opening RealSense {device.name} {device.serial}: {args.width}x{args.height} @ {args.fps} FPS")
feed.connect()
print(f"Connected {device.serial}")
except Exception as exc:
feed.error = f"{type(exc).__name__}: {exc}"
print(f"Failed to connect {device.serial}: {feed.error}")
feeds.append(feed)
initially_selected = requested or ([feeds[0].device.serial] if feeds else [])
state = _PreviewState(feeds, initially_selected, args.fps)
web_server = None
if backend == "web":
web_server = start_web_server(state, args.host, args.port)
elif backend == "opencv":
print("Press q or Esc to exit.")
try:
if backend == "web":
while True:
time.sleep(1.0)
elif backend == "none":
while True:
time.sleep(1.0)
statuses = [
f"{feed.device.serial}:{'OK' if feed.connected and feed.error is None else 'ERROR'}"
for feed in feeds
]
print(" | ".join(statuses), flush=True)
else:
while True:
mosaic = _make_mosaic(feeds)
if mosaic is not None:
cv2.imshow("RealSense Color", mosaic)
key = cv2.waitKey(1) & 0xFF
if key in (ord("q"), 27):
break
except KeyboardInterrupt:
pass
finally:
state.stop()
if web_server is not None:
web_server.shutdown()
web_server.server_close()
if backend == "opencv":
try:
cv2.destroyAllWindows()
except cv2.error:
pass
print("RealSense preview stopped.")
if __name__ == "__main__":
main()

130
src/lerobot_robot_ufactory/utils/action_safety.py Normal file
View File

@ -0,0 +1,130 @@
"""Model-agnostic action safety checks.
Validates the final action dict right before it is sent to the robot; any
failed check triggers an emergency halt. Only relies on the action key
naming convention: `[prefix.]pose.x/y/z/rx/ry/rz`, `*.gripper.pos`, so it
works with any policy (ACT / DP / pi0 / custom backbones) and with both
single-arm and multi-arm setups.
"""
import logging
import math
from dataclasses import dataclass
import numpy as np
from lerobot_robot_ufactory.devices.umi.vive_tracker.transformations import Transformations
POSE_AXES = ("x", "y", "z", "rx", "ry", "rz")
@dataclass
class ActionSafetyConfig:
"""Configuration for ActionSafetyGuard.
All limits are compared against the robot's current actual pose per
control step. `workspace_min`/`workspace_max` use the same unit as the
pose commands (e.g. mm) and must be provided together.
"""
enabled: bool = True
max_step_mm: float = 25.0
max_rot_step: float = 0.35
workspace_min: list | None = None
workspace_max: list | None = None
def __post_init__(self):
if (self.workspace_min is None) != (self.workspace_max is None):
raise ValueError("workspace_min and workspace_max must be provided together")
def _rotation_delta_norm(rot_prev, rot_curr) -> float:
"""Relative rotation angle (rad) between two axis-angle rotations.
Uses rotation matrices for the diff to avoid the ±π discontinuity
of subtracting raw rotvecs.
"""
R_prev = Transformations.rxryrz_to_rotation_matrix(*rot_prev)
R_curr = Transformations.rxryrz_to_rotation_matrix(*rot_curr)
R_delta = R_prev.T @ R_curr
delta = Transformations.rotation_matrix_to_rxryrz(R_delta)
return float(np.linalg.norm(delta))
class ActionSafetyGuard:
"""Safety checks for the action dict about to be sent to the robot.
check() returns None when the action is safe; otherwise it returns a
human-readable violation reason (the caller should trigger an e-stop).
"""
def __init__(self, config: ActionSafetyConfig | None = None):
config = config or ActionSafetyConfig()
self.config = config
self.enabled = config.enabled
self.max_step_mm = config.max_step_mm
self.max_rot_step = config.max_rot_step
self.workspace_min = None if config.workspace_min is None else np.asarray(config.workspace_min, dtype=np.float64)
self.workspace_max = None if config.workspace_max is None else np.asarray(config.workspace_max, dtype=np.float64)
def check(self, action: dict, curr_robot_dict: dict, keys) -> str | None:
"""Check the action dict. `action` holds absolute pose commands and
`curr_robot_dict` holds each arm's current actual pose."""
if not self.enabled:
return None
for key in keys:
prefix = f".{key}" if key else ""
pose_keys = [f"{prefix}pose.{axis}" for axis in POSE_AXES]
if not all(k in action for k in pose_keys):
continue # not TCP-pose controlled (e.g. joint-only arm), skip pose checks
target = np.array([action[k] for k in pose_keys], dtype=np.float64)
# 1. Numeric validity: NaN / Inf
if not np.all(np.isfinite(target)):
return f"[{key or 'arm'}] action contains NaN/Inf: {target.tolist()}"
# 2. Single-step delta limit, measured from the robot's current actual pose
curr = curr_robot_dict.get(key)
if curr is not None and curr.get("type") == 1:
curr_pose = np.asarray(curr["pose"], dtype=np.float64)
pos_dist = float(np.linalg.norm(target[:3] - curr_pose[:3]))
if pos_dist > self.max_step_mm:
return (
f"[{key or 'arm'}] position step {pos_dist:.1f}mm exceeds limit "
f"{self.max_step_mm}mm (current {curr_pose[:3].tolist()} -> target {target[:3].tolist()})"
)
rot_delta = _rotation_delta_norm(curr_pose[3:6], target[3:6])
if rot_delta > self.max_rot_step:
return (
f"[{key or 'arm'}] rotation step {rot_delta:.3f}rad exceeds limit "
f"{self.max_rot_step}rad"
)
# 3. Workspace bounding box (optional)
if self.workspace_min is not None:
pos = target[:3]
if np.any(pos < self.workspace_min) or np.any(pos > self.workspace_max):
return (
f"[{key or 'arm'}] target position {pos.tolist()} outside workspace "
f"[{self.workspace_min.tolist()}, {self.workspace_max.tolist()}]"
)
# Gripper numeric check
for k, v in action.items():
if "gripper" in k and isinstance(v, (int, float)) and not math.isfinite(v):
return f"[{k}] gripper action contains NaN/Inf: {v}"
return None
def log_config(self):
ws = (
f"[{self.workspace_min.tolist()}, {self.workspace_max.tolist()}]"
if self.workspace_min is not None
else "not set"
)
logging.info(
f"ActionSafetyGuard: max_step={self.max_step_mm}mm, "
f"max_rot_step={self.max_rot_step}rad, workspace={ws}"
)

48
start_manual_record.sh
View File

@ -2,49 +2,9 @@
set -euo pipefail set -euo pipefail
repo_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
cd "$repo_root"
config_path="config/manual_mode/xarm7_manual_record_config.yaml"
dataset_root="$(
sed -n '/^dataset:/,/^[^[:space:]]/s/^[[:space:]]*root:[[:space:]]*//p' "$config_path" \
| head -n 1 \
| tr -d "\"'"
)"
if [[ -z "$dataset_root" ]]; then
printf 'Could not read dataset.root from %s\n' "$config_path" >&2
exit 1
fi
if [[ "$dataset_root" != /* ]]; then
dataset_root="$repo_root/$dataset_root"
fi
record_args=("$@")
resume_requested=false
for arg in "${record_args[@]}"; do
if [[ "$arg" == "-r" ]]; then
resume_requested=true
break
fi
done
if [[ -e "$dataset_root" ]]; then
if [[ ! -f "$dataset_root/meta/info.json" ]]; then
printf 'Dataset directory exists but is not a valid LeRobot dataset: %s\n' "$dataset_root" >&2
printf 'Choose a new dataset.root, or remove this empty/incomplete directory before recording.\n' >&2
exit 1
fi
if [[ "$resume_requested" == false ]]; then
record_args=("-r" "${record_args[@]}")
fi
elif [[ "$resume_requested" == true ]]; then
printf 'Cannot resume because the dataset directory does not exist: %s\n' "$dataset_root" >&2
exit 1
fi
exec uv run uf-lerobot-record \ exec uv run uf-lerobot-record \
--config_path "$config_path" \ --config_path config/manual_mode/xarm7_manual_record_config.yaml \
"${record_args[@]}" "$@"

42
tests/test_manual_mode.py
View File

@ -1,5 +1,6 @@
import sys import sys
from pathlib import Path from pathlib import Path
from types import SimpleNamespace
import numpy as np import numpy as np
import pytest import pytest
@ -11,6 +12,7 @@ from lerobot_robot_ufactory.scripts.uf_lerobot_record import (
_manual_action_from_observation, _manual_action_from_observation,
_update_manual_gripper_key_state, _update_manual_gripper_key_state,
_update_manual_gripper_target, _update_manual_gripper_target,
_prepare_dataset_root,
_prepare_recording_episode, _prepare_recording_episode,
get_cfg, get_cfg,
) )
@ -294,6 +296,46 @@ def test_manual_record_config_has_no_teleop(monkeypatch):
assert config.dataset.fps == 30 assert config.dataset.fps == 30
def test_prepare_dataset_root_leaves_new_root_for_lerobot_create(tmp_path):
root = tmp_path / "nested" / "dataset"
cfg = SimpleNamespace(dataset=SimpleNamespace(root=root), resume=False)
_prepare_dataset_root(cfg)
assert not root.exists()
def test_prepare_dataset_root_rejects_incomplete_resume(tmp_path):
root = tmp_path / "dataset"
(root / "meta").mkdir(parents=True)
(root / "meta" / "info.json").write_text("{}")
cfg = SimpleNamespace(dataset=SimpleNamespace(root=root), resume=True)
with pytest.raises(RuntimeError, match="meta/tasks.parquet"):
_prepare_dataset_root(cfg)
def test_prepare_dataset_root_rejects_resume_when_root_is_missing(tmp_path):
cfg = SimpleNamespace(dataset=SimpleNamespace(root=tmp_path / "missing"), resume=True)
with pytest.raises(RuntimeError, match="does not exist"):
_prepare_dataset_root(cfg)
def test_prepare_dataset_root_resumes_complete_dataset_without_prompt(tmp_path, monkeypatch):
root = tmp_path / "dataset"
(root / "meta" / "episodes" / "chunk-000").mkdir(parents=True)
(root / "data" / "chunk-000").mkdir(parents=True)
(root / "meta" / "info.json").write_text("{}")
(root / "meta" / "tasks.parquet").write_bytes(b"tasks")
(root / "meta" / "episodes" / "chunk-000" / "file-000.parquet").write_bytes(b"episodes")
(root / "data" / "chunk-000" / "file-000.parquet").write_bytes(b"data")
cfg = SimpleNamespace(dataset=SimpleNamespace(root=root), resume=True)
monkeypatch.setattr(record_module.sys.stdin, "isatty", lambda: True)
_prepare_dataset_root(cfg)
def test_manual_record_loop_writes_actual_state_as_action(tmp_path): def test_manual_record_loop_writes_actual_state_as_action(tmp_path):
class FakeRobot: class FakeRobot:
name = "fake_manual_robot" name = "fake_manual_robot"