Include video

This commit is contained in:
Minna 2026-06-23 17:29:10 +08:00
parent 1282a073ec
commit fe42559905
2 changed files with 280 additions and 37 deletions

164
README.md
View File

@ -4,17 +4,59 @@
UFACTORY robot arm integration with the LeRobot framework for robot learning, data collection, and policy deployment. UFACTORY robot arm integration with the LeRobot framework for robot learning, data collection, and policy deployment.
Reference project: [ufactory_teleop](https://github.com/xArm-Developer/ufactory_teleop) ## Training & Inference Results
[Test datasets](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY) used during development, **for reference only, do NOT reuse**. As the robot arm and camera positions during development differ from user setups.
<table>
<tr>
<td width="50%">
<a href="https://www.youtube.com/watch?v=wTiWLiHciT8" target="_blank">
<div style="position: relative;">
<img src="https://img.youtube.com/vi/wTiWLiHciT8/maxresdefault.jpg" width="100%">
<svg style="position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); pointer-events: none;" width="44" height="44" viewBox="0 0 64 64">
<circle cx="32" cy="32" r="30" fill="rgba(0,0,0,0.5)" stroke="white" stroke-width="2"/>
<polygon points="24,18 24,46 46,32" fill="white"/>
</svg>
</div>
</a>
</td>
<td width="50%">
<a href="https://www.youtube.com/watch?v=IiyvewZh5OY" target="_blank">
<div style="position: relative;">
<img src="https://img.youtube.com/vi/IiyvewZh5OY/maxresdefault.jpg" width="100%">
<svg style="position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); pointer-events: none;" width="44" height="44" viewBox="0 0 64 64">
<circle cx="32" cy="32" r="30" fill="rgba(0,0,0,0.5)" stroke="white" stroke-width="2"/>
<polygon points="24,18 24,46 46,32" fill="white"/>
</svg>
</div>
</a>
</td>
</tr>
<tr>
<td width="50%">
<a href="https://www.bilibili.com/video/BV1xGEy6mE3i" target="_blank">
<div style="position: relative;">
<img src="http://i2.hdslb.com/bfs/archive/c8bbac04a736b7043a20b753e11afea7627bdae2.jpg" width="100%">
<svg style="position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); pointer-events: none;" width="44" height="44" viewBox="0 0 64 64">
<circle cx="32" cy="32" r="30" fill="rgba(0,0,0,0.5)" stroke="white" stroke-width="2"/>
<polygon points="24,18 24,46 46,32" fill="white"/>
</svg>
</div>
</a>
</td>
</tr>
</table>
## Features ## Features
- xArm robot control (xArm series) - 🤖 UFACTORY robot control ([xArm series](https://www.ufactory.cc/))
- Multiple teleop modes: GELLO / Pika / UMI / SpaceMouse - 🎮 Multiple teleop modes: GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/)
- Multi-camera data collection (RealSense / UMI camera) - 📷 Multi-camera data collection ([RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI camera)
- Dataset recording & management (LeRobot-compatible) - 📊 Dataset recording & management (LeRobot-compatible)
- Imitation learning training (ACT / Diffusion Policy / etc.) - 🧠 Imitation learning training (ACT / Diffusion Policy / etc.)
- Policy evaluation & real-time inference - 🚀 Policy evaluation & real-time inference
- Mock mode (no physical robot needed) - 🔧 Mock mode (teleop device only, no physical robot needed)
## Requirements ## Requirements
@ -48,6 +90,8 @@ Peripheral dependencies are available as optional extras via `[module]` install.
#### GELLO Teleop #### GELLO Teleop
Dynamixel-based leader arm, joint-space control. Dynamixel-based leader arm, joint-space control.
* Once data collection starts, the **relative position** between the robot arm and camera (D435 / D435i) **must remain unchanged**.
* The camera position during inference must match the collection setup. If the robot arm or camera changes, previously collected data becomes invalid.
```bash ```bash
# 1. Install GELLO module # 1. Install GELLO module
@ -59,7 +103,9 @@ sudo usermod -aG dialout $USER
#### Pika Teleop #### Pika Teleop
Pika Sense handheld + Vive Tracker, task-space control. Pika Sense handheld + Vive Tracker, Cartesian-space control.
* No requirement for the relative position of the two base stations and the robot arm. Only need to ensure the Pika Sense is within base station range during collection, but **base stations must be recalibrated after moving**.
* Base station positions for collection and inference do not need to be the same.
```bash ```bash
# 1. Install peripheral deps (skip transitive deps) # 1. Install peripheral deps (skip transitive deps)
@ -122,8 +168,12 @@ sudo udevadm control --reload-rules && sudo udevadm trigger
Test teleop-to-robot control loop without recording. Test teleop-to-robot control loop without recording.
```bash ```bash
# Generic usage
uf-robot-teleop -c path/to/config.yaml uf-robot-teleop -c path/to/config.yaml
uf-robot-teleop -c path/to/config.yaml -f 60 # specify frequency uf-robot-teleop -c path/to/config.yaml -f 60 # specify frequency
# Example: xArm6 + UMI teleop
uf-robot-teleop -c config/umi/xarm6_umi_record_config.yaml
``` ```
### 2. Data Collection ### 2. Data Collection
@ -131,8 +181,12 @@ uf-robot-teleop -c path/to/config.yaml -f 60 # specify frequency
Record datasets via teleop. Record datasets via teleop.
```bash ```bash
# Generic usage
uf-lerobot-record -c path/to/record_config.yaml uf-lerobot-record -c path/to/record_config.yaml
uf-lerobot-record -c path/to/config.yaml --resume # resume recording uf-lerobot-record -c path/to/config.yaml --resume # resume recording
# Example: xArm6 + UMI data collection
uf-lerobot-record -c config/umi/xarm6_umi_record_config.yaml
``` ```
### 3. Policy Training ### 3. Policy Training
@ -140,18 +194,47 @@ uf-lerobot-record -c path/to/config.yaml --resume # resume recording
Train imitation learning policies on collected data. Train imitation learning policies on collected data.
```bash ```bash
# Generic usage
lerobot-train --policy act --dataset your_dataset_name lerobot-train --policy act --dataset your_dataset_name
# Example: train ACT on xArm6 UMI dataset
lerobot-train --policy act --dataset ufactory/xarm6_umi_datas
``` ```
### 4. Policy Evaluation Full parameters example:
Evaluate trained policies.
```bash ```bash
uf-lerobot-eval -c path/to/eval_config.yaml # Note: repo_id is the same as in the record config
# Policy type: ACT, training steps: 800k
# Checkpoints saved every 20k steps, output to lerobot_datas/train (sibling of lerobot directory)
lerobot-train \
--dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \
--dataset.repo_id=ufactory/xarm6_umi_datas \
--policy.type=act \
--policy.device=cuda \
--policy.repo_id=ufactory/xarm6_umi_datas \
--output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \
--job_name=xarm6_umi_datas \
--steps=800000 \
--batch_size=8 \
--save_freq=20000
``` ```
### 5. Camera Viewer ### 4. Inference & Evaluation
Run inference with a trained policy.
```bash
# Generic usage
uf-lerobot-eval -c path/to/config.yaml --policy.path your_train_path
# Example: run inference with trained ACT policy
uf-lerobot-eval -c config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
```
## Tools
### 1. Camera Viewer
View and stitch multiple camera feeds. View and stitch multiple camera feeds.
@ -163,18 +246,45 @@ uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # specify format
uf-camera-view -T other # view other camera types uf-camera-view -T other # view other camera types
``` ```
### Mock Mode (no physical robot) ### 2. LeRobot Dataset Tools
```yaml LeRobot provides dataset utilities for inspecting, editing and managing collected datasets.
type: "uf::mock_robot" # single arm simulation
type: "uf::multiple_mock_robot" # dual arm simulation #### View an episode:
e.g. view episode index 17:
```bash
lerobot-dataset-viz \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
--repo-id ufactory/xarm7_record_datas \
--display-compressed-images true \
--episode-index 17
```
#### Delete specific episodes:
e.g. delete episodes 18 and 19:
```bash
lerobot-edit-dataset \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
--repo_id ufactory/xarm7_record_datas \
--new_repo_id ../xarm7_record_datas_new \
--operation.type delete_episodes \
--operation.episode_indices "[18, 19]"
```
#### Merge datasets:
```bash
lerobot-edit-dataset \
--root=../../../../lerobot_datas/record \
--repo_id ufactory/xarm7_record_datas_merge_1_2 \
--operation.type merge \
--operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']"
``` ```
## Teleop Comparison ## Teleop Comparison
| Feature | GELLO | Pika | UMI | SpaceMouse | | Feature | GELLO | Pika | UMI | SpaceMouse |
|---------|-------|------|-----|------------| |---------|-------|------|-----|------------|
| Control space | Joint space | Task space | Task space | Task space | | Control space | Joint space | Cartesian space | Cartesian space | Cartesian space |
| Tracking | Dynamixel servos | Vive Tracker | UMI SLAM / Vive | 3D mouse | | Tracking | Dynamixel servos | Vive Tracker | UMI SLAM / Vive | 3D mouse |
| Dual-arm | ❌ | ❌ | ✅ | ❌ | | Dual-arm | ❌ | ❌ | ✅ | ❌ |
| System dep | dialout group | — | XVSDK deb | — | | System dep | dialout group | — | XVSDK deb | — |
@ -188,7 +298,6 @@ ufactory_lerobot/
│ │ ├── robots/ # Robot control │ │ ├── robots/ # Robot control
│ │ │ ├── uf_robot/ # xArm physical robot │ │ │ ├── uf_robot/ # xArm physical robot
│ │ │ ├── uf_mock_robot/ # Mock robot simulator │ │ │ ├── uf_mock_robot/ # Mock robot simulator
│ │ │ └── utils.py # make_robot_from_config patch
│ │ ├── teleoperators/ # Teleop drivers │ │ ├── teleoperators/ # Teleop drivers
│ │ │ ├── gello_teleop/ # GELLO (Dynamixel leader) │ │ │ ├── gello_teleop/ # GELLO (Dynamixel leader)
│ │ │ ├── pika_teleop/ # Pika Sense (handheld + Vive) │ │ │ ├── pika_teleop/ # Pika Sense (handheld + Vive)
@ -203,20 +312,27 @@ ufactory_lerobot/
│ │ │ ├── uf_robot_teleop.py # Teleop testing │ │ │ ├── uf_robot_teleop.py # Teleop testing
│ │ │ ├── uf_lerobot_record.py # Data recording │ │ │ ├── uf_lerobot_record.py # Data recording
│ │ │ ├── uf_lerobot_eval.py # Policy evaluation │ │ │ ├── uf_lerobot_eval.py # Policy evaluation
│ │ │ ├── uf_camera_view.py # camera viewer tool │ │ │ ├── uf_camera_view.py # Camera viewer tool
│ │ │ └── vive_calibrate.py # Vive Tracker calibration │ │ │ └── vive_calibrate.py # Vive Tracker calibration
│ │ └── utils/ # Utilities │ │ └── utils/ # Utilities
│ ├── rules/ # udev device rules
│ └── xvsdk/ # XVSDK system dependency
├── config/ # YAML config files ├── config/ # YAML config files
│ ├── gello/ │ ├── gello/
│ ├── pika/ │ ├── pika/
│ ├── umi/ │ ├── umi/
│ └── spacemouse/ │ └── spacemouse/
├── rules/ # udev device rules
├── xvsdk/ # XVSDK system dependency
├── pyproject.toml ├── pyproject.toml
└── LICENSE └── README.md
``` ```
## Important Notes
Users are expected to thoroughly study the codebase and configuration parameters.
The provided configurations are **not guaranteed to work for all scenarios** and must be adjusted based on actual hardware setups and task requirements.
In particular, for **diffusion policies**, the default parameters in LeRobot are primarily designed for simulation and **are not optimized for real-world robots**.
## License ## License
This project is released under the Apache License 2.0. See [LICENSE](LICENSE). This project is released under the Apache License 2.0. See [LICENSE](LICENSE).

View File

@ -2,13 +2,57 @@
> [English Version](README.md) > [English Version](README.md)
UFACTORY 机械臂与 LeRobot 框架集成项目,支持多种遥操作方式的数据采集、策略训练和部署推理。 UFACTORY(深圳市众为创造科技有限公司) 机械臂与 LeRobot 框架集成项目,支持多种遥操作方式的数据采集、策略训练和部署推理。
## 训练推理效果
点击下载开发时[采集的数据集](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY)**仅供参考,不可复用**。因为用户机械臂和摄像头位置和开发测试时不一致。
<table>
<tr>
<td width="50%">
<a href="https://www.bilibili.com/video/BV12xFjzzEaX" target="_blank">
<div style="position: relative;">
<img src="http://i2.hdslb.com/bfs/archive/7b325df5fb4c16e922b66d27b56d8fb6534f8b46.jpg" width="100%">
<svg style="position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); pointer-events: none;" width="44" height="44" viewBox="0 0 64 64">
<circle cx="32" cy="32" r="30" fill="rgba(0,0,0,0.5)" stroke="white" stroke-width="2"/>
<polygon points="24,18 24,46 46,32" fill="white"/>
</svg>
</div>
</a>
</td>
<td width="50%">
<a href="https://www.bilibili.com/video/BV16ccizHE2P" target="_blank">
<div style="position: relative;">
<img src="http://i2.hdslb.com/bfs/archive/8c5d5e9370577fa89d06a175aceea282d9b2eb9a.jpg" width="100%">
<svg style="position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); pointer-events: none;" width="44" height="44" viewBox="0 0 64 64">
<circle cx="32" cy="32" r="30" fill="rgba(0,0,0,0.5)" stroke="white" stroke-width="2"/>
<polygon points="24,18 24,46 46,32" fill="white"/>
</svg>
</div>
</a>
</td>
</tr>
<tr>
<td width="50%">
<a href="https://www.bilibili.com/video/BV1xGEy6mE3i" target="_blank">
<div style="position: relative;">
<img src="http://i2.hdslb.com/bfs/archive/c8bbac04a736b7043a20b753e11afea7627bdae2.jpg" width="100%">
<svg style="position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); pointer-events: none;" width="44" height="44" viewBox="0 0 64 64">
<circle cx="32" cy="32" r="30" fill="rgba(0,0,0,0.5)" stroke="white" stroke-width="2"/>
<polygon points="24,18 24,46 46,32" fill="white"/>
</svg>
</div>
</a>
</td>
</tr>
</table>
## 功能特性 ## 功能特性
- 🤖 UFACTORY 机械臂控制xArm 系列) - 🤖 UFACTORY 机械臂控制([xArm 系列](https://www.ufactory.cc/)
- 🎮 多种遥操作方式GELLO / Pika / UMI / SpaceMouse - 🎮 多种遥操作方式GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/)
- 📷 多摄像头数据采集RealSense / UMI 相机) - 📷 多摄像头数据采集([RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI 相机)
- 📊 数据集录制与管理(兼容 LeRobot 格式) - 📊 数据集录制与管理(兼容 LeRobot 格式)
- 🧠 模仿学习训练ACT / Diffusion Policy 等) - 🧠 模仿学习训练ACT / Diffusion Policy 等)
- 🚀 策略评估与实时推理 - 🚀 策略评估与实时推理
@ -46,6 +90,8 @@ pip install -e .
#### GELLO 遥操作 #### GELLO 遥操作
适用于 GELLO 示教臂Dynamixel 舵机方案),控制空间为关节空间。 适用于 GELLO 示教臂Dynamixel 舵机方案),控制空间为关节空间。
* 一旦开始数据采集机械臂与摄像头D435 / D435i的**相对位置必须保持不变**。
* 推理时的摄像头位置必须与采集时相同。若机械臂或摄像头发生变化,此前采集的数据将无效。
```bash ```bash
# 1. 安装 GELLO 模块 # 1. 安装 GELLO 模块
@ -58,6 +104,8 @@ sudo usermod -aG dialout $USER
#### Pika 遥操作 #### Pika 遥操作
适用于 Pika Sense 手持示教器 + Vive Tracker控制空间为笛卡尔空间。 适用于 Pika Sense 手持示教器 + Vive Tracker控制空间为笛卡尔空间。
* 两个基站和机械臂相对位置没有要求只需要保证采集时pika sense在基站范围内但**基站移动后需要重新校准**。
* 采集和推理时基站位置可不相同。
```bash ```bash
# 1. 安装外设依赖(不需要它们的间接依赖) # 1. 安装外设依赖(不需要它们的间接依赖)
@ -112,7 +160,6 @@ sudo cp rules/*.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger sudo udevadm control --reload-rules && sudo udevadm trigger
``` ```
## 使用 ## 使用
### 1. 遥操作测试 ### 1. 遥操作测试
@ -120,8 +167,12 @@ sudo udevadm control --reload-rules && sudo udevadm trigger
测试遥操作设备与机械臂的联动,不录制数据。 测试遥操作设备与机械臂的联动,不录制数据。
```bash ```bash
# 通用格式
uf-robot-teleop -c path/to/config.yaml uf-robot-teleop -c path/to/config.yaml
uf-robot-teleop -c path/to/config.yaml -f 60 # 指定频率 uf-robot-teleop -c path/to/config.yaml -f 60 # 指定频率
# 示例xArm6 + UMI 遥操作
uf-robot-teleop -c config/umi/xarm6_umi_record_config.yaml
``` ```
### 2. 数据采集 ### 2. 数据采集
@ -129,36 +180,107 @@ uf-robot-teleop -c path/to/config.yaml -f 60 # 指定频率
通过遥操作录制数据集。 通过遥操作录制数据集。
```bash ```bash
# 通用格式
uf-lerobot-record -c path/to/record_config.yaml uf-lerobot-record -c path/to/record_config.yaml
uf-lerobot-record -c path/to/config.yaml --resume # 续录 uf-lerobot-record -c path/to/config.yaml --resume # 续录
# 示例xArm6 + UMI 数据采集
uf-lerobot-record -c config/umi/xarm6_umi_record_config.yaml
``` ```
### 3. 策略训练 ### 3. Lerobot训练
采集数据后,使用 LeRobot 训练管道进行模仿学习训练。 采集数据后,使用 LeRobot 训练管道进行模仿学习训练。
```bash ```bash
# 通用格式
lerobot-train --policy act --dataset your_dataset_name lerobot-train --policy act --dataset your_dataset_name
``` ```
### 4. 策略评估 参数示例:
```bash ```bash
uf-lerobot-eval -c path/to/eval_config.yaml # 注意: repo_id就是采集时配置文件里面的repo_id
# 这里训练策略policy.type选用act训练steps为80w次
# 训练过程每2w次保存一次结果结果输出到和lerobot同级目录下的lerobot_datas/train里面
lerobot-train \
--dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \
--dataset.repo_id=ufactory/xarm6_umi_datas \
--policy.type=act \
--policy.device=cuda \
--policy.repo_id=ufactory/xarm6_umi_datas \
--output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \
--job_name=xarm6_umi_datas \
--steps=800000 \
--batch_size=8 \
--save_freq=20000
``` ```
### Mock 模式(无实体机械臂) ### 4. 推理
```yaml 指定模型进行推理
type: "uf::mock_robot" # 单臂模拟
type: "uf::multiple_mock_robot" # 双臂模拟 ```bash
# 通用格式
uf-lerobot-eval -c path/to/config.yaml --policy.path your_train_path
# 示例:使用训练好的 ACT 策略进行推理
uf-lerobot-eval -c config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
``` ```
## 工具集
### 1. 摄像头查看器
查看和拼接多路摄像头画面。
```bash
uf-camera-view -l # 列出所有摄像头
uf-camera-view -l -T xvisio # 仅列出 XVisio 摄像头
uf-camera-view -T xvisio # 查看 XVisio 摄像头(默认 1280x1280 YU12
uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # 指定格式
uf-camera-view -T other # 查看其他类型摄像头
```
### 2. Lerobot数据集工具
Lerobot提供一些数据集工具方便对采集的数据集进行增删查操作。
### 查看某个索引的episode:
例如查看索引号为17的episode:
```bash
lerobot-dataset-viz \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
--repo-id ufactory/xarm7_record_datas \
--display-compressed-images true \
--episode-index 17
```
### 删除某些索引的episodes:
例如删除索引号为18和19的episode:
```bash
lerobot-edit-dataset \
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
--repo_id ufactory/xarm7_record_datas \
--new_repo_id ../xarm7_record_datas_new \
--operation.type delete_episodes \
--operation.episode_indices "[18, 19]"
```
### 合并数据集
```bash
lerobot-edit-dataset \
--root=../../../../lerobot_datas/record \
--repo_id ufactory/xarm7_record_datas_merge_1_2 \
--operation.type merge \
--operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']"
```
## 遥操作方式对比 ## 遥操作方式对比
| 特性 | GELLO | Pika | UMI | SpaceMouse | | 特性 | GELLO | Pika | UMI | SpaceMouse |
|------|-------|------|-----|------------| |------|-------|------|-----|------------|
| 控制空间 | 关节空间 | 任务空间 | 任务空间 | 任务空间 | | 控制空间 | 关节空间 | 笛卡尔空间 | 笛卡尔空间 | 笛卡尔空间 |
| 跟踪方式 | Dynamixel 舵机 | Vive Tracker | UMI SLAM / Vive | 3D 鼠标 | | 跟踪方式 | Dynamixel 舵机 | Vive Tracker | UMI SLAM / Vive | 3D 鼠标 |
| 双臂支持 | ❌ | ❌ | ✅ | ❌ | | 双臂支持 | ❌ | ❌ | ✅ | ❌ |
| 系统依赖 | dialout 组 | — | XVSDK deb | — | | 系统依赖 | dialout 组 | — | XVSDK deb | — |
@ -200,6 +322,11 @@ ufactory_lerobot/
└── README.md └── README.md
``` ```
## 重要提示
用户需要全面研究整个代码库,并了解相关的配置参数,因为代码中所写的配置并非适用于所有使用场景和设置,所以用户需要研究代码或相关理论,以获取相关知识,并自行进行修改和调整。特别是对于扩散策略(diffusion policy)LeRobot 中的默认参数可能仅用于模拟,并未针对实际机器人场景进行优化。
## 许可证 ## 许可证
本项目基于 Apache License 2.0 发布,详见 [LICENSE](LICENSE) 文件。 本项目基于 Apache License 2.0 发布,详见 [LICENSE](LICENSE) 文件。