Merge pull request #1 from MinnaZhong/main

Include video
2026-06-23 18:28:41 +08:00 · 2026-06-23 18:28:41 +08:00 · 57e57358b6
commit 57e57358b6
parent 1282a073ec 17ac84448f
2 changed files with 244 additions and 37 deletions
--- a/README.md
+++ b/README.md
@ -4,17 +4,41 @@

 UFACTORY robot arm integration with the LeRobot framework for robot learning, data collection, and policy deployment.

-Reference project: [ufactory_teleop](https://github.com/xArm-Developer/ufactory_teleop)
+## Training & Inference Results
+
+[Test datasets](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY) used during development, **for reference only, do NOT reuse**. As the robot arm and camera positions during development differ from user setups.
+
+<table>
+<tr>
+  <td width="50%">
+    <a href="https://www.youtube.com/watch?v=wTiWLiHciT8" target="_blank">
+      <img src="https://img.youtube.com/vi/wTiWLiHciT8/maxresdefault.jpg" width="100%">
+    </a>
+  </td>
+  <td width="50%">
+    <a href="https://www.youtube.com/watch?v=IiyvewZh5OY" target="_blank">
+      <img src="https://img.youtube.com/vi/IiyvewZh5OY/maxresdefault.jpg" width="100%">
+    </a>
+  </td>
+</tr>
+<tr>
+  <td width="50%">
+    <a href="https://youtu.be/wBwZH6POk38" target="_blank">
+      <img src="https://img.youtube.com/vi/wBwZH6POk38/maxresdefault.jpg" width="100%">
+    </a>
+  </td>
+</tr>
+</table>

 ## Features

- xArm robot control (xArm series)
- Multiple teleop modes: GELLO / Pika / UMI / SpaceMouse
- Multi-camera data collection (RealSense / UMI camera)
- Dataset recording & management (LeRobot-compatible)
- Imitation learning training (ACT / Diffusion Policy / etc.)
- Policy evaluation & real-time inference
- Mock mode (no physical robot needed)
+- 🤖 UFACTORY robot control ([xArm series](https://www.ufactory.cc/))
+- 🎮 Multiple teleop modes: GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/)
+- 📷 Multi-camera data collection ([RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI camera)
+- 📊 Dataset recording & management (LeRobot-compatible)
+- 🧠 Imitation learning training (ACT / Diffusion Policy / etc.)
+- 🚀 Policy evaluation & real-time inference
+- 🔧 Mock mode (teleop device only, no physical robot needed)

 ## Requirements

@ -48,6 +72,8 @@ Peripheral dependencies are available as optional extras via `[module]` install.
 #### GELLO Teleop

 Dynamixel-based leader arm, joint-space control.
+* Once data collection starts, the **relative position** between the robot arm and camera (D435 / D435i) **must remain unchanged**.
+* The camera position during inference must match the collection setup. If the robot arm or camera changes, previously collected data becomes invalid.

 ```bash
 # 1. Install GELLO module
@ -59,7 +85,9 @@ sudo usermod -aG dialout $USER

 #### Pika Teleop

-Pika Sense handheld + Vive Tracker, task-space control.
+Pika Sense handheld + Vive Tracker, Cartesian-space control.
+* No requirement for the relative position of the two base stations and the robot arm. Only need to ensure the Pika Sense is within base station range during collection, but **base stations must be recalibrated after moving**.
+* Base station positions for collection and inference do not need to be the same.

 ```bash
 # 1. Install peripheral deps (skip transitive deps)
@ -122,8 +150,12 @@ sudo udevadm control --reload-rules && sudo udevadm trigger
 Test teleop-to-robot control loop without recording.

 ```bash
+# Generic usage
 uf-robot-teleop -c path/to/config.yaml
 uf-robot-teleop -c path/to/config.yaml -f 60  # specify frequency
+
+# Example: xArm6 + UMI teleop
+uf-robot-teleop -c config/umi/xarm6_umi_record_config.yaml
 ```

 ### 2. Data Collection
@ -131,8 +163,12 @@ uf-robot-teleop -c path/to/config.yaml -f 60  # specify frequency
 Record datasets via teleop.

 ```bash
+# Generic usage
 uf-lerobot-record -c path/to/record_config.yaml
 uf-lerobot-record -c path/to/config.yaml --resume  # resume recording
+
+# Example: xArm6 + UMI data collection
+uf-lerobot-record -c config/umi/xarm6_umi_record_config.yaml
 ```

 ### 3. Policy Training
@ -140,18 +176,47 @@ uf-lerobot-record -c path/to/config.yaml --resume  # resume recording
 Train imitation learning policies on collected data.

 ```bash
+# Generic usage
 lerobot-train --policy act --dataset your_dataset_name
+
+# Example: train ACT on xArm6 UMI dataset
+lerobot-train --policy act --dataset ufactory/xarm6_umi_datas
 ```

-### 4. Policy Evaluation
-
-Evaluate trained policies.
+Important parameters:

 ```bash
-uf-lerobot-eval -c path/to/eval_config.yaml
+# Note: repo_id is the same as in the record config
+# Policy type: ACT, training steps: 800k
+# Checkpoints saved every 20k steps, output to lerobot_datas/train (sibling of lerobot directory)
+lerobot-train \
+  --dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \
+  --dataset.repo_id=ufactory/xarm6_umi_datas \
+  --policy.type=act \
+  --policy.device=cuda \
+  --policy.repo_id=ufactory/xarm6_umi_datas \
+  --output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \
+  --job_name=xarm6_umi_datas \
+  --steps=800000 \
+  --batch_size=8 \
+  --save_freq=20000
 ```

-### 5. Camera Viewer
+### 4. Inference & Evaluation
+
+Run inference with a trained policy.
+
+```bash
+# Generic usage
+uf-lerobot-eval -c path/to/config.yaml --policy.path your_train_path
+
+# Example: run inference with trained ACT policy
+uf-lerobot-eval -c config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
+```
+
+## Tools
+
+### 1. Camera Viewer

 View and stitch multiple camera feeds.

@ -163,18 +228,45 @@ uf-camera-view -T xvisio -W 640 -H 1920 -F NV12  # specify format
 uf-camera-view -T other                     # view other camera types
 ```

-### Mock Mode (no physical robot)
+### 2. LeRobot Dataset Tools

-```yaml
-type: "uf::mock_robot"           # single arm simulation
-type: "uf::multiple_mock_robot"  # dual arm simulation
+LeRobot provides dataset utilities for inspecting, editing and managing collected datasets.
+
+#### View an episode:
+e.g. view episode index 17:
+```bash
+lerobot-dataset-viz \
+  --root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
+  --repo-id ufactory/xarm7_record_datas \
+  --display-compressed-images true \
+  --episode-index 17
+```
+
+#### Delete specific episodes:
+e.g. delete episodes 18 and 19:
+```bash
+lerobot-edit-dataset \
+  --root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
+  --repo_id ufactory/xarm7_record_datas \
+  --new_repo_id ../xarm7_record_datas_new \
+  --operation.type delete_episodes \
+  --operation.episode_indices "[18, 19]"
+```
+
+#### Merge datasets:
+```bash
+lerobot-edit-dataset \
+  --root=../../../../lerobot_datas/record \
+  --repo_id ufactory/xarm7_record_datas_merge_1_2 \
+  --operation.type merge \
+  --operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']"
 ```

 ## Teleop Comparison

 | Feature | GELLO | Pika | UMI | SpaceMouse |
 |---------|-------|------|-----|------------|
-| Control space | Joint space | Task space | Task space | Task space |
+| Control space | Joint space | Cartesian space | Cartesian space | Cartesian space |
 | Tracking | Dynamixel servos | Vive Tracker | UMI SLAM / Vive | 3D mouse |
 | Dual-arm | ❌ | ❌ | ✅ | ❌ |
 | System dep | dialout group | — | XVSDK deb | — |
@ -188,7 +280,6 @@ ufactory_lerobot/
 │   │   ├── robots/                 # Robot control
 │   │   │   ├── uf_robot/           #   xArm physical robot
 │   │   │   ├── uf_mock_robot/      #   Mock robot simulator
-│   │   │   └── utils.py            #   make_robot_from_config patch
 │   │   ├── teleoperators/          # Teleop drivers
 │   │   │   ├── gello_teleop/       #   GELLO (Dynamixel leader)
 │   │   │   ├── pika_teleop/        #   Pika Sense (handheld + Vive)
@ -203,20 +294,27 @@ ufactory_lerobot/
 │   │   │   ├── uf_robot_teleop.py     # Teleop testing
 │   │   │   ├── uf_lerobot_record.py   # Data recording
 │   │   │   ├── uf_lerobot_eval.py     # Policy evaluation
-│   │   │   ├── uf_camera_view.py      # camera viewer tool
+│   │   │   ├── uf_camera_view.py      # Camera viewer tool
 │   │   │   └── vive_calibrate.py      # Vive Tracker calibration
 │   │   └── utils/                  # Utilities
-│   ├── rules/                      # udev device rules
-│   └── xvsdk/                      # XVSDK system dependency
 ├── config/                         # YAML config files
 │   ├── gello/
 │   ├── pika/
 │   ├── umi/
 │   └── spacemouse/
+├── rules/                          # udev device rules
+├── xvsdk/                          # XVSDK system dependency
 ├── pyproject.toml
-└── LICENSE
+└── README.md
 ```

+## Important Notes
+
+Users are expected to thoroughly study the codebase and configuration parameters.  
+The provided configurations are **not guaranteed to work for all scenarios** and must be adjusted based on actual hardware setups and task requirements.
+
+In particular, for **diffusion policies**, the default parameters in LeRobot are primarily designed for simulation and **are not optimized for real-world robots**.
+
 ## License

 This project is released under the Apache License 2.0. See [LICENSE](LICENSE).
--- a/README_ZH.md
+++ b/README_ZH.md
@ -2,13 +2,39 @@

 > [English Version](README.md)

-UFACTORY 机械臂与 LeRobot 框架集成项目，支持多种遥操作方式的数据采集、策略训练和部署推理。
+UFACTORY(深圳市众为创造科技有限公司) 机械臂与 LeRobot 框架集成项目，支持多种遥操作方式的数据采集、策略训练和部署推理。
+
+## 训练推理效果
+
+点击下载开发时[采集的数据集](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY)，**仅供参考，不可复用**。因为用户机械臂和摄像头位置和开发测试时不一致。
+
+<table>
+<tr>
+  <td width="50%">
+    <a href="https://www.bilibili.com/video/BV12xFjzzEaX" target="_blank">
+      <img src="https://i2.hdslb.com/bfs/archive/7b325df5fb4c16e922b66d27b56d8fb6534f8b46.jpg" width="100%">
+    </a>
+  </td>
+  <td width="50%">
+    <a href="https://www.bilibili.com/video/BV16ccizHE2P" target="_blank">
+      <img src="https://i2.hdslb.com/bfs/archive/8c5d5e9370577fa89d06a175aceea282d9b2eb9a.jpg" width="100%">
+    </a>
+  </td>
+</tr>
+<tr>
+  <td width="50%">
+    <a href="https://www.bilibili.com/video/BV1xGEy6mE3i" target="_blank">
+      <img src="https://i2.hdslb.com/bfs/archive/c8bbac04a736b7043a20b753e11afea7627bdae2.jpg" width="100%">
+    </a>
+  </td>
+</tr>
+</table>

 ## 功能特性

- 🤖 UFACTORY 机械臂控制（xArm 系列）
- 🎮 多种遥操作方式：GELLO / Pika / UMI / SpaceMouse
- 📷 多摄像头数据采集（RealSense / UMI 相机）
+- 🤖 UFACTORY 机械臂控制（[xArm 系列](https://www.ufactory.cc/)）
+- 🎮 多种遥操作方式：GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/)
+- 📷 多摄像头数据采集（[RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI 相机）
 - 📊 数据集录制与管理（兼容 LeRobot 格式）
 - 🧠 模仿学习训练（ACT / Diffusion Policy 等）
 - 🚀 策略评估与实时推理
@ -46,6 +72,8 @@ pip install -e .
 #### GELLO 遥操作

 适用于 GELLO 示教臂（Dynamixel 舵机方案），控制空间为关节空间。
+* 一旦开始数据采集，机械臂与摄像头（D435 / D435i）的**相对位置必须保持不变**。
+* 推理时的摄像头位置必须与采集时相同。若机械臂或摄像头发生变化，此前采集的数据将无效。

 ```bash
 # 1. 安装 GELLO 模块
@ -58,6 +86,8 @@ sudo usermod -aG dialout $USER
 #### Pika 遥操作

 适用于 Pika Sense 手持示教器 + Vive Tracker，控制空间为笛卡尔空间。
+* 两个基站和机械臂相对位置没有要求，只需要保证采集时pika sense在基站范围内，但**基站移动后需要重新校准**。
+* 采集和推理时基站位置可不相同。

 ```bash
 # 1. 安装外设依赖（不需要它们的间接依赖）
@ -112,7 +142,6 @@ sudo cp rules/*.rules /etc/udev/rules.d/
 sudo udevadm control --reload-rules && sudo udevadm trigger
 ```

-
 ## 使用

 ### 1. 遥操作测试
@ -120,8 +149,12 @@ sudo udevadm control --reload-rules && sudo udevadm trigger
 测试遥操作设备与机械臂的联动，不录制数据。

 ```bash
+# 通用格式
 uf-robot-teleop -c path/to/config.yaml
 uf-robot-teleop -c path/to/config.yaml -f 60  # 指定频率
+
+# 示例：xArm6 + UMI 遥操作
+uf-robot-teleop -c config/umi/xarm6_umi_record_config.yaml
 ```

 ### 2. 数据采集
@ -129,36 +162,107 @@ uf-robot-teleop -c path/to/config.yaml -f 60  # 指定频率
 通过遥操作录制数据集。

 ```bash
+# 通用格式
 uf-lerobot-record -c path/to/record_config.yaml
 uf-lerobot-record -c path/to/config.yaml --resume  # 续录
+
+# 示例：xArm6 + UMI 数据采集
+uf-lerobot-record -c config/umi/xarm6_umi_record_config.yaml
 ```

-### 3. 策略训练
+### 3. Lerobot训练

 采集数据后，使用 LeRobot 训练管道进行模仿学习训练。

 ```bash
+# 通用格式
 lerobot-train --policy act --dataset your_dataset_name
 ```

-### 4. 策略评估
+参数示例：

 ```bash
-uf-lerobot-eval -c path/to/eval_config.yaml
+# 注意: repo_id就是采集时配置文件里面的repo_id
+# 这里训练策略policy.type选用act，训练steps为80w次
+# 训练过程每2w次保存一次结果，结果输出到和lerobot同级目录下的lerobot_datas/train里面
+lerobot-train \
+  --dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \
+  --dataset.repo_id=ufactory/xarm6_umi_datas \
+  --policy.type=act \
+  --policy.device=cuda \
+  --policy.repo_id=ufactory/xarm6_umi_datas \
+  --output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \
+  --job_name=xarm6_umi_datas \
+  --steps=800000 \
+  --batch_size=8 \
+  --save_freq=20000 
 ```

-### Mock 模式（无实体机械臂）
+### 4. 推理

-```yaml
-type: "uf::mock_robot"           # 单臂模拟
-type: "uf::multiple_mock_robot"  # 双臂模拟
+指定模型进行推理
+
+```bash
+# 通用格式
+uf-lerobot-eval -c path/to/config.yaml --policy.path your_train_path
+
+# 示例：使用训练好的 ACT 策略进行推理
+uf-lerobot-eval -c config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
 ```

+## 工具集
+
+### 1. 摄像头查看器
+
+查看和拼接多路摄像头画面。
+
+```bash
+uf-camera-view -l                           # 列出所有摄像头
+uf-camera-view -l -T xvisio                 # 仅列出 XVisio 摄像头
+uf-camera-view -T xvisio                    # 查看 XVisio 摄像头（默认 1280x1280 YU12）
+uf-camera-view -T xvisio -W 640 -H 1920 -F NV12  # 指定格式
+uf-camera-view -T other                     # 查看其他类型摄像头
+```
+
+### 2. Lerobot数据集工具
+Lerobot提供一些数据集工具，方便对采集的数据集进行增删查操作。
+
+### 查看某个索引的episode:
+例如查看索引号为17的episode:
+```bash
+lerobot-dataset-viz \
+  --root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
+  --repo-id ufactory/xarm7_record_datas \
+  --display-compressed-images true \
+  --episode-index 17
+```
+
+### 删除某些索引的episodes:
+例如删除索引号为18和19的episode:
+```bash
+lerobot-edit-dataset \
+  --root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
+  --repo_id ufactory/xarm7_record_datas \
+  --new_repo_id ../xarm7_record_datas_new \
+  --operation.type delete_episodes \
+  --operation.episode_indices "[18, 19]"
+```
+
+### 合并数据集
+```bash
+lerobot-edit-dataset \
+  --root=../../../../lerobot_datas/record \
+  --repo_id ufactory/xarm7_record_datas_merge_1_2 \
+  --operation.type merge \
+  --operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']"
+```
+
+
 ## 遥操作方式对比

 | 特性 | GELLO | Pika | UMI | SpaceMouse |
 |------|-------|------|-----|------------|
-| 控制空间 | 关节空间 | 任务空间 | 任务空间 | 任务空间 |
+| 控制空间 | 关节空间 | 笛卡尔空间 | 笛卡尔空间 | 笛卡尔空间 |
 | 跟踪方式 | Dynamixel 舵机 | Vive Tracker | UMI SLAM / Vive | 3D 鼠标 |
 | 双臂支持 | ❌ | ❌ | ✅ | ❌ |
 | 系统依赖 | dialout 组 | — | XVSDK deb | — |
@ -200,6 +304,11 @@ ufactory_lerobot/
 └── README.md
 ```

+## 重要提示
+用户需要全面研究整个代码库，并了解相关的配置参数，因为代码中所写的配置并非适用于所有使用场景和设置，所以用户需要研究代码或相关理论，以获取相关知识，并自行进行修改和调整。特别是对于扩散策略(diffusion policy)，LeRobot 中的默认参数可能仅用于模拟，并未针对实际机器人场景进行优化。
+
+
 ## 许可证

 本项目基于 Apache License 2.0 发布，详见 [LICENSE](LICENSE) 文件。
+