From fe42559905886129ea79c872a1f80ccebc88473c Mon Sep 17 00:00:00 2001 From: Minna <43405238+MinnaZhong@users.noreply.github.com> Date: Tue, 23 Jun 2026 17:29:10 +0800 Subject: [PATCH] Include video --- README.md | 164 +++++++++++++++++++++++++++++++++++++++++++-------- README_ZH.md | 153 +++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 280 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index c57511d..b402326 100644 --- a/README.md +++ b/README.md @@ -4,17 +4,59 @@ UFACTORY robot arm integration with the LeRobot framework for robot learning, data collection, and policy deployment. -Reference project: [ufactory_teleop](https://github.com/xArm-Developer/ufactory_teleop) +## Training & Inference Results + +[Test datasets](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY) used during development, **for reference only, do NOT reuse**. As the robot arm and camera positions during development differ from user setups. + + + + + + + + + +
+ +
+ + + + + +
+ + 		+ +
+ + + + + +
+ +
+ +
+ + + + + +
+ +
## Features -- xArm robot control (xArm series) -- Multiple teleop modes: GELLO / Pika / UMI / SpaceMouse -- Multi-camera data collection (RealSense / UMI camera) -- Dataset recording & management (LeRobot-compatible) -- Imitation learning training (ACT / Diffusion Policy / etc.) -- Policy evaluation & real-time inference -- Mock mode (no physical robot needed) +- 🤖 UFACTORY robot control ([xArm series](https://www.ufactory.cc/)) +- 🎮 Multiple teleop modes: GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/) +- 📷 Multi-camera data collection ([RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI camera) +- 📊 Dataset recording & management (LeRobot-compatible) +- 🧠 Imitation learning training (ACT / Diffusion Policy / etc.) +- 🚀 Policy evaluation & real-time inference +- 🔧 Mock mode (teleop device only, no physical robot needed) ## Requirements @@ -48,6 +90,8 @@ Peripheral dependencies are available as optional extras via `[module]` install. #### GELLO Teleop Dynamixel-based leader arm, joint-space control. +* Once data collection starts, the **relative position** between the robot arm and camera (D435 / D435i) **must remain unchanged**. +* The camera position during inference must match the collection setup. If the robot arm or camera changes, previously collected data becomes invalid. ```bash # 1. Install GELLO module @@ -59,7 +103,9 @@ sudo usermod -aG dialout $USER #### Pika Teleop -Pika Sense handheld + Vive Tracker, task-space control. +Pika Sense handheld + Vive Tracker, Cartesian-space control. +* No requirement for the relative position of the two base stations and the robot arm. Only need to ensure the Pika Sense is within base station range during collection, but **base stations must be recalibrated after moving**. +* Base station positions for collection and inference do not need to be the same. ```bash # 1. Install peripheral deps (skip transitive deps) @@ -122,8 +168,12 @@ sudo udevadm control --reload-rules && sudo udevadm trigger Test teleop-to-robot control loop without recording. ```bash +# Generic usage uf-robot-teleop -c path/to/config.yaml uf-robot-teleop -c path/to/config.yaml -f 60 # specify frequency + +# Example: xArm6 + UMI teleop +uf-robot-teleop -c config/umi/xarm6_umi_record_config.yaml ``` ### 2. Data Collection @@ -131,8 +181,12 @@ uf-robot-teleop -c path/to/config.yaml -f 60 # specify frequency Record datasets via teleop. ```bash +# Generic usage uf-lerobot-record -c path/to/record_config.yaml uf-lerobot-record -c path/to/config.yaml --resume # resume recording + +# Example: xArm6 + UMI data collection +uf-lerobot-record -c config/umi/xarm6_umi_record_config.yaml ``` ### 3. Policy Training @@ -140,18 +194,47 @@ uf-lerobot-record -c path/to/config.yaml --resume # resume recording Train imitation learning policies on collected data. ```bash +# Generic usage lerobot-train --policy act --dataset your_dataset_name + +# Example: train ACT on xArm6 UMI dataset +lerobot-train --policy act --dataset ufactory/xarm6_umi_datas ``` -### 4. Policy Evaluation - -Evaluate trained policies. +Full parameters example: ```bash -uf-lerobot-eval -c path/to/eval_config.yaml +# Note: repo_id is the same as in the record config +# Policy type: ACT, training steps: 800k +# Checkpoints saved every 20k steps, output to lerobot_datas/train (sibling of lerobot directory) +lerobot-train \ + --dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \ + --dataset.repo_id=ufactory/xarm6_umi_datas \ + --policy.type=act \ + --policy.device=cuda \ + --policy.repo_id=ufactory/xarm6_umi_datas \ + --output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \ + --job_name=xarm6_umi_datas \ + --steps=800000 \ + --batch_size=8 \ + --save_freq=20000 ``` -### 5. Camera Viewer +### 4. Inference & Evaluation + +Run inference with a trained policy. + +```bash +# Generic usage +uf-lerobot-eval -c path/to/config.yaml --policy.path your_train_path + +# Example: run inference with trained ACT policy +uf-lerobot-eval -c config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/ +``` + +## Tools + +### 1. Camera Viewer View and stitch multiple camera feeds. @@ -163,18 +246,45 @@ uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # specify format uf-camera-view -T other # view other camera types ``` -### Mock Mode (no physical robot) +### 2. LeRobot Dataset Tools -```yaml -type: "uf::mock_robot" # single arm simulation -type: "uf::multiple_mock_robot" # dual arm simulation +LeRobot provides dataset utilities for inspecting, editing and managing collected datasets. + +#### View an episode: +e.g. view episode index 17: +```bash +lerobot-dataset-viz \ + --root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \ + --repo-id ufactory/xarm7_record_datas \ + --display-compressed-images true \ + --episode-index 17 +``` + +#### Delete specific episodes: +e.g. delete episodes 18 and 19: +```bash +lerobot-edit-dataset \ + --root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \ + --repo_id ufactory/xarm7_record_datas \ + --new_repo_id ../xarm7_record_datas_new \ + --operation.type delete_episodes \ + --operation.episode_indices "[18, 19]" +``` + +#### Merge datasets: +```bash +lerobot-edit-dataset \ + --root=../../../../lerobot_datas/record \ + --repo_id ufactory/xarm7_record_datas_merge_1_2 \ + --operation.type merge \ + --operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']" ``` ## Teleop Comparison | Feature | GELLO | Pika | UMI | SpaceMouse | |---------|-------|------|-----|------------| -| Control space | Joint space | Task space | Task space | Task space | +| Control space | Joint space | Cartesian space | Cartesian space | Cartesian space | | Tracking | Dynamixel servos | Vive Tracker | UMI SLAM / Vive | 3D mouse | | Dual-arm | ❌ | ❌ | ✅ | ❌ | | System dep | dialout group | — | XVSDK deb | — | @@ -188,7 +298,6 @@ ufactory_lerobot/ │ │ ├── robots/ # Robot control │ │ │ ├── uf_robot/ # xArm physical robot │ │ │ ├── uf_mock_robot/ # Mock robot simulator -│ │ │ └── utils.py # make_robot_from_config patch │ │ ├── teleoperators/ # Teleop drivers │ │ │ ├── gello_teleop/ # GELLO (Dynamixel leader) │ │ │ ├── pika_teleop/ # Pika Sense (handheld + Vive) @@ -203,20 +312,27 @@ ufactory_lerobot/ │ │ │ ├── uf_robot_teleop.py # Teleop testing │ │ │ ├── uf_lerobot_record.py # Data recording │ │ │ ├── uf_lerobot_eval.py # Policy evaluation -│ │ │ ├── uf_camera_view.py # camera viewer tool +│ │ │ ├── uf_camera_view.py # Camera viewer tool │ │ │ └── vive_calibrate.py # Vive Tracker calibration │ │ └── utils/ # Utilities -│ ├── rules/ # udev device rules -│ └── xvsdk/ # XVSDK system dependency ├── config/ # YAML config files │ ├── gello/ │ ├── pika/ │ ├── umi/ │ └── spacemouse/ +├── rules/ # udev device rules +├── xvsdk/ # XVSDK system dependency ├── pyproject.toml -└── LICENSE +└── README.md ``` +## Important Notes + +Users are expected to thoroughly study the codebase and configuration parameters. +The provided configurations are **not guaranteed to work for all scenarios** and must be adjusted based on actual hardware setups and task requirements. + +In particular, for **diffusion policies**, the default parameters in LeRobot are primarily designed for simulation and **are not optimized for real-world robots**. + ## License This project is released under the Apache License 2.0. See [LICENSE](LICENSE). diff --git a/README_ZH.md b/README_ZH.md index 39a237f..afcf01a 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -2,13 +2,57 @@ > [English Version](README.md) -UFACTORY 机械臂与 LeRobot 框架集成项目，支持多种遥操作方式的数据采集、策略训练和部署推理。 +UFACTORY(深圳市众为创造科技有限公司) 机械臂与 LeRobot 框架集成项目，支持多种遥操作方式的数据采集、策略训练和部署推理。 + +## 训练推理效果 + +点击下载开发时[采集的数据集](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY)，**仅供参考，不可复用**。因为用户机械臂和摄像头位置和开发测试时不一致。 + + + + + + + + + +
+ +
+ + + + + +
+ + 		+ +
+ + + + + +
+ +
+ +
+ + + + + +
+ +
## 功能特性 -- 🤖 UFACTORY 机械臂控制（xArm 系列） -- 🎮 多种遥操作方式：GELLO / Pika / UMI / SpaceMouse -- 📷 多摄像头数据采集（RealSense / UMI 相机） +- 🤖 UFACTORY 机械臂控制（[xArm 系列](https://www.ufactory.cc/)） +- 🎮 多种遥操作方式：GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/) +- 📷 多摄像头数据采集（[RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI 相机） - 📊 数据集录制与管理（兼容 LeRobot 格式） - 🧠 模仿学习训练（ACT / Diffusion Policy 等） - 🚀 策略评估与实时推理 @@ -46,6 +90,8 @@ pip install -e . #### GELLO 遥操作 适用于 GELLO 示教臂（Dynamixel 舵机方案），控制空间为关节空间。 +* 一旦开始数据采集，机械臂与摄像头（D435 / D435i）的**相对位置必须保持不变**。 +* 推理时的摄像头位置必须与采集时相同。若机械臂或摄像头发生变化，此前采集的数据将无效。 ```bash # 1. 安装 GELLO 模块 @@ -58,6 +104,8 @@ sudo usermod -aG dialout $USER #### Pika 遥操作 适用于 Pika Sense 手持示教器 + Vive Tracker，控制空间为笛卡尔空间。 +* 两个基站和机械臂相对位置没有要求，只需要保证采集时pika sense在基站范围内，但**基站移动后需要重新校准**。 +* 采集和推理时基站位置可不相同。 ```bash # 1. 安装外设依赖（不需要它们的间接依赖） @@ -112,7 +160,6 @@ sudo cp rules/*.rules /etc/udev/rules.d/ sudo udevadm control --reload-rules && sudo udevadm trigger ``` - ## 使用 ### 1. 遥操作测试 @@ -120,8 +167,12 @@ sudo udevadm control --reload-rules && sudo udevadm trigger 测试遥操作设备与机械臂的联动，不录制数据。 ```bash +# 通用格式 uf-robot-teleop -c path/to/config.yaml uf-robot-teleop -c path/to/config.yaml -f 60 # 指定频率 + +# 示例：xArm6 + UMI 遥操作 +uf-robot-teleop -c config/umi/xarm6_umi_record_config.yaml ``` ### 2. 数据采集 @@ -129,36 +180,107 @@ uf-robot-teleop -c path/to/config.yaml -f 60 # 指定频率 通过遥操作录制数据集。 ```bash +# 通用格式 uf-lerobot-record -c path/to/record_config.yaml uf-lerobot-record -c path/to/config.yaml --resume # 续录 + +# 示例：xArm6 + UMI 数据采集 +uf-lerobot-record -c config/umi/xarm6_umi_record_config.yaml ``` -### 3. 策略训练 +### 3. Lerobot训练 采集数据后，使用 LeRobot 训练管道进行模仿学习训练。 ```bash +# 通用格式 lerobot-train --policy act --dataset your_dataset_name ``` -### 4. 策略评估 +参数示例： ```bash -uf-lerobot-eval -c path/to/eval_config.yaml +# 注意: repo_id就是采集时配置文件里面的repo_id +# 这里训练策略policy.type选用act，训练steps为80w次 +# 训练过程每2w次保存一次结果，结果输出到和lerobot同级目录下的lerobot_datas/train里面 +lerobot-train \ + --dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \ + --dataset.repo_id=ufactory/xarm6_umi_datas \ + --policy.type=act \ + --policy.device=cuda \ + --policy.repo_id=ufactory/xarm6_umi_datas \ + --output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \ + --job_name=xarm6_umi_datas \ + --steps=800000 \ + --batch_size=8 \ + --save_freq=20000 ``` -### Mock 模式（无实体机械臂） +### 4. 推理 -```yaml -type: "uf::mock_robot" # 单臂模拟 -type: "uf::multiple_mock_robot" # 双臂模拟 +指定模型进行推理 + +```bash +# 通用格式 +uf-lerobot-eval -c path/to/config.yaml --policy.path your_train_path + +# 示例：使用训练好的 ACT 策略进行推理 +uf-lerobot-eval -c config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/ ``` +## 工具集 + +### 1. 摄像头查看器 + +查看和拼接多路摄像头画面。 + +```bash +uf-camera-view -l # 列出所有摄像头 +uf-camera-view -l -T xvisio # 仅列出 XVisio 摄像头 +uf-camera-view -T xvisio # 查看 XVisio 摄像头（默认 1280x1280 YU12） +uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # 指定格式 +uf-camera-view -T other # 查看其他类型摄像头 +``` + +### 2. Lerobot数据集工具 +Lerobot提供一些数据集工具，方便对采集的数据集进行增删查操作。 + +### 查看某个索引的episode: +例如查看索引号为17的episode: +```bash +lerobot-dataset-viz \ + --root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \ + --repo-id ufactory/xarm7_record_datas \ + --display-compressed-images true \ + --episode-index 17 +``` + +### 删除某些索引的episodes: +例如删除索引号为18和19的episode: +```bash +lerobot-edit-dataset \ + --root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \ + --repo_id ufactory/xarm7_record_datas \ + --new_repo_id ../xarm7_record_datas_new \ + --operation.type delete_episodes \ + --operation.episode_indices "[18, 19]" +``` + +### 合并数据集 +```bash +lerobot-edit-dataset \ + --root=../../../../lerobot_datas/record \ + --repo_id ufactory/xarm7_record_datas_merge_1_2 \ + --operation.type merge \ + --operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']" +``` + + ## 遥操作方式对比 | 特性 | GELLO | Pika | UMI | SpaceMouse | |------|-------|------|-----|------------| -| 控制空间 | 关节空间 | 任务空间 | 任务空间 | 任务空间 | +| 控制空间 | 关节空间 | 笛卡尔空间 | 笛卡尔空间 | 笛卡尔空间 | | 跟踪方式 | Dynamixel 舵机 | Vive Tracker | UMI SLAM / Vive | 3D 鼠标 | | 双臂支持 | ❌ | ❌ | ✅ | ❌ | | 系统依赖 | dialout 组 | — | XVSDK deb | — | @@ -200,6 +322,11 @@ ufactory_lerobot/ └── README.md ``` +## 重要提示 +用户需要全面研究整个代码库，并了解相关的配置参数，因为代码中所写的配置并非适用于所有使用场景和设置，所以用户需要研究代码或相关理论，以获取相关知识，并自行进行修改和调整。特别是对于扩散策略(diffusion policy)，LeRobot 中的默认参数可能仅用于模拟，并未针对实际机器人场景进行优化。 + + ## 许可证 本项目基于 Apache License 2.0 发布，详见 [LICENSE](LICENSE) 文件。 +