main #1
427
README.md
427
README.md
@ -1,328 +1,303 @@
|
||||
# UFACTORY LeRobot
|
||||
# UFACTORY xArm7 · LeRobot (GELLO / Manual Drag)
|
||||
|
||||
> [中文版本](README_ZH.md)
|
||||
|
||||
UFACTORY robot arm integration with the LeRobot framework for robot learning, data collection, and policy deployment.
|
||||
UFACTORY xArm integration with the [LeRobot](https://github.com/huggingface/lerobot) framework, focused on two data-collection workflows:
|
||||
|
||||
## Training & Inference Results
|
||||
- **GELLO** — joint-space teleoperation with a Dynamixel leader arm
|
||||
- **Manual drag** — record demonstrations by freely moving the arm in xArm teach mode
|
||||
|
||||
[Test datasets](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY) used during development, **for reference only, do NOT reuse**. As the robot arm and camera positions during development differ from user setups.
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<a href="https://www.youtube.com/watch?v=wTiWLiHciT8" target="_blank">
|
||||
<img src="https://img.youtube.com/vi/wTiWLiHciT8/maxresdefault.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<a href="https://www.youtube.com/watch?v=IiyvewZh5OY" target="_blank">
|
||||
<img src="https://img.youtube.com/vi/IiyvewZh5OY/maxresdefault.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<a href="https://youtu.be/wBwZH6POk38" target="_blank">
|
||||
<img src="https://img.youtube.com/vi/wBwZH6POk38/maxresdefault.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
Collected data is stored in the standard LeRobot dataset format and can be used for imitation learning (ACT / Diffusion Policy, etc.) and real-time policy inference.
|
||||
|
||||
## Features
|
||||
|
||||
- 🤖 UFACTORY robot control ([xArm series](https://www.ufactory.cc/))
|
||||
- 🎮 Multiple teleop modes: GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/)
|
||||
- 📷 Multi-camera data collection ([RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI camera)
|
||||
- 📊 Dataset recording & management (LeRobot-compatible)
|
||||
- 🧠 Imitation learning training (ACT / Diffusion Policy / etc.)
|
||||
- 🚀 Policy evaluation & real-time inference
|
||||
- 🔧 Mock mode (teleop device only, no physical robot needed)
|
||||
- 🤖 UFACTORY xArm7 control
|
||||
- 🎮 GELLO joint-space teleoperation (Dynamixel leader arm)
|
||||
- ✋ Manual drag recording via xArm teach mode
|
||||
- 📷 Intel RealSense camera observation (D435 / D435i)
|
||||
- 📊 LeRobot-compatible dataset recording & management
|
||||
- 🧠 Imitation learning training and policy inference
|
||||
- ▶️ Episode replay for recorded manual demonstrations
|
||||
|
||||
## Requirements
|
||||
|
||||
- Ubuntu 22.04 / 24.04
|
||||
- Python >= 3.10
|
||||
- CUDA >= 12.0 (recommended for GPU training)
|
||||
- UFACTORY xArm (optional)
|
||||
- UFACTORY xArm7 and its controller
|
||||
- GELLO arm (FTDI USB serial)
|
||||
- Intel RealSense D435 / D435i (for camera observations)
|
||||
|
||||
## Installation
|
||||
|
||||
### Base Install
|
||||
|
||||
```bash
|
||||
git clone https://github.com/xArm-Developer/lerobot_robot_ufactory.git
|
||||
cd lerobot_robot_ufactory
|
||||
git clone https://git.weiyantech.cn/wangshuxun/Xarm-DataCollection.git lerobot_xarm7
|
||||
cd lerobot_xarm7
|
||||
|
||||
# Create a uv virtual environment and sync project dependencies
|
||||
uv venv --python 3.10
|
||||
uv sync
|
||||
uv sync --extra gello
|
||||
```
|
||||
|
||||
Includes: `lerobot==0.4.3`, `xarm-python-sdk`, `numpy`, `pyyaml`. LeRobot already pulls in torch, opencv, wandb, etc.
|
||||
The base dependencies include `lerobot==0.4.3` (with Intel RealSense support), `xarm-python-sdk`, `numpy`, `pyyaml`, and `opencv-python`. The `gello` extra adds the GELLO software and Dynamixel SDK.
|
||||
|
||||
### Peripheral Modules
|
||||
### Serial port permission
|
||||
|
||||
Peripheral dependencies are available as optional extras via `[module]` install.
|
||||
|
||||
#### GELLO Teleop
|
||||
|
||||
Dynamixel-based leader arm, joint-space control.
|
||||
* Once data collection starts, the **relative position** between the robot arm and camera (D435 / D435i) **must remain unchanged**.
|
||||
* The camera position during inference must match the collection setup. If the robot arm or camera changes, previously collected data becomes invalid.
|
||||
The GELLO arm connects over a serial port, so add your user to the `dialout` group (re-login afterwards):
|
||||
|
||||
```bash
|
||||
# 1. Install GELLO module
|
||||
uv sync --extra gello
|
||||
|
||||
# 2. Add serial port permissions (re-login required)
|
||||
sudo usermod -aG dialout $USER
|
||||
```
|
||||
|
||||
#### Pika Teleop
|
||||
|
||||
Pika Sense handheld + Vive Tracker, Cartesian-space control.
|
||||
* No requirement for the relative position of the two base stations and the robot arm. Only need to ensure the Pika Sense is within base station range during collection, but **base stations must be recalibrated after moving**.
|
||||
* Base station positions for collection and inference do not need to be the same.
|
||||
Find the GELLO serial port path (used as `teleop.port` in the configs):
|
||||
|
||||
```bash
|
||||
# 1. Install peripheral deps (skip transitive deps)
|
||||
uv pip install pysurvive agx-pypika --no-deps
|
||||
|
||||
# 2. Install udev rules (re-plug devices afterwards)
|
||||
sudo cp src/rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
ls /dev/serial/by-id/
|
||||
```
|
||||
|
||||
> Calibrate Vive Tracker before first use: `uf-vive-calibrate`
|
||||
## Configuration
|
||||
|
||||
#### UMI Teleop
|
||||
Predefined configs are provided under `config/`:
|
||||
|
||||
Universal Manipulation Interface + Vive Tracker, supports dual-arm.
|
||||
| Workflow | Config |
|
||||
|---|---|
|
||||
| GELLO · xArm7 | `config/gello/xarm7_gello_record_config.yaml` |
|
||||
| Manual drag · xArm7 | `config/manual_mode/xarm7_manual_record_config.yaml` |
|
||||
|
||||
### GELLO config
|
||||
|
||||
- `robot.robot_ip` — xArm controller IP (e.g. `192.168.1.245`)
|
||||
- `robot.robot_dof` — `7`
|
||||
- `robot.gripper_type` — `1` for the xArm gripper
|
||||
- `teleop.port` — GELLO serial port (`/dev/serial/by-id/...`)
|
||||
- `teleop.joint_ids` / `teleop.joint_signs` — per-arm servo mapping and direction
|
||||
- `teleop.start_joints` — GELLO calibration reference, should match the xArm SDK initial point (degrees)
|
||||
- `teleop.gripper_id` — GELLO gripper servo ID (`8`; `-1` disables it)
|
||||
- `dataset.root` / `dataset.repo_id` — where the dataset is stored
|
||||
- `dataset.single_task` — task description saved with each frame
|
||||
- `dataset.fps` / `episode_time_s` / `reset_time_s` — recording timing
|
||||
|
||||
> The xArm7 config already contains the correct joint mapping; only edit the port, IP, and dataset fields for your setup.
|
||||
|
||||
### Manual-drag config
|
||||
|
||||
- `robot.manual_mode: true` — enable xArm teach mode (joint free-drive)
|
||||
- `robot.teach_sensitivity` — teaching sensitivity, valid range 1–5
|
||||
- `robot.manual_gripper_speed` — gripper velocity in normalized position per second (default `0.5`)
|
||||
- `robot.observe_joint_vel` — record joint velocities in observations (`false` by default)
|
||||
- `robot.cameras.camera` — Intel RealSense camera (`serial_number_or_name`, resolution, fps)
|
||||
- `dataset.root` / `dataset.repo_id` / `single_task` / `fps` / `episode_time_s` / `reset_time_s` / `num_episodes` — dataset settings
|
||||
|
||||
### Camera configuration
|
||||
|
||||
When adding a camera to the robot config, use the template in
|
||||
`config/manual_mode/xarm7_manual_record_config.yaml`:
|
||||
|
||||
```yaml
|
||||
robot:
|
||||
cameras:
|
||||
camera:
|
||||
type: intelrealsense # RealSense type, NOT opencv
|
||||
serial_number_or_name: "148522072685"
|
||||
width: 640
|
||||
height: 480
|
||||
fps: 30
|
||||
```
|
||||
|
||||
- `type` must be `intelrealsense` (RealSense), **not** `opencv` (generic USB camera).
|
||||
- `serial_number_or_name` must be filled with the actual RealSense serial number **obtained beforehand**, otherwise connecting/recording fails. Get it with:
|
||||
|
||||
```bash
|
||||
# 1. Install XVSDK (system-level, Ubuntu Focal only)
|
||||
curl -sL https://raw.githubusercontent.com/xArm-Developer/ufactory_resources/main/fastumi/sdk/XVSDK_focal_amd64.deb -o /tmp/xvsdk.deb && sudo dpkg -i /tmp/xvsdk.deb
|
||||
sudo apt install -y --fix-broken
|
||||
|
||||
# 2. Install peripheral deps
|
||||
uv pip install pysurvive --no-deps
|
||||
|
||||
# 3. Install udev rules (re-plug devices afterwards)
|
||||
sudo cp src/rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
```
|
||||
|
||||
> Calibrate Vive Tracker before first use: `uf-vive-calibrate`
|
||||
|
||||
**Multi-UMI device configuration** (two or more devices):
|
||||
|
||||
```bash
|
||||
# Increase USB buffer size
|
||||
sudo sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/s/quiet splash/quiet splash usbcore.usbfs_memory_mb=128/' /etc/default/grub
|
||||
sync
|
||||
sudo update-grub
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
#### SpaceMouse Teleop
|
||||
|
||||
3Dconnexion SpaceMouse / SpaceNavigator.
|
||||
|
||||
```bash
|
||||
# 1. Install SpaceMouse module
|
||||
uv sync --extra spacemouse
|
||||
|
||||
# 2. Install udev rules (re-plug device afterwards)
|
||||
sudo cp src/rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
uv run uf-camera-view -l -T realsense # prints each camera's serial number
|
||||
```
|
||||
|
||||
or with the librealsense tool `rs-enumerate-devices`.
|
||||
|
||||
## Usage
|
||||
|
||||
### 1. Teleop Testing
|
||||
### 1. GELLO teleop test
|
||||
|
||||
Test teleop-to-robot control loop without recording.
|
||||
Test the GELLO → robot control loop without recording:
|
||||
|
||||
```bash
|
||||
# Generic usage
|
||||
uv run uf-robot-teleop --config_path path/to/config.yaml
|
||||
uv run uf-robot-teleop --config_path path/to/config.yaml --fps 60 # specify frequency
|
||||
|
||||
# Example: xArm6 + UMI teleop
|
||||
uv run uf-robot-teleop --config_path config/umi/xarm6_umi_record_config.yaml
|
||||
uv run uf-robot-teleop --config_path config/gello/xarm7_gello_record_config.yaml
|
||||
uv run uf-robot-teleop --config_path config/gello/xarm7_gello_record_config.yaml --fps 60 # optional loop rate
|
||||
```
|
||||
|
||||
### 2. Manual Drag Data Collection
|
||||
`Space` reset & start, `←` reset, `Esc` exit.
|
||||
|
||||
Manual drag recording uses `manual_mode: true` in the robot configuration and does not configure a teleoperator. During recording, the actual joint state is written as both the observation and action in the LeRobot dataset. When a gripper is configured, hold `C` to close it slowly and `O` to open it slowly. Adjust the speed with `manual_gripper_speed`, which defaults to `0.5`:
|
||||
### 2. GELLO data collection
|
||||
|
||||
```bash
|
||||
# Record a new dataset
|
||||
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml
|
||||
|
||||
# Continue recording on an existing dataset
|
||||
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml -r
|
||||
|
||||
# Optional: save episodes in the background
|
||||
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml -a
|
||||
```
|
||||
|
||||
Controls: `Space` start the episode, `→` save it, `←` discard and re-record it, `Esc` stop recording. The arm resets to its initial point between episodes.
|
||||
|
||||
> During collection the **relative position between the robot arm and the camera must not change**, and the camera setup at inference time must match the one used during collection. If the arm or camera moves, previously collected data becomes invalid.
|
||||
|
||||
> If the dataset root already exists and `-r` is not given, the script asks whether to overwrite it, resume, or cancel.
|
||||
|
||||
### 3. Manual drag recording
|
||||
|
||||
```bash
|
||||
./start_manual_record.sh
|
||||
./start_manual_record.sh -r
|
||||
./start_manual_record.sh -r # force resume; fails if the dataset directory does not exist
|
||||
```
|
||||
|
||||
The launcher creates a dataset on its first run and automatically resumes an existing valid dataset. If it reports an empty or incomplete dataset directory, choose a new `dataset.root` or remove that directory after confirming it contains no data. Press `Space` to start an episode, `Right` to save it, `Left` to discard and re-record it, and `Esc` to stop recording. Reset the arm manually between episodes.
|
||||
The record script (`uf-lerobot-record`) reads `dataset.root` from the config and checks the path before recording, then:
|
||||
|
||||
### 3. Teleop Data Collection
|
||||
- Directory does not exist → records a new dataset.
|
||||
- Directory already exists (valid LeRobot dataset) and no `-r` was given → asks interactively:
|
||||
- `o` overwrite: delete the existing dataset and record a new one
|
||||
- `r` resume: keep existing episodes and continue recording
|
||||
- `c` cancel
|
||||
- Directory exists but is incomplete (missing required metadata or data parquet files) → asks to overwrite it or cancel; non-interactive runs error out instead.
|
||||
|
||||
Record datasets via teleop.
|
||||
Pass `-r` to resume directly without asking. Note that `./start_manual_record.sh` keeps its original launcher behavior — it automatically resumes an existing valid dataset (equivalent to `-r`), so run `uv run uf-lerobot-record --config_path config/manual_mode/xarm7_manual_record_config.yaml` directly if you want to see the overwrite/resume prompt.
|
||||
|
||||
During recording the arm is in teach mode: the actual joint state is written as both the observation and the action. Hold `C` to slowly close the gripper and `O` to slowly open it. Controls: `Space` start, `→` save, `←` discard & re-record, `Esc` stop. Reset the arm manually between episodes.
|
||||
|
||||
> **IMPORTANT: At the start of every episode, wait until the arm has finished resetting and then wait another 5 seconds before operating it, or wait until the console prints `Start Recording` before operating it.** This prevents reset commands from conflicting with operation commands and causing errors.
|
||||
|
||||
### 4. Policy training
|
||||
|
||||
```bash
|
||||
# Generic usage
|
||||
uv run uf-lerobot-record --config_path path/to/record_config.yaml
|
||||
uv run uf-lerobot-record --config_path path/to/config.yaml --resume true # resume recording
|
||||
|
||||
# Example: xArm6 + UMI data collection
|
||||
uv run uf-lerobot-record --config_path config/umi/xarm6_umi_record_config.yaml
|
||||
uv run lerobot-train --policy act --dataset ufactory/xarm7_gello_datas
|
||||
```
|
||||
|
||||
### 5. Policy Training
|
||||
|
||||
Train imitation learning policies on collected data.
|
||||
Example with explicit training parameters (checkpoints are saved every `save_freq` steps into `output_dir`):
|
||||
|
||||
```bash
|
||||
# Generic usage
|
||||
uv run lerobot-train --policy act --dataset your_dataset_name
|
||||
|
||||
# Example: train ACT on xArm6 UMI dataset
|
||||
uv run lerobot-train --policy act --dataset ufactory/xarm6_umi_datas
|
||||
```
|
||||
|
||||
Important parameters:
|
||||
|
||||
```bash
|
||||
# Note: repo_id is the same as in the record config
|
||||
# Policy type: ACT, training steps: 800k
|
||||
# Checkpoints saved every 20k steps, output to lerobot_datas/train (sibling of lerobot directory)
|
||||
uv run lerobot-train \
|
||||
--dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \
|
||||
--dataset.repo_id=ufactory/xarm6_umi_datas \
|
||||
--dataset.root=/home/<user>/lerobot_datas/record/ufactory/xarm7_gello_datas \
|
||||
--dataset.repo_id=ufactory/xarm7_gello_datas \
|
||||
--policy.type=act \
|
||||
--policy.device=cuda \
|
||||
--policy.repo_id=ufactory/xarm6_umi_datas \
|
||||
--output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \
|
||||
--job_name=xarm6_umi_datas \
|
||||
--policy.repo_id=ufactory/xarm7_gello_datas \
|
||||
--output_dir=/home/<user>/lerobot_datas/train/xarm7_gello_datas \
|
||||
--job_name=xarm7_gello_datas \
|
||||
--steps=800000 \
|
||||
--batch_size=8 \
|
||||
--save_freq=20000
|
||||
```
|
||||
|
||||
### 6. Inference & Evaluation
|
||||
|
||||
Run inference with a trained policy.
|
||||
### 5. Policy inference
|
||||
|
||||
```bash
|
||||
# Generic usage
|
||||
uv run uf-lerobot-eval --config_path path/to/config.yaml --policy.path your_train_path
|
||||
|
||||
# Example: run inference with trained ACT policy
|
||||
uv run uf-lerobot-eval --config_path config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
|
||||
uv run uf-lerobot-eval \
|
||||
--config_path config/gello/xarm7_gello_record_config.yaml \
|
||||
--policy.path /path/to/train/output/checkpoints/last/pretrained_model/
|
||||
```
|
||||
|
||||
`←` / `→` reset, `Esc` stop.
|
||||
|
||||
### 6. Replay recorded episodes
|
||||
|
||||
Replay the absolute joint states (`observation.state`) of a manual-drag episode on an xArm7. States are sent as absolute targets at the dataset FPS (default 30), so the motion matches the recording:
|
||||
|
||||
```bash
|
||||
uv run uf-lerobot-replay \
|
||||
--dataset-root /path/to/xarm7_manual_datas \
|
||||
--robot-ip 192.168.1.245
|
||||
|
||||
# Skip the interactive confirmation (non-interactive use)
|
||||
uv run uf-lerobot-replay --dataset-root /path/to/xarm7_manual_datas --robot-ip 192.168.1.245 --yes
|
||||
|
||||
# Replay another episode
|
||||
uv run uf-lerobot-replay --dataset-root /path/to/xarm7_manual_datas --robot-ip 192.168.1.245 --episode-index 3
|
||||
```
|
||||
|
||||
The robot first moves to the xArm SDK initial point, then replays the episode and stays at the last state. Make sure the workspace is clear and the recorded initial pose matches the current arm setup.
|
||||
|
||||
## IMPORTANT: Robot Power-On and Power-Off
|
||||
|
||||
### Power-on
|
||||
|
||||
1. **Connect the computer to the robot controller with an Ethernet cable.**
|
||||
2. **Configure the computer's Ethernet interface to the same IP subnet as the robot IP shown on the controller** (for example, `192.168.1.xxx`).
|
||||
3. **Open `http://192.168.1.245:18333/` in a browser.** The controller console should be displayed.
|
||||
4. **Release the emergency-stop button before operating the robot.**
|
||||
|
||||
### Power-off
|
||||
|
||||
1. **Use the web console to return the arm to its initial position.**
|
||||
2. **Press the emergency-stop button.**
|
||||
3. **Turn off the controller power.**
|
||||
|
||||
## Tools
|
||||
|
||||
### 1. Camera Viewer
|
||||
|
||||
View and stitch multiple camera feeds.
|
||||
### Camera viewer
|
||||
|
||||
```bash
|
||||
uv run uf-camera-view -l # list all cameras
|
||||
uv run uf-camera-view -l -T xvisio # list XVisio cameras only
|
||||
uv run uf-camera-view -T xvisio # view XVisio cameras (default 1280x1280 YU12)
|
||||
uv run uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # specify format
|
||||
uv run uf-camera-view -T other # view other camera types
|
||||
uv run uf-camera-view -l # list cameras
|
||||
uv run uf-camera-view -T realsense # view RealSense cameras
|
||||
```
|
||||
|
||||
### 2. LeRobot Dataset Tools
|
||||
### LeRobot dataset tools
|
||||
|
||||
LeRobot provides dataset utilities for inspecting, editing and managing collected datasets.
|
||||
|
||||
#### View an episode:
|
||||
e.g. view episode index 17:
|
||||
```bash
|
||||
# View episode 17
|
||||
uv run lerobot-dataset-viz \
|
||||
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
|
||||
--repo-id ufactory/xarm7_record_datas \
|
||||
--root=/path/to/record/ufactory/xarm7_manual_datas \
|
||||
--repo-id ufactory/xarm7_manual_datas \
|
||||
--display-compressed-images true \
|
||||
--episode-index 17
|
||||
```
|
||||
|
||||
#### Delete specific episodes:
|
||||
e.g. delete episodes 18 and 19:
|
||||
```bash
|
||||
# Delete episodes 18 and 19
|
||||
uv run lerobot-edit-dataset \
|
||||
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
|
||||
--repo_id ufactory/xarm7_record_datas \
|
||||
--new_repo_id ../xarm7_record_datas_new \
|
||||
--root=/path/to/record/ufactory/xarm7_manual_datas \
|
||||
--repo_id ufactory/xarm7_manual_datas \
|
||||
--new_repo_id ../xarm7_manual_datas_new \
|
||||
--operation.type delete_episodes \
|
||||
--operation.episode_indices "[18, 19]"
|
||||
```
|
||||
|
||||
#### Merge datasets:
|
||||
```bash
|
||||
# Merge datasets
|
||||
uv run lerobot-edit-dataset \
|
||||
--root=../../../../lerobot_datas/record \
|
||||
--repo_id ufactory/xarm7_record_datas_merge_1_2 \
|
||||
--root=/path/to/record \
|
||||
--repo_id ufactory/xarm7_datas_merge \
|
||||
--operation.type merge \
|
||||
--operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']"
|
||||
--operation.repo_ids "['ufactory/xarm7_datas_1', 'ufactory/xarm7_datas_2']"
|
||||
```
|
||||
|
||||
## Teleop Comparison
|
||||
|
||||
| Feature | GELLO | Pika | UMI | SpaceMouse |
|
||||
|---------|-------|------|-----|------------|
|
||||
| Control space | Joint space | Cartesian space | Cartesian space | Cartesian space |
|
||||
| Tracking | Dynamixel servos | Vive Tracker | UMI SLAM / Vive | 3D mouse |
|
||||
| Dual-arm | ❌ | ❌ | ✅ | ❌ |
|
||||
| System dep | dialout group | — | XVSDK deb | — |
|
||||
|
||||
## Project Structure
|
||||
## Project structure
|
||||
|
||||
```
|
||||
lerobot_robot_ufactory/
|
||||
├── src/
|
||||
│ ├── lerobot_robot_ufactory/ # LeRobot plugin package
|
||||
│ │ ├── robots/ # Robot control
|
||||
│ │ │ ├── uf_robot/ # xArm physical robot
|
||||
│ │ │ ├── uf_mock_robot/ # Mock robot simulator
|
||||
│ │ ├── teleoperators/ # Teleop drivers
|
||||
│ │ │ ├── base_teleop/ # Shared base class
|
||||
│ │ │ ├── gello_teleop/ # GELLO (Dynamixel leader)
|
||||
│ │ │ ├── pika_teleop/ # Pika Sense (handheld + Vive)
|
||||
│ │ │ ├── umi_teleop/ # UMI (dual-arm support)
|
||||
│ │ │ └── space_mouse/ # SpaceMouse (3D mouse)
|
||||
│ │ ├── cameras/ # Camera modules
|
||||
│ │ │ └── umi_camera/ # UMI camera
|
||||
│ │ ├── devices/ # External device drivers
|
||||
│ │ │ ├── pika/ # Pika serial driver
|
||||
│ │ │ └── umi/ # XVLib / Vive Tracker
|
||||
│ │ ├── scripts/ # Entry-point scripts
|
||||
│ │ │ ├── uf_robot_teleop.py # Teleop testing
|
||||
│ │ │ ├── uf_lerobot_record.py # Data recording
|
||||
│ │ │ ├── uf_lerobot_eval.py # Policy evaluation
|
||||
│ │ │ ├── uf_camera_view.py # Camera viewer tool
|
||||
│ │ │ └── vive_calibrate.py # Vive Tracker calibration
|
||||
│ │ ├── context.py # Teleop context registry
|
||||
│ │ └── utils/ # Utilities
|
||||
├── config/ # YAML config files
|
||||
│ ├── gello/
|
||||
│ ├── pika/
|
||||
│ ├── umi/
|
||||
│ └── spacemouse/
|
||||
├── rules/ # udev device rules
|
||||
lerobot_xarm7/
|
||||
├── config/
|
||||
│ ├── gello/ # xArm7 GELLO record config
|
||||
│ └── manual_mode/ # xArm7 manual-drag record config
|
||||
├── src/lerobot_robot_ufactory/
|
||||
│ ├── robots/
|
||||
│ │ └── uf_robot/ # xArm control (joint/cartesian, teach mode)
|
||||
│ ├── teleoperators/
|
||||
│ │ ├── base_teleop/ # shared teleop base class
|
||||
│ │ └── gello_teleop/ # GELLO (Dynamixel leader arm)
|
||||
│ ├── scripts/
|
||||
│ │ ├── uf_robot_teleop.py # teleop test loop
|
||||
│ │ ├── uf_lerobot_record.py # data collection (incl. manual mode)
|
||||
│ │ ├── uf_lerobot_eval.py # policy inference
|
||||
│ │ ├── uf_lerobot_replay.py # episode replay
|
||||
│ │ └── uf_camera_view.py # camera viewer
|
||||
│ └── configs/parser.py # config loading / CLI overrides
|
||||
├── start_manual_record.sh # manual-drag launcher
|
||||
├── pyproject.toml
|
||||
└── README.md
|
||||
├── README.md
|
||||
└── README_ZH.md
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
## Important notes
|
||||
|
||||
Users are expected to thoroughly study the codebase and configuration parameters.
|
||||
The provided configurations are **not guaranteed to work for all scenarios** and must be adjusted based on actual hardware setups and task requirements.
|
||||
|
||||
In particular, for **diffusion policies**, the default parameters in LeRobot are primarily designed for simulation and **are not optimized for real-world robots**.
|
||||
- The provided configs are **examples**: edit IPs, serial ports, camera serials, dataset paths, and task descriptions to match your hardware.
|
||||
- For GELLO data, keep the robot–camera relative pose identical between collection and inference.
|
||||
- The LeRobot default parameters for diffusion policies are mostly designed for simulation and are **not optimized for real robots** — tune them for your task.
|
||||
- Check the workspace for obstacles and keep the arm's initial pose consistent with the recorded data before replay or inference.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
455
README_ZH.md
455
README_ZH.md
@ -1,337 +1,328 @@
|
||||
# UFACTORY LeRobot
|
||||
# UFACTORY xArm7 · LeRobot（GELLO / 手动拖拽）
|
||||
|
||||
> [English Version](README.md)
|
||||
|
||||
UFACTORY(深圳市众为创造科技有限公司) 机械臂与 LeRobot 框架集成项目，支持多种遥操作方式的数据采集、策略训练和部署推理。
|
||||
UFACTORY xArm 与 [LeRobot](https://github.com/huggingface/lerobot) 框架的集成项目，专注于两种数据采集方式：
|
||||
|
||||
## 训练推理效果
|
||||
- **GELLO** — 使用 Dynamixel 示教臂的关节空间遥操作
|
||||
- **手动拖拽** — 在 xArm 示教模式下自由拖动机械臂录制演示
|
||||
|
||||
点击下载开发时[采集的数据集](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY)，**仅供参考，不可复用**。因为用户机械臂和摄像头位置和开发测试时不一致。
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<a href="https://www.bilibili.com/video/BV12xFjzzEaX" target="_blank">
|
||||
<img src="https://i2.hdslb.com/bfs/archive/7b325df5fb4c16e922b66d27b56d8fb6534f8b46.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<a href="https://www.bilibili.com/video/BV16ccizHE2P" target="_blank">
|
||||
<img src="https://i2.hdslb.com/bfs/archive/8c5d5e9370577fa89d06a175aceea282d9b2eb9a.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<a href="https://www.bilibili.com/video/BV1xGEy6mE3i" target="_blank">
|
||||
<img src="https://i2.hdslb.com/bfs/archive/c8bbac04a736b7043a20b753e11afea7627bdae2.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
采集的数据以标准 LeRobot 数据集格式保存，可用于模仿学习训练（ACT / Diffusion Policy 等）和实时策略推理。
|
||||
|
||||
## 功能特性
|
||||
|
||||
- 🤖 UFACTORY 机械臂控制（[xArm 系列](https://www.ufactory.cc/)）
|
||||
- 🎮 多种遥操作方式：GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/)
|
||||
- 📷 多摄像头数据采集（[RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI 相机）
|
||||
- 📊 数据集录制与管理（兼容 LeRobot 格式）
|
||||
- 🧠 模仿学习训练（ACT / Diffusion Policy 等）
|
||||
- 🚀 策略评估与实时推理
|
||||
- 🔧 Mock 机器人模拟（只用遥操作设备采集数据）
|
||||
- 🤖 UFACTORY xArm7 控制
|
||||
- 🎮 GELLO 关节空间遥操作（Dynamixel 示教臂）
|
||||
- ✋ xArm 示教模式手动拖拽采集
|
||||
- 📷 Intel RealSense 相机观测（D435 / D435i）
|
||||
- 📊 兼容 LeRobot 格式的数据集录制与管理
|
||||
- 🧠 模仿学习训练与策略推理
|
||||
- ▶️ 手动演示数据的 episode 回放
|
||||
|
||||
## 环境要求
|
||||
|
||||
- Ubuntu 22.04 / 24.04
|
||||
- Python >= 3.10
|
||||
- CUDA >= 12.0（GPU 训练推荐）
|
||||
- UFACTORY 机械臂（xArm 系列，可选）
|
||||
- UFACTORY xArm7 及控制器
|
||||
- GELLO 示教臂（FTDI USB 串口）
|
||||
- Intel RealSense D435 / D435i（需要相机观测时）
|
||||
|
||||
## 安装
|
||||
|
||||
### 基础项目安装
|
||||
|
||||
```bash
|
||||
git clone https://github.com/xArm-Developer/lerobot_robot_ufactory.git
|
||||
cd lerobot_robot_ufactory
|
||||
git clone https://git.weiyantech.cn/wangshuxun/Xarm-DataCollection.git lerobot_xarm7
|
||||
cd lerobot_xarm7
|
||||
|
||||
# 创建 uv 虚拟环境并同步项目依赖
|
||||
uv venv --python 3.10
|
||||
uv sync
|
||||
uv sync --extra gello
|
||||
```
|
||||
|
||||
包含：`lerobot==0.4.3`、`xarm-python-sdk`、`numpy`、`pyyaml`（lerobot 已自动携带 torch、opencv、wandb 等训练相关依赖）。
|
||||
基础依赖包含 `lerobot==0.4.3`（带 Intel RealSense 支持）、`xarm-python-sdk`、`numpy`、`pyyaml` 和 `opencv-python`。`gello` 可选依赖会额外安装 GELLO 软件和 Dynamixel SDK。
|
||||
|
||||
### 外设模块安装
|
||||
### 串口权限
|
||||
|
||||
外设依赖以可选模块形式提供，通过 `[模块名]` 安装。
|
||||
|
||||
#### GELLO 遥操作
|
||||
|
||||
适用于 GELLO 示教臂（Dynamixel 舵机方案），控制空间为关节空间。
|
||||
* 一旦开始数据采集，机械臂与摄像头（D435 / D435i）的**相对位置必须保持不变**。
|
||||
* 推理时的摄像头位置必须与采集时相同。若机械臂或摄像头发生变化，此前采集的数据将无效。
|
||||
GELLO 示教臂通过串口连接，需要将当前用户加入 `dialout` 组（重新登录后生效）：
|
||||
|
||||
```bash
|
||||
# 1. 安装 GELLO 模块
|
||||
uv sync --extra gello
|
||||
|
||||
# 2. 添加串口权限（重新登录后生效）
|
||||
sudo usermod -aG dialout $USER
|
||||
```
|
||||
|
||||
#### Pika 遥操作
|
||||
|
||||
适用于 Pika Sense 手持示教器 + Vive Tracker，控制空间为笛卡尔空间。
|
||||
* 两个基站和机械臂相对位置没有要求，只需要保证采集时pika sense在基站范围内，但**基站移动后需要重新校准**。
|
||||
* 采集和推理时基站位置可不相同。
|
||||
查看 GELLO 串口路径（用于配置文件中的 `teleop.port`）：
|
||||
|
||||
```bash
|
||||
# 1. 安装外设依赖（不需要它们的间接依赖）
|
||||
uv pip install pysurvive agx-pypika --no-deps
|
||||
|
||||
# 2. 安装 udev 规则（重新插拔设备后生效）
|
||||
sudo cp rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
ls /dev/serial/by-id/
|
||||
```
|
||||
|
||||
> Vive Tracker 首次使用前需校准：`uf-vive-calibrate`
|
||||
## 配置
|
||||
|
||||
#### UMI 遥操作
|
||||
项目在 `config/` 下提供了预置配置文件：
|
||||
|
||||
适用于 UMI（Universal Manipulation Interface）方案，含 Vive Tracker 追踪，支持双机械臂。
|
||||
| 采集方式 | 配置文件 |
|
||||
|---|---|
|
||||
| GELLO · xArm7 | `config/gello/xarm7_gello_record_config.yaml` |
|
||||
| 手动拖拽 · xArm7 | `config/manual_mode/xarm7_manual_record_config.yaml` |
|
||||
|
||||
### GELLO 配置说明
|
||||
|
||||
- `robot.robot_ip` — xArm 控制器 IP（如 `192.168.1.245`）
|
||||
- `robot.robot_dof` — `7`
|
||||
- `robot.gripper_type` — `1` 表示 xArm 夹爪
|
||||
- `teleop.port` — GELLO 串口路径（`/dev/serial/by-id/...`）
|
||||
- `teleop.joint_ids` / `teleop.joint_signs` — 各型号机械臂的舵机映射与方向
|
||||
- `teleop.start_joints` — GELLO 校准参考值（角度），应与 xArm SDK 初始点一致
|
||||
- `teleop.gripper_id` — GELLO 夹爪舵机 ID（`8`；`-1` 表示无夹爪）
|
||||
- `dataset.root` / `dataset.repo_id` — 数据集保存位置
|
||||
- `dataset.single_task` — 随每一帧保存的任务描述
|
||||
- `dataset.fps` / `episode_time_s` / `reset_time_s` — 录制时序参数
|
||||
|
||||
> xArm7 的配置已包含正确的关节映射，一般只需要修改串口、IP 和数据集路径。
|
||||
|
||||
### 手动拖拽配置说明
|
||||
|
||||
- `robot.manual_mode: true` — 开启 xArm 示教模式（关节自由拖动）
|
||||
- `robot.teach_sensitivity` — 示教灵敏度，有效范围 1–5
|
||||
- `robot.manual_gripper_speed` — 夹爪速度（每秒归一化位置变化，默认 `0.5`）
|
||||
- `robot.observe_joint_vel` — 是否在观测中记录关节速度（默认 `false`）
|
||||
- `robot.cameras.camera` — Intel RealSense 相机配置（`serial_number_or_name`、分辨率、fps）
|
||||
- `dataset.root` / `dataset.repo_id` / `single_task` / `fps` / `episode_time_s` / `reset_time_s` / `num_episodes` — 数据集配置
|
||||
|
||||
### 相机配置说明
|
||||
|
||||
需要给机器人配置添加相机时，参考 `config/manual_mode/xarm7_manual_record_config.yaml` 中的模板：
|
||||
|
||||
```yaml
|
||||
robot:
|
||||
cameras:
|
||||
camera:
|
||||
type: intelrealsense # RealSense 类型，不是 opencv
|
||||
serial_number_or_name: "148522072685"
|
||||
width: 640
|
||||
height: 480
|
||||
fps: 30
|
||||
```
|
||||
|
||||
- `type` 必须是 `intelrealsense`（RealSense 类型），**不能**写成 `opencv`（普通 USB 相机类型）。
|
||||
- `serial_number_or_name` 需要先获取 RealSense 相机序列号再填写，否则连接/录制会报错。获取序列号：
|
||||
|
||||
```bash
|
||||
# 1. 安装 XVSDK（系统级依赖，仅支持 Ubuntu Focal）
|
||||
curl -sL https://raw.githubusercontent.com/xArm-Developer/ufactory_resources/main/fastumi/sdk/XVSDK_focal_amd64.deb -o /tmp/xvsdk.deb && sudo dpkg -i /tmp/xvsdk.deb
|
||||
sudo apt install -y --fix-broken
|
||||
|
||||
# 2. 安装外设依赖
|
||||
uv pip install pysurvive --no-deps
|
||||
|
||||
# 3. 安装 udev 规则（重新插拔设备后生效）
|
||||
sudo cp rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
uv run uf-camera-view -l -T realsense # 列出每台相机的序列号
|
||||
```
|
||||
|
||||
> Vive Tracker 首次使用前需校准：`uf-vive-calibrate`
|
||||
|
||||
**多 UMI 设备配置**（使用两台及以上时）：
|
||||
|
||||
```bash
|
||||
# 增加 USB 缓冲区大小
|
||||
sudo sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/s/quiet splash/quiet splash usbcore.usbfs_memory_mb=128/' /etc/default/grub
|
||||
sync
|
||||
sudo update-grub
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
#### SpaceMouse 遥操作
|
||||
|
||||
适用于 3Dconnexion SpaceMouse / SpaceNavigator。
|
||||
|
||||
```bash
|
||||
# 1. 安装 SpaceMouse 模块
|
||||
uv sync --extra spacemouse
|
||||
|
||||
# 2. 安装 udev 规则（重新插拔设备后生效）
|
||||
sudo cp rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
```
|
||||
也可以使用 librealsense 自带的 `rs-enumerate-devices`。
|
||||
|
||||
## 使用
|
||||
|
||||
### 1. 遥操作测试
|
||||
### 1. GELLO 遥操作测试
|
||||
|
||||
测试遥操作设备与机械臂的联动，不录制数据。
|
||||
不录制数据，仅测试 GELLO 与机械臂的联动：
|
||||
|
||||
```bash
|
||||
# 通用格式
|
||||
uv run uf-robot-teleop --config_path path/to/config.yaml
|
||||
uv run uf-robot-teleop --config_path path/to/config.yaml --fps 60 # 指定频率
|
||||
|
||||
# 示例: xArm6 + UMI 遥操作
|
||||
uv run uf-robot-teleop --config_path config/umi/xarm6_umi_record_config.yaml
|
||||
uv run uf-robot-teleop --config_path config/gello/xarm7_gello_record_config.yaml
|
||||
uv run uf-robot-teleop --config_path config/gello/xarm7_gello_record_config.yaml --fps 60 # 可选，指定循环频率
|
||||
```
|
||||
|
||||
### 2. 人工拖拽数据采集
|
||||
`Space` 复位并开始，`←` 复位，`Esc` 退出。
|
||||
|
||||
人工拖拽录制使用 robot 配置中的 `manual_mode: true`，不需要配置 teleop。录制过程中，机械臂的实际关节状态会作为 observation 和 action 写入 LeRobot 数据集；如果配置了夹爪，还可以按住 `C` 缓慢闭合、按住 `O` 缓慢张开。夹爪速度通过 `manual_gripper_speed` 配置，默认值为 `0.5`：
|
||||
### 2. GELLO 数据采集
|
||||
|
||||
```bash
|
||||
# 录制新数据集
|
||||
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml
|
||||
|
||||
# 在已有数据集上续录
|
||||
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml -r
|
||||
|
||||
# 可选：后台异步保存 episode
|
||||
uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.yaml -a
|
||||
```
|
||||
|
||||
按键控制：`Space` 开始当前 episode，`→` 保存，`←` 放弃并重录，`Esc` 停止录制。每个 episode 之间机械臂会自动复位到初始点。
|
||||
|
||||
> 采集过程中**机械臂与相机（D435 / D435i）的相对位置必须保持不变**，推理时的相机位置必须与采集时一致。若机械臂或相机发生变化，此前采集的数据将失效。
|
||||
|
||||
> 如果数据集目录已存在且未加 `-r`，脚本会询问是覆盖、续录还是取消。
|
||||
|
||||
### 3. 手动拖拽数据采集
|
||||
|
||||
```bash
|
||||
./start_manual_record.sh
|
||||
./start_manual_record.sh -r
|
||||
./start_manual_record.sh -r # 强制续录；数据集目录不存在时会报错
|
||||
```
|
||||
|
||||
启动脚本首次运行时创建数据集，检测到有效数据集时会自动续录。如果提示数据目录为空或不完整，请修改 `dataset.root`，或者确认没有数据后删除该目录。按 `Space` 开始当前 episode，按 `Right` 保存，按 `Left` 放弃并重录当前 episode，按 `Esc` 停止录制。episode 之间可以手动复位机械臂。
|
||||
录制脚本（`uf-lerobot-record`）会从配置读取 `dataset.root` 并在录制前检查路径，然后：
|
||||
|
||||
### 3. 遥操作数据采集
|
||||
- 目录不存在：直接录制新数据集。
|
||||
- 目录已存在（有效 LeRobot 数据集）且未加 `-r`：交互询问：
|
||||
- `o` 覆盖：删除已有数据集，重新录制
|
||||
- `r` 续录：保留已有 episode，继续录制
|
||||
- `c` 取消
|
||||
- 目录存在但不完整（缺少必要元数据或数据 parquet 文件）：询问覆盖或取消；非交互运行时直接报错。
|
||||
|
||||
通过遥操作录制数据集。
|
||||
加 `-r` 可跳过询问直接续录。注意 `./start_manual_record.sh` 保持原有启动脚本行为——检测到有效数据集会自动续录（相当于 `-r`），如果想看到覆盖/续录的询问，请直接用 `uv run uf-lerobot-record --config_path config/manual_mode/xarm7_manual_record_config.yaml` 运行。
|
||||
|
||||
录制时机械臂处于示教模式，实际关节状态会同时作为 observation 和 action 写入数据集。按住 `C` 缓慢闭合夹爪，按住 `O` 缓慢张开。按键控制：`Space` 开始，`→` 保存，`←` 放弃并重录，`Esc` 停止。episode 之间手动复位机械臂。
|
||||
|
||||
> **重要：每个 episode 开始时，必须等待机械臂复位完成后再等待 5 秒，然后才能开始操作；或者确认控制台打印 `Start Recording` 后再开始操作。** 这样可以避免机械臂复位控制指令与操作指令冲突导致报错。
|
||||
|
||||
### 4. 策略训练
|
||||
|
||||
```bash
|
||||
# 通用格式
|
||||
uv run uf-lerobot-record --config_path path/to/record_config.yaml
|
||||
uv run uf-lerobot-record --config_path path/to/config.yaml --resume true # 续录
|
||||
|
||||
# 示例: xArm6 + UMI 数据采集
|
||||
uv run uf-lerobot-record --config_path config/umi/xarm6_umi_record_config.yaml
|
||||
uv run lerobot-train --policy act --dataset ufactory/xarm7_gello_datas
|
||||
```
|
||||
|
||||
### 4. 数据重放
|
||||
|
||||
`datasets/xarm7_manual_replay` 是人工拖拽录制的 LeRobot 数据集。回放脚本使用其中的
|
||||
`observation.state`，将 7 个关节弧度值和归一化夹爪位置作为**绝对目标值**发送给 xArm7，
|
||||
不会将相邻帧相减，也不会累加成相对动作。脚本默认按数据集的 30 FPS 播放一个 episode。
|
||||
|
||||
启动前会要求确认，连接后会先自动移动到 xArm SDK 初始点；播放结束后保持最后一帧姿态并断开连接：
|
||||
带完整训练参数的示例（每 `save_freq` 步保存一次 checkpoint 到 `output_dir`）：
|
||||
|
||||
```bash
|
||||
uv run uf-lerobot-replay \
|
||||
--dataset-root /home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay \
|
||||
--robot-ip 192.168.1.245
|
||||
```
|
||||
|
||||
无人值守运行时可以使用 `--yes` 跳过确认。执行前请确认机械臂工作空间无障碍物，且数据中的初始姿态与当前设备匹配。
|
||||
|
||||
### 5. Lerobot训练
|
||||
|
||||
采集数据后，使用 LeRobot 训练管道进行模仿学习训练。
|
||||
|
||||
```bash
|
||||
# 通用格式
|
||||
uv run lerobot-train --policy act --dataset your_dataset_name
|
||||
```
|
||||
|
||||
参数示例：
|
||||
|
||||
```bash
|
||||
# 注意: repo_id就是采集时配置文件里面的repo_id
|
||||
# 这里训练策略policy.type选用act，训练steps为80w次
|
||||
# 训练过程每2w次保存一次结果，结果输出到和lerobot同级目录下的lerobot_datas/train里面
|
||||
uv run lerobot-train \
|
||||
--dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \
|
||||
--dataset.repo_id=ufactory/xarm6_umi_datas \
|
||||
--dataset.root=/home/<user>/lerobot_datas/record/ufactory/xarm7_gello_datas \
|
||||
--dataset.repo_id=ufactory/xarm7_gello_datas \
|
||||
--policy.type=act \
|
||||
--policy.device=cuda \
|
||||
--policy.repo_id=ufactory/xarm6_umi_datas \
|
||||
--output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \
|
||||
--job_name=xarm6_umi_datas \
|
||||
--policy.repo_id=ufactory/xarm7_gello_datas \
|
||||
--output_dir=/home/<user>/lerobot_datas/train/xarm7_gello_datas \
|
||||
--job_name=xarm7_gello_datas \
|
||||
--steps=800000 \
|
||||
--batch_size=8 \
|
||||
--save_freq=20000
|
||||
```
|
||||
|
||||
### 6. 推理
|
||||
|
||||
指定模型进行推理
|
||||
### 5. 策略推理
|
||||
|
||||
```bash
|
||||
# 通用格式
|
||||
uv run uf-lerobot-eval --config_path path/to/config.yaml --policy.path your_train_path
|
||||
|
||||
# 示例：使用训练好的 ACT 策略进行推理
|
||||
uv run uf-lerobot-eval --config_path config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
|
||||
uv run uf-lerobot-eval \
|
||||
--config_path config/gello/xarm7_gello_record_config.yaml \
|
||||
--policy.path /path/to/train/output/checkpoints/last/pretrained_model/
|
||||
```
|
||||
|
||||
## 工具集
|
||||
`←` / `→` 复位，`Esc` 停止。
|
||||
|
||||
### 1. 摄像头查看器
|
||||
### 6. 回放已录制 episode
|
||||
|
||||
查看和拼接多路摄像头画面。
|
||||
将手动拖拽数据集的绝对关节状态（`observation.state`）回放到 xArm7。脚本按数据集 FPS（默认 30）将状态作为**绝对目标值**发送，不做差分或累加，因此运动轨迹与录制时一致：
|
||||
|
||||
```bash
|
||||
# 默认回放第一条
|
||||
uv run uf-lerobot-replay \
|
||||
--dataset-root /path/to/xarm7_manual_datas \
|
||||
--robot-ip 192.168.1.245
|
||||
|
||||
# 跳过交互确认（无人值守）
|
||||
uv run uf-lerobot-replay --dataset-root /path/to/xarm7_manual_datas --robot-ip 192.168.1.245 --yes
|
||||
|
||||
# 回放其他 episode
|
||||
uv run uf-lerobot-replay --dataset-root /path/to/xarm7_manual_datas --robot-ip 192.168.1.245 --episode-index 3
|
||||
```
|
||||
|
||||
回放开始前机械臂会先移动到 xArm SDK 初始点，播放结束后保持最后一帧姿态并断开连接。执行前请确认工作空间无障碍物，且数据中的初始姿态与当前设备一致。
|
||||
|
||||
## 重要：机械臂开关机事项
|
||||
|
||||
### 开机
|
||||
|
||||
1. **使用网线将电脑连接到机械臂控制器。**
|
||||
2. **参考控制器上标注的机械臂 IP，将电脑以太网接口配置到同一网段**（例如 `192.168.1.xxx`）。
|
||||
3. **在浏览器中访问 `http://192.168.1.245:18333/`**，应出现控制台界面。
|
||||
4. **操作机械臂前，抬起急停按钮。**
|
||||
|
||||
### 关机
|
||||
|
||||
1. **先通过网页控制将机械臂返回到初始位置。**
|
||||
2. **再按下急停按钮。**
|
||||
3. **关闭控制器电源。**
|
||||
|
||||
## 工具
|
||||
|
||||
### 摄像头查看器
|
||||
|
||||
```bash
|
||||
uv run uf-camera-view -l # 列出所有摄像头
|
||||
uv run uf-camera-view -l -T xvisio # 仅列出 XVisio 摄像头
|
||||
uv run uf-camera-view -T xvisio # 查看 XVisio 摄像头（默认 1280x1280 YU12）
|
||||
uv run uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # 指定格式
|
||||
uv run uf-camera-view -T other # 查看其他类型摄像头
|
||||
uv run uf-camera-view -T realsense # 查看 RealSense 摄像头
|
||||
```
|
||||
|
||||
### 2. Lerobot数据集工具
|
||||
Lerobot提供一些数据集工具，方便对采集的数据集进行增删查操作。
|
||||
网页预览会扫描所有 RealSense 摄像头，默认使用 `640x480`、`30fps`，页面中可勾选设备切换或同时显示多路画面：
|
||||
|
||||
### 查看某个索引的episode:
|
||||
例如查看索引号为17的episode:
|
||||
```bash
|
||||
uv run uf-realsense-view
|
||||
```
|
||||
|
||||
脚本使用 LeRobot 的 `RealSenseCamera`。如果当前环境中的 OpenCV 是 LeRobot 默认的
|
||||
headless 版本，会自动启动网页预览。同一台机器上打开
|
||||
`http://127.0.0.1:8765/`；从其他机器访问时，请将 `127.0.0.1` 替换为运行脚本机器的实际 IP。
|
||||
网页服务默认监听 `0.0.0.0`，也可以用 `--host 127.0.0.1` 限制为本机访问；还可以通过
|
||||
`--backend opencv` 强制使用 OpenCV 窗口。
|
||||
|
||||
也可以只打开指定设备；需要多个设备时重复 `--serial`：
|
||||
|
||||
```bash
|
||||
uv run uf-realsense-view \
|
||||
--serial 148522072685 --width 640 --height 480 --fps 30
|
||||
|
||||
uv run uf-realsense-view \
|
||||
--serial 148522072685 --serial SECOND_CAMERA_SERIAL
|
||||
```
|
||||
|
||||
OpenCV 窗口按 `q` 或 `Esc` 退出；网页模式按 `Ctrl+C` 退出。无桌面环境时可以用
|
||||
`--no-display` 检查是否能持续取帧。
|
||||
|
||||
### LeRobot 数据集工具
|
||||
|
||||
```bash
|
||||
# 查看索引为 17 的 episode
|
||||
uv run lerobot-dataset-viz \
|
||||
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
|
||||
--repo-id ufactory/xarm7_record_datas \
|
||||
--root=/path/to/record/ufactory/xarm7_manual_datas \
|
||||
--repo-id ufactory/xarm7_manual_datas \
|
||||
--display-compressed-images true \
|
||||
--episode-index 17
|
||||
```
|
||||
|
||||
### 删除某些索引的episodes:
|
||||
例如删除索引号为18和19的episode:
|
||||
```bash
|
||||
# 删除索引为 18 和 19 的 episode
|
||||
uv run lerobot-edit-dataset \
|
||||
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
|
||||
--repo_id ufactory/xarm7_record_datas \
|
||||
--new_repo_id ../xarm7_record_datas_new \
|
||||
--root=/path/to/record/ufactory/xarm7_manual_datas \
|
||||
--repo_id ufactory/xarm7_manual_datas \
|
||||
--new_repo_id ../xarm7_manual_datas_new \
|
||||
--operation.type delete_episodes \
|
||||
--operation.episode_indices "[18, 19]"
|
||||
```
|
||||
|
||||
### 合并数据集
|
||||
```bash
|
||||
# 合并数据集
|
||||
uv run lerobot-edit-dataset \
|
||||
--root=../../../../lerobot_datas/record \
|
||||
--repo_id ufactory/xarm7_record_datas_merge_1_2 \
|
||||
--root=/path/to/record \
|
||||
--repo_id ufactory/xarm7_datas_merge \
|
||||
--operation.type merge \
|
||||
--operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']"
|
||||
--operation.repo_ids "['ufactory/xarm7_datas_1', 'ufactory/xarm7_datas_2']"
|
||||
```
|
||||
|
||||
|
||||
## 遥操作方式对比
|
||||
|
||||
| 特性 | GELLO | Pika | UMI | SpaceMouse |
|
||||
|------|-------|------|-----|------------|
|
||||
| 控制空间 | 关节空间 | 笛卡尔空间 | 笛卡尔空间 | 笛卡尔空间 |
|
||||
| 跟踪方式 | Dynamixel 舵机 | Vive Tracker | UMI SLAM / Vive | 3D 鼠标 |
|
||||
| 双臂支持 | ❌ | ❌ | ✅ | ❌ |
|
||||
| 系统依赖 | dialout 组 | — | XVSDK deb | — |
|
||||
|
||||
## 项目结构
|
||||
|
||||
```
|
||||
lerobot_robot_ufactory/
|
||||
├── src/
|
||||
│ ├── lerobot_robot_ufactory/ # LeRobot 插件包
|
||||
│ │ ├── robots/ # 机器人控制
|
||||
│ │ │ ├── uf_robot/ # xArm 实体机器人
|
||||
│ │ │ ├── uf_mock_robot/ # 仿真 Mock 机器人
|
||||
│ │ ├── teleoperators/ # 遥操作器
|
||||
│ │ │ ├── base_teleop/ # 共享基类
|
||||
│ │ │ ├── gello_teleop/ # GELLO (Dynamixel 示教臂)
|
||||
│ │ │ ├── pika_teleop/ # Pika Sense (手持示教器 + Vive)
|
||||
│ │ │ ├── umi_teleop/ # UMI (含双机械臂)
|
||||
│ │ │ └── space_mouse/ # SpaceMouse (3D 鼠标)
|
||||
│ │ ├── cameras/ # 摄像头模块
|
||||
│ │ │ └── umi_camera/ # UMI 相机
|
||||
│ │ ├── devices/ # 外部设备驱动
|
||||
│ │ │ ├── pika/ # Pika 串口驱动
|
||||
│ │ │ └── umi/ # XVLib / Vive Tracker
|
||||
│ │ ├── scripts/ # 执行脚本
|
||||
│ │ │ ├── uf_robot_teleop.py # 遥操作测试
|
||||
│ │ │ ├── uf_lerobot_record.py # 数据采集
|
||||
│ │ │ ├── uf_lerobot_eval.py # 策略评估
|
||||
│ │ │ ├── uf_camera_view.py # 摄像头查看工具
|
||||
│ │ │ └── vive_calibrate.py # Vive Tracker 校准
|
||||
│ │ ├── context.py # Teleop 上下文注册
|
||||
│ │ └── utils/ # 工具函数
|
||||
├── config/ # YAML 配置文件
|
||||
│ ├── gello/
|
||||
│ ├── pika/
|
||||
│ ├── umi/
|
||||
│ └── spacemouse/
|
||||
├── rules/ # udev 设备规则
|
||||
lerobot_xarm7/
|
||||
├── config/
|
||||
│ ├── gello/ # xArm7 GELLO 录制配置
|
||||
│ └── manual_mode/ # xArm7 手动拖拽录制配置
|
||||
├── src/lerobot_robot_ufactory/
|
||||
│ ├── robots/
|
||||
│ │ └── uf_robot/ # xArm 控制（关节/笛卡尔空间、示教模式）
|
||||
│ ├── teleoperators/
|
||||
│ │ ├── base_teleop/ # 遥操作基类
|
||||
│ │ └── gello_teleop/ # GELLO（Dynamixel 示教臂）
|
||||
│ ├── scripts/
|
||||
│ │ ├── uf_robot_teleop.py # 遥操作测试
|
||||
│ │ ├── uf_lerobot_record.py # 数据采集（含手动模式）
|
||||
│ │ ├── uf_lerobot_eval.py # 策略推理
|
||||
│ │ ├── uf_lerobot_replay.py # episode 回放
|
||||
│ │ └── uf_camera_view.py # 摄像头查看器
|
||||
│ └── configs/parser.py # 配置加载 / CLI 覆盖
|
||||
├── start_manual_record.sh # 手动拖拽启动脚本
|
||||
├── pyproject.toml
|
||||
└── README.md
|
||||
├── README.md
|
||||
└── README_ZH.md
|
||||
```
|
||||
|
||||
## 重要提示
|
||||
用户需要全面研究整个代码库，并了解相关的配置参数，因为代码中所写的配置并非适用于所有使用场景和设置，所以用户需要研究代码或相关理论，以获取相关知识，并自行进行修改和调整。特别是对于扩散策略(diffusion policy)，LeRobot 中的默认参数可能仅用于模拟，并未针对实际机器人场景进行优化。
|
||||
|
||||
- 提供的配置都是**示例**：请根据实际硬件修改 IP、串口、相机序列号、数据集路径和任务描述。
|
||||
- GELLO 数据采集与推理时，机械臂与相机的相对位姿必须保持一致。
|
||||
- LeRobot 中扩散策略（Diffusion Policy）的默认参数主要面向仿真，**未针对真实机器人优化**，需要根据任务自行调整。
|
||||
- 回放或推理前，请确认工作空间无障碍物，并保证机械臂初始姿态与录制数据一致。
|
||||
|
||||
## 许可证
|
||||
|
||||
|
||||
16
backup/README.md
Normal file
16
backup/README.md
Normal file
@ -0,0 +1,16 @@
|
||||
# Backup
|
||||
|
||||
Previous full versions of the project READMEs, kept before the documentation was
|
||||
rewritten (2026-08) to focus only on the currently used workflows
|
||||
(GELLO teleop and manual drag).
|
||||
|
||||
- `README_old.md` — previous English README (Pika / UMI / SpaceMouse / demo videos, etc.)
|
||||
- `README_ZH_old.md` — previous Chinese README
|
||||
|
||||
The backups are verbatim copies of the files as they existed at commit `b5d46c4`
|
||||
and can be restored with:
|
||||
|
||||
```bash
|
||||
cp backup/README_old.md README.md
|
||||
cp backup/README_ZH_old.md README_ZH.md
|
||||
```
|
||||
338
backup/README_ZH_old.md
Normal file
338
backup/README_ZH_old.md
Normal file
@ -0,0 +1,338 @@
|
||||
# UFACTORY LeRobot
|
||||
|
||||
> [English Version](README.md)
|
||||
|
||||
UFACTORY(深圳市众为创造科技有限公司) 机械臂与 LeRobot 框架集成项目，支持多种遥操作方式的数据采集、策略训练和部署推理。
|
||||
|
||||
## 训练推理效果
|
||||
|
||||
点击下载开发时[采集的数据集](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY)，**仅供参考，不可复用**。因为用户机械臂和摄像头位置和开发测试时不一致。
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<a href="https://www.bilibili.com/video/BV12xFjzzEaX" target="_blank">
|
||||
<img src="https://i2.hdslb.com/bfs/archive/7b325df5fb4c16e922b66d27b56d8fb6534f8b46.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<a href="https://www.bilibili.com/video/BV16ccizHE2P" target="_blank">
|
||||
<img src="https://i2.hdslb.com/bfs/archive/8c5d5e9370577fa89d06a175aceea282d9b2eb9a.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<a href="https://www.bilibili.com/video/BV1xGEy6mE3i" target="_blank">
|
||||
<img src="https://i2.hdslb.com/bfs/archive/c8bbac04a736b7043a20b753e11afea7627bdae2.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
## 功能特性
|
||||
|
||||
- 🤖 UFACTORY 机械臂控制（[xArm 系列](https://www.ufactory.cc/)）
|
||||
- 🎮 多种遥操作方式：GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/)
|
||||
- 📷 多摄像头数据采集（[RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI 相机）
|
||||
- 📊 数据集录制与管理（兼容 LeRobot 格式）
|
||||
- 🧠 模仿学习训练（ACT / Diffusion Policy 等）
|
||||
- 🚀 策略评估与实时推理
|
||||
- 🔧 Mock 机器人模拟（只用遥操作设备采集数据）
|
||||
|
||||
## 环境要求
|
||||
|
||||
- Ubuntu 22.04 / 24.04
|
||||
- Python >= 3.10
|
||||
- CUDA >= 12.0（GPU 训练推荐）
|
||||
- UFACTORY 机械臂（xArm 系列，可选）
|
||||
|
||||
## 安装
|
||||
|
||||
### 基础项目安装
|
||||
|
||||
```bash
|
||||
git clone https://github.com/xArm-Developer/lerobot_robot_ufactory.git
|
||||
cd lerobot_robot_ufactory
|
||||
|
||||
# 创建 uv 虚拟环境并同步项目依赖
|
||||
uv venv --python 3.10
|
||||
uv sync
|
||||
```
|
||||
|
||||
包含：`lerobot==0.4.3`、`xarm-python-sdk`、`numpy`、`pyyaml`（lerobot 已自动携带 torch、opencv、wandb 等训练相关依赖）。
|
||||
|
||||
### 外设模块安装
|
||||
|
||||
外设依赖以可选模块形式提供，通过 `[模块名]` 安装。
|
||||
|
||||
#### GELLO 遥操作
|
||||
|
||||
适用于 GELLO 示教臂（Dynamixel 舵机方案），控制空间为关节空间。
|
||||
* 一旦开始数据采集，机械臂与摄像头（D435 / D435i）的**相对位置必须保持不变**。
|
||||
* 推理时的摄像头位置必须与采集时相同。若机械臂或摄像头发生变化，此前采集的数据将无效。
|
||||
|
||||
```bash
|
||||
# 1. 安装 GELLO 模块
|
||||
uv sync --extra gello
|
||||
|
||||
# 2. 添加串口权限（重新登录后生效）
|
||||
sudo usermod -aG dialout $USER
|
||||
```
|
||||
|
||||
#### Pika 遥操作
|
||||
|
||||
适用于 Pika Sense 手持示教器 + Vive Tracker，控制空间为笛卡尔空间。
|
||||
* 两个基站和机械臂相对位置没有要求，只需要保证采集时pika sense在基站范围内，但**基站移动后需要重新校准**。
|
||||
* 采集和推理时基站位置可不相同。
|
||||
|
||||
```bash
|
||||
# 1. 安装外设依赖（不需要它们的间接依赖）
|
||||
uv pip install pysurvive agx-pypika --no-deps
|
||||
|
||||
# 2. 安装 udev 规则（重新插拔设备后生效）
|
||||
sudo cp rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
```
|
||||
|
||||
> Vive Tracker 首次使用前需校准：`uf-vive-calibrate`
|
||||
|
||||
#### UMI 遥操作
|
||||
|
||||
适用于 UMI（Universal Manipulation Interface）方案，含 Vive Tracker 追踪，支持双机械臂。
|
||||
|
||||
```bash
|
||||
# 1. 安装 XVSDK（系统级依赖，仅支持 Ubuntu Focal）
|
||||
curl -sL https://raw.githubusercontent.com/xArm-Developer/ufactory_resources/main/fastumi/sdk/XVSDK_focal_amd64.deb -o /tmp/xvsdk.deb && sudo dpkg -i /tmp/xvsdk.deb
|
||||
sudo apt install -y --fix-broken
|
||||
|
||||
# 2. 安装外设依赖
|
||||
uv pip install pysurvive --no-deps
|
||||
|
||||
# 3. 安装 udev 规则（重新插拔设备后生效）
|
||||
sudo cp rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
```
|
||||
|
||||
> Vive Tracker 首次使用前需校准：`uf-vive-calibrate`
|
||||
|
||||
**多 UMI 设备配置**（使用两台及以上时）：
|
||||
|
||||
```bash
|
||||
# 增加 USB 缓冲区大小
|
||||
sudo sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/s/quiet splash/quiet splash usbcore.usbfs_memory_mb=128/' /etc/default/grub
|
||||
sync
|
||||
sudo update-grub
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
#### SpaceMouse 遥操作
|
||||
|
||||
适用于 3Dconnexion SpaceMouse / SpaceNavigator。
|
||||
|
||||
```bash
|
||||
# 1. 安装 SpaceMouse 模块
|
||||
uv sync --extra spacemouse
|
||||
|
||||
# 2. 安装 udev 规则（重新插拔设备后生效）
|
||||
sudo cp rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
```
|
||||
|
||||
## 使用
|
||||
|
||||
### 1. 遥操作测试
|
||||
|
||||
测试遥操作设备与机械臂的联动，不录制数据。
|
||||
|
||||
```bash
|
||||
# 通用格式
|
||||
uv run uf-robot-teleop --config_path path/to/config.yaml
|
||||
uv run uf-robot-teleop --config_path path/to/config.yaml --fps 60 # 指定频率
|
||||
|
||||
# 示例: xArm6 + UMI 遥操作
|
||||
uv run uf-robot-teleop --config_path config/umi/xarm6_umi_record_config.yaml
|
||||
```
|
||||
|
||||
### 2. 人工拖拽数据采集
|
||||
|
||||
人工拖拽录制使用 robot 配置中的 `manual_mode: true`，不需要配置 teleop。录制过程中，机械臂的实际关节状态会作为 observation 和 action 写入 LeRobot 数据集；如果配置了夹爪，还可以按住 `C` 缓慢闭合、按住 `O` 缓慢张开。夹爪速度通过 `manual_gripper_speed` 配置，默认值为 `0.5`：
|
||||
|
||||
```bash
|
||||
./start_manual_record.sh
|
||||
./start_manual_record.sh -r
|
||||
```
|
||||
|
||||
启动脚本首次运行时创建数据集，检测到有效数据集时会自动续录。如果提示数据目录为空或不完整，请修改 `dataset.root`，或者确认没有数据后删除该目录。按 `Space` 开始当前 episode，按 `Right` 保存，按 `Left` 放弃并重录当前 episode，按 `Esc` 停止录制。episode 之间可以手动复位机械臂。
|
||||
|
||||
### 3. 遥操作数据采集
|
||||
|
||||
通过遥操作录制数据集。
|
||||
|
||||
```bash
|
||||
# 通用格式
|
||||
uv run uf-lerobot-record --config_path path/to/record_config.yaml
|
||||
uv run uf-lerobot-record --config_path path/to/config.yaml --resume true # 续录
|
||||
|
||||
# 示例: xArm6 + UMI 数据采集
|
||||
uv run uf-lerobot-record --config_path config/umi/xarm6_umi_record_config.yaml
|
||||
```
|
||||
|
||||
### 4. 数据重放
|
||||
|
||||
`datasets/xarm7_manual_replay` 是人工拖拽录制的 LeRobot 数据集。回放脚本使用其中的
|
||||
`observation.state`，将 7 个关节弧度值和归一化夹爪位置作为**绝对目标值**发送给 xArm7，
|
||||
不会将相邻帧相减，也不会累加成相对动作。脚本默认按数据集的 30 FPS 播放一个 episode。
|
||||
|
||||
启动前会要求确认，连接后会先自动移动到 xArm SDK 初始点；播放结束后保持最后一帧姿态并断开连接：
|
||||
|
||||
```bash
|
||||
uv run uf-lerobot-replay \
|
||||
--dataset-root /home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay \
|
||||
--robot-ip 192.168.1.245
|
||||
```
|
||||
|
||||
无人值守运行时可以使用 `--yes` 跳过确认。执行前请确认机械臂工作空间无障碍物，且数据中的初始姿态与当前设备匹配。
|
||||
|
||||
### 5. Lerobot训练
|
||||
|
||||
采集数据后，使用 LeRobot 训练管道进行模仿学习训练。
|
||||
|
||||
```bash
|
||||
# 通用格式
|
||||
uv run lerobot-train --policy act --dataset your_dataset_name
|
||||
```
|
||||
|
||||
参数示例：
|
||||
|
||||
```bash
|
||||
# 注意: repo_id就是采集时配置文件里面的repo_id
|
||||
# 这里训练策略policy.type选用act，训练steps为80w次
|
||||
# 训练过程每2w次保存一次结果，结果输出到和lerobot同级目录下的lerobot_datas/train里面
|
||||
uv run lerobot-train \
|
||||
--dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \
|
||||
--dataset.repo_id=ufactory/xarm6_umi_datas \
|
||||
--policy.type=act \
|
||||
--policy.device=cuda \
|
||||
--policy.repo_id=ufactory/xarm6_umi_datas \
|
||||
--output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \
|
||||
--job_name=xarm6_umi_datas \
|
||||
--steps=800000 \
|
||||
--batch_size=8 \
|
||||
--save_freq=20000
|
||||
```
|
||||
|
||||
### 6. 推理
|
||||
|
||||
指定模型进行推理
|
||||
|
||||
```bash
|
||||
# 通用格式
|
||||
uv run uf-lerobot-eval --config_path path/to/config.yaml --policy.path your_train_path
|
||||
|
||||
# 示例：使用训练好的 ACT 策略进行推理
|
||||
uv run uf-lerobot-eval --config_path config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
|
||||
```
|
||||
|
||||
## 工具集
|
||||
|
||||
### 1. 摄像头查看器
|
||||
|
||||
查看和拼接多路摄像头画面。
|
||||
|
||||
```bash
|
||||
uv run uf-camera-view -l # 列出所有摄像头
|
||||
uv run uf-camera-view -l -T xvisio # 仅列出 XVisio 摄像头
|
||||
uv run uf-camera-view -T xvisio # 查看 XVisio 摄像头（默认 1280x1280 YU12）
|
||||
uv run uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # 指定格式
|
||||
uv run uf-camera-view -T other # 查看其他类型摄像头
|
||||
```
|
||||
|
||||
### 2. Lerobot数据集工具
|
||||
Lerobot提供一些数据集工具，方便对采集的数据集进行增删查操作。
|
||||
|
||||
### 查看某个索引的episode:
|
||||
例如查看索引号为17的episode:
|
||||
```bash
|
||||
uv run lerobot-dataset-viz \
|
||||
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
|
||||
--repo-id ufactory/xarm7_record_datas \
|
||||
--display-compressed-images true \
|
||||
--episode-index 17
|
||||
```
|
||||
|
||||
### 删除某些索引的episodes:
|
||||
例如删除索引号为18和19的episode:
|
||||
```bash
|
||||
uv run lerobot-edit-dataset \
|
||||
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
|
||||
--repo_id ufactory/xarm7_record_datas \
|
||||
--new_repo_id ../xarm7_record_datas_new \
|
||||
--operation.type delete_episodes \
|
||||
--operation.episode_indices "[18, 19]"
|
||||
```
|
||||
|
||||
### 合并数据集
|
||||
```bash
|
||||
uv run lerobot-edit-dataset \
|
||||
--root=../../../../lerobot_datas/record \
|
||||
--repo_id ufactory/xarm7_record_datas_merge_1_2 \
|
||||
--operation.type merge \
|
||||
--operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']"
|
||||
```
|
||||
|
||||
|
||||
## 遥操作方式对比
|
||||
|
||||
| 特性 | GELLO | Pika | UMI | SpaceMouse |
|
||||
|------|-------|------|-----|------------|
|
||||
| 控制空间 | 关节空间 | 笛卡尔空间 | 笛卡尔空间 | 笛卡尔空间 |
|
||||
| 跟踪方式 | Dynamixel 舵机 | Vive Tracker | UMI SLAM / Vive | 3D 鼠标 |
|
||||
| 双臂支持 | ❌ | ❌ | ✅ | ❌ |
|
||||
| 系统依赖 | dialout 组 | — | XVSDK deb | — |
|
||||
|
||||
## 项目结构
|
||||
|
||||
```
|
||||
lerobot_robot_ufactory/
|
||||
├── src/
|
||||
│ ├── lerobot_robot_ufactory/ # LeRobot 插件包
|
||||
│ │ ├── robots/ # 机器人控制
|
||||
│ │ │ ├── uf_robot/ # xArm 实体机器人
|
||||
│ │ │ ├── uf_mock_robot/ # 仿真 Mock 机器人
|
||||
│ │ ├── teleoperators/ # 遥操作器
|
||||
│ │ │ ├── base_teleop/ # 共享基类
|
||||
│ │ │ ├── gello_teleop/ # GELLO (Dynamixel 示教臂)
|
||||
│ │ │ ├── pika_teleop/ # Pika Sense (手持示教器 + Vive)
|
||||
│ │ │ ├── umi_teleop/ # UMI (含双机械臂)
|
||||
│ │ │ └── space_mouse/ # SpaceMouse (3D 鼠标)
|
||||
│ │ ├── cameras/ # 摄像头模块
|
||||
│ │ │ └── umi_camera/ # UMI 相机
|
||||
│ │ ├── devices/ # 外部设备驱动
|
||||
│ │ │ ├── pika/ # Pika 串口驱动
|
||||
│ │ │ └── umi/ # XVLib / Vive Tracker
|
||||
│ │ ├── scripts/ # 执行脚本
|
||||
│ │ │ ├── uf_robot_teleop.py # 遥操作测试
|
||||
│ │ │ ├── uf_lerobot_record.py # 数据采集
|
||||
│ │ │ ├── uf_lerobot_eval.py # 策略评估
|
||||
│ │ │ ├── uf_camera_view.py # 摄像头查看工具
|
||||
│ │ │ └── vive_calibrate.py # Vive Tracker 校准
|
||||
│ │ ├── context.py # Teleop 上下文注册
|
||||
│ │ └── utils/ # 工具函数
|
||||
├── config/ # YAML 配置文件
|
||||
│ ├── gello/
|
||||
│ ├── pika/
|
||||
│ ├── umi/
|
||||
│ └── spacemouse/
|
||||
├── rules/ # udev 设备规则
|
||||
├── pyproject.toml
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## 重要提示
|
||||
用户需要全面研究整个代码库，并了解相关的配置参数，因为代码中所写的配置并非适用于所有使用场景和设置，所以用户需要研究代码或相关理论，以获取相关知识，并自行进行修改和调整。特别是对于扩散策略(diffusion policy)，LeRobot 中的默认参数可能仅用于模拟，并未针对实际机器人场景进行优化。
|
||||
|
||||
|
||||
## 许可证
|
||||
|
||||
本项目基于 Apache License 2.0 发布，详见 [LICENSE](LICENSE) 文件。
|
||||
329
backup/README_old.md
Normal file
329
backup/README_old.md
Normal file
@ -0,0 +1,329 @@
|
||||
# UFACTORY LeRobot
|
||||
|
||||
> [中文版本](README_ZH.md)
|
||||
|
||||
UFACTORY robot arm integration with the LeRobot framework for robot learning, data collection, and policy deployment.
|
||||
|
||||
## Training & Inference Results
|
||||
|
||||
[Test datasets](https://drive.google.com/drive/folders/1Ms25rd2YYGdh3tHPEsTTMU-m1fE7uNYY) used during development, **for reference only, do NOT reuse**. As the robot arm and camera positions during development differ from user setups.
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<a href="https://www.youtube.com/watch?v=wTiWLiHciT8" target="_blank">
|
||||
<img src="https://img.youtube.com/vi/wTiWLiHciT8/maxresdefault.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
<td width="50%">
|
||||
<a href="https://www.youtube.com/watch?v=IiyvewZh5OY" target="_blank">
|
||||
<img src="https://img.youtube.com/vi/IiyvewZh5OY/maxresdefault.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td width="50%">
|
||||
<a href="https://youtu.be/wBwZH6POk38" target="_blank">
|
||||
<img src="https://img.youtube.com/vi/wBwZH6POk38/maxresdefault.jpg" width="100%">
|
||||
</a>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
## Features
|
||||
|
||||
- 🤖 UFACTORY robot control ([xArm series](https://www.ufactory.cc/))
|
||||
- 🎮 Multiple teleop modes: GELLO / [Pika](https://global.agilex.ai/products/pika) / [UMI](https://lumosumi.lumosbot.tech/pro/) / [SpaceMouse](https://3dconnexion.com/sg/product/spacemouse-wireless/)
|
||||
- 📷 Multi-camera data collection ([RealSense](https://www.realsenseai.com/products/depth-camera-d435i/) / UMI camera)
|
||||
- 📊 Dataset recording & management (LeRobot-compatible)
|
||||
- 🧠 Imitation learning training (ACT / Diffusion Policy / etc.)
|
||||
- 🚀 Policy evaluation & real-time inference
|
||||
- 🔧 Mock mode (teleop device only, no physical robot needed)
|
||||
|
||||
## Requirements
|
||||
|
||||
- Ubuntu 22.04 / 24.04
|
||||
- Python >= 3.10
|
||||
- CUDA >= 12.0 (recommended for GPU training)
|
||||
- UFACTORY xArm (optional)
|
||||
|
||||
## Installation
|
||||
|
||||
### Base Install
|
||||
|
||||
```bash
|
||||
git clone https://github.com/xArm-Developer/lerobot_robot_ufactory.git
|
||||
cd lerobot_robot_ufactory
|
||||
|
||||
# Create a uv virtual environment and sync project dependencies
|
||||
uv venv --python 3.10
|
||||
uv sync
|
||||
```
|
||||
|
||||
Includes: `lerobot==0.4.3`, `xarm-python-sdk`, `numpy`, `pyyaml`. LeRobot already pulls in torch, opencv, wandb, etc.
|
||||
|
||||
### Peripheral Modules
|
||||
|
||||
Peripheral dependencies are available as optional extras via `[module]` install.
|
||||
|
||||
#### GELLO Teleop
|
||||
|
||||
Dynamixel-based leader arm, joint-space control.
|
||||
* Once data collection starts, the **relative position** between the robot arm and camera (D435 / D435i) **must remain unchanged**.
|
||||
* The camera position during inference must match the collection setup. If the robot arm or camera changes, previously collected data becomes invalid.
|
||||
|
||||
```bash
|
||||
# 1. Install GELLO module
|
||||
uv sync --extra gello
|
||||
|
||||
# 2. Add serial port permissions (re-login required)
|
||||
sudo usermod -aG dialout $USER
|
||||
```
|
||||
|
||||
#### Pika Teleop
|
||||
|
||||
Pika Sense handheld + Vive Tracker, Cartesian-space control.
|
||||
* No requirement for the relative position of the two base stations and the robot arm. Only need to ensure the Pika Sense is within base station range during collection, but **base stations must be recalibrated after moving**.
|
||||
* Base station positions for collection and inference do not need to be the same.
|
||||
|
||||
```bash
|
||||
# 1. Install peripheral deps (skip transitive deps)
|
||||
uv pip install pysurvive agx-pypika --no-deps
|
||||
|
||||
# 2. Install udev rules (re-plug devices afterwards)
|
||||
sudo cp src/rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
```
|
||||
|
||||
> Calibrate Vive Tracker before first use: `uf-vive-calibrate`
|
||||
|
||||
#### UMI Teleop
|
||||
|
||||
Universal Manipulation Interface + Vive Tracker, supports dual-arm.
|
||||
|
||||
```bash
|
||||
# 1. Install XVSDK (system-level, Ubuntu Focal only)
|
||||
curl -sL https://raw.githubusercontent.com/xArm-Developer/ufactory_resources/main/fastumi/sdk/XVSDK_focal_amd64.deb -o /tmp/xvsdk.deb && sudo dpkg -i /tmp/xvsdk.deb
|
||||
sudo apt install -y --fix-broken
|
||||
|
||||
# 2. Install peripheral deps
|
||||
uv pip install pysurvive --no-deps
|
||||
|
||||
# 3. Install udev rules (re-plug devices afterwards)
|
||||
sudo cp src/rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
```
|
||||
|
||||
> Calibrate Vive Tracker before first use: `uf-vive-calibrate`
|
||||
|
||||
**Multi-UMI device configuration** (two or more devices):
|
||||
|
||||
```bash
|
||||
# Increase USB buffer size
|
||||
sudo sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/s/quiet splash/quiet splash usbcore.usbfs_memory_mb=128/' /etc/default/grub
|
||||
sync
|
||||
sudo update-grub
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
#### SpaceMouse Teleop
|
||||
|
||||
3Dconnexion SpaceMouse / SpaceNavigator.
|
||||
|
||||
```bash
|
||||
# 1. Install SpaceMouse module
|
||||
uv sync --extra spacemouse
|
||||
|
||||
# 2. Install udev rules (re-plug device afterwards)
|
||||
sudo cp src/rules/*.rules /etc/udev/rules.d/
|
||||
sudo udevadm control --reload-rules && sudo udevadm trigger
|
||||
```
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
### 1. Teleop Testing
|
||||
|
||||
Test teleop-to-robot control loop without recording.
|
||||
|
||||
```bash
|
||||
# Generic usage
|
||||
uv run uf-robot-teleop --config_path path/to/config.yaml
|
||||
uv run uf-robot-teleop --config_path path/to/config.yaml --fps 60 # specify frequency
|
||||
|
||||
# Example: xArm6 + UMI teleop
|
||||
uv run uf-robot-teleop --config_path config/umi/xarm6_umi_record_config.yaml
|
||||
```
|
||||
|
||||
### 2. Manual Drag Data Collection
|
||||
|
||||
Manual drag recording uses `manual_mode: true` in the robot configuration and does not configure a teleoperator. During recording, the actual joint state is written as both the observation and action in the LeRobot dataset. When a gripper is configured, hold `C` to close it slowly and `O` to open it slowly. Adjust the speed with `manual_gripper_speed`, which defaults to `0.5`:
|
||||
|
||||
```bash
|
||||
./start_manual_record.sh
|
||||
./start_manual_record.sh -r
|
||||
```
|
||||
|
||||
The launcher creates a dataset on its first run and automatically resumes an existing valid dataset. If it reports an empty or incomplete dataset directory, choose a new `dataset.root` or remove that directory after confirming it contains no data. Press `Space` to start an episode, `Right` to save it, `Left` to discard and re-record it, and `Esc` to stop recording. Reset the arm manually between episodes.
|
||||
|
||||
### 3. Teleop Data Collection
|
||||
|
||||
Record datasets via teleop.
|
||||
|
||||
```bash
|
||||
# Generic usage
|
||||
uv run uf-lerobot-record --config_path path/to/record_config.yaml
|
||||
uv run uf-lerobot-record --config_path path/to/config.yaml --resume true # resume recording
|
||||
|
||||
# Example: xArm6 + UMI data collection
|
||||
uv run uf-lerobot-record --config_path config/umi/xarm6_umi_record_config.yaml
|
||||
```
|
||||
|
||||
### 5. Policy Training
|
||||
|
||||
Train imitation learning policies on collected data.
|
||||
|
||||
```bash
|
||||
# Generic usage
|
||||
uv run lerobot-train --policy act --dataset your_dataset_name
|
||||
|
||||
# Example: train ACT on xArm6 UMI dataset
|
||||
uv run lerobot-train --policy act --dataset ufactory/xarm6_umi_datas
|
||||
```
|
||||
|
||||
Important parameters:
|
||||
|
||||
```bash
|
||||
# Note: repo_id is the same as in the record config
|
||||
# Policy type: ACT, training steps: 800k
|
||||
# Checkpoints saved every 20k steps, output to lerobot_datas/train (sibling of lerobot directory)
|
||||
uv run lerobot-train \
|
||||
--dataset.root=../../../../lerobot_datas/record/ufactory/xarm6_umi_datas \
|
||||
--dataset.repo_id=ufactory/xarm6_umi_datas \
|
||||
--policy.type=act \
|
||||
--policy.device=cuda \
|
||||
--policy.repo_id=ufactory/xarm6_umi_datas \
|
||||
--output_dir=../../../../lerobot_datas/train/xarm6_umi_datas \
|
||||
--job_name=xarm6_umi_datas \
|
||||
--steps=800000 \
|
||||
--batch_size=8 \
|
||||
--save_freq=20000
|
||||
```
|
||||
|
||||
### 6. Inference & Evaluation
|
||||
|
||||
Run inference with a trained policy.
|
||||
|
||||
```bash
|
||||
# Generic usage
|
||||
uv run uf-lerobot-eval --config_path path/to/config.yaml --policy.path your_train_path
|
||||
|
||||
# Example: run inference with trained ACT policy
|
||||
uv run uf-lerobot-eval --config_path config/umi/xarm6_umi_record_config.yaml --policy.path ../../../../lerobot_datas/train/xarm6_umi_datas/checkpoints/last/pretrained_model/
|
||||
```
|
||||
|
||||
## Tools
|
||||
|
||||
### 1. Camera Viewer
|
||||
|
||||
View and stitch multiple camera feeds.
|
||||
|
||||
```bash
|
||||
uv run uf-camera-view -l # list all cameras
|
||||
uv run uf-camera-view -l -T xvisio # list XVisio cameras only
|
||||
uv run uf-camera-view -T xvisio # view XVisio cameras (default 1280x1280 YU12)
|
||||
uv run uf-camera-view -T xvisio -W 640 -H 1920 -F NV12 # specify format
|
||||
uv run uf-camera-view -T other # view other camera types
|
||||
```
|
||||
|
||||
### 2. LeRobot Dataset Tools
|
||||
|
||||
LeRobot provides dataset utilities for inspecting, editing and managing collected datasets.
|
||||
|
||||
#### View an episode:
|
||||
e.g. view episode index 17:
|
||||
```bash
|
||||
uv run lerobot-dataset-viz \
|
||||
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
|
||||
--repo-id ufactory/xarm7_record_datas \
|
||||
--display-compressed-images true \
|
||||
--episode-index 17
|
||||
```
|
||||
|
||||
#### Delete specific episodes:
|
||||
e.g. delete episodes 18 and 19:
|
||||
```bash
|
||||
uv run lerobot-edit-dataset \
|
||||
--root=../../../../lerobot_datas/record/ufactory/xarm7_record_datas \
|
||||
--repo_id ufactory/xarm7_record_datas \
|
||||
--new_repo_id ../xarm7_record_datas_new \
|
||||
--operation.type delete_episodes \
|
||||
--operation.episode_indices "[18, 19]"
|
||||
```
|
||||
|
||||
#### Merge datasets:
|
||||
```bash
|
||||
uv run lerobot-edit-dataset \
|
||||
--root=../../../../lerobot_datas/record \
|
||||
--repo_id ufactory/xarm7_record_datas_merge_1_2 \
|
||||
--operation.type merge \
|
||||
--operation.repo_ids "['ufactory/xarm7_record_datas_1', 'ufactory/xarm7_record_datas_2']"
|
||||
```
|
||||
|
||||
## Teleop Comparison
|
||||
|
||||
| Feature | GELLO | Pika | UMI | SpaceMouse |
|
||||
|---------|-------|------|-----|------------|
|
||||
| Control space | Joint space | Cartesian space | Cartesian space | Cartesian space |
|
||||
| Tracking | Dynamixel servos | Vive Tracker | UMI SLAM / Vive | 3D mouse |
|
||||
| Dual-arm | ❌ | ❌ | ✅ | ❌ |
|
||||
| System dep | dialout group | — | XVSDK deb | — |
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
lerobot_robot_ufactory/
|
||||
├── src/
|
||||
│ ├── lerobot_robot_ufactory/ # LeRobot plugin package
|
||||
│ │ ├── robots/ # Robot control
|
||||
│ │ │ ├── uf_robot/ # xArm physical robot
|
||||
│ │ │ ├── uf_mock_robot/ # Mock robot simulator
|
||||
│ │ ├── teleoperators/ # Teleop drivers
|
||||
│ │ │ ├── base_teleop/ # Shared base class
|
||||
│ │ │ ├── gello_teleop/ # GELLO (Dynamixel leader)
|
||||
│ │ │ ├── pika_teleop/ # Pika Sense (handheld + Vive)
|
||||
│ │ │ ├── umi_teleop/ # UMI (dual-arm support)
|
||||
│ │ │ └── space_mouse/ # SpaceMouse (3D mouse)
|
||||
│ │ ├── cameras/ # Camera modules
|
||||
│ │ │ └── umi_camera/ # UMI camera
|
||||
│ │ ├── devices/ # External device drivers
|
||||
│ │ │ ├── pika/ # Pika serial driver
|
||||
│ │ │ └── umi/ # XVLib / Vive Tracker
|
||||
│ │ ├── scripts/ # Entry-point scripts
|
||||
│ │ │ ├── uf_robot_teleop.py # Teleop testing
|
||||
│ │ │ ├── uf_lerobot_record.py # Data recording
|
||||
│ │ │ ├── uf_lerobot_eval.py # Policy evaluation
|
||||
│ │ │ ├── uf_camera_view.py # Camera viewer tool
|
||||
│ │ │ └── vive_calibrate.py # Vive Tracker calibration
|
||||
│ │ ├── context.py # Teleop context registry
|
||||
│ │ └── utils/ # Utilities
|
||||
├── config/ # YAML config files
|
||||
│ ├── gello/
|
||||
│ ├── pika/
|
||||
│ ├── umi/
|
||||
│ └── spacemouse/
|
||||
├── rules/ # udev device rules
|
||||
├── pyproject.toml
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Important Notes
|
||||
|
||||
Users are expected to thoroughly study the codebase and configuration parameters.
|
||||
The provided configurations are **not guaranteed to work for all scenarios** and must be adjusted based on actual hardware setups and task requirements.
|
||||
|
||||
In particular, for **diffusion policies**, the default parameters in LeRobot are primarily designed for simulation and **are not optimized for real-world robots**.
|
||||
|
||||
## License
|
||||
|
||||
This project is released under the Apache License 2.0. See [LICENSE](LICENSE).
|
||||
@ -24,10 +24,10 @@ robot:
|
||||
fps: 30
|
||||
|
||||
dataset:
|
||||
root: "/home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay_pick_pen"
|
||||
repo_id: "ufactory/xarm7_manual_datas"
|
||||
root: "/home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay_pick_bottle"
|
||||
repo_id: "zjuscl/xarm7_manual_replay_pick_bottle"
|
||||
# Task description stored with each recorded frame.
|
||||
single_task: "Describe the task being demonstrated."
|
||||
single_task: "Pick up the black bottle and place it on the blue bag"
|
||||
fps: 30
|
||||
episode_time_s: 60
|
||||
# Reserved reset duration between episodes in seconds.
|
||||
|
||||
@ -39,6 +39,7 @@ uf-lerobot-eval = "lerobot_robot_ufactory.scripts.uf_lerobot_eval:main"
|
||||
uf-lerobot-replay = "lerobot_robot_ufactory.scripts.uf_lerobot_replay:main"
|
||||
uf-vive-calibrate = "lerobot_robot_ufactory.scripts.vive_calibrate:main"
|
||||
uf-camera-view = "lerobot_robot_ufactory.scripts.uf_camera_view:main"
|
||||
uf-realsense-view = "lerobot_robot_ufactory.scripts.uf_realsense_view:main"
|
||||
uf-camera-test = "lerobot_robot_ufactory.scripts.uf_camera_test:main"
|
||||
|
||||
[project.optional-dependencies]
|
||||
|
||||
4
replay_trajectory.sh
Executable file
4
replay_trajectory.sh
Executable file
@ -0,0 +1,4 @@
|
||||
#!/bin/bash
|
||||
uv run uf-lerobot-replay \
|
||||
--dataset-root /home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay_pick_bottle \
|
||||
--robot-ip 192.168.1.245
|
||||
@ -449,9 +449,9 @@ class UFRobot(Robot, Thread):
|
||||
warn_code = getattr(arm, "warn_code", "unknown")
|
||||
return f"mode={mode}, state={state}, error_code={error_code}, warn_code={warn_code}"
|
||||
|
||||
def _check_motion_code(self, command: str, code: int) -> None:
|
||||
"""Fail loudly when the SDK rejects a joint command."""
|
||||
if code != 0:
|
||||
def _check_motion_code(self, command: str, code: int | None) -> None:
|
||||
"""Fail loudly when the SDK explicitly rejects a motion command."""
|
||||
if code is not None and code != 0:
|
||||
raise RuntimeError(f"{command} failed, code={code}, {self._motion_status()}")
|
||||
|
||||
def send_action(self, action: dict) -> np.ndarray:
|
||||
|
||||
@ -38,6 +38,7 @@ from lerobot.configs.policies import PreTrainedConfig
|
||||
from lerobot.scripts.lerobot_record import DatasetRecordConfig
|
||||
from lerobot.datasets.lerobot_dataset import LeRobotDataset
|
||||
from lerobot_robot_ufactory.utils.utils import init_keyboard_listener
|
||||
from lerobot_robot_ufactory.utils.action_safety import ActionSafetyConfig, ActionSafetyGuard
|
||||
from lerobot_robot_ufactory.devices.umi.vive_tracker.transformations import Transformations
|
||||
|
||||
|
||||
@ -120,9 +121,11 @@ class EvalConfig:
|
||||
return ["policy"]
|
||||
|
||||
|
||||
def eval_loop(cfg: EvalConfig, relative=False, rx_continuous=False):
|
||||
def eval_loop(cfg: EvalConfig, relative=False, rx_continuous=False, safety_guard: ActionSafetyGuard | None = None):
|
||||
init_logging()
|
||||
logging.info(pformat(asdict(cfg)))
|
||||
if safety_guard is not None:
|
||||
safety_guard.log_config()
|
||||
|
||||
robot = make_robot_from_config(cfg.robot)
|
||||
|
||||
@ -261,6 +264,8 @@ def eval_loop(cfg: EvalConfig, relative=False, rx_continuous=False):
|
||||
prev_robot_dict[key] = {'type': 1 if is_tcp else 0, 'pose': np.array(pose)}
|
||||
prev_action_dict[key] = {'type': 1 if is_tcp else 0, 'pose': np.array(pose)}
|
||||
|
||||
safety_halted = False
|
||||
|
||||
while True:
|
||||
start_loop_t = time.perf_counter()
|
||||
|
||||
@ -269,6 +274,12 @@ def eval_loop(cfg: EvalConfig, relative=False, rx_continuous=False):
|
||||
print("\n********** Policy Eval Episode (Reset) **********")
|
||||
break
|
||||
|
||||
# Safety halt: stop inference and stop sending actions,
|
||||
# wait for the operator to press the right arrow key to resume
|
||||
if safety_halted:
|
||||
precise_sleep(sleep_time_s)
|
||||
continue
|
||||
|
||||
# Get robot observation
|
||||
obs = robot.get_observation()
|
||||
|
||||
@ -398,6 +409,16 @@ def eval_loop(cfg: EvalConfig, relative=False, rx_continuous=False):
|
||||
# gripper_raw = (future_gripper_norm + 1) / 2 * (_gripper_max - _gripper_min) + _gripper_min
|
||||
# robot_action_to_send['right.gripper.pos'] = 1.0 if gripper_raw > 0.4 else 0.0
|
||||
|
||||
# Safety check: any violation triggers an e-stop; the action is
|
||||
# NOT sent and the loop waits for the operator to press right arrow
|
||||
if safety_guard is not None:
|
||||
violation = safety_guard.check(robot_action_to_send, curr_robot_dict, keys)
|
||||
if violation is not None:
|
||||
safety_halted = True
|
||||
logging.error(f"*** SAFETY HALT *** {violation}")
|
||||
print(f"\n*** SAFETY HALT *** {violation}\nAction was NOT sent. Press right arrow (->) to reset and resume, ESC to exit.")
|
||||
continue
|
||||
|
||||
robot.send_action(robot_action_to_send)
|
||||
|
||||
dt_s = time.perf_counter() - start_loop_t
|
||||
@ -418,11 +439,15 @@ def main():
|
||||
parser = argparse.ArgumentParser(description='configuration args')
|
||||
parser.add_argument('--relative', action='store_true', help='is relative motion or not')
|
||||
parser.add_argument('--rx_continuous', action='store_true', help='rx continuous or not')
|
||||
parser.add_argument('--enable_safety', action='store_true', help='enable action safety guard (thresholds in ActionSafetyConfig)')
|
||||
args, unknown = parser.parse_known_args()
|
||||
sys.argv = [sys.argv[0]] + unknown
|
||||
register_third_party_plugins()
|
||||
cfg = get_cfg()
|
||||
eval_loop(cfg, args.relative, args.rx_continuous)
|
||||
# Action safety guard: tune thresholds in ActionSafetyConfig directly.
|
||||
# Any violation triggers an e-stop; press right arrow to resume.
|
||||
safety_guard = ActionSafetyGuard(ActionSafetyConfig(enabled=args.enable_safety))
|
||||
eval_loop(cfg, args.relative, args.rx_continuous, safety_guard)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@ -493,12 +493,102 @@ def _print_record_controls(is_recorded, manual_mode):
|
||||
print(f'⌨ {controls}')
|
||||
|
||||
|
||||
def _ask_choice(prompt: str, options: dict[str, str]) -> str:
|
||||
"""Prompt the user to pick one of the given options (lowercase keys)."""
|
||||
keys = "/".join(options)
|
||||
while True:
|
||||
print(f"\n{prompt}")
|
||||
for key, description in options.items():
|
||||
print(f" [{key}] {description}")
|
||||
try:
|
||||
choice = input(f"Choose [{keys}]: ").strip().lower()
|
||||
except EOFError:
|
||||
print("No input available, cancelling.")
|
||||
raise SystemExit(1)
|
||||
if choice in options:
|
||||
return choice
|
||||
print(f"Invalid choice, please enter {keys}.")
|
||||
|
||||
|
||||
def _missing_dataset_files(root: Path) -> list[str]:
|
||||
"""Return the local files required before a dataset can be resumed."""
|
||||
missing = []
|
||||
for relative_path in ("meta/info.json", "meta/tasks.parquet"):
|
||||
if not (root / relative_path).is_file():
|
||||
missing.append(relative_path)
|
||||
if not any((root / "meta" / "episodes").glob("*/*.parquet")):
|
||||
missing.append("meta/episodes/*/*.parquet")
|
||||
if not any((root / "data").glob("*/*.parquet")):
|
||||
missing.append("data/*/*.parquet")
|
||||
return missing
|
||||
|
||||
|
||||
def _prepare_dataset_root(cfg: UFRecordConfig) -> None:
|
||||
"""Prepare an existing dataset root without pre-creating a new one."""
|
||||
root = Path(cfg.dataset.root)
|
||||
existed = root.exists()
|
||||
|
||||
if not existed:
|
||||
if cfg.resume:
|
||||
raise RuntimeError(f"Cannot resume because the dataset directory does not exist: {root}")
|
||||
return
|
||||
|
||||
missing = _missing_dataset_files(root)
|
||||
if missing:
|
||||
missing_text = ", ".join(missing)
|
||||
message = (
|
||||
f"Dataset directory is incomplete and cannot be resumed: {root}\n"
|
||||
f"Missing: {missing_text}"
|
||||
)
|
||||
if cfg.resume or not sys.stdin.isatty():
|
||||
raise RuntimeError(message)
|
||||
choice = _ask_choice(
|
||||
message,
|
||||
options={
|
||||
"o": "Overwrite: remove this directory and record a new dataset",
|
||||
"c": "Cancel",
|
||||
},
|
||||
)
|
||||
if choice == "o":
|
||||
shutil.rmtree(root)
|
||||
else:
|
||||
raise SystemExit("Recording cancelled.")
|
||||
return
|
||||
|
||||
if cfg.resume:
|
||||
return
|
||||
|
||||
# A valid LeRobot dataset already exists.
|
||||
if not sys.stdin.isatty():
|
||||
# Non-interactive run: keep the previous auto-resume behaviour.
|
||||
cfg.resume = True
|
||||
print(f"Existing dataset found, resuming recording (non-interactive): {root}")
|
||||
return
|
||||
|
||||
choice = _ask_choice(
|
||||
f"Dataset directory already exists: {root}",
|
||||
options={
|
||||
"o": "Overwrite: delete the existing dataset and record a new one",
|
||||
"r": "Resume: keep existing episodes and continue recording",
|
||||
"c": "Cancel",
|
||||
},
|
||||
)
|
||||
if choice == "o":
|
||||
shutil.rmtree(root)
|
||||
elif choice == "r":
|
||||
cfg.resume = True
|
||||
else:
|
||||
raise SystemExit("Recording cancelled.")
|
||||
|
||||
|
||||
def record(cfg: UFRecordConfig, async_save: bool = False) -> LeRobotDataset:
|
||||
init_logging()
|
||||
logging.info(pformat(asdict(cfg)))
|
||||
if cfg.display_data:
|
||||
init_rerun(session_name="recording")
|
||||
|
||||
_prepare_dataset_root(cfg)
|
||||
|
||||
robot = make_robot_from_config(cfg.robot)
|
||||
teleop = make_teleoperator_from_config(cfg.teleop) if cfg.teleop is not None else None
|
||||
manual_mode = bool(getattr(cfg.robot, "manual_mode", False))
|
||||
|
||||
570
src/lerobot_robot_ufactory/scripts/uf_realsense_view.py
Normal file
570
src/lerobot_robot_ufactory/scripts/uf_realsense_view.py
Normal file
@ -0,0 +1,570 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Web viewer for one or more Intel RealSense color streams."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import http.server
|
||||
import json
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
from urllib.parse import parse_qs, urlsplit
|
||||
|
||||
import cv2
|
||||
import pyrealsense2 as rs
|
||||
|
||||
from lerobot.cameras import ColorMode
|
||||
from lerobot.cameras.realsense import RealSenseCamera, RealSenseCameraConfig
|
||||
|
||||
|
||||
DEFAULT_SERIAL = None
|
||||
DEFAULT_WIDTH = 640
|
||||
DEFAULT_HEIGHT = 480
|
||||
DEFAULT_FPS = 30
|
||||
DEFAULT_HOST = "0.0.0.0"
|
||||
DEFAULT_PORT = 8765
|
||||
|
||||
|
||||
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Preview one or more Intel RealSense color streams in a browser"
|
||||
)
|
||||
parser.add_argument(
|
||||
"-s",
|
||||
"--serial",
|
||||
action="append",
|
||||
default=DEFAULT_SERIAL,
|
||||
help="camera serial number; repeat to limit the device list to selected cameras",
|
||||
)
|
||||
parser.add_argument("-W", "--width", type=int, default=DEFAULT_WIDTH, help="color width")
|
||||
parser.add_argument("-H", "--height", type=int, default=DEFAULT_HEIGHT, help="color height")
|
||||
parser.add_argument(
|
||||
"-F", "--fps", type=int, default=DEFAULT_FPS, help="capture and preview FPS (default: 30)"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--backend",
|
||||
choices=("auto", "opencv", "web"),
|
||||
default="web",
|
||||
help="preview backend (default: web; web provides device selection)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--host",
|
||||
default=DEFAULT_HOST,
|
||||
help="web server bind address (default: 0.0.0.0; use 127.0.0.1 for local-only access)",
|
||||
)
|
||||
parser.add_argument("--port", type=int, default=DEFAULT_PORT, help="web server port")
|
||||
parser.add_argument(
|
||||
"--no-display",
|
||||
action="store_true",
|
||||
help="capture frames without opening a preview",
|
||||
)
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def opencv_has_gui() -> bool:
|
||||
return next(
|
||||
("NONE" not in line for line in cv2.getBuildInformation().splitlines() if "GUI:" in line),
|
||||
False,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RealSenseDevice:
|
||||
serial: str
|
||||
name: str
|
||||
product_line: str
|
||||
|
||||
|
||||
def discover_devices() -> list[RealSenseDevice]:
|
||||
"""Discover connected RealSense devices without opening their streams."""
|
||||
devices: list[RealSenseDevice] = []
|
||||
context = rs.context()
|
||||
for device in context.query_devices():
|
||||
devices.append(
|
||||
RealSenseDevice(
|
||||
serial=device.get_info(rs.camera_info.serial_number),
|
||||
name=device.get_info(rs.camera_info.name),
|
||||
product_line=device.get_info(rs.camera_info.product_line),
|
||||
)
|
||||
)
|
||||
return devices
|
||||
|
||||
|
||||
class _CameraFeed:
|
||||
"""Own one camera connection and publish its latest JPEG frame."""
|
||||
|
||||
def __init__(self, device: RealSenseDevice, width: int, height: int, fps: int) -> None:
|
||||
self.device = device
|
||||
self.width = width
|
||||
self.height = height
|
||||
self.fps = fps
|
||||
self.camera = RealSenseCamera(
|
||||
RealSenseCameraConfig(
|
||||
serial_number_or_name=device.serial,
|
||||
width=width,
|
||||
height=height,
|
||||
fps=fps,
|
||||
color_mode=ColorMode.BGR,
|
||||
)
|
||||
)
|
||||
self.condition = threading.Condition()
|
||||
self.jpeg: bytes | None = None
|
||||
self.frame: Any | None = None
|
||||
self.frame_id = 0
|
||||
self.frame_count = 0
|
||||
self.error: str | None = None
|
||||
self.connected = False
|
||||
self.stopped = False
|
||||
self._stop_event = threading.Event()
|
||||
self._thread: threading.Thread | None = None
|
||||
|
||||
def connect(self) -> None:
|
||||
self.camera.connect()
|
||||
self.connected = True
|
||||
self._thread = threading.Thread(
|
||||
target=self._capture_loop,
|
||||
name=f"realsense-{self.device.serial}",
|
||||
daemon=True,
|
||||
)
|
||||
self._thread.start()
|
||||
|
||||
def _set_error(self, error: Exception) -> None:
|
||||
with self.condition:
|
||||
self.error = f"{type(error).__name__}: {error}"
|
||||
self.condition.notify_all()
|
||||
|
||||
def _capture_loop(self) -> None:
|
||||
period = 1.0 / self.fps
|
||||
next_deadline = time.monotonic()
|
||||
while not self._stop_event.is_set():
|
||||
try:
|
||||
frame = self.camera.read()
|
||||
encoded, buffer = cv2.imencode(".jpg", frame)
|
||||
if not encoded:
|
||||
raise RuntimeError("JPEG encoding failed")
|
||||
with self.condition:
|
||||
self.frame = frame
|
||||
self.jpeg = buffer.tobytes()
|
||||
self.frame_id += 1
|
||||
self.frame_count += 1
|
||||
self.error = None
|
||||
self.condition.notify_all()
|
||||
except Exception as exc:
|
||||
self._set_error(exc)
|
||||
if self._stop_event.wait(0.1):
|
||||
break
|
||||
|
||||
next_deadline += period
|
||||
delay = next_deadline - time.monotonic()
|
||||
if delay > 0:
|
||||
self._stop_event.wait(delay)
|
||||
elif -delay > period * 2:
|
||||
next_deadline = time.monotonic()
|
||||
|
||||
def wait_for_frame(self, last_frame_id: int) -> tuple[bytes | None, int, bool]:
|
||||
with self.condition:
|
||||
self.condition.wait_for(
|
||||
lambda: self.frame_id > last_frame_id or self.stopped,
|
||||
timeout=1.0,
|
||||
)
|
||||
return self.jpeg, self.frame_id, self.stopped
|
||||
|
||||
def latest_frame(self) -> Any | None:
|
||||
with self.condition:
|
||||
return None if self.frame is None else self.frame.copy()
|
||||
|
||||
def status(self, selected: bool) -> dict[str, Any]:
|
||||
with self.condition:
|
||||
return {
|
||||
"serial": self.device.serial,
|
||||
"name": self.device.name,
|
||||
"product_line": self.device.product_line,
|
||||
"width": self.width,
|
||||
"height": self.height,
|
||||
"fps": self.fps,
|
||||
"connected": self.connected,
|
||||
"selected": selected,
|
||||
"frame_count": self.frame_count,
|
||||
"error": self.error,
|
||||
}
|
||||
|
||||
def stop(self) -> None:
|
||||
self._stop_event.set()
|
||||
with self.condition:
|
||||
self.stopped = True
|
||||
self.condition.notify_all()
|
||||
if self._thread is not None:
|
||||
self._thread.join(timeout=2.0)
|
||||
if self.connected:
|
||||
try:
|
||||
self.camera.disconnect()
|
||||
except Exception as exc:
|
||||
self._set_error(exc)
|
||||
self.connected = False
|
||||
|
||||
|
||||
class _PreviewState:
|
||||
def __init__(self, feeds: list[_CameraFeed], selected: list[str], fps: int) -> None:
|
||||
self.feeds = {feed.device.serial: feed for feed in feeds}
|
||||
self.selected = set(selected)
|
||||
self.fps = fps
|
||||
|
||||
def get_feed(self, serial: str | None) -> _CameraFeed | None:
|
||||
return self.feeds.get(serial) if serial else None
|
||||
|
||||
def device_status(self) -> list[dict[str, Any]]:
|
||||
return [
|
||||
feed.status(feed.device.serial in self.selected)
|
||||
for feed in self.feeds.values()
|
||||
]
|
||||
|
||||
def stop(self) -> None:
|
||||
for feed in self.feeds.values():
|
||||
feed.stop()
|
||||
|
||||
|
||||
WEB_PAGE = """<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>RealSense Viewer</title>
|
||||
<style>
|
||||
:root { color-scheme: dark; font-family: system-ui, sans-serif; }
|
||||
* { box-sizing: border-box; }
|
||||
body { margin: 0; background: #101214; color: #e8eaed; }
|
||||
header { position: sticky; top: 0; z-index: 2; padding: 14px 18px; background: #191c20; border-bottom: 1px solid #30343a; }
|
||||
h1 { margin: 0 0 10px; font-size: 20px; font-weight: 600; }
|
||||
#status { color: #9da5af; font-size: 13px; }
|
||||
.toolbar { display: flex; flex-wrap: wrap; gap: 8px; align-items: center; margin-top: 12px; }
|
||||
button { border: 1px solid #4a515b; border-radius: 5px; padding: 7px 11px; color: #e8eaed; background: #262b31; cursor: pointer; }
|
||||
button:hover { background: #323941; }
|
||||
#devices { display: flex; flex-wrap: wrap; gap: 8px; margin-top: 12px; }
|
||||
.device { display: flex; align-items: center; gap: 7px; padding: 7px 9px; border: 1px solid #3b4149; border-radius: 5px; background: #20242a; font-size: 13px; }
|
||||
.device input { width: 16px; height: 16px; accent-color: #4da3ff; }
|
||||
.device small { color: #9da5af; }
|
||||
.device.error { border-color: #a44d4d; }
|
||||
#grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(360px, 1fr)); gap: 12px; padding: 14px; }
|
||||
.tile { min-width: 0; overflow: hidden; background: #191c20; border: 1px solid #30343a; border-radius: 6px; }
|
||||
.tile h2 { margin: 0; padding: 9px 11px; font-size: 14px; font-weight: 500; }
|
||||
.tile img { display: block; width: 100%; height: auto; background: #08090a; }
|
||||
.empty { padding: 36px 18px; color: #9da5af; text-align: center; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>RealSense Viewer</h1>
|
||||
<div id="status">Loading devices...</div>
|
||||
<div class="toolbar">
|
||||
<button id="all" type="button">Select all</button>
|
||||
<button id="none" type="button">Clear</button>
|
||||
</div>
|
||||
<div id="devices"></div>
|
||||
</header>
|
||||
<main id="grid"><div class="empty">Loading...</div></main>
|
||||
<script>
|
||||
const devicesEl = document.getElementById('devices');
|
||||
const gridEl = document.getElementById('grid');
|
||||
const statusEl = document.getElementById('status');
|
||||
let devices = [];
|
||||
let selected = new Set();
|
||||
|
||||
function deviceLabel(device) {
|
||||
return `${device.name} (${device.serial})`;
|
||||
}
|
||||
|
||||
function streamUrl(serial) {
|
||||
return `/stream?serial=${encodeURIComponent(serial)}`;
|
||||
}
|
||||
|
||||
function renderGrid() {
|
||||
gridEl.replaceChildren();
|
||||
const visible = devices.filter(device => selected.has(device.serial) && device.connected);
|
||||
if (!visible.length) {
|
||||
const empty = document.createElement('div');
|
||||
empty.className = 'empty';
|
||||
empty.textContent = 'Select a connected device to display.';
|
||||
gridEl.appendChild(empty);
|
||||
return;
|
||||
}
|
||||
for (const device of visible) {
|
||||
const tile = document.createElement('section');
|
||||
tile.className = 'tile';
|
||||
const title = document.createElement('h2');
|
||||
title.textContent = deviceLabel(device);
|
||||
const image = document.createElement('img');
|
||||
image.alt = deviceLabel(device);
|
||||
image.src = streamUrl(device.serial);
|
||||
tile.append(title, image);
|
||||
gridEl.appendChild(tile);
|
||||
}
|
||||
}
|
||||
|
||||
function renderDevices() {
|
||||
devicesEl.replaceChildren();
|
||||
for (const device of devices) {
|
||||
const label = document.createElement('label');
|
||||
label.className = `device${device.error ? ' error' : ''}`;
|
||||
const checkbox = document.createElement('input');
|
||||
checkbox.type = 'checkbox';
|
||||
checkbox.checked = selected.has(device.serial);
|
||||
checkbox.disabled = !device.connected;
|
||||
checkbox.addEventListener('change', () => {
|
||||
if (checkbox.checked) selected.add(device.serial);
|
||||
else selected.delete(device.serial);
|
||||
renderGrid();
|
||||
});
|
||||
const text = document.createElement('span');
|
||||
text.textContent = deviceLabel(device);
|
||||
const details = document.createElement('small');
|
||||
details.textContent = device.connected ? `${device.width}x${device.height} @ ${device.fps} Hz` : (device.error || 'offline');
|
||||
label.append(checkbox, text, details);
|
||||
devicesEl.appendChild(label);
|
||||
}
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
try {
|
||||
const response = await fetch('/api/devices', { cache: 'no-store' });
|
||||
const payload = await response.json();
|
||||
devices = payload.devices;
|
||||
if (!selected.size) {
|
||||
for (const device of devices.filter(device => device.selected && device.connected)) selected.add(device.serial);
|
||||
}
|
||||
statusEl.textContent = `${devices.filter(device => device.connected).length}/${devices.length} devices connected · ${payload.fps} Hz`;
|
||||
renderDevices();
|
||||
renderGrid();
|
||||
} catch (error) {
|
||||
statusEl.textContent = `Device service unavailable: ${error}`;
|
||||
}
|
||||
}
|
||||
|
||||
document.getElementById('all').addEventListener('click', () => {
|
||||
for (const device of devices) if (device.connected) selected.add(device.serial);
|
||||
renderDevices(); renderGrid();
|
||||
});
|
||||
document.getElementById('none').addEventListener('click', () => {
|
||||
selected.clear(); renderDevices(); renderGrid();
|
||||
});
|
||||
refresh();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
""".encode("utf-8")
|
||||
|
||||
|
||||
class _PreviewHandler(http.server.BaseHTTPRequestHandler):
|
||||
state: _PreviewState
|
||||
|
||||
def _send_bytes(self, status: int, content_type: str, body: bytes) -> None:
|
||||
try:
|
||||
self.send_response(status)
|
||||
self.send_header("Content-Type", content_type)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
self.wfile.flush()
|
||||
except (BrokenPipeError, ConnectionResetError):
|
||||
pass
|
||||
|
||||
def do_GET(self) -> None:
|
||||
parsed = urlsplit(self.path)
|
||||
path = parsed.path
|
||||
|
||||
if path == "/":
|
||||
self._send_bytes(200, "text/html; charset=utf-8", WEB_PAGE)
|
||||
return
|
||||
if path == "/favicon.ico":
|
||||
self._send_bytes(204, "text/plain; charset=utf-8", b"")
|
||||
return
|
||||
if path == "/health":
|
||||
self._send_bytes(200, "application/json; charset=utf-8", self._json(self.state.device_status()))
|
||||
return
|
||||
if path == "/api/devices":
|
||||
self._send_bytes(
|
||||
200,
|
||||
"application/json; charset=utf-8",
|
||||
self._json(self.state.device_status(), include_fps=True),
|
||||
)
|
||||
return
|
||||
if path == "/stream":
|
||||
serial = parse_qs(parsed.query).get("serial", [None])[0]
|
||||
self._stream(serial)
|
||||
return
|
||||
try:
|
||||
self.send_error(404)
|
||||
except (BrokenPipeError, ConnectionResetError):
|
||||
pass
|
||||
|
||||
def _json(self, devices: list[dict[str, Any]], include_fps: bool = False) -> bytes:
|
||||
payload: dict[str, Any] = {"ok": True, "devices": devices}
|
||||
if include_fps:
|
||||
payload["fps"] = self.state.fps
|
||||
return json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
||||
|
||||
def _stream(self, serial: str | None) -> None:
|
||||
feed = self.state.get_feed(serial)
|
||||
if feed is None:
|
||||
self._send_bytes(404, "text/plain; charset=utf-8", b"Unknown camera serial\n")
|
||||
return
|
||||
if not feed.connected:
|
||||
self._send_bytes(503, "text/plain; charset=utf-8", b"Camera is not connected\n")
|
||||
return
|
||||
|
||||
try:
|
||||
self.send_response(200)
|
||||
self.send_header("Cache-Control", "no-cache, private")
|
||||
self.send_header("Pragma", "no-cache")
|
||||
self.send_header("Content-Type", "multipart/x-mixed-replace; boundary=frame")
|
||||
self.end_headers()
|
||||
last_frame_id = 0
|
||||
while True:
|
||||
jpeg, frame_id, stopped = feed.wait_for_frame(last_frame_id)
|
||||
if stopped:
|
||||
return
|
||||
if jpeg is None or frame_id == last_frame_id:
|
||||
continue
|
||||
last_frame_id = frame_id
|
||||
self.wfile.write(
|
||||
b"--frame\r\nContent-Type: image/jpeg\r\n"
|
||||
+ f"Content-Length: {len(jpeg)}\r\n\r\n".encode()
|
||||
+ jpeg
|
||||
+ b"\r\n"
|
||||
)
|
||||
self.wfile.flush()
|
||||
except (BrokenPipeError, ConnectionResetError):
|
||||
pass
|
||||
|
||||
def log_message(self, format: str, *args: Any) -> None:
|
||||
pass
|
||||
|
||||
|
||||
def start_web_server(
|
||||
state: _PreviewState, host: str, port: int
|
||||
) -> http.server.ThreadingHTTPServer:
|
||||
handler = type("RealSensePreviewHandler", (_PreviewHandler,), {"state": state})
|
||||
server = http.server.ThreadingHTTPServer((host, port), handler)
|
||||
server.daemon_threads = True
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
if host == "0.0.0.0":
|
||||
print(f"Open http://127.0.0.1:{port}/ on this machine to view the streams.")
|
||||
print(f"For another machine, use http://<server-ip>:{port}/")
|
||||
else:
|
||||
print(f"Open http://{host}:{port}/ in a browser to view the streams.")
|
||||
print("Press Ctrl+C to exit.")
|
||||
return server
|
||||
|
||||
|
||||
def _make_mosaic(feeds: list[_CameraFeed]) -> Any | None:
|
||||
tiles = []
|
||||
for feed in feeds:
|
||||
frame = feed.latest_frame()
|
||||
if frame is None:
|
||||
continue
|
||||
cv2.putText(
|
||||
frame,
|
||||
f"{feed.device.name} {feed.device.serial}",
|
||||
(10, 28),
|
||||
cv2.FONT_HERSHEY_SIMPLEX,
|
||||
0.65,
|
||||
(0, 255, 255),
|
||||
2,
|
||||
cv2.LINE_AA,
|
||||
)
|
||||
tiles.append(frame)
|
||||
if not tiles:
|
||||
return None
|
||||
height = max(tile.shape[0] for tile in tiles)
|
||||
aligned = [
|
||||
cv2.resize(tile, (int(tile.shape[1] * height / tile.shape[0]), height))
|
||||
if tile.shape[0] != height
|
||||
else tile
|
||||
for tile in tiles
|
||||
]
|
||||
return cv2.hconcat(aligned)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
if args.fps <= 0:
|
||||
raise SystemExit("FPS must be positive")
|
||||
|
||||
if args.no_display:
|
||||
backend = "none"
|
||||
elif args.backend == "auto":
|
||||
backend = "opencv" if opencv_has_gui() else "web"
|
||||
else:
|
||||
backend = args.backend
|
||||
|
||||
devices = discover_devices()
|
||||
requested = list(dict.fromkeys(args.serial or []))
|
||||
if requested:
|
||||
requested_set = set(requested)
|
||||
devices = [device for device in devices if device.serial in requested_set]
|
||||
missing = sorted(requested_set - {device.serial for device in devices})
|
||||
if missing:
|
||||
print(f"Requested cameras not found: {', '.join(missing)}")
|
||||
if not devices:
|
||||
raise SystemExit("No RealSense cameras found.")
|
||||
|
||||
feeds: list[_CameraFeed] = []
|
||||
for device in devices:
|
||||
feed = _CameraFeed(device, args.width, args.height, args.fps)
|
||||
try:
|
||||
print(f"Opening RealSense {device.name} {device.serial}: {args.width}x{args.height} @ {args.fps} FPS")
|
||||
feed.connect()
|
||||
print(f"Connected {device.serial}")
|
||||
except Exception as exc:
|
||||
feed.error = f"{type(exc).__name__}: {exc}"
|
||||
print(f"Failed to connect {device.serial}: {feed.error}")
|
||||
feeds.append(feed)
|
||||
|
||||
initially_selected = requested or ([feeds[0].device.serial] if feeds else [])
|
||||
state = _PreviewState(feeds, initially_selected, args.fps)
|
||||
web_server = None
|
||||
|
||||
if backend == "web":
|
||||
web_server = start_web_server(state, args.host, args.port)
|
||||
elif backend == "opencv":
|
||||
print("Press q or Esc to exit.")
|
||||
|
||||
try:
|
||||
if backend == "web":
|
||||
while True:
|
||||
time.sleep(1.0)
|
||||
elif backend == "none":
|
||||
while True:
|
||||
time.sleep(1.0)
|
||||
statuses = [
|
||||
f"{feed.device.serial}:{'OK' if feed.connected and feed.error is None else 'ERROR'}"
|
||||
for feed in feeds
|
||||
]
|
||||
print(" | ".join(statuses), flush=True)
|
||||
else:
|
||||
while True:
|
||||
mosaic = _make_mosaic(feeds)
|
||||
if mosaic is not None:
|
||||
cv2.imshow("RealSense Color", mosaic)
|
||||
key = cv2.waitKey(1) & 0xFF
|
||||
if key in (ord("q"), 27):
|
||||
break
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
finally:
|
||||
state.stop()
|
||||
if web_server is not None:
|
||||
web_server.shutdown()
|
||||
web_server.server_close()
|
||||
if backend == "opencv":
|
||||
try:
|
||||
cv2.destroyAllWindows()
|
||||
except cv2.error:
|
||||
pass
|
||||
print("RealSense preview stopped.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
130
src/lerobot_robot_ufactory/utils/action_safety.py
Normal file
130
src/lerobot_robot_ufactory/utils/action_safety.py
Normal file
@ -0,0 +1,130 @@
|
||||
"""Model-agnostic action safety checks.
|
||||
|
||||
Validates the final action dict right before it is sent to the robot; any
|
||||
failed check triggers an emergency halt. Only relies on the action key
|
||||
naming convention: `[prefix.]pose.x/y/z/rx/ry/rz`, `*.gripper.pos`, so it
|
||||
works with any policy (ACT / DP / pi0 / custom backbones) and with both
|
||||
single-arm and multi-arm setups.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
|
||||
import numpy as np
|
||||
|
||||
from lerobot_robot_ufactory.devices.umi.vive_tracker.transformations import Transformations
|
||||
|
||||
POSE_AXES = ("x", "y", "z", "rx", "ry", "rz")
|
||||
|
||||
|
||||
@dataclass
|
||||
class ActionSafetyConfig:
|
||||
"""Configuration for ActionSafetyGuard.
|
||||
|
||||
All limits are compared against the robot's current actual pose per
|
||||
control step. `workspace_min`/`workspace_max` use the same unit as the
|
||||
pose commands (e.g. mm) and must be provided together.
|
||||
"""
|
||||
|
||||
enabled: bool = True
|
||||
max_step_mm: float = 25.0
|
||||
max_rot_step: float = 0.35
|
||||
workspace_min: list | None = None
|
||||
workspace_max: list | None = None
|
||||
|
||||
def __post_init__(self):
|
||||
if (self.workspace_min is None) != (self.workspace_max is None):
|
||||
raise ValueError("workspace_min and workspace_max must be provided together")
|
||||
|
||||
|
||||
def _rotation_delta_norm(rot_prev, rot_curr) -> float:
|
||||
"""Relative rotation angle (rad) between two axis-angle rotations.
|
||||
|
||||
Uses rotation matrices for the diff to avoid the ±π discontinuity
|
||||
of subtracting raw rotvecs.
|
||||
"""
|
||||
R_prev = Transformations.rxryrz_to_rotation_matrix(*rot_prev)
|
||||
R_curr = Transformations.rxryrz_to_rotation_matrix(*rot_curr)
|
||||
R_delta = R_prev.T @ R_curr
|
||||
delta = Transformations.rotation_matrix_to_rxryrz(R_delta)
|
||||
return float(np.linalg.norm(delta))
|
||||
|
||||
|
||||
class ActionSafetyGuard:
|
||||
"""Safety checks for the action dict about to be sent to the robot.
|
||||
|
||||
check() returns None when the action is safe; otherwise it returns a
|
||||
human-readable violation reason (the caller should trigger an e-stop).
|
||||
"""
|
||||
|
||||
def __init__(self, config: ActionSafetyConfig | None = None):
|
||||
config = config or ActionSafetyConfig()
|
||||
self.config = config
|
||||
self.enabled = config.enabled
|
||||
self.max_step_mm = config.max_step_mm
|
||||
self.max_rot_step = config.max_rot_step
|
||||
self.workspace_min = None if config.workspace_min is None else np.asarray(config.workspace_min, dtype=np.float64)
|
||||
self.workspace_max = None if config.workspace_max is None else np.asarray(config.workspace_max, dtype=np.float64)
|
||||
|
||||
def check(self, action: dict, curr_robot_dict: dict, keys) -> str | None:
|
||||
"""Check the action dict. `action` holds absolute pose commands and
|
||||
`curr_robot_dict` holds each arm's current actual pose."""
|
||||
if not self.enabled:
|
||||
return None
|
||||
|
||||
for key in keys:
|
||||
prefix = f".{key}" if key else ""
|
||||
pose_keys = [f"{prefix}pose.{axis}" for axis in POSE_AXES]
|
||||
if not all(k in action for k in pose_keys):
|
||||
continue # not TCP-pose controlled (e.g. joint-only arm), skip pose checks
|
||||
|
||||
target = np.array([action[k] for k in pose_keys], dtype=np.float64)
|
||||
|
||||
# 1. Numeric validity: NaN / Inf
|
||||
if not np.all(np.isfinite(target)):
|
||||
return f"[{key or 'arm'}] action contains NaN/Inf: {target.tolist()}"
|
||||
|
||||
# 2. Single-step delta limit, measured from the robot's current actual pose
|
||||
curr = curr_robot_dict.get(key)
|
||||
if curr is not None and curr.get("type") == 1:
|
||||
curr_pose = np.asarray(curr["pose"], dtype=np.float64)
|
||||
pos_dist = float(np.linalg.norm(target[:3] - curr_pose[:3]))
|
||||
if pos_dist > self.max_step_mm:
|
||||
return (
|
||||
f"[{key or 'arm'}] position step {pos_dist:.1f}mm exceeds limit "
|
||||
f"{self.max_step_mm}mm (current {curr_pose[:3].tolist()} -> target {target[:3].tolist()})"
|
||||
)
|
||||
rot_delta = _rotation_delta_norm(curr_pose[3:6], target[3:6])
|
||||
if rot_delta > self.max_rot_step:
|
||||
return (
|
||||
f"[{key or 'arm'}] rotation step {rot_delta:.3f}rad exceeds limit "
|
||||
f"{self.max_rot_step}rad"
|
||||
)
|
||||
|
||||
# 3. Workspace bounding box (optional)
|
||||
if self.workspace_min is not None:
|
||||
pos = target[:3]
|
||||
if np.any(pos < self.workspace_min) or np.any(pos > self.workspace_max):
|
||||
return (
|
||||
f"[{key or 'arm'}] target position {pos.tolist()} outside workspace "
|
||||
f"[{self.workspace_min.tolist()}, {self.workspace_max.tolist()}]"
|
||||
)
|
||||
|
||||
# Gripper numeric check
|
||||
for k, v in action.items():
|
||||
if "gripper" in k and isinstance(v, (int, float)) and not math.isfinite(v):
|
||||
return f"[{k}] gripper action contains NaN/Inf: {v}"
|
||||
|
||||
return None
|
||||
|
||||
def log_config(self):
|
||||
ws = (
|
||||
f"[{self.workspace_min.tolist()}, {self.workspace_max.tolist()}]"
|
||||
if self.workspace_min is not None
|
||||
else "not set"
|
||||
)
|
||||
logging.info(
|
||||
f"ActionSafetyGuard: max_step={self.max_step_mm}mm, "
|
||||
f"max_rot_step={self.max_rot_step}rad, workspace={ws}"
|
||||
)
|
||||
@ -2,49 +2,9 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
repo_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$repo_root"
|
||||
|
||||
config_path="config/manual_mode/xarm7_manual_record_config.yaml"
|
||||
dataset_root="$(
|
||||
sed -n '/^dataset:/,/^[^[:space:]]/s/^[[:space:]]*root:[[:space:]]*//p' "$config_path" \
|
||||
| head -n 1 \
|
||||
| tr -d "\"'"
|
||||
)"
|
||||
|
||||
if [[ -z "$dataset_root" ]]; then
|
||||
printf 'Could not read dataset.root from %s\n' "$config_path" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$dataset_root" != /* ]]; then
|
||||
dataset_root="$repo_root/$dataset_root"
|
||||
fi
|
||||
|
||||
record_args=("$@")
|
||||
resume_requested=false
|
||||
for arg in "${record_args[@]}"; do
|
||||
if [[ "$arg" == "-r" ]]; then
|
||||
resume_requested=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ -e "$dataset_root" ]]; then
|
||||
if [[ ! -f "$dataset_root/meta/info.json" ]]; then
|
||||
printf 'Dataset directory exists but is not a valid LeRobot dataset: %s\n' "$dataset_root" >&2
|
||||
printf 'Choose a new dataset.root, or remove this empty/incomplete directory before recording.\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$resume_requested" == false ]]; then
|
||||
record_args=("-r" "${record_args[@]}")
|
||||
fi
|
||||
elif [[ "$resume_requested" == true ]]; then
|
||||
printf 'Cannot resume because the dataset directory does not exist: %s\n' "$dataset_root" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exec uv run uf-lerobot-record \
|
||||
--config_path "$config_path" \
|
||||
"${record_args[@]}"
|
||||
--config_path config/manual_mode/xarm7_manual_record_config.yaml \
|
||||
"$@"
|
||||
|
||||
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
@ -11,6 +12,7 @@ from lerobot_robot_ufactory.scripts.uf_lerobot_record import (
|
||||
_manual_action_from_observation,
|
||||
_update_manual_gripper_key_state,
|
||||
_update_manual_gripper_target,
|
||||
_prepare_dataset_root,
|
||||
_prepare_recording_episode,
|
||||
get_cfg,
|
||||
)
|
||||
@ -294,6 +296,46 @@ def test_manual_record_config_has_no_teleop(monkeypatch):
|
||||
assert config.dataset.fps == 30
|
||||
|
||||
|
||||
def test_prepare_dataset_root_leaves_new_root_for_lerobot_create(tmp_path):
|
||||
root = tmp_path / "nested" / "dataset"
|
||||
cfg = SimpleNamespace(dataset=SimpleNamespace(root=root), resume=False)
|
||||
|
||||
_prepare_dataset_root(cfg)
|
||||
|
||||
assert not root.exists()
|
||||
|
||||
|
||||
def test_prepare_dataset_root_rejects_incomplete_resume(tmp_path):
|
||||
root = tmp_path / "dataset"
|
||||
(root / "meta").mkdir(parents=True)
|
||||
(root / "meta" / "info.json").write_text("{}")
|
||||
cfg = SimpleNamespace(dataset=SimpleNamespace(root=root), resume=True)
|
||||
|
||||
with pytest.raises(RuntimeError, match="meta/tasks.parquet"):
|
||||
_prepare_dataset_root(cfg)
|
||||
|
||||
|
||||
def test_prepare_dataset_root_rejects_resume_when_root_is_missing(tmp_path):
|
||||
cfg = SimpleNamespace(dataset=SimpleNamespace(root=tmp_path / "missing"), resume=True)
|
||||
|
||||
with pytest.raises(RuntimeError, match="does not exist"):
|
||||
_prepare_dataset_root(cfg)
|
||||
|
||||
|
||||
def test_prepare_dataset_root_resumes_complete_dataset_without_prompt(tmp_path, monkeypatch):
|
||||
root = tmp_path / "dataset"
|
||||
(root / "meta" / "episodes" / "chunk-000").mkdir(parents=True)
|
||||
(root / "data" / "chunk-000").mkdir(parents=True)
|
||||
(root / "meta" / "info.json").write_text("{}")
|
||||
(root / "meta" / "tasks.parquet").write_bytes(b"tasks")
|
||||
(root / "meta" / "episodes" / "chunk-000" / "file-000.parquet").write_bytes(b"episodes")
|
||||
(root / "data" / "chunk-000" / "file-000.parquet").write_bytes(b"data")
|
||||
cfg = SimpleNamespace(dataset=SimpleNamespace(root=root), resume=True)
|
||||
monkeypatch.setattr(record_module.sys.stdin, "isatty", lambda: True)
|
||||
|
||||
_prepare_dataset_root(cfg)
|
||||
|
||||
|
||||
def test_manual_record_loop_writes_actual_state_as_action(tmp_path):
|
||||
class FakeRobot:
|
||||
name = "fake_manual_robot"
|
||||
|
||||
Loading…
Reference in New Issue
Block a user