Compare commits

..

No commits in common. "03775cefc08e0ca7be1631c9ac12e83f985c3dff" and "6d813e53c0b74c22e33e750eb3d2a60c21466765" have entirely different histories.

10 changed files with 67 additions and 721 deletions

View File

@ -151,21 +151,19 @@ Controls: `Space` start the episode, `→` save it, `←` discard and re-record
./start_manual_record.sh -r # force resume; fails if the dataset directory does not exist ./start_manual_record.sh -r # force resume; fails if the dataset directory does not exist
``` ```
The record script (`uf-lerobot-record`) reads `dataset.root` from the config and checks the path before recording, then: The record script (`uf-lerobot-record`) reads `dataset.root` from the config and creates the path with `mkdir -p` first, then:
- Directory does not exist → records a new dataset. - Directory does not exist → records a new dataset.
- Directory already exists (valid LeRobot dataset) and no `-r` was given → asks interactively: - Directory already exists (valid LeRobot dataset) and no `-r` was given → asks interactively:
- `o` overwrite: delete the existing dataset and record a new one - `o` overwrite: delete the existing dataset and record a new one
- `r` resume: keep existing episodes and continue recording - `r` resume: keep existing episodes and continue recording
- `c` cancel - `c` cancel
- Directory exists but is incomplete (missing required metadata or data parquet files) → asks to overwrite it or cancel; non-interactive runs error out instead. - Directory exists but is not a valid LeRobot dataset (missing `meta/info.json`) → asks to overwrite it or cancel; non-interactive runs error out instead.
Pass `-r` to resume directly without asking. Note that `./start_manual_record.sh` keeps its original launcher behavior — it automatically resumes an existing valid dataset (equivalent to `-r`), so run `uv run uf-lerobot-record --config_path config/manual_mode/xarm7_manual_record_config.yaml` directly if you want to see the overwrite/resume prompt. Pass `-r` to resume directly without asking. Note that `./start_manual_record.sh` keeps its original launcher behavior — it automatically resumes an existing valid dataset (equivalent to `-r`), so run `uv run uf-lerobot-record --config_path config/manual_mode/xarm7_manual_record_config.yaml` directly if you want to see the overwrite/resume prompt.
During recording the arm is in teach mode: the actual joint state is written as both the observation and the action. Hold `C` to slowly close the gripper and `O` to slowly open it. Controls: `Space` start, `→` save, `←` discard & re-record, `Esc` stop. Reset the arm manually between episodes. During recording the arm is in teach mode: the actual joint state is written as both the observation and the action. Hold `C` to slowly close the gripper and `O` to slowly open it. Controls: `Space` start, `→` save, `←` discard & re-record, `Esc` stop. Reset the arm manually between episodes.
> **IMPORTANT: At the start of every episode, wait until the arm has finished resetting and then wait another 5 seconds before operating it, or wait until the console prints `Start Recording` before operating it.** This prevents reset commands from conflicting with operation commands and causing errors.
### 4. Policy training ### 4. Policy training
```bash ```bash
@ -216,21 +214,6 @@ uv run uf-lerobot-replay --dataset-root /path/to/xarm7_manual_datas --robot-ip 1
The robot first moves to the xArm SDK initial point, then replays the episode and stays at the last state. Make sure the workspace is clear and the recorded initial pose matches the current arm setup. The robot first moves to the xArm SDK initial point, then replays the episode and stays at the last state. Make sure the workspace is clear and the recorded initial pose matches the current arm setup.
## IMPORTANT: Robot Power-On and Power-Off
### Power-on
1. **Connect the computer to the robot controller with an Ethernet cable.**
2. **Configure the computer's Ethernet interface to the same IP subnet as the robot IP shown on the controller** (for example, `192.168.1.xxx`).
3. **Open `http://192.168.1.245:18333/` in a browser.** The controller console should be displayed.
4. **Release the emergency-stop button before operating the robot.**
### Power-off
1. **Use the web console to return the arm to its initial position.**
2. **Press the emergency-stop button.**
3. **Turn off the controller power.**
## Tools ## Tools
### Camera viewer ### Camera viewer

View File

@ -150,21 +150,19 @@ uv run uf-lerobot-record --config_path config/gello/xarm7_gello_record_config.ya
./start_manual_record.sh -r # 强制续录;数据集目录不存在时会报错 ./start_manual_record.sh -r # 强制续录;数据集目录不存在时会报错
``` ```
录制脚本(`uf-lerobot-record`)会从配置读取 `dataset.root`在录制前检查路径,然后: 录制脚本(`uf-lerobot-record`)会从配置读取 `dataset.root`先执行 `mkdir -p` 创建路径,然后:
- 目录不存在:直接录制新数据集。 - 目录不存在:直接录制新数据集。
- 目录已存在(有效 LeRobot 数据集)且未加 `-r`:交互询问: - 目录已存在(有效 LeRobot 数据集)且未加 `-r`:交互询问:
- `o` 覆盖:删除已有数据集,重新录制 - `o` 覆盖:删除已有数据集,重新录制
- `r` 续录:保留已有 episode继续录制 - `r` 续录:保留已有 episode继续录制
- `c` 取消 - `c` 取消
- 目录存在但不完整(缺少必要元数据或数据 parquet 文件):询问覆盖或取消;非交互运行时直接报错。 - 目录存在但不是有效 LeRobot 数据集(缺少 `meta/info.json`):询问覆盖或取消;非交互运行时直接报错。
`-r` 可跳过询问直接续录。注意 `./start_manual_record.sh` 保持原有启动脚本行为——检测到有效数据集会自动续录(相当于 `-r`),如果想看到覆盖/续录的询问,请直接用 `uv run uf-lerobot-record --config_path config/manual_mode/xarm7_manual_record_config.yaml` 运行。 `-r` 可跳过询问直接续录。注意 `./start_manual_record.sh` 保持原有启动脚本行为——检测到有效数据集会自动续录(相当于 `-r`),如果想看到覆盖/续录的询问,请直接用 `uv run uf-lerobot-record --config_path config/manual_mode/xarm7_manual_record_config.yaml` 运行。
录制时机械臂处于示教模式,实际关节状态会同时作为 observation 和 action 写入数据集。按住 `C` 缓慢闭合夹爪,按住 `O` 缓慢张开。按键控制:`Space` 开始,`→` 保存,`←` 放弃并重录,`Esc` 停止。episode 之间手动复位机械臂。 录制时机械臂处于示教模式,实际关节状态会同时作为 observation 和 action 写入数据集。按住 `C` 缓慢闭合夹爪,按住 `O` 缓慢张开。按键控制:`Space` 开始,`→` 保存,`←` 放弃并重录,`Esc` 停止。episode 之间手动复位机械臂。
> **重要:每个 episode 开始时,必须等待机械臂复位完成后再等待 5 秒,然后才能开始操作;或者确认控制台打印 `Start Recording` 后再开始操作。** 这样可以避免机械臂复位控制指令与操作指令冲突导致报错。
### 4. 策略训练 ### 4. 策略训练
```bash ```bash
@ -202,7 +200,6 @@ uv run uf-lerobot-eval \
将手动拖拽数据集的绝对关节状态(`observation.state`)回放到 xArm7。脚本按数据集 FPS默认 30将状态作为**绝对目标值**发送,不做差分或累加,因此运动轨迹与录制时一致: 将手动拖拽数据集的绝对关节状态(`observation.state`)回放到 xArm7。脚本按数据集 FPS默认 30将状态作为**绝对目标值**发送,不做差分或累加,因此运动轨迹与录制时一致:
```bash ```bash
# 默认回放第一条
uv run uf-lerobot-replay \ uv run uf-lerobot-replay \
--dataset-root /path/to/xarm7_manual_datas \ --dataset-root /path/to/xarm7_manual_datas \
--robot-ip 192.168.1.245 --robot-ip 192.168.1.245
@ -216,21 +213,6 @@ uv run uf-lerobot-replay --dataset-root /path/to/xarm7_manual_datas --robot-ip 1
回放开始前机械臂会先移动到 xArm SDK 初始点,播放结束后保持最后一帧姿态并断开连接。执行前请确认工作空间无障碍物,且数据中的初始姿态与当前设备一致。 回放开始前机械臂会先移动到 xArm SDK 初始点,播放结束后保持最后一帧姿态并断开连接。执行前请确认工作空间无障碍物,且数据中的初始姿态与当前设备一致。
## 重要:机械臂开关机事项
### 开机
1. **使用网线将电脑连接到机械臂控制器。**
2. **参考控制器上标注的机械臂 IP将电脑以太网接口配置到同一网段**(例如 `192.168.1.xxx`)。
3. **在浏览器中访问 `http://192.168.1.245:18333/`**,应出现控制台界面。
4. **操作机械臂前,抬起急停按钮。**
### 关机
1. **先通过网页控制将机械臂返回到初始位置。**
2. **再按下急停按钮。**
3. **关闭控制器电源。**
## 工具 ## 工具
### 摄像头查看器 ### 摄像头查看器
@ -240,31 +222,6 @@ uv run uf-camera-view -l # 列出所有摄像头
uv run uf-camera-view -T realsense # 查看 RealSense 摄像头 uv run uf-camera-view -T realsense # 查看 RealSense 摄像头
``` ```
网页预览会扫描所有 RealSense 摄像头,默认使用 `640x480`、`30fps`,页面中可勾选设备切换或同时显示多路画面:
```bash
uv run uf-realsense-view
```
脚本使用 LeRobot 的 `RealSenseCamera`。如果当前环境中的 OpenCV 是 LeRobot 默认的
headless 版本,会自动启动网页预览。同一台机器上打开
`http://127.0.0.1:8765/`;从其他机器访问时,请将 `127.0.0.1` 替换为运行脚本机器的实际 IP。
网页服务默认监听 `0.0.0.0`,也可以用 `--host 127.0.0.1` 限制为本机访问;还可以通过
`--backend opencv` 强制使用 OpenCV 窗口。
也可以只打开指定设备;需要多个设备时重复 `--serial`
```bash
uv run uf-realsense-view \
--serial 148522072685 --width 640 --height 480 --fps 30
uv run uf-realsense-view \
--serial 148522072685 --serial SECOND_CAMERA_SERIAL
```
OpenCV 窗口按 `q``Esc` 退出;网页模式按 `Ctrl+C` 退出。无桌面环境时可以用
`--no-display` 检查是否能持续取帧。
### LeRobot 数据集工具 ### LeRobot 数据集工具
```bash ```bash

View File

@ -24,10 +24,10 @@ robot:
fps: 30 fps: 30
dataset: dataset:
root: "/home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay_pick_bottle" root: "/home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay_pick_pen"
repo_id: "zjuscl/xarm7_manual_replay_pick_bottle" repo_id: "ufactory/xarm7_manual_datas"
# Task description stored with each recorded frame. # Task description stored with each recorded frame.
single_task: "Pick up the black bottle and place it on the blue bag" single_task: "Describe the task being demonstrated."
fps: 30 fps: 30
episode_time_s: 60 episode_time_s: 60
# Reserved reset duration between episodes in seconds. # Reserved reset duration between episodes in seconds.

View File

@ -39,7 +39,6 @@ uf-lerobot-eval = "lerobot_robot_ufactory.scripts.uf_lerobot_eval:main"
uf-lerobot-replay = "lerobot_robot_ufactory.scripts.uf_lerobot_replay:main" uf-lerobot-replay = "lerobot_robot_ufactory.scripts.uf_lerobot_replay:main"
uf-vive-calibrate = "lerobot_robot_ufactory.scripts.vive_calibrate:main" uf-vive-calibrate = "lerobot_robot_ufactory.scripts.vive_calibrate:main"
uf-camera-view = "lerobot_robot_ufactory.scripts.uf_camera_view:main" uf-camera-view = "lerobot_robot_ufactory.scripts.uf_camera_view:main"
uf-realsense-view = "lerobot_robot_ufactory.scripts.uf_realsense_view:main"
uf-camera-test = "lerobot_robot_ufactory.scripts.uf_camera_test:main" uf-camera-test = "lerobot_robot_ufactory.scripts.uf_camera_test:main"
[project.optional-dependencies] [project.optional-dependencies]

View File

@ -1,4 +0,0 @@
#!/bin/bash
uv run uf-lerobot-replay \
--dataset-root /home/wsx/code/lerobot_robot_ufactory/datasets/xarm7_manual_replay_pick_bottle \
--robot-ip 192.168.1.245

View File

@ -449,9 +449,9 @@ class UFRobot(Robot, Thread):
warn_code = getattr(arm, "warn_code", "unknown") warn_code = getattr(arm, "warn_code", "unknown")
return f"mode={mode}, state={state}, error_code={error_code}, warn_code={warn_code}" return f"mode={mode}, state={state}, error_code={error_code}, warn_code={warn_code}"
def _check_motion_code(self, command: str, code: int | None) -> None: def _check_motion_code(self, command: str, code: int) -> None:
"""Fail loudly when the SDK explicitly rejects a motion command.""" """Fail loudly when the SDK rejects a joint command."""
if code is not None and code != 0: if code != 0:
raise RuntimeError(f"{command} failed, code={code}, {self._motion_status()}") raise RuntimeError(f"{command} failed, code={code}, {self._motion_status()}")
def send_action(self, action: dict) -> np.ndarray: def send_action(self, action: dict) -> np.ndarray:

View File

@ -510,40 +510,26 @@ def _ask_choice(prompt: str, options: dict[str, str]) -> str:
print(f"Invalid choice, please enter {keys}.") print(f"Invalid choice, please enter {keys}.")
def _missing_dataset_files(root: Path) -> list[str]:
"""Return the local files required before a dataset can be resumed."""
missing = []
for relative_path in ("meta/info.json", "meta/tasks.parquet"):
if not (root / relative_path).is_file():
missing.append(relative_path)
if not any((root / "meta" / "episodes").glob("*/*.parquet")):
missing.append("meta/episodes/*/*.parquet")
if not any((root / "data").glob("*/*.parquet")):
missing.append("data/*/*.parquet")
return missing
def _prepare_dataset_root(cfg: UFRecordConfig) -> None: def _prepare_dataset_root(cfg: UFRecordConfig) -> None:
"""Prepare an existing dataset root without pre-creating a new one.""" """Create the dataset root and ask how to handle an existing dataset."""
root = Path(cfg.dataset.root) root = Path(cfg.dataset.root)
existed = root.exists() existed = root.exists()
if not existed: # Create the dataset root (and any parent directories) first.
if cfg.resume: root.mkdir(parents=True, exist_ok=True)
raise RuntimeError(f"Cannot resume because the dataset directory does not exist: {root}")
if not existed or cfg.resume:
return return
missing = _missing_dataset_files(root) if not (root / "meta" / "info.json").is_file():
if missing: # The directory exists but is not a valid LeRobot dataset.
missing_text = ", ".join(missing) if not sys.stdin.isatty():
message = ( raise RuntimeError(
f"Dataset directory is incomplete and cannot be resumed: {root}\n" f"Dataset directory exists but is not a valid LeRobot dataset: {root}\n"
f"Missing: {missing_text}" "Choose a new dataset.root, or remove this empty/incomplete directory before recording."
) )
if cfg.resume or not sys.stdin.isatty():
raise RuntimeError(message)
choice = _ask_choice( choice = _ask_choice(
message, f"Directory exists but is not a valid LeRobot dataset: {root}",
options={ options={
"o": "Overwrite: remove this directory and record a new dataset", "o": "Overwrite: remove this directory and record a new dataset",
"c": "Cancel", "c": "Cancel",
@ -555,9 +541,6 @@ def _prepare_dataset_root(cfg: UFRecordConfig) -> None:
raise SystemExit("Recording cancelled.") raise SystemExit("Recording cancelled.")
return return
if cfg.resume:
return
# A valid LeRobot dataset already exists. # A valid LeRobot dataset already exists.
if not sys.stdin.isatty(): if not sys.stdin.isatty():
# Non-interactive run: keep the previous auto-resume behaviour. # Non-interactive run: keep the previous auto-resume behaviour.

View File

@ -1,570 +0,0 @@
#!/usr/bin/env python3
"""Web viewer for one or more Intel RealSense color streams."""
from __future__ import annotations
import argparse
import http.server
import json
import threading
import time
from dataclasses import dataclass
from typing import Any
from urllib.parse import parse_qs, urlsplit
import cv2
import pyrealsense2 as rs
from lerobot.cameras import ColorMode
from lerobot.cameras.realsense import RealSenseCamera, RealSenseCameraConfig
DEFAULT_SERIAL = None
DEFAULT_WIDTH = 640
DEFAULT_HEIGHT = 480
DEFAULT_FPS = 30
DEFAULT_HOST = "0.0.0.0"
DEFAULT_PORT = 8765
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Preview one or more Intel RealSense color streams in a browser"
)
parser.add_argument(
"-s",
"--serial",
action="append",
default=DEFAULT_SERIAL,
help="camera serial number; repeat to limit the device list to selected cameras",
)
parser.add_argument("-W", "--width", type=int, default=DEFAULT_WIDTH, help="color width")
parser.add_argument("-H", "--height", type=int, default=DEFAULT_HEIGHT, help="color height")
parser.add_argument(
"-F", "--fps", type=int, default=DEFAULT_FPS, help="capture and preview FPS (default: 30)"
)
parser.add_argument(
"--backend",
choices=("auto", "opencv", "web"),
default="web",
help="preview backend (default: web; web provides device selection)",
)
parser.add_argument(
"--host",
default=DEFAULT_HOST,
help="web server bind address (default: 0.0.0.0; use 127.0.0.1 for local-only access)",
)
parser.add_argument("--port", type=int, default=DEFAULT_PORT, help="web server port")
parser.add_argument(
"--no-display",
action="store_true",
help="capture frames without opening a preview",
)
return parser.parse_args(argv)
def opencv_has_gui() -> bool:
return next(
("NONE" not in line for line in cv2.getBuildInformation().splitlines() if "GUI:" in line),
False,
)
@dataclass(frozen=True)
class RealSenseDevice:
serial: str
name: str
product_line: str
def discover_devices() -> list[RealSenseDevice]:
"""Discover connected RealSense devices without opening their streams."""
devices: list[RealSenseDevice] = []
context = rs.context()
for device in context.query_devices():
devices.append(
RealSenseDevice(
serial=device.get_info(rs.camera_info.serial_number),
name=device.get_info(rs.camera_info.name),
product_line=device.get_info(rs.camera_info.product_line),
)
)
return devices
class _CameraFeed:
"""Own one camera connection and publish its latest JPEG frame."""
def __init__(self, device: RealSenseDevice, width: int, height: int, fps: int) -> None:
self.device = device
self.width = width
self.height = height
self.fps = fps
self.camera = RealSenseCamera(
RealSenseCameraConfig(
serial_number_or_name=device.serial,
width=width,
height=height,
fps=fps,
color_mode=ColorMode.BGR,
)
)
self.condition = threading.Condition()
self.jpeg: bytes | None = None
self.frame: Any | None = None
self.frame_id = 0
self.frame_count = 0
self.error: str | None = None
self.connected = False
self.stopped = False
self._stop_event = threading.Event()
self._thread: threading.Thread | None = None
def connect(self) -> None:
self.camera.connect()
self.connected = True
self._thread = threading.Thread(
target=self._capture_loop,
name=f"realsense-{self.device.serial}",
daemon=True,
)
self._thread.start()
def _set_error(self, error: Exception) -> None:
with self.condition:
self.error = f"{type(error).__name__}: {error}"
self.condition.notify_all()
def _capture_loop(self) -> None:
period = 1.0 / self.fps
next_deadline = time.monotonic()
while not self._stop_event.is_set():
try:
frame = self.camera.read()
encoded, buffer = cv2.imencode(".jpg", frame)
if not encoded:
raise RuntimeError("JPEG encoding failed")
with self.condition:
self.frame = frame
self.jpeg = buffer.tobytes()
self.frame_id += 1
self.frame_count += 1
self.error = None
self.condition.notify_all()
except Exception as exc:
self._set_error(exc)
if self._stop_event.wait(0.1):
break
next_deadline += period
delay = next_deadline - time.monotonic()
if delay > 0:
self._stop_event.wait(delay)
elif -delay > period * 2:
next_deadline = time.monotonic()
def wait_for_frame(self, last_frame_id: int) -> tuple[bytes | None, int, bool]:
with self.condition:
self.condition.wait_for(
lambda: self.frame_id > last_frame_id or self.stopped,
timeout=1.0,
)
return self.jpeg, self.frame_id, self.stopped
def latest_frame(self) -> Any | None:
with self.condition:
return None if self.frame is None else self.frame.copy()
def status(self, selected: bool) -> dict[str, Any]:
with self.condition:
return {
"serial": self.device.serial,
"name": self.device.name,
"product_line": self.device.product_line,
"width": self.width,
"height": self.height,
"fps": self.fps,
"connected": self.connected,
"selected": selected,
"frame_count": self.frame_count,
"error": self.error,
}
def stop(self) -> None:
self._stop_event.set()
with self.condition:
self.stopped = True
self.condition.notify_all()
if self._thread is not None:
self._thread.join(timeout=2.0)
if self.connected:
try:
self.camera.disconnect()
except Exception as exc:
self._set_error(exc)
self.connected = False
class _PreviewState:
def __init__(self, feeds: list[_CameraFeed], selected: list[str], fps: int) -> None:
self.feeds = {feed.device.serial: feed for feed in feeds}
self.selected = set(selected)
self.fps = fps
def get_feed(self, serial: str | None) -> _CameraFeed | None:
return self.feeds.get(serial) if serial else None
def device_status(self) -> list[dict[str, Any]]:
return [
feed.status(feed.device.serial in self.selected)
for feed in self.feeds.values()
]
def stop(self) -> None:
for feed in self.feeds.values():
feed.stop()
WEB_PAGE = """<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>RealSense Viewer</title>
<style>
:root { color-scheme: dark; font-family: system-ui, sans-serif; }
* { box-sizing: border-box; }
body { margin: 0; background: #101214; color: #e8eaed; }
header { position: sticky; top: 0; z-index: 2; padding: 14px 18px; background: #191c20; border-bottom: 1px solid #30343a; }
h1 { margin: 0 0 10px; font-size: 20px; font-weight: 600; }
#status { color: #9da5af; font-size: 13px; }
.toolbar { display: flex; flex-wrap: wrap; gap: 8px; align-items: center; margin-top: 12px; }
button { border: 1px solid #4a515b; border-radius: 5px; padding: 7px 11px; color: #e8eaed; background: #262b31; cursor: pointer; }
button:hover { background: #323941; }
#devices { display: flex; flex-wrap: wrap; gap: 8px; margin-top: 12px; }
.device { display: flex; align-items: center; gap: 7px; padding: 7px 9px; border: 1px solid #3b4149; border-radius: 5px; background: #20242a; font-size: 13px; }
.device input { width: 16px; height: 16px; accent-color: #4da3ff; }
.device small { color: #9da5af; }
.device.error { border-color: #a44d4d; }
#grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(360px, 1fr)); gap: 12px; padding: 14px; }
.tile { min-width: 0; overflow: hidden; background: #191c20; border: 1px solid #30343a; border-radius: 6px; }
.tile h2 { margin: 0; padding: 9px 11px; font-size: 14px; font-weight: 500; }
.tile img { display: block; width: 100%; height: auto; background: #08090a; }
.empty { padding: 36px 18px; color: #9da5af; text-align: center; }
</style>
</head>
<body>
<header>
<h1>RealSense Viewer</h1>
<div id="status">Loading devices...</div>
<div class="toolbar">
<button id="all" type="button">Select all</button>
<button id="none" type="button">Clear</button>
</div>
<div id="devices"></div>
</header>
<main id="grid"><div class="empty">Loading...</div></main>
<script>
const devicesEl = document.getElementById('devices');
const gridEl = document.getElementById('grid');
const statusEl = document.getElementById('status');
let devices = [];
let selected = new Set();
function deviceLabel(device) {
return `${device.name} (${device.serial})`;
}
function streamUrl(serial) {
return `/stream?serial=${encodeURIComponent(serial)}`;
}
function renderGrid() {
gridEl.replaceChildren();
const visible = devices.filter(device => selected.has(device.serial) && device.connected);
if (!visible.length) {
const empty = document.createElement('div');
empty.className = 'empty';
empty.textContent = 'Select a connected device to display.';
gridEl.appendChild(empty);
return;
}
for (const device of visible) {
const tile = document.createElement('section');
tile.className = 'tile';
const title = document.createElement('h2');
title.textContent = deviceLabel(device);
const image = document.createElement('img');
image.alt = deviceLabel(device);
image.src = streamUrl(device.serial);
tile.append(title, image);
gridEl.appendChild(tile);
}
}
function renderDevices() {
devicesEl.replaceChildren();
for (const device of devices) {
const label = document.createElement('label');
label.className = `device${device.error ? ' error' : ''}`;
const checkbox = document.createElement('input');
checkbox.type = 'checkbox';
checkbox.checked = selected.has(device.serial);
checkbox.disabled = !device.connected;
checkbox.addEventListener('change', () => {
if (checkbox.checked) selected.add(device.serial);
else selected.delete(device.serial);
renderGrid();
});
const text = document.createElement('span');
text.textContent = deviceLabel(device);
const details = document.createElement('small');
details.textContent = device.connected ? `${device.width}x${device.height} @ ${device.fps} Hz` : (device.error || 'offline');
label.append(checkbox, text, details);
devicesEl.appendChild(label);
}
}
async function refresh() {
try {
const response = await fetch('/api/devices', { cache: 'no-store' });
const payload = await response.json();
devices = payload.devices;
if (!selected.size) {
for (const device of devices.filter(device => device.selected && device.connected)) selected.add(device.serial);
}
statusEl.textContent = `${devices.filter(device => device.connected).length}/${devices.length} devices connected · ${payload.fps} Hz`;
renderDevices();
renderGrid();
} catch (error) {
statusEl.textContent = `Device service unavailable: ${error}`;
}
}
document.getElementById('all').addEventListener('click', () => {
for (const device of devices) if (device.connected) selected.add(device.serial);
renderDevices(); renderGrid();
});
document.getElementById('none').addEventListener('click', () => {
selected.clear(); renderDevices(); renderGrid();
});
refresh();
</script>
</body>
</html>
""".encode("utf-8")
class _PreviewHandler(http.server.BaseHTTPRequestHandler):
state: _PreviewState
def _send_bytes(self, status: int, content_type: str, body: bytes) -> None:
try:
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
self.wfile.flush()
except (BrokenPipeError, ConnectionResetError):
pass
def do_GET(self) -> None:
parsed = urlsplit(self.path)
path = parsed.path
if path == "/":
self._send_bytes(200, "text/html; charset=utf-8", WEB_PAGE)
return
if path == "/favicon.ico":
self._send_bytes(204, "text/plain; charset=utf-8", b"")
return
if path == "/health":
self._send_bytes(200, "application/json; charset=utf-8", self._json(self.state.device_status()))
return
if path == "/api/devices":
self._send_bytes(
200,
"application/json; charset=utf-8",
self._json(self.state.device_status(), include_fps=True),
)
return
if path == "/stream":
serial = parse_qs(parsed.query).get("serial", [None])[0]
self._stream(serial)
return
try:
self.send_error(404)
except (BrokenPipeError, ConnectionResetError):
pass
def _json(self, devices: list[dict[str, Any]], include_fps: bool = False) -> bytes:
payload: dict[str, Any] = {"ok": True, "devices": devices}
if include_fps:
payload["fps"] = self.state.fps
return json.dumps(payload, ensure_ascii=False).encode("utf-8")
def _stream(self, serial: str | None) -> None:
feed = self.state.get_feed(serial)
if feed is None:
self._send_bytes(404, "text/plain; charset=utf-8", b"Unknown camera serial\n")
return
if not feed.connected:
self._send_bytes(503, "text/plain; charset=utf-8", b"Camera is not connected\n")
return
try:
self.send_response(200)
self.send_header("Cache-Control", "no-cache, private")
self.send_header("Pragma", "no-cache")
self.send_header("Content-Type", "multipart/x-mixed-replace; boundary=frame")
self.end_headers()
last_frame_id = 0
while True:
jpeg, frame_id, stopped = feed.wait_for_frame(last_frame_id)
if stopped:
return
if jpeg is None or frame_id == last_frame_id:
continue
last_frame_id = frame_id
self.wfile.write(
b"--frame\r\nContent-Type: image/jpeg\r\n"
+ f"Content-Length: {len(jpeg)}\r\n\r\n".encode()
+ jpeg
+ b"\r\n"
)
self.wfile.flush()
except (BrokenPipeError, ConnectionResetError):
pass
def log_message(self, format: str, *args: Any) -> None:
pass
def start_web_server(
state: _PreviewState, host: str, port: int
) -> http.server.ThreadingHTTPServer:
handler = type("RealSensePreviewHandler", (_PreviewHandler,), {"state": state})
server = http.server.ThreadingHTTPServer((host, port), handler)
server.daemon_threads = True
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
if host == "0.0.0.0":
print(f"Open http://127.0.0.1:{port}/ on this machine to view the streams.")
print(f"For another machine, use http://<server-ip>:{port}/")
else:
print(f"Open http://{host}:{port}/ in a browser to view the streams.")
print("Press Ctrl+C to exit.")
return server
def _make_mosaic(feeds: list[_CameraFeed]) -> Any | None:
tiles = []
for feed in feeds:
frame = feed.latest_frame()
if frame is None:
continue
cv2.putText(
frame,
f"{feed.device.name} {feed.device.serial}",
(10, 28),
cv2.FONT_HERSHEY_SIMPLEX,
0.65,
(0, 255, 255),
2,
cv2.LINE_AA,
)
tiles.append(frame)
if not tiles:
return None
height = max(tile.shape[0] for tile in tiles)
aligned = [
cv2.resize(tile, (int(tile.shape[1] * height / tile.shape[0]), height))
if tile.shape[0] != height
else tile
for tile in tiles
]
return cv2.hconcat(aligned)
def main() -> None:
args = parse_args()
if args.fps <= 0:
raise SystemExit("FPS must be positive")
if args.no_display:
backend = "none"
elif args.backend == "auto":
backend = "opencv" if opencv_has_gui() else "web"
else:
backend = args.backend
devices = discover_devices()
requested = list(dict.fromkeys(args.serial or []))
if requested:
requested_set = set(requested)
devices = [device for device in devices if device.serial in requested_set]
missing = sorted(requested_set - {device.serial for device in devices})
if missing:
print(f"Requested cameras not found: {', '.join(missing)}")
if not devices:
raise SystemExit("No RealSense cameras found.")
feeds: list[_CameraFeed] = []
for device in devices:
feed = _CameraFeed(device, args.width, args.height, args.fps)
try:
print(f"Opening RealSense {device.name} {device.serial}: {args.width}x{args.height} @ {args.fps} FPS")
feed.connect()
print(f"Connected {device.serial}")
except Exception as exc:
feed.error = f"{type(exc).__name__}: {exc}"
print(f"Failed to connect {device.serial}: {feed.error}")
feeds.append(feed)
initially_selected = requested or ([feeds[0].device.serial] if feeds else [])
state = _PreviewState(feeds, initially_selected, args.fps)
web_server = None
if backend == "web":
web_server = start_web_server(state, args.host, args.port)
elif backend == "opencv":
print("Press q or Esc to exit.")
try:
if backend == "web":
while True:
time.sleep(1.0)
elif backend == "none":
while True:
time.sleep(1.0)
statuses = [
f"{feed.device.serial}:{'OK' if feed.connected and feed.error is None else 'ERROR'}"
for feed in feeds
]
print(" | ".join(statuses), flush=True)
else:
while True:
mosaic = _make_mosaic(feeds)
if mosaic is not None:
cv2.imshow("RealSense Color", mosaic)
key = cv2.waitKey(1) & 0xFF
if key in (ord("q"), 27):
break
except KeyboardInterrupt:
pass
finally:
state.stop()
if web_server is not None:
web_server.shutdown()
web_server.server_close()
if backend == "opencv":
try:
cv2.destroyAllWindows()
except cv2.error:
pass
print("RealSense preview stopped.")
if __name__ == "__main__":
main()

View File

@ -2,9 +2,49 @@
set -euo pipefail set -euo pipefail
repo_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
cd "$repo_root"
config_path="config/manual_mode/xarm7_manual_record_config.yaml"
dataset_root="$(
sed -n '/^dataset:/,/^[^[:space:]]/s/^[[:space:]]*root:[[:space:]]*//p' "$config_path" \
| head -n 1 \
| tr -d "\"'"
)"
if [[ -z "$dataset_root" ]]; then
printf 'Could not read dataset.root from %s\n' "$config_path" >&2
exit 1
fi
if [[ "$dataset_root" != /* ]]; then
dataset_root="$repo_root/$dataset_root"
fi
record_args=("$@")
resume_requested=false
for arg in "${record_args[@]}"; do
if [[ "$arg" == "-r" ]]; then
resume_requested=true
break
fi
done
if [[ -e "$dataset_root" ]]; then
if [[ ! -f "$dataset_root/meta/info.json" ]]; then
printf 'Dataset directory exists but is not a valid LeRobot dataset: %s\n' "$dataset_root" >&2
printf 'Choose a new dataset.root, or remove this empty/incomplete directory before recording.\n' >&2
exit 1
fi
if [[ "$resume_requested" == false ]]; then
record_args=("-r" "${record_args[@]}")
fi
elif [[ "$resume_requested" == true ]]; then
printf 'Cannot resume because the dataset directory does not exist: %s\n' "$dataset_root" >&2
exit 1
fi
exec uv run uf-lerobot-record \ exec uv run uf-lerobot-record \
--config_path config/manual_mode/xarm7_manual_record_config.yaml \ --config_path "$config_path" \
"$@" "${record_args[@]}"

View File

@ -1,6 +1,5 @@
import sys import sys
from pathlib import Path from pathlib import Path
from types import SimpleNamespace
import numpy as np import numpy as np
import pytest import pytest
@ -12,7 +11,6 @@ from lerobot_robot_ufactory.scripts.uf_lerobot_record import (
_manual_action_from_observation, _manual_action_from_observation,
_update_manual_gripper_key_state, _update_manual_gripper_key_state,
_update_manual_gripper_target, _update_manual_gripper_target,
_prepare_dataset_root,
_prepare_recording_episode, _prepare_recording_episode,
get_cfg, get_cfg,
) )
@ -296,46 +294,6 @@ def test_manual_record_config_has_no_teleop(monkeypatch):
assert config.dataset.fps == 30 assert config.dataset.fps == 30
def test_prepare_dataset_root_leaves_new_root_for_lerobot_create(tmp_path):
root = tmp_path / "nested" / "dataset"
cfg = SimpleNamespace(dataset=SimpleNamespace(root=root), resume=False)
_prepare_dataset_root(cfg)
assert not root.exists()
def test_prepare_dataset_root_rejects_incomplete_resume(tmp_path):
root = tmp_path / "dataset"
(root / "meta").mkdir(parents=True)
(root / "meta" / "info.json").write_text("{}")
cfg = SimpleNamespace(dataset=SimpleNamespace(root=root), resume=True)
with pytest.raises(RuntimeError, match="meta/tasks.parquet"):
_prepare_dataset_root(cfg)
def test_prepare_dataset_root_rejects_resume_when_root_is_missing(tmp_path):
cfg = SimpleNamespace(dataset=SimpleNamespace(root=tmp_path / "missing"), resume=True)
with pytest.raises(RuntimeError, match="does not exist"):
_prepare_dataset_root(cfg)
def test_prepare_dataset_root_resumes_complete_dataset_without_prompt(tmp_path, monkeypatch):
root = tmp_path / "dataset"
(root / "meta" / "episodes" / "chunk-000").mkdir(parents=True)
(root / "data" / "chunk-000").mkdir(parents=True)
(root / "meta" / "info.json").write_text("{}")
(root / "meta" / "tasks.parquet").write_bytes(b"tasks")
(root / "meta" / "episodes" / "chunk-000" / "file-000.parquet").write_bytes(b"episodes")
(root / "data" / "chunk-000" / "file-000.parquet").write_bytes(b"data")
cfg = SimpleNamespace(dataset=SimpleNamespace(root=root), resume=True)
monkeypatch.setattr(record_module.sys.stdin, "isatty", lambda: True)
_prepare_dataset_root(cfg)
def test_manual_record_loop_writes_actual_state_as_action(tmp_path): def test_manual_record_loop_writes_actual_state_as_action(tmp_path):
class FakeRobot: class FakeRobot:
name = "fake_manual_robot" name = "fake_manual_robot"