-
Notifications
You must be signed in to change notification settings - Fork 428
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Wandelbots NOVA bridge as example #9161
Changes from 15 commits
383d25e
87afd25
e515c7b
a8a8a67
cbeaa0f
76bc290
66628bc
074f118
f8651db
1717108
bff1d8b
fea6057
dded2d6
f3cce2c
3077d8d
c83e2cb
0c07268
8ae54bb
e104328
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
NOVA_API= | ||
NOVA_ACCESS_TOKEN= | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Missing newline at EOF will be rejected by our CI. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I removed the example code as it requires login |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
models | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I removed the example code as it requires login |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,171 @@ | ||
<!--[metadata] | ||
title = "Wandelbots NOVA Bridge" | ||
source = "https://github.com/wandelbotsgmbh/wandelbots-nova" | ||
tags = ["3D", "Robot"] | ||
thumbnail = "https://github.com/user-attachments/assets/526a3cff-3d27-4963-8d7b-e7cd98a758f9" | ||
stefanwagnerdev marked this conversation as resolved.
Show resolved
Hide resolved
|
||
thumbnail_dimensions = [480, 480] | ||
--> | ||
|
||
A visualization extension for [wandelbots-nova](https://github.com/wandelbotsgmbh/wandelbots-nova) that enables real-time 3D visualization of robot trajectories using [rerun.io](https://rerun.io). | ||
|
||
https://vimeo.com/1060763904?autoplay=1&loop=1&autopause=0&background=1&muted=1&ratio=10000:7627 | ||
|
||
## Background | ||
|
||
[Wandelbots NOVA](https://www.wandelbots.com/) is an agnostic robot operating system that enables developers to virtually plan their industrial six-axis robot fleet, as well as to program, control and operate your robots on the shopfloor - all independent on the robot brand and through a unified API. It combines modern development tools (Python, JavaScript APIs) with an AI-driven approach to robot control and motion planning, enabling developers to build applications like gluing, grinding, welding, and palletizing without worrying about underlying hardware differences. The holistic software offers a variety of tools to create unique automation solutions along the whole automation process. | ||
|
||
This example demonstrates how to use Rerun to visualize and analyze NOVA capabilities through: | ||
|
||
- Trajectory visualization and motion planning | ||
- Robot state monitoring and digital twin visualization | ||
- Collision scene inspection, avoidance and validation | ||
- Motion timing and performance analysis | ||
|
||
### Run the code | ||
|
||
To use the bridge you need to install the [wandelbots-nova](https://github.com/wandelbotsgmbh/wandelbots-nova) package and access to the Wandelbots NOVA platform. | ||
|
||
Apply at: [wandelbots.com/contact](https://www.wandelbots.com/contact). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is unideal. See main review comment. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since it is in another project should I leave this here or should I remove it and let the explanation be in https://github.com/wandelbotsgmbh/wandelbots-nova ? |
||
|
||
```bash | ||
uv run download-models # Download the required models | ||
uv run main.py | ||
``` | ||
|
||
The example demonstrates how to use Wandelbots NOVA with Rerun for visualizing planned robot trajectories. | ||
|
||
```python | ||
from nova_rerun_bridge import NovaRerunBridge | ||
from nova import Nova | ||
from nova import api | ||
from nova.actions import jnt, ptp | ||
from nova.types import Pose | ||
import asyncio | ||
|
||
async def main(): | ||
# Connect to your Nova instance (or use .env file) | ||
nova = Nova( | ||
host="https://your-instance.wandelbots.io", | ||
access_token="your-access-token" | ||
) | ||
bridge = NovaRerunBridge(nova) | ||
|
||
# Setup visualization | ||
await bridge.setup_blueprint() | ||
|
||
# Setup robot | ||
cell = nova.cell() | ||
controller = await cell.ensure_virtual_robot_controller( | ||
"ur", | ||
api.models.VirtualControllerTypes.UNIVERSALROBOTS_MINUS_UR10E, | ||
api.models.Manufacturer.UNIVERSALROBOTS, | ||
) | ||
|
||
# Connect to the controller and activate motion groups | ||
async with controller[0] as motion_group: | ||
home_joints = await motion_group.joints() | ||
tcp_names = await motion_group.tcp_names() | ||
tcp = tcp_names[0] | ||
|
||
# Get current TCP pose and offset it slightly along the x-axis | ||
current_pose = await motion_group.tcp_pose(tcp) | ||
target_pose = current_pose @ Pose((1, 0, 0, 0, 0, 0)) | ||
|
||
actions = [ | ||
jnt(home_joints), | ||
ptp(target_pose), | ||
jnt(home_joints), | ||
] | ||
|
||
# Plan trajectory | ||
joint_trajectory = await motion_group.plan(actions, tcp) | ||
|
||
# Log a trajectory | ||
await bridge.log_trajectory(joint_trajectory, tcp, motion_group) | ||
|
||
|
||
if __name__ == "__main__": | ||
asyncio.run(main()) | ||
``` | ||
|
||
### Features | ||
|
||
- **Real-time 3D robot visualization** | ||
See a [list of supported robots](https://wandelbotsgmbh.github.io/wandelbots-js-react-components/?path=/story/3d-view-robot-supported-models--abb-1010-037-15) and get real-time playback of robot movements. | ||
- **Trajectory playback and analysis** | ||
Easily log trajectories and visualize them in the Rerun viewer. | ||
- **Collision scene visualization** | ||
Inspect collision objects, plan safe paths, and avoid unexpected contact points. | ||
- **Continuous monitoring mode** | ||
Stream live robot states and keep an eye on actual motion in real time. | ||
|
||
<picture> | ||
<img src="https://github.com/user-attachments/assets/617dd2c5-ea51-472d-84d5-77aa25f6c2b6" alt=""/> | ||
<source media="(max-width: 1200px)" srcset="https://github.com/user-attachments/assets/617dd2c5-ea51-472d-84d5-77aa25f6c2b6"> | ||
</picture> | ||
|
||
### Usage Examples | ||
|
||
Below are some common usage patterns. For more detailed examples, see the [example repository.](https://github.com/wandelbotsgmbh/wandelbots-nova/tree/main/nova_rerun_bridge/examples) | ||
|
||
#### Basic Motion Logging | ||
|
||
```python | ||
# Log a pre-planned trajectory | ||
await bridge.log_trajectory(joint_trajectory, tcp, motion_group) | ||
``` | ||
|
||
#### Collision free movements | ||
|
||
Apart from the usual movement commands like `point to point`, `joint point to point`, `linear` and `circular` the plattform also supports collision free movements. You need to setup a collision scene beforehand and pass it to the action. | ||
|
||
```python | ||
actions = [ | ||
collision_free( | ||
target=Pose((-500, -400, 200, np.pi, 0, 0)), | ||
collision_scene=collision_scene, | ||
settings=MotionSettings(tcp_velocity_limit=30), | ||
) | ||
] | ||
|
||
trajectory = await motion_group.plan( | ||
actions, | ||
tcp=tcp | ||
) | ||
await bridge.log_actions(actions) | ||
await bridge.log_trajectory(trajectory_plan_combined, tcp, motion_group) | ||
``` | ||
|
||
https://vimeo.com/1060763933?autoplay=1&loop=1&autopause=0&background=1&muted=1&ratio=10000:7627 | ||
|
||
#### Real-time robot state streaming | ||
|
||
The bridge also supports continuous monitoring of robot states: | ||
|
||
```python | ||
# Start streaming robot state | ||
await bridge.start_streaming(motion_group) | ||
|
||
# ... do something with your robot ... | ||
|
||
# Stop streaming all robot states | ||
await bridge.stop_streaming() | ||
``` | ||
|
||
#### Log Actions | ||
|
||
If you’d like to log the planned actions themselves: | ||
|
||
```python | ||
# Log planned actions | ||
await bridge.log_actions(actions) | ||
``` | ||
|
||
### More Examples | ||
|
||
Check out the [examples folder](https://github.com/wandelbotsgmbh/wandelbots-nova/tree/main/nova_rerun_bridge/examples) for more detailed usage scenarios (e.g., advanced collision scene management, streaming multiple robots simultaneously, etc.). | ||
|
||
<picture> | ||
<img src="https://github.com/user-attachments/assets/586811cc-278c-484d-8a2a-9abcba6ab5d3" alt=""/> | ||
<source media="(max-width: 1200px)" srcset="https://github.com/user-attachments/assets/586811cc-278c-484d-8a2a-9abcba6ab5d3"> | ||
</picture> |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. these needs to be named There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I removed the example code as it requires login |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,179 @@ | ||
""" | ||
Example of planning a collision free PTP motion. A sphere is placed in the robot's path and the robot uses collision free p2p to move around it. | ||
""" | ||
|
||
import asyncio | ||
|
||
import numpy as np | ||
import rerun as rr | ||
from nova import MotionSettings | ||
from nova.actions.motions import collision_free, ptp | ||
from nova.api import models | ||
from nova.core.exceptions import PlanTrajectoryFailed | ||
from nova.core.nova import Nova | ||
from nova.types import Pose | ||
from nova_rerun_bridge import NovaRerunBridge | ||
from wandelbots_api_client.models import ( | ||
CoordinateSystem, | ||
RotationAngles, | ||
RotationAngleTypes, | ||
Vector3d, | ||
) | ||
|
||
|
||
DESCRIPTION = """ | ||
# Nova rerun bridge | ||
[Wandelbots Nova](https://www.wandelbots.com/) is a robot-agnostic operating system that | ||
enables programming and controlling various industrial robots through a unified interface. | ||
This example demonstrates how to use Nova and Rerun to visualize robot trajectories and | ||
real-time states for any supported industrial robot. | ||
|
||
More examples can be found | ||
[on GitHub](https://github.com/wandelbotsgmbh/wandelbots-nova). | ||
""".strip() | ||
|
||
|
||
async def build_collision_world(nova: Nova, cell_name: str, robot_setup: models.OptimizerSetup) -> str: | ||
collision_api = nova._api_client.store_collision_components_api | ||
scene_api = nova._api_client.store_collision_scenes_api | ||
|
||
# define annoying obstacle | ||
sphere_collider = models.Collider( | ||
shape=models.ColliderShape(models.Sphere2(radius=100, shape_type="sphere")), | ||
pose=models.Pose2(position=[-100, -500, 200]), | ||
) | ||
await collision_api.store_collider(cell=cell_name, collider="annoying_obstacle", collider2=sphere_collider) | ||
|
||
# define TCP collider geometry | ||
tool_collider = models.Collider( | ||
shape=models.ColliderShape(models.Box2(size_x=100, size_y=100, size_z=100, shape_type="box", box_type="FULL")) | ||
) | ||
await collision_api.store_collision_tool( | ||
cell=cell_name, tool="tool_box", request_body={"tool_collider": tool_collider} | ||
) | ||
|
||
# define robot link geometries | ||
robot_link_colliders = await collision_api.get_default_link_chain( | ||
cell=cell_name, motion_group_model=robot_setup.motion_group_type | ||
) | ||
await collision_api.store_collision_link_chain( | ||
cell=cell_name, link_chain="robot_links", collider=robot_link_colliders | ||
) | ||
|
||
# assemble scene | ||
scene = models.CollisionScene( | ||
colliders={"annoying_obstacle": sphere_collider}, | ||
motion_groups={ | ||
robot_setup.motion_group_type: models.CollisionMotionGroup( | ||
tool={"tool_geometry": tool_collider}, link_chain=robot_link_colliders | ||
) | ||
}, | ||
) | ||
scene_id = "collision_scene" | ||
await scene_api.store_collision_scene(cell_name, scene_id, models.CollisionSceneAssembly(scene=scene)) | ||
return scene_id | ||
|
||
|
||
async def test(): | ||
async with Nova() as nova, NovaRerunBridge(nova) as bridge: | ||
await bridge.setup_blueprint() | ||
|
||
rr.log("description", rr.TextDocument(DESCRIPTION, media_type=rr.MediaType.MARKDOWN), static=True) | ||
cell = nova.cell() | ||
controller = await cell.ensure_virtual_robot_controller( | ||
"ur5", | ||
models.VirtualControllerTypes.UNIVERSALROBOTS_MINUS_UR5E, | ||
models.Manufacturer.UNIVERSALROBOTS, | ||
) | ||
|
||
await nova._api_client.virtual_robot_setup_api.set_virtual_robot_mounting( | ||
cell="cell", | ||
controller=controller.controller_id, | ||
id=0, | ||
coordinate_system=CoordinateSystem( | ||
coordinate_system="world", | ||
name="mounting", | ||
reference_uid="", | ||
position=Vector3d(x=0, y=0, z=0), | ||
rotation=RotationAngles(angles=[0, 0, 0], type=RotationAngleTypes.EULER_ANGLES_EXTRINSIC_XYZ), | ||
), | ||
) | ||
|
||
# NC-1047 | ||
await asyncio.sleep(5) | ||
|
||
# Connect to the controller and activate motion groups | ||
async with controller[0] as motion_group: | ||
await bridge.log_saftey_zones(motion_group) | ||
|
||
tcp = "Flange" | ||
|
||
robot_setup: models.OptimizerSetup = await motion_group._get_optimizer_setup(tcp=tcp) | ||
robot_setup.safety_setup.global_limits.tcp_velocity_limit = 200 | ||
|
||
collision_scene_id = await build_collision_world(nova, "cell", robot_setup) | ||
|
||
await bridge.log_collision_scenes() | ||
|
||
# Use default planner to move to the right of the sphere | ||
home = await motion_group.tcp_pose(tcp) | ||
actions = [ptp(home), ptp(target=Pose((300, -400, 200, np.pi, 0, 0)))] | ||
|
||
for action in actions: | ||
action.settings = MotionSettings(tcp_velocity_limit=200) | ||
|
||
try: | ||
joint_trajectory = await motion_group.plan( | ||
actions, tcp, start_joint_position=(0, -np.pi / 2, np.pi / 2, 0, 0, 0) | ||
) | ||
await bridge.log_actions(actions) | ||
await bridge.log_trajectory(joint_trajectory, tcp, motion_group) | ||
except PlanTrajectoryFailed as e: | ||
await bridge.log_actions(actions) | ||
await bridge.log_trajectory(e.error.joint_trajectory, tcp, motion_group) | ||
await bridge.log_error_feedback(e.error.error_feedback) | ||
|
||
rr.log("motion/target_", rr.Points3D([[-500, -400, 200]], radii=[10], colors=[(0, 255, 0)])) | ||
|
||
# Use default planner to move to the left of the sphere | ||
# -> this will collide | ||
# only plan don't move | ||
actions = [ptp(target=Pose((-500, -400, 200, np.pi, 0, 0)))] | ||
|
||
for action in actions: | ||
action.settings = MotionSettings(tcp_velocity_limit=200) | ||
|
||
try: | ||
joint_trajectory_with_collision = await motion_group.plan( | ||
actions, tcp, start_joint_position=joint_trajectory.joint_positions[-1].joints | ||
) | ||
await bridge.log_actions(actions) | ||
await bridge.log_trajectory(joint_trajectory_with_collision, tcp, motion_group) | ||
except PlanTrajectoryFailed as e: | ||
await bridge.log_actions(actions) | ||
await bridge.log_trajectory(e.error.joint_trajectory, tcp, motion_group) | ||
await bridge.log_error_feedback(e.error.error_feedback) | ||
|
||
# Plan collision free PTP motion around the sphere | ||
scene_api = nova._api_client.store_collision_scenes_api | ||
collision_scene = await scene_api.get_stored_collision_scene(cell="cell", scene=collision_scene_id) | ||
|
||
welding_actions = [ | ||
collision_free( | ||
target=Pose((-500, -400, 200, np.pi, 0, 0)), | ||
collision_scene=collision_scene, | ||
settings=MotionSettings(tcp_velocity_limit=30), | ||
) | ||
] | ||
|
||
trajectory_plan_combined = await motion_group.plan( | ||
welding_actions, | ||
tcp=tcp, | ||
start_joint_position=joint_trajectory.joint_positions[-1].joints, | ||
) | ||
await bridge.log_actions(welding_actions) | ||
await bridge.log_trajectory(trajectory_plan_combined, tcp, motion_group) | ||
|
||
|
||
if __name__ == "__main__": | ||
asyncio.run(test()) |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's not leak tooling specific stuff here. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I removed the example code as it requires login |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
[virtualenvs] | ||
in-project = true |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This needs to be fleshed out to be an installable There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I removed the example code as it requires login |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
[project] | ||
name = "nova_rerun" | ||
version = "0.0.1" | ||
description = "Minimal example of a project using the nova_rerun_bridge" | ||
authors = [ | ||
{ name = "Wandelbots GmbH" }, | ||
{ name = "Stefan Wagner", email = "stefan.wagner@wandelbots.com" } | ||
] | ||
readme = "README.md" | ||
requires-python = ">=3.10" | ||
dependencies = [ | ||
"wandelbots-nova[nova-rerun-bridge]>=0.12", | ||
"python-dotenv>=1.0.0" | ||
] | ||
|
||
[project.optional-dependencies] | ||
dev = [ | ||
"ruff>=0.9.2" | ||
] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please rename the directory
wandelbots_nova
. The_rerun
part is superfluous given it's in the rerun repository :)