Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Wandelbots NOVA bridge as example #9161

Closed
wants to merge 19 commits into from
Closed
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
383d25e
Add wandelbots nova example
stefanwagnerdev Feb 26, 2025
87afd25
Add images to example
stefanwagnerdev Feb 26, 2025
e515c7b
Update README.md
stefanwagnerdev Feb 26, 2025
a8a8a67
Enhance Nova Rerun example with detailed documentation and real-time …
stefanwagnerdev Feb 26, 2025
cbeaa0f
Update README.md
stefanwagnerdev Feb 26, 2025
76bc290
Update Nova Rerun example README to streamline model download instruc…
stefanwagnerdev Feb 26, 2025
66628bc
Remove unnecessary blank line in Nova Rerun example README
stefanwagnerdev Feb 26, 2025
074f118
Enhance README.md with improved feature descriptions for real-time vi…
stefanwagnerdev Feb 26, 2025
f8651db
Update README.md
stefanwagnerdev Feb 26, 2025
1717108
Update README.md
stefanwagnerdev Feb 27, 2025
bff1d8b
Update README.md
stefanwagnerdev Feb 27, 2025
fea6057
Update Nova Rerun example README with enhanced descriptions and acces…
stefanwagnerdev Feb 27, 2025
dded2d6
Add initial files for Wandelbots NOVA Rerun example, including .gitig…
stefanwagnerdev Feb 27, 2025
f3cce2c
Clarify README instructions for accessing the Wandelbots NOVA platform
stefanwagnerdev Feb 27, 2025
3077d8d
Merge branch 'rerun-io:main' into main
stefanwagnerdev Feb 27, 2025
c83e2cb
Remove optional dependencies and ruff configuration files from the Wa…
stefanwagnerdev Feb 28, 2025
0c07268
fix: formatting in nova example
stefanwagnerdev Mar 3, 2025
8ae54bb
Update examples/python/wandelbots_nova_rerun/README.md
stefanwagnerdev Mar 3, 2025
e104328
remove example code as it requries login
stefanwagnerdev Mar 3, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions examples/python/wandelbots_nova_rerun/.env
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rename the directory wandelbots_nova. The _rerun part is superfluous given it's in the rerun repository :)

Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
NOVA_API=
NOVA_ACCESS_TOKEN=
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing newline at EOF will be rejected by our CI.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the example code as it requires login

1 change: 1 addition & 0 deletions examples/python/wandelbots_nova_rerun/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
models
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the example code as it requires login

171 changes: 171 additions & 0 deletions examples/python/wandelbots_nova_rerun/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
<!--[metadata]
title = "Wandelbots NOVA Bridge"
source = "https://github.com/wandelbotsgmbh/wandelbots-nova"
tags = ["3D", "Robot"]
thumbnail = "https://github.com/user-attachments/assets/526a3cff-3d27-4963-8d7b-e7cd98a758f9"
thumbnail_dimensions = [480, 480]
-->

A visualization extension for [wandelbots-nova](https://github.com/wandelbotsgmbh/wandelbots-nova) that enables real-time 3D visualization of robot trajectories using [rerun.io](https://rerun.io).

https://vimeo.com/1060763904?autoplay=1&loop=1&autopause=0&background=1&muted=1&ratio=10000:7627

## Background

[Wandelbots NOVA](https://www.wandelbots.com/) is an agnostic robot operating system that enables developers to virtually plan their industrial six-axis robot fleet, as well as to program, control and operate your robots on the shopfloor - all independent on the robot brand and through a unified API. It combines modern development tools (Python, JavaScript APIs) with an AI-driven approach to robot control and motion planning, enabling developers to build applications like gluing, grinding, welding, and palletizing without worrying about underlying hardware differences. The holistic software offers a variety of tools to create unique automation solutions along the whole automation process.

This example demonstrates how to use Rerun to visualize and analyze NOVA capabilities through:

- Trajectory visualization and motion planning
- Robot state monitoring and digital twin visualization
- Collision scene inspection, avoidance and validation
- Motion timing and performance analysis

### Run the code

To use the bridge you need to install the [wandelbots-nova](https://github.com/wandelbotsgmbh/wandelbots-nova) package and access to the Wandelbots NOVA platform.

Apply at: [wandelbots.com/contact](https://www.wandelbots.com/contact).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is unideal. See main review comment.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it is in another project should I leave this here or should I remove it and let the explanation be in https://github.com/wandelbotsgmbh/wandelbots-nova ?


```bash
uv run download-models # Download the required models
uv run main.py
```

The example demonstrates how to use Wandelbots NOVA with Rerun for visualizing planned robot trajectories.

```python
from nova_rerun_bridge import NovaRerunBridge
from nova import Nova
from nova import api
from nova.actions import jnt, ptp
from nova.types import Pose
import asyncio

async def main():
# Connect to your Nova instance (or use .env file)
nova = Nova(
host="https://your-instance.wandelbots.io",
access_token="your-access-token"
)
bridge = NovaRerunBridge(nova)

# Setup visualization
await bridge.setup_blueprint()

# Setup robot
cell = nova.cell()
controller = await cell.ensure_virtual_robot_controller(
"ur",
api.models.VirtualControllerTypes.UNIVERSALROBOTS_MINUS_UR10E,
api.models.Manufacturer.UNIVERSALROBOTS,
)

# Connect to the controller and activate motion groups
async with controller[0] as motion_group:
home_joints = await motion_group.joints()
tcp_names = await motion_group.tcp_names()
tcp = tcp_names[0]

# Get current TCP pose and offset it slightly along the x-axis
current_pose = await motion_group.tcp_pose(tcp)
target_pose = current_pose @ Pose((1, 0, 0, 0, 0, 0))

actions = [
jnt(home_joints),
ptp(target_pose),
jnt(home_joints),
]

# Plan trajectory
joint_trajectory = await motion_group.plan(actions, tcp)

# Log a trajectory
await bridge.log_trajectory(joint_trajectory, tcp, motion_group)


if __name__ == "__main__":
asyncio.run(main())
```

### Features

- **Real-time 3D robot visualization**
See a [list of supported robots](https://wandelbotsgmbh.github.io/wandelbots-js-react-components/?path=/story/3d-view-robot-supported-models--abb-1010-037-15) and get real-time playback of robot movements.
- **Trajectory playback and analysis**
Easily log trajectories and visualize them in the Rerun viewer.
- **Collision scene visualization**
Inspect collision objects, plan safe paths, and avoid unexpected contact points.
- **Continuous monitoring mode**
Stream live robot states and keep an eye on actual motion in real time.

<picture>
<img src="https://github.com/user-attachments/assets/617dd2c5-ea51-472d-84d5-77aa25f6c2b6" alt=""/>
<source media="(max-width: 1200px)" srcset="https://github.com/user-attachments/assets/617dd2c5-ea51-472d-84d5-77aa25f6c2b6">
</picture>

### Usage Examples

Below are some common usage patterns. For more detailed examples, see the [example repository.](https://github.com/wandelbotsgmbh/wandelbots-nova/tree/main/nova_rerun_bridge/examples)

#### Basic Motion Logging

```python
# Log a pre-planned trajectory
await bridge.log_trajectory(joint_trajectory, tcp, motion_group)
```

#### Collision free movements

Apart from the usual movement commands like `point to point`, `joint point to point`, `linear` and `circular` the plattform also supports collision free movements. You need to setup a collision scene beforehand and pass it to the action.

```python
actions = [
collision_free(
target=Pose((-500, -400, 200, np.pi, 0, 0)),
collision_scene=collision_scene,
settings=MotionSettings(tcp_velocity_limit=30),
)
]

trajectory = await motion_group.plan(
actions,
tcp=tcp
)
await bridge.log_actions(actions)
await bridge.log_trajectory(trajectory_plan_combined, tcp, motion_group)
```

https://vimeo.com/1060763933?autoplay=1&loop=1&autopause=0&background=1&muted=1&ratio=10000:7627

#### Real-time robot state streaming

The bridge also supports continuous monitoring of robot states:

```python
# Start streaming robot state
await bridge.start_streaming(motion_group)

# ... do something with your robot ...

# Stop streaming all robot states
await bridge.stop_streaming()
```

#### Log Actions

If you’d like to log the planned actions themselves:

```python
# Log planned actions
await bridge.log_actions(actions)
```

### More Examples

Check out the [examples folder](https://github.com/wandelbotsgmbh/wandelbots-nova/tree/main/nova_rerun_bridge/examples) for more detailed usage scenarios (e.g., advanced collision scene management, streaming multiple robots simultaneously, etc.).

<picture>
<img src="https://github.com/user-attachments/assets/586811cc-278c-484d-8a2a-9abcba6ab5d3" alt=""/>
<source media="(max-width: 1200px)" srcset="https://github.com/user-attachments/assets/586811cc-278c-484d-8a2a-9abcba6ab5d3">
</picture>
179 changes: 179 additions & 0 deletions examples/python/wandelbots_nova_rerun/main.py
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these needs to be named wandelbots_nova.py, since it should be a valid installable python package (no need to make a subdir + __init__.py though, as long as it fits a single file).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the example code as it requires login

Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
"""
Example of planning a collision free PTP motion. A sphere is placed in the robot's path and the robot uses collision free p2p to move around it.
"""

import asyncio

import numpy as np
import rerun as rr
from nova import MotionSettings
from nova.actions.motions import collision_free, ptp
from nova.api import models
from nova.core.exceptions import PlanTrajectoryFailed
from nova.core.nova import Nova
from nova.types import Pose
from nova_rerun_bridge import NovaRerunBridge
from wandelbots_api_client.models import (
CoordinateSystem,
RotationAngles,
RotationAngleTypes,
Vector3d,
)


DESCRIPTION = """
# Nova rerun bridge
[Wandelbots Nova](https://www.wandelbots.com/) is a robot-agnostic operating system that
enables programming and controlling various industrial robots through a unified interface.
This example demonstrates how to use Nova and Rerun to visualize robot trajectories and
real-time states for any supported industrial robot.

More examples can be found
[on GitHub](https://github.com/wandelbotsgmbh/wandelbots-nova).
""".strip()


async def build_collision_world(nova: Nova, cell_name: str, robot_setup: models.OptimizerSetup) -> str:
collision_api = nova._api_client.store_collision_components_api
scene_api = nova._api_client.store_collision_scenes_api

# define annoying obstacle
sphere_collider = models.Collider(
shape=models.ColliderShape(models.Sphere2(radius=100, shape_type="sphere")),
pose=models.Pose2(position=[-100, -500, 200]),
)
await collision_api.store_collider(cell=cell_name, collider="annoying_obstacle", collider2=sphere_collider)

# define TCP collider geometry
tool_collider = models.Collider(
shape=models.ColliderShape(models.Box2(size_x=100, size_y=100, size_z=100, shape_type="box", box_type="FULL"))
)
await collision_api.store_collision_tool(
cell=cell_name, tool="tool_box", request_body={"tool_collider": tool_collider}
)

# define robot link geometries
robot_link_colliders = await collision_api.get_default_link_chain(
cell=cell_name, motion_group_model=robot_setup.motion_group_type
)
await collision_api.store_collision_link_chain(
cell=cell_name, link_chain="robot_links", collider=robot_link_colliders
)

# assemble scene
scene = models.CollisionScene(
colliders={"annoying_obstacle": sphere_collider},
motion_groups={
robot_setup.motion_group_type: models.CollisionMotionGroup(
tool={"tool_geometry": tool_collider}, link_chain=robot_link_colliders
)
},
)
scene_id = "collision_scene"
await scene_api.store_collision_scene(cell_name, scene_id, models.CollisionSceneAssembly(scene=scene))
return scene_id


async def test():
async with Nova() as nova, NovaRerunBridge(nova) as bridge:
await bridge.setup_blueprint()

rr.log("description", rr.TextDocument(DESCRIPTION, media_type=rr.MediaType.MARKDOWN), static=True)
cell = nova.cell()
controller = await cell.ensure_virtual_robot_controller(
"ur5",
models.VirtualControllerTypes.UNIVERSALROBOTS_MINUS_UR5E,
models.Manufacturer.UNIVERSALROBOTS,
)

await nova._api_client.virtual_robot_setup_api.set_virtual_robot_mounting(
cell="cell",
controller=controller.controller_id,
id=0,
coordinate_system=CoordinateSystem(
coordinate_system="world",
name="mounting",
reference_uid="",
position=Vector3d(x=0, y=0, z=0),
rotation=RotationAngles(angles=[0, 0, 0], type=RotationAngleTypes.EULER_ANGLES_EXTRINSIC_XYZ),
),
)

# NC-1047
await asyncio.sleep(5)

# Connect to the controller and activate motion groups
async with controller[0] as motion_group:
await bridge.log_saftey_zones(motion_group)

tcp = "Flange"

robot_setup: models.OptimizerSetup = await motion_group._get_optimizer_setup(tcp=tcp)
robot_setup.safety_setup.global_limits.tcp_velocity_limit = 200

collision_scene_id = await build_collision_world(nova, "cell", robot_setup)

await bridge.log_collision_scenes()

# Use default planner to move to the right of the sphere
home = await motion_group.tcp_pose(tcp)
actions = [ptp(home), ptp(target=Pose((300, -400, 200, np.pi, 0, 0)))]

for action in actions:
action.settings = MotionSettings(tcp_velocity_limit=200)

try:
joint_trajectory = await motion_group.plan(
actions, tcp, start_joint_position=(0, -np.pi / 2, np.pi / 2, 0, 0, 0)
)
await bridge.log_actions(actions)
await bridge.log_trajectory(joint_trajectory, tcp, motion_group)
except PlanTrajectoryFailed as e:
await bridge.log_actions(actions)
await bridge.log_trajectory(e.error.joint_trajectory, tcp, motion_group)
await bridge.log_error_feedback(e.error.error_feedback)

rr.log("motion/target_", rr.Points3D([[-500, -400, 200]], radii=[10], colors=[(0, 255, 0)]))

# Use default planner to move to the left of the sphere
# -> this will collide
# only plan don't move
actions = [ptp(target=Pose((-500, -400, 200, np.pi, 0, 0)))]

for action in actions:
action.settings = MotionSettings(tcp_velocity_limit=200)

try:
joint_trajectory_with_collision = await motion_group.plan(
actions, tcp, start_joint_position=joint_trajectory.joint_positions[-1].joints
)
await bridge.log_actions(actions)
await bridge.log_trajectory(joint_trajectory_with_collision, tcp, motion_group)
except PlanTrajectoryFailed as e:
await bridge.log_actions(actions)
await bridge.log_trajectory(e.error.joint_trajectory, tcp, motion_group)
await bridge.log_error_feedback(e.error.error_feedback)

# Plan collision free PTP motion around the sphere
scene_api = nova._api_client.store_collision_scenes_api
collision_scene = await scene_api.get_stored_collision_scene(cell="cell", scene=collision_scene_id)

welding_actions = [
collision_free(
target=Pose((-500, -400, 200, np.pi, 0, 0)),
collision_scene=collision_scene,
settings=MotionSettings(tcp_velocity_limit=30),
)
]

trajectory_plan_combined = await motion_group.plan(
welding_actions,
tcp=tcp,
start_joint_position=joint_trajectory.joint_positions[-1].joints,
)
await bridge.log_actions(welding_actions)
await bridge.log_trajectory(trajectory_plan_combined, tcp, motion_group)


if __name__ == "__main__":
asyncio.run(test())
2 changes: 2 additions & 0 deletions examples/python/wandelbots_nova_rerun/poetry.toml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not leak tooling specific stuff here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the example code as it requires login

Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[virtualenvs]
in-project = true
19 changes: 19 additions & 0 deletions examples/python/wandelbots_nova_rerun/pyproject.toml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be fleshed out to be an installable pyproject.toml, so it needs to specify a build backend. See our other examples. I would strongly suggest using hatch for that purpose, for consistency reason.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the example code as it requires login

Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
[project]
name = "nova_rerun"
version = "0.0.1"
description = "Minimal example of a project using the nova_rerun_bridge"
authors = [
{ name = "Wandelbots GmbH" },
{ name = "Stefan Wagner", email = "stefan.wagner@wandelbots.com" }
]
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
"wandelbots-nova[nova-rerun-bridge]>=0.12",
"python-dotenv>=1.0.0"
]

[project.optional-dependencies]
dev = [
"ruff>=0.9.2"
]
Loading
Loading