Skip to content

v0.5

Latest
Latest
Compare
Choose a tag to compare
@j20001970 j20001970 released this 13 Feb 08:25
· 14 commits to master since this release
v0.5
a8874b8

GDMP v0.5 is here. All vision-related tasks are included in this release, as well as improvements of existing framework classes. Initial support for macOS and Web are also included. There are both newly introduced and deprecated classes in this release, please refer to the changelog below.

What's changed in this release

General

  • godot-cpp update to v4.3
  • MediaPipe update to v0.10.20
  • build.py now build release libraries by default.

Framework improvements

  • MediaPipeProto now handles nested set/get.
  • MediaPipeGraphConfig is now the subclass of MediaPipeProto
  • MediaPipePacket is more simplified.
  • MediaPipeGraph reworked to match original mediapipe::CalculatorGraph usage more.

New classes

  • Two new classes MediaPipeGraphBuilder and MediaPipeGraphNode are introduced using MediaPipe's api2, which allows building MediaPipeGraphConfig directly from code.
  • MediaPipeTaskRunner is the higher-level class for running calculator graphs, you can use it to run graphs that do not have task API yet (e.g. HolisticLandmarkerGraph).
  • MediaPipeExternalFiles is the class autoload for downloading MediaPipe assets and models from Google Cloud Storage via Godot's HTTPRequest, developers can use it to download additional model assets on demand instead of packaging them on export.

New vision tasks

The following new vision tasks and graphs has been added, marking the completion of all currently available vision tasks:

  • HolisticLandmarkerGraph
  • MediaPipeFaceStylizer
  • MediaPipeInteractiveSegmenter

Deprecated classes

MediaPipeCameraHelper is considered deprecated since v0.5, and will be removed in future releases. 3.x (GDNative) versions are not affected by this change unless platforms support for CameraServer are backported to Godot 3.x in future. No new features will be developed for camera helper from now on.

Android

  • CI now also build x86_64 architecture
  • Camera permissions are removed from AAR manifest, you will need to enable camera permission upon exporting Godot projects if accessing cameras is required.

iOS

  • CI now builds both x86_64 and arm64 architectures.

Linux

  • CI build linux libraries with Freedesktop SDK.
  • OpenCV is now built from source to avoid linking entire libraries.

macOS support

macOS CPU build has been added, available in both arm64 and x86_64 architectures, please help us test if it works!

Web support

GDMP for web support has been added via WebAssembly module. While the library can be loaded, most tasks are still not working as expected, and some tasks are not included at all due to OpenCV disabled. GDMP for web will be excluded from version release libraries until most tasks start working, you can still build web library with build.py if you would like to try anyway.

Q&A

Q: Audio/Text tasks when?
A: Audio and text tasks are planned to be added in next (v0.6) release.

Q: Why deprecate MediaPipeCameraHelper?
A:
If I had a nickel for every time I included a computer vision-related Godot plugin that also does camera I/O in the same project, forget the nickels because those plugins alone will be as heavy as node_modules first.

While the statement above is definitely for the joke, including camera I/O functionalities for GDMP does certainly increase project complexity, and hinder distribution due to libraries size, and more importantly, the lack of standardized ways to access camera frames on Godot Engine leads to the possibility of camera code duplication if multiple CV libraries are used. Therefore camera helper is planned to be removed in future releases, so that we can focus on providing MediaPipe framework and solutions to Godot.

Your next line is..."How do I access camera frames after camera helper is removed?"

Q: How do I access camera frames after camera helper is removed...?!
A: CameraServer-based approach is recommended as its usage is common across Godot Engine, while the platform support for CameraServer is incomplete (only macOS and Linux in v4.4), there has been effort to extend the support to more platforms via GDExtension.

CameraServerExtension is a Godot v4.4+ plugin (also made by me) that aim to bring CameraServer support to more platforms, while only Android and Linux is available at the time of writing, support for more platforms will come along the way. We'll try to cover all officially-supported platforms by the time MediaPipeCameraHelper is completely removed.

Q: Where is MediaPipeHolisticLandmarker?
A: As of MediaPipe v0.10.20, there are no official task API available for holistic landmarker, you can instead use MediaPipeGraphBuilder to create a calculator graph with mediapipe.tasks.vision.holistic_landmarker.HolisticLandmarkerGraph node, then use MediaPipeTaskRunner to run it. MediaPipeHolisticLandmarker will be added after MediaPipe getting the task API for holistic landmarker.

Q: Generative AI tasks (e.g. Image Generator, LLM Inference) when?
A: Generative AI tasks are planned after the support of audio and text tasks, it may or may not added in the same release of audio/text tasks.

Full Changelog: v0.4...v0.5

Assets 3
Loading