Bacalhau is an open-source distributed compute orchestration framework designed to bring compute to the data. Instead of moving large datasets around networks, Bacalhau makes it easy to execute jobs close to the data's location, drastically reducing latency and resource overhead.
- ⚡ Fast job processing: Jobs in Bacalhau are processed where the data was created and all jobs are parallel by default
- 💰 Low cost: Reduce (or eliminate) ingress/egress costs since jobs are processed closer to the source
- 🔒 Secure: Data scrubbing and security can happen before migration, with a granular, code-based permission model
- 🚛 Large-scale data: Process petabytes of data efficiently without massive data transfers
- 🏢 Data sovereignty: Process sensitive data within security boundaries without requiring it to leave your premises
- 🤝 Cross-organizational computation: Allow specific vetted computations on protected datasets without exposing raw data
-
Single Binary Simplicity: Bacalhau is a single self-contained binary that functions as a client, orchestrator, and compute node—making it incredibly easy to set up and scale
-
Modular Architecture: Support for multiple execution engines (Docker, WebAssembly) and storage providers through clean interfaces
-
Orchestrator-Compute Model: A dedicated orchestrator coordinates job scheduling, while compute nodes run tasks
-
Flexible Storage Integrations: Integrates with S3, HTTP/HTTPS, IPFS, and local storage systems
-
Multiple Job Types: Support for batch, ops, daemon, and service job types for different workflow requirements
-
Declarative & Imperative Submissions: Define jobs in YAML (declarative) or pass arguments via CLI (imperative)
-
Publisher Support: Output results to local volumes, S3, or other storage backends
# Install Bacalhau CLI (Linux/macOS)
curl -sL https://get.bacalhau.org/install.sh | bash
# Verify installation
bacalhau versionFor the complete quick start guide, including running your first job, see our Quick Start Documentation.
Bacalhau's distributed compute framework enables a wide range of applications:
- Log Processing: Process logs efficiently at scale by running distributed jobs directly at the source
- Distributed Data Warehousing: Query and analyze data across multiple regions without moving large datasets
- Fleet Management: Efficiently manage distributed nodes across multiple environments
- Distributed Machine Learning: Train and deploy ML models across a distributed compute fleet
- Edge Computing: Run compute tasks closer to the data source for applications requiring low latency
📚 Read the Bacalhau docs guide here! 📚
The Bacalhau documentation contains all the information you need to get started:
Bacalhau has a very friendly community, and we are always happy to help:
- Join the Slack Community and go to the
#generalchannel - it is the easiest way to engage with other members in the community and get help
If you are interested in contributing to the Bacalhau project:
- Set up your local environment
- Check out our Contributing Guide
- For issues and feature requests, please open a GitHub issue
We are excited to hear your feedback!
This repository contains the Bacalhau software, covered under the Apache-2.0 license, except where noted (any Bacalhau logos or trademarks are not covered under the Apache License, and should be explicitly noted by a LICENSE file.)
Bacalhau is a product produced from this open source software, exclusively by Expanso, Inc. It is distributed under our commercial terms.
Others are allowed to make their own distribution of the software, but they cannot use any of the Bacalhau trademarks, cloud services, etc.
We explicitly grant permission for you to make a build that includes our trademarks while developing Bacalhau software itself. You may not publish or share the build, and you may not use that build to run Bacalhau software for any other purpose.
We have borrowed the above Open Source clause from the excellent System Initiative
![@dependabot[bot]](https://avatars.githubusercontent.com/in/29110?s=64&v=4){kind=small}
