README.md

Spice.ai OSS Cookbook

The Spice.ai OSS Cookbook is a collection of recipes for building and deploying data & AI applications using Spice.ai. Each recipe is a self-contained example that demonstrates a specific use case or feature of Spice.ai.

Recipes

Guides

• Real-time Data Access Pattern Analysis - Use AI to analyze query patterns and detect potential security risks.

Core scenarios

• Federated SQL Query - Query data from S3, PostgreSQL, and Dremio in a single query.

Sample Applications

• Command Query Responsibility Segregation (CQRS) - Sample application implementing the CQRS pattern with Spice.

Models & AI - Connect data to hosted or local AI models

• Azure OpenAI Models
• Running Llama3 Locally - Use the Llama family of models locally from HuggingFace using Spice.
• OpenAI Models - Use OpenAI LLM and embedding models.
• OpenAI SDK - Use the OpenAI SDK to connect to models hosted on Spice.
• LLM Memory - Persistent memory for language models.
• Text to SQL (Tools)
• Nvidia NIM on Kubernetes - Deploy Nvidia NIM infrastructure, on Kubernetes, with GPUs connected to Spice.
• Nvidia NIM on AWS EC2 - Deploy Nvidia NIM on AWS GPU-optimized EC2 instances connected to Spice.
• Searching GitHub Files - Search GitHub files with embeddings and vector similarity search.
• xAI Models - Use xAI models such as Grok.
• DeepSeek Model - Use DeepSeek model through Spice.
• Filesystem Hosted Model - Use models hosted directly on filesystems.
• [Web Search Tools using Perplexity)[./websearch/README.md) - Provide LLMs with web search access for more informed answers.

Data Acceleration - Materializing & accelerating data locally with Data Accelerators

• DuckDB Data Accelerator
• PostgreSQL Data Accelerator
• SQLite Data Accelerator
• Apache Arrow Data Accelerator

Consuming and visualizing data with clients

• Sales BI (Apache Superset) - Visualize data in Spice with Apache Superset.
• Grafana Datasource - Add Spice as a Grafana datasource.

Connecting to Data Sources with Data Connectors

• Postgres Data Connector
  • AWS RDS PostgreSQL
  • Supabase
• MySQL Data Connector
  • AWS RDS Aurora (MySQL Compatible)
  • PlanetScale
• Clickhouse Data Connector
• Databricks Connector - Delta Lake and Spark Connect.
• Delta Lake Connector - Query data from Delta Lake tables.
• Debezium Change Data Capture (CDC) Data Connector from Postgres - Stream changes from a Postgres database to Spice.
  • Debezium CDC SASL/SCRAM Authentication from MySQL - Stream changes from a MySQL database to Spice using SASL/SCRAM authentication.
• Dremio Data Connector
• DuckDB Data Connector - Use a DuckDB database with sample TPCH data.
• File Data Connector - Query data from local files.
• FTP Data Connector - Query data from an FTP server.
• GitHub Data Connector
• GraphQL Data Connector
• MSSQL (Microsoft SQL Server) Data Connector
• ODBC Data Connector
• S3 Data Connector
• SharePoint/OneDrive for Business Data Connector
• Snowflake Data Connector
• Spice.ai Cloud Platform Data Connector
• Apache Spark Data Connector

Connecting to Data Sources with Catalog Connectors

• Spice.ai Cloud Platform Catalog Connector
• Databricks Unity Catalog Connector
• Unity Catalog Connector
• Iceberg Catalog Connector

Deployment and Installation

• Deploying to Kubernetes
• Running in Docker
• Sidecar Deployment Architecture
• Microservice Deployment Architecture

Performance

• TPC-H Benchmarking
• Results Caching
• Indexes on Accelerated Data

Acceleration Data Configuration

• Data Retention Policy
• Refresh Data Window
• Advanced Data Refresh
• Data Quality with Constraints

Client SDKs - Recipes for querying data from Spice with language-specific SDKs

• Rust SDK
• Python SDK
• Go SDK
• JavaScript SDK
• Java SDK

Security

• Encryption in transit using TLS
• API Key Authentication

Advanced Topics

• Local dataset replication - Link datasets in a parent/child relationship within the current Spicepod