​

Mixpeek Pipelines Guide

Mixpeek pipelines serve as serverless orchestrators, coordinating various methods into a single atomic unit that directly interacts with your database. They are designed to streamline operations by automatically managing tasks like create, upsert, and delete through generated Lambdas. Below, you will find a step-by-step guide on creating and using Mixpeek pipelines.

​

Setup Options

You can set up your Mixpeek pipeline using the API, through the Mixpeek Dashboard, or by integrating with GitHub. Here are the options:

    from mixpeek import Mixpeek

    mixpeek = Mixpeek("API_KEY")


    pipeline_id = mixpeek.pipelines.create(
    alias="MyPipeline",
    code="function execute() { return true; }",
    source={
        "connection_id": "conn_456",
        "filters": {
        "foo": "bar"
        }
    },
    destination={
        "connection_id": "conn_431",
        "collection": "data_bucket",
        "metadata": {
        "type": "storage"
        }
    }
    )

    my-repo/
    ├── .github/
    │   └── workflows/
    │       └── build-and-push-docker.yml
    ├── pipeline1/
    │   ├── handler.py
    │   └── config.json
    ├── pipeline2/
    │   ├── handler.py
    │   └── config.json
    └── pipeline3/
        ├── handler.py
        └── config.json

​

Creating a Pipeline

To create a pipeline in Mixpeek, you first need to set up your environment with the necessary credentials and import the Mixpeek library.

from mixpeek import Mixpeek

# Initialize Mixpeek with your API key
mixpeek = Mixpeek("YOUR_API_KEY")

​

Define and Create the Pipeline via API

You can define and create a pipeline by specifying the source, the operations to perform, and the destination for the data. Here’s an example of how to create a pipeline:

pipeline_id = mixpeek.pipelines.create(
    alias="MyPipeline",
    code="function execute() { return true; }",
    source={
        "connection_id": "conn_456",
        "filters": {
            "foo": "bar"
        }
    },
    destination={
        "connection_id": "conn_431",
        "collection": "data_bucket",
        "metadata": {
            "type": "storage"
        }
    }
)

​

Parameters:

• alias: A friendly name for the pipeline.
• code: Python code to execute within the pipeline.
• source: Configuration of the data source including connection identifiers and filters.
• destination: Configuration of the destination where the results will be stored.