Get Batch Configuration

Headers

Authorization

string

required

REQUIRED: Bearer token authentication using your API key. Format: 'Bearer sk_xxxxxxxxxxxxx'. You can create API keys in the Mixpeek dashboard under Organization Settings.

X-Namespace

string

required

REQUIRED: Namespace identifier for scoping this request. All resources (collections, buckets, taxonomies, etc.) are scoped to a namespace. You can provide either the namespace name or namespace ID. Format: ns_xxxxxxxxxxxxx (ID) or a custom name like 'my-namespace'

Path Parameters

bucket_identifier

string

required

The unique identifier of the bucket.

batch_id

string

required

The unique identifier of the batch.

Response

Successful Response

Model representing a batch of objects for processing through collections.

A batch groups bucket objects together for processing through one or more collections. Batches support multi-tier processing where collections are processed in dependency order (e.g., bucket → chunks → frames → scenes). Each tier has independent task tracking.

Use Cases: - Process multiple objects through collections in a single batch - Track progress of multi-tier decomposition pipelines - Monitor and retry individual processing tiers - Query batch status and tier-specific task information

Lifecycle: 1. Created in DRAFT status with object_ids 2. Submitted for processing → status changes to PENDING 3. Each tier processes sequentially (tier 0 → tier 1 → ... → tier N) 4. Batch completes when all tiers finish (status=COMPLETED) or any tier fails (status=FAILED)

Multi-Tier Processing: - Tier 0: Bucket objects → Collections (bucket as source) - Tier N (N > 0): Collection documents → Collections (upstream collection as source) - Each tier gets independent task tracking via tier_tasks array - Processing proceeds tier-by-tier with automatic chaining

Requirements: - batch_id: OPTIONAL (auto-generated if not provided) - bucket_id: REQUIRED - status: OPTIONAL (defaults to DRAFT) - object_ids: REQUIRED for processing (must have at least 1 object) - collection_ids: OPTIONAL (discovered via DAG resolution) - tier_tasks: OPTIONAL (populated during processing) - current_tier: OPTIONAL (set during processing) - total_tiers: OPTIONAL (defaults to 1, set during DAG resolution) - dag_tiers: OPTIONAL (populated during DAG resolution)

bucket_id

string

required

REQUIRED. Unique identifier of the bucket containing the objects to process. Must be a valid bucket ID that exists in the system. All object_ids must belong to this bucket. Format: Bucket ID as defined when bucket was created.

batch_id

string

OPTIONAL (auto-generated if not provided). Unique identifier for this batch. Format: 'btch_' prefix followed by 12-character secure token. Generated using generate_secure_token() from shared.utilities.helpers. Used to query batch status and track processing across tiers. Immutable after creation.

status

enum<string>

default:DRAFT

OPTIONAL (defaults to DRAFT). Current processing status of the batch. Lifecycle: DRAFT → PENDING → IN_PROGRESS → COMPLETED/FAILED. DRAFT: Batch created but not yet submitted. PENDING: Batch submitted and queued for processing. IN_PROGRESS: Batch currently processing (one or more tiers active). COMPLETED: All tiers successfully completed. FAILED: One or more tiers failed. Aggregated from tier_tasks statuses during multi-tier processing.

Available options:

PENDING,

IN_PROGRESS,

PROCESSING,

COMPLETED,

COMPLETED_WITH_ERRORS,

FAILED,

CANCELED,

UNKNOWN,

SKIPPED,

DRAFT,

ACTIVE,

ARCHIVED,

SUSPENDED

object_ids

string[]

REQUIRED for processing (must have at least 1). List of object IDs to include in this batch. All objects must exist in the specified bucket_id. These objects are the source data for tier 0 processing. Minimum 1 object, no maximum limit. Objects are processed in parallel within each tier.

Minimum array length: 1

collection_ids

string[] | null

OPTIONAL. List of all collection IDs involved in this batch's processing. Automatically populated during DAG resolution from dag_tiers. Includes collections from all tiers (flattened view of dag_tiers). Used for quick lookups without traversing tier structure. Format: List of collection IDs across all tiers.

Example:

["col_chunks"]

error

string | null

OPTIONAL. Legacy error message field for backward compatibility. None if batch succeeded or is still processing. Contains human-readable error description from first failed tier. DEPRECATED: Use tier_tasks[].errors for detailed error information. For multi-tier batches, typically contains the error from the first failed tier. Check tier_tasks array for tier-specific error details and error_summary for aggregation.

Example:

"Failed to process batch: Object not found"

error_summary

Error Summary · object

OPTIONAL. Aggregated summary of errors across ALL tiers in the batch. None if batch succeeded or is still processing. Maps error_type (category) to total count of affected documents across all tiers. Provides quick batch-wide overview of error distribution. Example: {'dependency': 15, 'authentication': 25, 'validation': 5} means across all tiers, 15 documents failed with dependency errors, 25 with auth errors, 5 with validation errors. Automatically aggregated from tier_tasks[].error_summary. Used for batch health dashboard and error trend analysis.

Show child attributes

error_summary.{key}

integer

Example:

null

type

enum<string>

default:BUCKET

OPTIONAL (defaults to BUCKET). Type of batch. BUCKET: Standard batch processing bucket objects through collections. COLLECTION: Reserved for future collection-only batch processing. Currently only BUCKET type is supported.

Available options:

BUCKET,

COLLECTION

manifest_key

string | null

OPTIONAL. S3 key where the batch manifest is stored. Contains metadata and row data (Parquet) for Engine processing. For tier 0, points to bucket object manifest. For tier N+, points to collection document manifest. Format: S3 path (e.g., 'namespace_id/internal_id/manifests/tier_0.parquet'). Generated during batch submission.

Example:

"ns_abc/org_123/manifests/tier_0.parquet"

task_id

string | null

OPTIONAL. Primary task ID for the batch (typically tier 0 task). Used for backward compatibility with single-tier batch tracking. For multi-tier batches, prefer querying tier_tasks array for granular tracking. Format: Task ID as generated for tier 0.

Example:

"task_tier0_abc123"

loaded_object_ids

string[] | null

OPTIONAL. List of object IDs that were successfully validated and loaded into the batch. Subset of object_ids that passed validation. Used to track which objects are ready for processing. None if batch hasn't been validated yet.

Example:

["obj_video_001", "obj_video_002"]

internal_metadata

Internal Metadata · object

OPTIONAL. Internal engine/job metadata for system use. May contain: job_id (provider-specific), engine_version, processing hints, last_health_check. last_health_check: Most recent health check results with health_status, enriched_documents, vector_populated_count, stall_duration_seconds, recommendations, missing_features. Populated asynchronously via Celery task (non-blocking, best-effort). Used for troubleshooting batch processing issues via API. Format: Free-form dictionary for internal tracking.

Example:

{
  "include_processing_history": true,
  "last_health_check": {
    "enriched_documents": 98,
    "health_status": "HEALTHY",
    "missing_features": ["text_embedding"],
    "processed_documents": 100,
    "recommendations": [],
    "stall_duration_seconds": 0,
    "timestamp": "2025-11-06T10:05:00Z",
    "total_documents": 100,
    "vector_populated_count": 98
  }
}

metadata

Metadata · object

OPTIONAL. User-defined metadata for the batch. Arbitrary key-value pairs for user tracking and organization. Persisted with the batch and returned in API responses. Not used by the system for processing logic. Examples: campaign_id, user_email, processing_notes.

tier_tasks

TierTaskInfo · object[]

OPTIONAL. List of tier task tracking information for multi-tier processing. Each element represents one tier in the processing pipeline. Empty array for simple single-tier batches. Populated during batch submission with tier 0 info, then appended as tiers progress. Each TierTaskInfo contains: tier_num, task_id, status, collection_ids, timestamps. Used for granular monitoring: 'Show me status of tier 2' or 'Retry tier 1'. Array index typically matches tier_num (tier_tasks[0] = tier 0, tier_tasks[1] = tier 1, etc.).

Show child attributes

tier_tasks.tier_num

integer

required

REQUIRED. Zero-based tier number indicating the processing stage. Tier 0 = initial bucket-to-collection processing (bucket objects as source). Tier N (N > 0) = collection-to-collection processing (upstream documents as source). Used to determine processing order and identify which stage a task represents. Example: In a 5-tier pipeline (bucket → chunks → frames → scenes → summaries), chunks=tier 0, frames=tier 1, scenes=tier 2, summaries=tier 3.

Required range: x >= 0

tier_tasks.source_type

string

required

REQUIRED. Type of data source for this tier's processing. 'bucket': Tier 0 processing where source is bucket objects from the objects table. 'collection': Tier N+ processing where source is documents from upstream collection(s). Determines how the API prepares the input dataset manifest for the Engine. Bucket sources query the objects table and include file blobs. Collection sources query the documents table and include processed features.

tier_tasks.task_id

string | null

OPTIONAL. Unique task identifier for this tier's processing task. None if tier has not yet started (status=PENDING). Assigned when tier processing begins (status=IN_PROGRESS). Used to query task status via GET /v1/tasks/{task_id}. Format: 'task_' prefix followed by secure token. Generated using generate_secure_token() from shared.utilities.helpers.

Example:

"task_tier0_abc123"

tier_tasks.status

enum<string>

default:PENDING

REQUIRED. Current processing status of this tier's task. Lifecycle: PENDING → IN_PROGRESS → COMPLETED/FAILED. PENDING: Tier scheduled but not yet started. IN_PROGRESS: Tier currently processing documents. COMPLETED: Tier successfully processed all documents. FAILED: Tier encountered an error and stopped processing. Used to determine overall batch status and whether to proceed to next tier.

Available options:

PENDING,

IN_PROGRESS,

PROCESSING,

COMPLETED,

COMPLETED_WITH_ERRORS,

FAILED,

CANCELED,

UNKNOWN,

SKIPPED,

DRAFT,

ACTIVE,

ARCHIVED,

SUSPENDED

tier_tasks.collection_ids

string[]

REQUIRED. List of collection IDs being processed in this tier. Each tier can process one or more collections in parallel. Collections in the same tier have no dependencies on each other. Format: Collection IDs as defined when collections were created. Minimum 1 collection per tier. Example: Tier 1 might process ['col_frames_30fps', 'col_frames_60fps'] in parallel.

Minimum array length: 1

tier_tasks.source_collection_ids

string[] | null

OPTIONAL for tier 0 (must be None). REQUIRED for tier N+ (N > 0). List of upstream collection IDs that provide documents as input to this tier. Typically contains collection IDs from the previous tier (tier_num - 1). Used by the API to query documents from these collections for processing. These upstream documents are converted to a Parquet manifest for the Engine. Example: If tier 1 processes 'col_frames' and sources from tier 0's 'col_chunks', then source_collection_ids=['col_chunks'].

Example:

["col_chunks"]

tier_tasks.parent_task_id

string | null

OPTIONAL. Task ID of the previous tier (tier_num - 1) that processed before this tier. Used to link tiers together for audit trail and lineage tracking. None for tier 0 (no parent). Enables queries like 'show all tiers that processed after tier 0' or 'trace back through all parent tiers to find the original batch'. Format: Same as task_id (e.g., 'task_tier0_abc123').

Example:

"task_tier0_abc123"

tier_tasks.started_at

string<date-time> | null

OPTIONAL. ISO 8601 timestamp when this tier began processing. None if tier has not yet started (status=PENDING). Set using current_time() from shared.utilities.helpers when tier starts. Used to calculate tier processing duration and identify long-running tiers. Example: '2025-11-03T10:00:00Z'.

Example:

"2025-11-03T10:00:00Z"

tier_tasks.completed_at

string<date-time> | null

OPTIONAL. ISO 8601 timestamp when this tier finished processing (success or failure). None if tier has not yet completed (status=PENDING or IN_PROGRESS). Set using current_time() from shared.utilities.helpers when tier completes. Used to calculate tier processing duration (completed_at - started_at). Set for both COMPLETED and FAILED statuses. Example: '2025-11-03T10:05:00Z'.

Example:

"2025-11-03T10:05:00Z"

tier_tasks.errors

BatchErrorDetail · object[]

OPTIONAL. List of detailed errors that occurred during tier processing. Empty list if tier succeeded or has not yet completed. Each error includes: error_type, message, component, stage, traceback, timestamp. Multiple errors may occur if different documents fail with different issues. Used for detailed error analysis, debugging, and intelligent retry logic. Example: Multiple documents failing with different errors (dependency vs auth). For backward compatibility, check if list is empty for success/in-progress status.

Show child attributes

tier_tasks.errors.error_type

enum<string>

required

REQUIRED. Category of error that occurred. Used for filtering, retry logic, and error analytics. DEPENDENCY: Missing packages/modules (e.g., google-genai not installed). AUTHENTICATION: Invalid credentials or expired tokens. VALIDATION: Schema mismatches or invalid input data. RUNTIME: Code exceptions or processing failures. NETWORK: Connectivity issues or timeouts. RESOURCE: Out of memory or disk space.

Available options:

dependency,

authentication,

validation,

runtime,

network,

resource

tier_tasks.errors.message

string

required

REQUIRED. Human-readable error message. Concise description of what went wrong. Should be actionable and help users understand the issue.

tier_tasks.errors.component

string | null

OPTIONAL. Component or service where the error occurred. Helps identify which part of the system failed. Examples: service class names, module names, or feature names.

Example:

"VertexMultimodalService"

tier_tasks.errors.stage

string | null

OPTIONAL. Processing stage where the error occurred. Identifies which pipeline stage failed. Examples: pipeline stage names from collection configuration.

Example:

"gemini_extraction"

tier_tasks.errors.traceback

string | null

OPTIONAL. Full Python traceback for debugging. Includes stack trace for code-level troubleshooting. Should be truncated if too long (e.g., max 2000 chars).

tier_tasks.errors.timestamp

string<date-time>

REQUIRED. ISO 8601 timestamp when the error occurred. Used for chronological error tracking and debugging.

tier_tasks.errors.affected_document_ids

string[]

OPTIONAL. List of document IDs affected by this error. For object-level errors: contains single document ID. For batch-level aggregation: contains all affected document IDs. Used to identify scope of impact.

tier_tasks.errors.affected_count

integer

default:1

REQUIRED. Number of documents affected by this error. For object-level: typically 1. For batch-level aggregation: total count of affected documents. Used for error impact analysis.

Required range: x >= 1

tier_tasks.errors.recovery_suggestion

string | null

OPTIONAL. Actionable suggestion for resolving the error. Helps users quickly fix common issues. Examples: install missing package, check credentials, update schema.

Example:

"Install google-genai package: pip install google-genai"

tier_tasks.errors.metadata

Metadata · object

OPTIONAL. Additional error context and metadata. Free-form dictionary for error-specific details. Examples: retry_count, last_retry_at, error_code, http_status.

tier_tasks.error_summary

Error Summary · object

OPTIONAL. Aggregated summary of errors by error type. None if tier succeeded or has not yet completed. Maps error_type (category) to count of affected documents. Provides quick overview of error distribution without parsing full error list. Example: {'dependency': 5, 'authentication': 10, 'validation': 3} means 5 documents failed with dependency errors, 10 with auth errors, 3 with validation. Automatically generated from errors list for convenience. Used for batch health monitoring and error trend analysis.

Show child attributes

tier_tasks.error_summary.{key}

integer

Example:

null

tier_tasks.performance

Performance · object

OPTIONAL. Performance metrics summary for this tier's execution. Automatically populated after tier completion by collecting data from ClickHouse analytics. Contains: total_time_ms (total execution time), avg_latency_ms (average operation latency), bottlenecks (list of slowest operations), stage_count (number of profiled stages). Used for troubleshooting performance issues and identifying bottlenecks. None if tier has not completed or performance data collection failed. Populated asynchronously via Celery task (non-blocking, best-effort).

Example:

{
  "avg_latency_ms": 234.56,
  "bottlenecks": [
    {
      "avg_time_ms": 113.58,
      "execution_count": 50,
      "max_time_ms": 234.56,
      "stage_name": "gcs_batch_upload_all_segments",
      "total_time_ms": 5678.9
    },
    {
      "avg_time_ms": 69.14,
      "execution_count": 50,
      "max_time_ms": 123.45,
      "stage_name": "pipeline_run",
      "total_time_ms": 3456.78
    }
  ],
  "stage_count": 5,
  "timestamp": "2025-11-06T10:05:00Z",
  "total_time_ms": 12345.67
}

tier_tasks.ray_job_id

string | null

OPTIONAL. Ray/Anyscale job ID for tracking the infrastructure-level processing job. None if tier has not yet started or if the job ID was not returned by the engine. Set when tier processing is submitted to Ray/Anyscale via the Engine. Used for cancelling running jobs and monitoring infrastructure-level status. Format: 'raysubmit_' prefix followed by job identifier (e.g., 'raysubmit_9pDAyZbd5MN281TB'). This is the job ID that appears in the Ray/Anyscale dashboard.

Example:

"raysubmit_9pDAyZbd5MN281TB"

current_tier

integer | null

OPTIONAL. Zero-based index of the currently processing tier. None if batch hasn't started processing (status=DRAFT or PENDING). Updated as batch progresses through tiers. Used to show processing progress: 'Processing tier 2 of 5'. Set to last tier number when batch completes. Example: If processing tier 1 (frames), current_tier=1.

Required range: x >= 0

Example:

0

total_tiers

integer

default:1

OPTIONAL (defaults to 1). Total number of tiers in the collection DAG. Minimum 1 (tier 0 only = bucket → collection). Set during DAG resolution when batch is submitted. Equals len(dag_tiers) if dag_tiers is populated. Used to calculate progress: current_tier / total_tiers. Example: 5-tier pipeline (bucket → chunks → frames → scenes → summaries) has total_tiers=5.

Required range: x >= 1

dag_tiers

string[][] | null

OPTIONAL. Complete DAG tier structure for this batch. List of tiers, where each tier is a list of collection IDs to process at that stage. Tier 0 = bucket-sourced collections. Tier N (N > 0) = collection-sourced collections. Collections within same tier have no dependencies (can run in parallel). Collections in tier N+1 depend on collections in tier N. Populated during DAG resolution at batch submission. Used for tier-by-tier processing orchestration. Example: [['col_chunks'], ['col_frames', 'col_objects'], ['col_scenes']] = 3 tiers where frames and objects run in parallel at tier 1.

Example:

[["col_chunks"]]

created_at

string<date-time>

OPTIONAL (auto-set on creation). ISO 8601 timestamp when batch was created. Set using current_time() from shared.utilities.helpers. Immutable after creation. Used for batch age tracking and cleanup of old batches.

updated_at

string<date-time>

OPTIONAL (auto-updated). ISO 8601 timestamp when batch was last modified. Updated using current_time() whenever batch status or tier_tasks change. Used to track batch activity and identify stale batches.

Health

Organizations

Namespaces

Buckets

Feature Extractors

Collections

Retrievers

Taxonomies

Clusters

Templates

Analytics

Resource Search

Inference

Agent Sessions

Tasks

Webhooks

Notifications

Triggers

Get Batch Configuration

Headers

Path Parameters

Response