System Overview
API Layer
FastAPI and Celery handle HTTP requests, validation, authorization, metadata updates, task creation, and webhook dispatch.
Engine Layer
Ray orchestrates distributed feature extraction, inference, taxonomy materialization, clustering, and other long-running compute.
Shared Storage
MongoDB, Qdrant, Redis, and S3 provide the contract between layers—no direct imports cross the API ↔ Engine boundary.
Data Flow
Ingestion (Object → Document)
1
Upload objects
Client uploads objects (
/v1/buckets/{bucket}/objects)—metadata lands in MongoDB, blobs in S3.2
Submit batch
Client submits a batch (
/v1/buckets/{bucket}/batches/{batch}/submit); the API flattens manifests into per-extractor artifacts in S3 and creates tasks.3
Process in Ray
Ray pollers pick up the batch, execute feature extractors tier-by-tier, write documents to Qdrant, and emit webhook events.
4
Dispatch events
Celery Beat processes webhook events, updates collection schemas, invalidates caches, and notifies external systems.
Retrieval (Query → Results)
1
Execute retriever
Client calls
/v1/retrievers/{id}/execute with structured inputs and optional filters.2
Validate & plan
FastAPI loads the retriever definition, validates inputs, and walks the configured stage pipeline.
3
Invoke services
Stages call Ray Serve for inference, Qdrant for search, MongoDB for joins, and Redis for cache signatures.
4
Return results
Response includes results, stage metrics, cache hints (
ETag), and budget usage.Communication Patterns
- Task Queue (API → Engine)
- Webhook Events (Engine → API)
- Real-time Inference (API → Engine)
MongoDB stores batch descriptors; Ray pollers submit jobs and track state on behalf of the Engine.
Further Reading
- Processing → Feature Extractors for extractor catalogs and configuration patterns
- Retrieval → Retrievers for stage pipelines, caching, and execution telemetry
- Operations → Deployment for local and production deployment topologies
- Operations → Observability for metrics, logging, and tracing integrations