Architecture

​

Overview

​

Core Components

• API (FastAPI)
  • Auth via API keys; tenancy resolved per request using Authorization and X-Namespace headers
  • Centralized errors with a consistent JSON envelope and documented status codes
  • Rate limiting and safety policies enforced per organization and API name
  • Asynchronous tasks (e.g., bucket deletes) scheduled via Celery
• Engine (FastAPI + Ray)
  • Distributed execution for feature extraction, taxonomy enrichment, and clustering
  • Parallel processing across extractors; resilient fan‑out/fan‑in with backpressure
  • Ensures Qdrant collections exist and writes vectors and payloads in a single‑write pattern
• Data stores
  • Qdrant: Primary store for documents, payloads, and feature vectors (per‑namespace collections)
  • MongoDB: Metadata/configuration for collections, retrievers, and taxonomies
  • Redis: Caching, counters, rate limiting, and lightweight task metadata
  • S3‑compatible storage: Source object blobs, batch manifests, and optional presigned URL generation

​

Tenancy and Security

• Organizations and Namespaces: Every request is scoped to an organization and, for most routes, a namespace. Namespaces map 1:1 to Qdrant collections.
• Headers: Authorization: Bearer <API_KEY> and X-Namespace: <id-or-name> are required for namespace‑scoped routes.
• Payload scoping: All Qdrant documents include internal_id for strict tenant filtering.
• Rate limiting: Redis‑backed limits per organization and API operation.

​

Ingestion and Processing

• Uploads
  • Register objects via the Buckets/Object API; optionally batch register many at once
  • Store raw assets in S3 (or any S3‑compatible storage)
• Batch extraction (Engine)
  • Engine reads batch manifests and collection configs
  • Runs feature extractors in parallel using Ray
  • Writes vectors and payloads to Qdrant with tenant metadata
  • Optionally materializes taxonomy enrichments post‑extraction
• Feature store
  • Multiple vector indexes per collection (e.g., text, image) with payload indexes for filtering

​

Retrieval

• Retrievers define inputs, stages, and target collections
• SearchFlow executes KNN, hybrid (RRF), and grouped searches over Qdrant
• Pre/post‑filters, sorting, pagination, and optional S3 URL presigning are supported
• Normalized results prefer Qdrant point id as document_id

​

Error Handling and Health

• Consistent error envelope with explicit status codes across all routes
• Health fan‑out checks Redis, MongoDB, Qdrant, Celery, and Engine endpoints

​

Operations

• Asynchronous tasks: Celery for API‑side async operations and housekeeping
• Configuration: All services configure via environment variables (e.g., QDRANT_URL, MONGO_URI)
• Observability: Structured logs and clear error surfaces; rate limiting in non‑local environments

​

Extensibility

• Self‑host or managed: Deploy core services in your cloud or use managed offerings
• Custom feature extractors: Plug in your own extractors and indexes
• API‑first: Everything is accessible via documented APIs