Course Content Extractor

​

Pipeline Steps

1. Filter Dataset (if collection_id provided)
   • Filter to specified collection
2. Content Detection & Routing
   • Auto-detect content type: video, PDF, or code archive
   • Route to appropriate processor
3. Video Segmentation (if video input)
   • Scene-based segmentation or SRT subtitle-based segmentation
   • Extract transcripts via Whisper ASR (or use provided SRT)
   • OCR video frames for screen text detection
4. PDF Decomposition (if PDF input)
   • Layout detection: paragraphs, headers, tables, lists, figures, code blocks
   • Layout-aware extraction per element or per page
   • Extract images and figures with bounding boxes
5. Code Archive Processing (if code input)
   • Extract source files from ZIP archive
   • Segment code into individual functions/classes
   • Auto-detect programming language
6. Multi-Modal Embedding Generation
   • E5-Large (1024D) for transcripts, PDF text, and captions
   • Jina Code v2 (768D) for code snippets and functions
   • SigLIP (768D) for figures, screenshots, diagrams (optional)
7. LLM Enrichment (optional: if enrich_with_llm=true)
   • Generate summaries using Gemini
   • Add semantic context and key concepts
8. Output
   • Learning units with text_content, code_content, screen_text
   • Layout types, timing info, language tags
   • Multiple embeddings per unit for diverse search scenarios

​

When to Use

​

When NOT to Use

​

Input Schema

{
  "video": "s3://my-bucket/lectures/intro-to-ml.mp4",
  "srt": "s3://my-bucket/lectures/intro-to-ml.srt"
}

​

Output Schema

{
  "unit_type": "video_segment",
  "doc_type": "transcript",
  "text_content": "In this section, we explore supervised learning algorithms...",
  "screen_text": "SUPERVISED LEARNING\n- Regression\n- Classification",
  "title": "Intro to ML: Supervised Learning",
  "start_time": 120.5,
  "end_time": 245.3,
  "segment_index": 3,
  "thumbnail_url": "s3://mixpeek/ns_123/thumbnails/seg_3.jpg",
  "intfloat__multilingual_e5_large_instruct": [0.023, -0.041, 0.018, ...],
  "llm_summary": "Introduction to supervised learning covering regression and classification techniques"
}

​

Parameters

​

Video Segmentation Parameters

​

Segmentation Methods

​

PDF Extraction Parameters

​

Code Extraction Parameters

​

Feature Extraction Parameters

​

LLM Enrichment Parameters

​

Configuration Examples

{
  "feature_extractor": {
    "feature_extractor_name": "course_content_extractor",
    "version": "v1",
    "input_mappings": {
      "video": "payload.lecture_url",
      "srt": "payload.subtitle_url"
    },
    "field_passthrough": [
      { "source_path": "metadata.course_id" },
      { "source_path": "metadata.lesson_number" }
    ],
    "parameters": {
      "target_segment_duration_ms": 120000,
      "segmentation_method": "scene",
      "scene_detection_threshold": 0.3,
      "use_whisper_asr": true,
      "expand_to_granular_docs": true,
      "ocr_frames_per_segment": 3,
      "run_text_embedding": true,
      "run_code_embedding": true,
      "run_visual_embedding": false,
      "generate_thumbnails": true
    }
  }
}

​

Performance & Costs

​

Vector Indexes

​

Comparison with Other Extractors

​

Limitations

• Video length: Optimized for videos up to 4 hours. Longer videos may require segmentation.
• Transcription quality: Whisper ASR works best with clear audio; noisy lectures may have reduced accuracy.
• Code extraction: Requires valid ZIP archives; loose files not supported.
• Language support: Code embedding works with common languages; domain-specific DSLs have reduced accuracy.
• PDF complexity: Complex layouts with nested tables may have reduced extraction quality.
• Visual embeddings: Optional and add significant processing cost.

​

Related

• Feature Extractors Overview
• Text Extractor
• Image Extractor
• Document Graph Extractor
• Multimodal Extractor