Deep Dive

Ingest Pipeline

From upload to node: type detection, extractor dispatch, splitting and the batch ID bug.

01
Extractor Dispatch

detect_file_type() maps extensions to types. EXTRACTORS dict maps types to functions. Each extractor returns a dict with source, type, full_text and optional split data.

".pdf" → "pdf" → extract_pdf
".docx" → "docx" → extract_docx
".xlsx" → "xlsx" → extract_xlsx
02
Splitting Logic

_has_parts() checks if extracted data has enough parts. Sections needs 2+ headings, pages needs 2+ pages. Otherwise fallback to single. Max 20 nodes per split.

if split == "sections" and len(sections) > 1:
  → One node per heading
else: → fallback: single
03
Batch ID & Duplicate Check

Per split node: duplicate check, then generate_id with in-memory index (bug fix!), then save. Index written to disk only at the end.

for nd in nodes:
  similar = find_similar(ws, title, content)
  node_id = generate_id(type, ws, index) # in-memory!
  save_node(ws, node_id, content)
  index["nodes"][node_id] = {...}
save_index(ws, index) # disk write at end