Deep Dive · Script Architecture

14 Scripts, One Core Library

cm_core.py as foundation. How ID generation, duplicate detection and temporal metadata work.

cm_core.py — The Foundation

All 13 other scripts import cm_core. It provides: workspace management, ID generation, index I/O, history logging, tag normalization, duplicate detection and the temporal metadata factory.

# Core functions in cm_core.py
get_workspace(project, base) → Path
generate_id(type, ws, index) → str # arch-001, dec-002...
load/save_index(ws) → dict
find_similar_nodes(ws, title, content) → list
make_temporal(source_date, ...) → dict
normalize_tags("api, REST") → ["api","rest"]
content_hash(text) → str # MD5[:12]

ID Generation — The Batch Bug

generate_id() normally reads the index from disk. Problem: cm_ingest with --split creates 10 nodes in a loop. The disk index is written only at the end → all IDs would be identical. Fix: optional in-memory index parameter.

# BEFORE (Bug): All IDs = api-001
for nd in nodes_to_create:
node_id = generate_id(type, ws) # reads disk → always same!

# AFTER (Fix): api-001, api-002, api-003...
for nd in nodes_to_create:
node_id = generate_id(type, ws, index) # in-memory!

Duplicate Detection — Word Overlap, Not Embeddings

find_similar_nodes compares title words and tags via set operations. Threshold 0.6 = 60% word overlap. No ML, no API calls — just sets. Fast, transparent, deterministic.

# Title similarity
title_words = {"postgresql", "as", "database"}
existing = {"postgresql", "instead", "mongodb"}
overlap = 1/3 = 0.33 → below 0.6 → not a duplicate

# Tag overlap (≥3 shared tags → similar)
new_tags = {"database", "postgresql", "sql", "migration"}
existing = {"database", "postgresql", "sql"}
overlap = 3 → ≥ 3 → ⚠️ duplicate warning

Temporal Metadata — make_temporal()

Each node can carry source_date, valid_from, valid_until and temporal_confidence. The factory function creates the dict, cm_add.py stores it in the index.

# cm_core.py
def make_temporal(source_date="", ...):
  return {"source_date": source_date, ...}

# cm_add.py — new fields alongside relevance
index["nodes"][node_id] = {
  "confidence": args.confidence, # NEW
  "temporal": temporal, # NEW
}