protocol-bicorder/analysis/INTEGRATION_GUIDE.md

Bicorder Classifier — Research Notes

Status: removed from the tool (v1.3.0). The formal/informal (bureaucratic↔relational) LDA analysis was removed from the bicorder itself in v1.3.0. The cluster classification survives as research only — the scripts in this directory can still train and apply the model to datasets, but the web app and ascii_bicorder.py no longer consume it. This document is retained as a historical record of how the integration worked and how to reproduce the research analysis.

Overview

The analysis directory contains a cluster classification system that was previously integrated into the Bicorder web application to provide:

1. Real-time cluster prediction as users filled out diagnostics
2. Smart form selection (short vs. long form based on classification confidence)
3. Visual feedback showing protocol family positioning

Original Design Philosophy

Version-based compatibility: The model included a bicorder_version field. The classifier checked that versions matched. When bicorder.json structure changed:

1. The version number in bicorder.json was incremented
2. The model was retrained with python3 scripts/export_model_for_js.py data/synthetic_20251116/readings.csv
3. The new model had the updated version

Files (research-only now)

• bicorder_model.json - Trained model parameters (~5KB), trained on the synthetic dataset (bicorder v1.2.6 structure — stale relative to v1.3.0; retrain before applying to new readings)
• scripts/bicorder_classifier.py - Python classifier (used by classify_readings.py)
• scripts/export_model_for_js.py - Retrain and export the model to JSON
• scripts/classify_readings.py - Apply the classifier to a readings CSV

Reproducing the research analysis

# Retrain the model on a (new) synthetic dataset
python3 scripts/export_model_for_js.py data/<dataset>/readings.csv

# Classify a dataset's readings
python3 scripts/classify_readings.py data/<dataset>/readings.csv --training data/<dataset>/readings.csv

The classifier predicts which of two protocol families a reading belongs to:

• Cluster 1: Relational/Cultural — community-based, emergent, voluntary protocols
• Cluster 2: Institutional/Bureaucratic — formal, top-down, externally enforced protocols

See analysis/README.md for the full multivariate analysis these clusters came from.

Historical integration patterns

The removed web-app integration supported progressive classification display, smart form selection (suggesting the long form when classification confidence was low), short-form optimization around the most discriminative dimensions, and readiness checks. The Python classifier API remains:

• predict(ratings, options) → cluster, clusterName, confidence, completeness, recommendedForm (detailed mode adds ldaScore, distanceToBoundary, dimension counts)
• explain_classification(ratings) → human-readable explanation
• get_key_dimensions() → the shortform/key dimensions from bicorder.json
• assess_short_form_readiness(ratings) (TS only, removed) — the Python recommended_form field remains

The shortform gradients themselves are defined in bicorder.json (shortform: true), derived from the original feature-importance analysis — that part of the research lives on in the tool.

Why it was removed

• The LDA sign convention was inverted in ascii_bicorder.py (never caught there because a term-rename also silently disabled the calculation), while the web app had been separately fixed — two divergent implementations.
• Compressing a two-family classification into a 1–9 gradient was semantically awkward and produced recurring bugs (see commit fd556d9).
• The version-mismatch handling differed between implementations (Python skipped; TypeScript continued with a stale model).
• The two-families finding is a research result, not a diagnostic — it belongs in analysis, not in the instrument itself.

The form-recommendation feature (suggesting long form when classification confidence was low) was also removed. Shortform/longform selection is now entirely the analyst's choice.