docs: command flow for analysis runs in README; keep WORKFLOW in sync

- README: new 'Command flow' section — dependency-ordered pipeline (setup →
  batch generation → univariate → multivariate → LDA/cluster visualizations →
  classify → cross-version compare → JSON/model export) with actual output
  paths per step, using data/synthetic_1.4.0/ as the example
- README: document chart production — univariate_analysis --img publishes the
  img/ summary charts; visualize_clusters added to the flow
- README: fix -b path in 'Adding a new run' (bicorder.json lives at repo root)
- WORKFLOW: script list and any-readings-CSV command block updated to match
This commit is contained in:
Nathan Schneider committed 2026-10-02 08:33:08 -06:00
1 parent 8c44064196
commit 5a84408587
2 files changed
+116 -11

No files matched your search

+97 -6
View File
@@ -14,19 +14,19 @@ Readings are organized under `data/<type>_<version>/`, each self-contained with
Runs:
- **`data/synthetic_1.2.6/`** — 411 synthetic LLM-generated readings scored against bicorder v1.2.6 (see detailed procedure below). Renamed from `synthetic_20251116` to make the bicorder version it was generated against explicit, since the gradient structure changes between versions.
- **`data/synthetic_1.4.0/`** - Same as above, but scored on v1.4.0, which has some changes to the gradients.
- **`data/manual_20260320/`** — manual readings collected at [git.medlab.host/ntnsndr/protocol-bicorder-data](https://git.medlab.host/ntnsndr/protocol-bicorder-data), continuously expanding
### Adding a new run (e.g. re-running on edited gradients)
1. Make a new run directory keyed by bicorder version, e.g. `data/synthetic_1.4.0/`.
2. Point `bicorder_batch.py` at the shared input and the new output:
2. Point `bicorder_batch.py` at the shared input and the new output (`-b` defaults to `../bicorder.json` in the repo root):
```bash
python3 scripts/bicorder_batch.py data/protocols_edited.csv \
-o data/synthetic_1.4.0/readings.csv \
-b bicorder.json \
-o data/synthetic_1.4.0/readings.csv -b ../bicorder.json \
-m <model> -a "<analyst>" -s "<standpoint>"
```
3. The output `readings.csv` now records a `bicorder_version` column, so every run is self-describing about which gradient structure produced it. Run the downstream analysis (`multivariate_analysis.py`, `classify_readings.py`, etc.) against the new `readings.csv` with the new run directory as `--output`.
3. The output `readings.csv` now records a `bicorder_version` column, so every run is self-describing about which gradient structure produced it. Run the downstream analysis (`univariate_analysis.py`, `multivariate_analysis.py`, `classify_readings.py`, etc.) against the new `readings.csv` with the new run directory as `--output`. The analysis scripts are version-agnostic: `classify_readings.py` auto-matches training data to the recorded version, and `compare_analyses.py` aligns renamed gradients across runs.
### Syncing the manual dataset
@@ -54,6 +54,87 @@ Many manual readings use the shortform bicorder (9 key dimensions rather than al
---
## Command flow
The analysis pipeline for a run, using `data/synthetic_1.4.0/` as the example. Commands run from this directory with the virtualenv activated. The flow is version-agnostic: each `readings.csv` records its `bicorder_version`, and every script reads the gradient structure from the file itself (shared helpers in `scripts/bicorder_common.py`), so the same sequence applies to any run and any future version.
### 0. Setup (once)
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```
### 1. Generate the readings (LLM batch diagnostic)
Queries the LLM for every gradient of every protocol, one chat each. Interrupted runs can be continued by re-running with `--resume`, which skips completed work. (For the manual dataset, `scripts/sync_readings.sh data/manual_20260320` replaces this step.)
```bash
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.4.0/readings.csv \
-m <model> -a "<analyst>" -s "<standpoint>"
```
→ `data/synthetic_1.4.0/readings.csv` — 411 protocols × 23 gradients, plus provenance columns (`bicorder_version`, `analyst`, `standpoint`)
### 2. Univariate analysis (averages and distributions)
```bash
python3 scripts/univariate_analysis.py data/synthetic_1.4.0/readings.csv
# add --img to also publish the three summary charts into img/ (the charts linked in this README's analysis narrative)
python3 scripts/univariate_analysis.py data/synthetic_1.2.6/readings.csv --img
```
→ `analysis/plots/` (protocol averages, distribution histogram, gradient averages), `analysis/data/` (per-protocol and per-gradient tables), and `analysis/reports/univariate_summary.txt` (mean, median, midpoint deviation, skewness)
### 3. Multivariate analysis (clustering, PCA, correlation, feature importance)
```bash
python3 scripts/multivariate_analysis.py data/synthetic_1.4.0/readings.csv
```
→ `analysis/plots/`, `analysis/data/` (including `kmeans_clusters.csv`, **required by steps 4–5**), and `analysis/reports/analysis_summary.txt`. Use `--analyses clustering pca` to run a subset, and `--min-coverage 0.8` for sparse (shortform) datasets.
### 4. Cluster visualizations (LDA + overlaid dimensional reductions)
```bash
# Cluster separation histogram and projection onto the discriminant axis
python3 scripts/lda_visualization.py data/synthetic_1.4.0/readings.csv
# Clusters overlaid on PCA / t-SNE / UMAP scatter plots
python3 scripts/visualize_clusters.py data/synthetic_1.4.0/readings.csv
```
→ `analysis/plots/lda_cluster_separation.png` (plus separation statistics on the console) and `pca_2d_clustered.png` / `tsne_2d_clustered.png` / `umap_2d_clustered.png`
### 5. Classify all readings
```bash
python3 scripts/classify_readings.py data/manual_20260320/readings.csv
```
Most typically applied to manual/shortform readings, with a synthetic run as training data. The training CSV is auto-selected to match the readings' recorded bicorder version (falls back to the most recent run and warns when versions differ, aligning renamed gradients); `--training` overrides. → `analysis/classifications.csv` (cluster, confidence, completeness, recommended form)
### 6. Compare runs (including across bicorder versions)
```bash
python3 scripts/compare_analyses.py data/synthetic_1.4.0/readings.csv data/synthetic_1.2.6/readings.csv
```
Canonicalizes old gradient names and compares only shared dimensions, so runs from different versions align. Console report: Euclidean distance, RMSE, correlation; with several comparison files it ranks them. With no arguments it executes the legacy model-audit comparison (see *Manual and alternate model audit* below).
### 7. Per-protocol JSONs and model export (optional)
```bash
python3 scripts/convert_csv_to_json.py data/synthetic_1.4.0/readings.csv
python3 scripts/export_model_for_js.py data/synthetic_1.4.0/readings.csv
```
→ `json/` (one bicorder.json-spec reading per protocol); `bicorder_model.json` in this directory, read by `bicorder-app` at build time (requires step 3 first; see `INTEGRATION_GUIDE.md`)
---
## Purpose
This analyses has several purposes:
@@ -120,7 +201,7 @@ python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.2.6/readings_gemma3-12b.csv -m gemma3:12b -a "Gemma3:12b" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
```
A Euclidean distance analysis (`python3 scripts/compare_analyses.py`) found that the `gpt-oss` model was closer to the manual example than the others. It was therefore selected to be the model used for conducting the bicorder diagnostic on the dataset.
A Euclidean distance analysis (`python3 scripts/compare_analyses.py`, with paths given as arguments: a reference CSV followed by comparison CSVs) found that the `gpt-oss` model was closer to the manual example than the others. It was therefore selected to be the model used for conducting the bicorder diagnostic on the dataset.
```
Average Euclidean Distance:
@@ -137,16 +218,26 @@ python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.
The result was a CSV-formatted list of protocols (`data/synthetic_1.2.6/readings.csv`, n=411).
The same command, with changed version names, was used to produce the `synthetic_1.4.0` readings.
### Further analysis
#### Basic averages
For any run, `scripts/univariate_analysis.py` computes per-protocol and per-gradient averages, the summary statistics (mean, median, deviation from the midpoint, skewness), and the plots below, saving them to the run's `analysis/` directory:
```bash
python3 scripts/univariate_analysis.py data/synthetic_1.4.0/readings.csv
```
Per-protocol values are meaningful for the bicorder because, despite varying levels of appropriateness, all of the gradients are structured as ranging from "hardness" to "softness"---with lower values associated with greater rigidity. The average value for a given protocol, therefore, provides a rough sense of the protocol's hardness.
Basic averages appear in `data/synthetic_1.2.6/readings-analysis.ods`.
#### Univariate analysis
The charts linked below are the repo-level copies in `img/`, reproduced from a run with `python3 scripts/univariate_analysis.py <readings.csv> --img` (these were produced from the 1.2.6 dataset).
First, a plot of average values for each protocol:
![Protocol averages plot](img/protocol_averages.png)
@@ -178,7 +269,7 @@ Expectations:
* There are some gradients whose values are highly correlated. These might point to redundancies in the bicorder design.
* Some correlations might be revealing about connections in the characteristics of protocols, but these should be considered carefully as they may be the result of design or LLM interpretation.
Claude Code created a `multivariate_analysis.py` tool to conduct this analysis. Usage:
I created a `multivariate_analysis.py` tool to conduct this analysis. Usage:
```bash
# Run all analyses (default)