protocol-bicorder/analysis/WORKFLOW.md

# Protocol Bicorder Analysis Workflow

This directory contains scripts for analyzing protocols using the Protocol Bicorder framework with LLM assistance.

The scripts automatically draw the gradients from the current state of the [bicorder.json](`../bicorder.json`) file.

## Scripts

### Diagnostic data generation (LLM-based)

1. **scripts/bicorder_batch.py** - **[RECOMMENDED]** Process entire CSV with one command
2. **scripts/bicorder_analyze.py** - Prepares CSV with gradient columns
3. **scripts/bicorder_query.py** - Queries LLM for each gradient value and updates CSV (each query is a new chat)

### Manual / JSON-based readings

4. **scripts/json_to_csv.py** - Convert a directory of individual bicorder JSON reading files into a `readings.csv`
5. **scripts/sync_readings.sh** - Sync a readings dataset from a remote git repository, then regenerate CSV and run analysis (see below)

### Analysis

6. **scripts/multivariate_analysis.py** - Run clustering, PCA, correlation, and feature importance analysis on a readings CSV
7. **scripts/lda_visualization.py** - Generate LDA cluster separation plot and projection data
8. **scripts/classify_readings.py** - Apply the synthetic-trained LDA classifier to all readings; saves `analysis/classifications.csv` (training data is auto-matched to the input's recorded bicorder version)
9. **scripts/univariate_analysis.py** - Per-protocol and per-gradient averages, distributions, and summary stats (replaces the ad-hoc averages workflow)
10. **scripts/compare_analyses.py** - Compare readings CSVs to a reference (Euclidean distance, RMSE, correlation); canonicalizes renamed gradients so versions can be compared
11. **scripts/visualize_clusters.py** - Additional cluster visualizations
12. **scripts/export_model_for_js.py** - Export trained model to `bicorder_model.json` (research-only since v1.3.0 — the app integration was removed)

Version-agnostic helpers shared by these scripts live in **scripts/bicorder_common.py**: the historical gradient rename map, version detection (from the `bicorder_version`/`version` column, falling back to the `data/<type>_<version>/` directory convention), and training-CSV selection. When gradients are renamed in `../bicorder.json`, update `COLUMN_RENAMES` in that one module.

## Syncing a manual readings dataset

If the dataset has a `.sync_source` file (e.g., `data/manual_20260320/`), one command handles everything:

```bash
scripts/sync_readings.sh data/manual_20260320
```

This fetches new JSON files from the remote repo, regenerates `readings.csv`, runs multivariate analysis (with `--min-coverage 0.8` to handle shortform readings), generates the LDA visualization, and saves cluster classifications to `analysis/classifications.csv`.

## Running analysis on any readings CSV

```bash
# Full analysis pipeline
python3 scripts/multivariate_analysis.py data/manual_20260320/readings.csv \
  --min-coverage 0.8 \
  --analyses clustering pca correlation importance

# LDA visualization (cluster separation plot)
python3 scripts/lda_visualization.py data/manual_20260320/readings.csv

# Classify all readings (training data auto-matched to the dataset's bicorder version)
python3 scripts/classify_readings.py data/manual_20260320/readings.csv

# Univariate averages (protocol and gradient plots + summary stats)
python3 scripts/univariate_analysis.py data/manual_20260320/readings.csv
# ... add --img to also publish its three summary charts into analysis/img/ (linked from README)

# Cluster overlaid PCA/t-SNE/UMAP plots (after multivariate analysis)
python3 scripts/visualize_clusters.py data/manual_20260320/readings.csv

# Compare two runs, including across bicorder versions (renamed gradients are aligned)
python3 scripts/compare_analyses.py data/synthetic_1.4.0/readings.csv data/synthetic_1.2.6/readings.csv
```

Use `--min-coverage` (0.0–1.0) to drop dimension columns below the given coverage fraction before analysis. This is important for datasets with many shortform readings where most dimensions are sparsely filled. `classify_readings.py` still accepts an explicit `--training` CSV to override the automatic selection.

## Converting JSON reading files to CSV

If you have a directory of individual bicorder JSON reading files:

```bash
python3 scripts/json_to_csv.py data/manual_20260320/json/ \
  -o data/manual_20260320/readings.csv
```

---

## Quick Start (Recommended, LLM-based)

### Process All Protocols with One Command

```bash
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv
```

This will:
1. Create the analysis CSV with gradient columns
2. For each protocol row, query all gradients (each query is a new chat with full protocol context)
3. Update the CSV automatically with the results
4. Show progress and summary

### Common Options

```bash
# Process only rows 1-5 (useful for testing)
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv --start 1 --end 5

# Use specific LLM model
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv -m mistral

# Add analyst metadata
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv \
  -a "Your Name" -s "Your analytical standpoint"
```

---

## Manual Workflow (Advanced)

### Step 1: Prepare the Analysis CSV

Create a CSV with empty gradient columns:

```bash
python3 scripts/bicorder_analyze.py data/protocols_edited.csv -o analysis_output.csv
```

Optional: Add analyst metadata:
```bash
python3 scripts/bicorder_analyze.py data/protocols_edited.csv -o analysis_output.csv \
  -a "Your Name" -s "Your analytical standpoint"
```

### Step 2: Query Gradients for a Protocol Row

Query all gradients for a specific protocol:

```bash
python3 scripts/bicorder_query.py analysis_output.csv 1
```

- Replace `1` with the row number you want to analyze
- Each gradient is queried in a new chat with full protocol context
- Each response is automatically parsed and written to the CSV
- Progress is shown for each gradient

Optional: Specify a model:
```bash
python3 scripts/bicorder_query.py analysis_output.csv 1 -m mistral
```

### Step 3: Repeat for All Protocols

For each protocol in your CSV:

```bash
python3 scripts/bicorder_query.py analysis_output.csv 1
python3 scripts/bicorder_query.py analysis_output.csv 2
python3 scripts/bicorder_query.py analysis_output.csv 3
# ... and so on

# OR: Use scripts/bicorder_batch.py to automate all of this!
```

## Architecture

### How It Works

Each gradient query is sent to the LLM as a **new, independent chat**. Every query includes:
- The protocol descriptor (name)
- The protocol description
- The gradient definition (left term, right term, and their descriptions)
- Instructions to rate 1-9

This approach:
- **Simplifies the code** - No conversation state management
- **Prevents bias** - Each evaluation is independent, not influenced by previous responses
- **Enables parallelization** - Queries could theoretically run concurrently
- **Makes debugging easier** - Each query/response pair is self-contained

## Tips

### Dry Run Mode

Test prompts without calling the LLM:

```bash
python3 scripts/bicorder_query.py analysis_output.csv 1 --dry-run
```

This shows you exactly what prompt will be sent for each gradient, including the full protocol context.

### Check Your Progress

View completed values:

```bash
python3 -c "
import csv
with open('analysis_output.csv') as f:
    reader = csv.DictReader(f)
    for i, row in enumerate(reader, 1):
        empty = sum(1 for k, v in row.items() if 'vs' in k and not v)
        print(f'Row {i}: {empty}/23 gradients empty')
"
```

### Batch Processing

Use the `scripts/bicorder_batch.py` script (see Quick Start section above) for processing multiple protocols.