ntnsndr/protocol-bicorder
1
0 Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: ntnsndr/protocol-bicorder:459015fe170ec8ab8fb3535a80b87ee0dcea5cd1
ntnsndr/protocol-bicorder:main
...
compare: ntnsndr/protocol-bicorder:161ba3b136496eb36ac72af9c9053ae8bd515b8d
ntnsndr/protocol-bicorder:main
2 Commits
459015fe17 ... 161ba3b136
Author SHA1 Message Date
Protocolbot 161ba3b136 fix: carry institutional→formal rename through analysis rename maps 
Nathan renamed the Design gradient 'institutional' → 'formal' (v1.4.0).
The analysis scripts carry historical rename maps that route old data
(elite/institutional) into the current bicorder.json gradient names, and
those maps still stopped at 'institutional' — so old readings would have
silently misaligned against the v1.4.0 structure.

- bicorder_classifier.py / export_model_for_js.py: add
  institutional_vs_vernacular → formal_vs_vernacular, keep elite→formal
- json_to_csv.py TERM_RENAMES: same two-step route
- convert_csv_to_json.py GRADIENT_MAPPINGS: elite_vs_vernacular now maps
  to 'formal'
- bicorder_classifier.py demo ratings: use the current dimension name
- analysis/README.md: example run name 1.3.0 → 1.4.0
 2026-09-23 14:27:51 -06:00
Protocolbot 6ae77a4f9b refactor: reorganize analysis data to support multiple runs 
Strategy: key runs by bicorder version (not date), promote shared inputs
to analysis/data/, and stamp every output with its bicorder_version so
a re-run on edited gradients is self-describing.

Data layout:
- Promote the shared protocol inputs out of the run directory:
    analysis/data/protocols_edited.csv  (411 cleaned protocols)
    analysis/data/protocols_raw.csv     (774 un-cleaned entries)
- Rename the v1.2.6 synthetic run:
    data/synthetic_20251116/ -> data/synthetic_1.2.6/
  so the bicorder version it was scored against is explicit (gradient
  structure changes between versions make date-based names ambiguous)

Provenance:
- bicorder_analyze.py now writes a 'bicorder_version' column into every
  output readings.csv, recording which gradient structure produced it

Scripts:
- Update the real code defaults that pointed at the old run path
  (bicorder_classifier.py, classify_readings.py, sync_readings.sh,
  compare_analyses.py) and refresh docstring/help examples
- Remove a stray committed __pycache__/.pyc

Docs: analysis/README.md documents the new layout + how to add a run;
WORKFLOW.md, TEST_COMMANDS.md, INTEGRATION_GUIDE.md paths updated.
 2026-09-23 14:23:17 -06:00
475 changed files with 100 additions and 73 deletions
Expand all files Collapse all files

No files matched your search

analysis/INTEGRATION_GUIDE.md
+1 -1
View File
@@ -22,7 +22,7 @@ previously integrated into the Bicorder web application to provide:
**Version-based compatibility**: The model included a `bicorder_version` field.
The classifier checked that versions matched. When bicorder.json structure changed:
1. The version number in bicorder.json was incremented
2. The model was retrained with `python3 scripts/export_model_for_js.py data/synthetic_20251116/readings.csv`
2. The model was retrained with `python3 scripts/export_model_for_js.py data/synthetic_1.2.6/readings.csv`
3. The new model had the updated version
## Files (research-only now)
analysis/README.md
+37 -20
View File
@@ -6,11 +6,28 @@ Scripts were created with the assistance of Claude Code. Data processing was don
## Datasets
Readings are organized under `data/<type>_<YYYYMMDD>/`, each self-contained with its own `readings.csv`, `analysis/`, and `json/` subdirectories:
Readings are organized under `data/<type>_<version>/`, each self-contained with its own `readings.csv`, `analysis/`, and `json/` subdirectories. The shared protocol inputs (the list of protocols themselves, before any diagnostic is run) live directly under `data/`:
- **`data/synthetic_20251116/`** — 411 protocols from synthetic LLM-generated readings (see detailed procedure below)
- **`data/protocols_edited.csv`** — the 411 cleaned protocol descriptors+descriptions (the input shared by every synthetic run)
- **`data/protocols_raw.csv`** — the 774 un-cleaned entries the chunking stage produced
Runs:
- **`data/synthetic_1.2.6/`** — 411 synthetic LLM-generated readings scored against bicorder v1.2.6 (see detailed procedure below). Renamed from `synthetic_20251116` to make the bicorder version it was generated against explicit, since the gradient structure changes between versions.
- **`data/manual_20260320/`** — manual readings collected at [git.medlab.host/ntnsndr/protocol-bicorder-data](https://git.medlab.host/ntnsndr/protocol-bicorder-data), continuously expanding
### Adding a new run (e.g. re-running on edited gradients)
1. Make a new run directory keyed by bicorder version, e.g. `data/synthetic_1.4.0/`.
2. Point `bicorder_batch.py` at the shared input and the new output:
```bash
python3 scripts/bicorder_batch.py data/protocols_edited.csv \
-o data/synthetic_1.4.0/readings.csv \
-b bicorder.json \
-m <model> -a "<analyst>" -s "<standpoint>"
```
3. The output `readings.csv` now records a `bicorder_version` column, so every run is self-describing about which gradient structure produced it. Run the downstream analysis (`multivariate_analysis.py`, `classify_readings.py`, etc.) against the new `readings.csv` with the new run directory as `--output`.
### Syncing the manual dataset
The manual dataset is kept current via a `.sync_source` config file and a one-command sync script:
@@ -25,7 +42,7 @@ Options:
```bash
scripts/sync_readings.sh data/manual_20260320 --min-coverage 0.8 # default
scripts/sync_readings.sh data/manual_20260320 --no-analysis # sync JSON only
scripts/sync_readings.sh data/manual_20260320 --training data/synthetic_20251116/readings.csv
scripts/sync_readings.sh data/manual_20260320 --training data/synthetic_1.2.6/readings.csv
```
### Handling shortform readings
@@ -59,7 +76,7 @@ context: "model running on ollama locally, accessed with llm on the command line
prompt: "Return csv-formatted data (with no markdown wrapper) that consists of a list of protocols discussed or referred to in the attached text. Protocols are defined extremely broadly as 'patterns of interaction,' and may be of a nontechnical nature. Protocols should be as specific as possible, such as 'Sacrament of Reconciliation' rather than 'Religious Protocols.' The first column should provide a brief descriptor of the protocol, and the second column should describe it in a substantial paragraph of 3-5 sentences, encapsulated in quotation marks to avoid breaking on commas. Be sure to paraphrase rather than quoting directly from the source text."
```
The result was a CSV-formatted list of protocols (`data/synthetic_20251116/protocols_raw.csv`, n=774 total protocols listed).
The result was a CSV-formatted list of protocols (`data/protocols_raw.csv`, n=774 total protocols listed).
### Dataset cleaning
@@ -74,7 +91,7 @@ The dataset was then manually reviewed. The review involved the following:
The cleaning process was carried out in a subjective manner, so some entries that meet the above criteria may remain in the dataset. The dataset also appears to include some LLM hallucinations---that is, protocols not in the texts---but the hallucinations are often acceptable examples and so some have been retained. Some degree of noise in the dataset was considered acceptable for the purposes of the study. Some degree of repetition, also, provides the dataset with a kind of control cases for evaluating the diagnostic process.
The result was a CSV-formatted list of protocols (`data/synthetic_20251116/protocols_edited.csv`, n=411).
The result was a CSV-formatted list of protocols (`data/protocols_edited.csv`, n=411).
### Initial diagnostic
@@ -83,24 +100,24 @@ This diagnostic used the file now at `bicorder_analyzed.json`, though the script
For each row in the dataset, and on each gradient, a series of scripts prompts the LLM to apply each gradient to the protocol. The outputs are then added to a CSV output file.
The result was a CSV-formatted list of protocols (`data/synthetic_20251116/readings.csv`, n=411).
The result was a CSV-formatted list of protocols (`data/synthetic_1.2.6/readings.csv`, n=411).
See detailed documentation of the scripts at `WORKFLOW.md`.
### Manual and alternate model audit
To test the output, a manual review of the first 10 protocols in the `data/synthetic_20251116/protocols_edited.csv` dataset was produced in the file `data/synthetic_20251116/readings_manual.csv`. (Alphabetization in this case seems a reasonable proxy for a random sample of protocols. It includes some partially overlapping protocols, as does the dataset as a whole.) Additionally, three models were tested on the same cases:
To test the output, a manual review of the first 10 protocols in the `data/protocols_edited.csv` dataset was produced in the file `data/synthetic_1.2.6/readings_manual.csv`. (Alphabetization in this case seems a reasonable proxy for a random sample of protocols. It includes some partially overlapping protocols, as does the dataset as a whole.) Additionally, three models were tested on the same cases:
```bash
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o data/synthetic_20251116/readings_mistral.csv -m mistral -a "Mistral" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.2.6/readings_mistral.csv -m mistral -a "Mistral" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
```
```bash
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o data/synthetic_20251116/readings_gpt-oss.csv -m gpt-oss -a "GPT-OSS" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.2.6/readings_gpt-oss.csv -m gpt-oss -a "GPT-OSS" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
```
```bash
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o data/synthetic_20251116/readings_gemma3-12b.csv -m gemma3:12b -a "Gemma3:12b" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.2.6/readings_gemma3-12b.csv -m gemma3:12b -a "Gemma3:12b" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
```
A Euclidean distance analysis (`python3 scripts/compare_analyses.py`) found that the `gpt-oss` model was closer to the manual example than the others. It was therefore selected to be the model used for conducting the bicorder diagnostic on the dataset.
@@ -112,13 +129,13 @@ Average Euclidean Distance:
3. readings_mistral.csv - Avg Distance: 13.33
```
Command used to produce `data/synthetic_20251116/readings.csv` (using the Ollama cloud service for the `gpt-oss` model):
Command used to produce `data/synthetic_1.2.6/readings.csv` (using the Ollama cloud service for the `gpt-oss` model):
```bash
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o data/synthetic_20251116/readings.csv -m gpt-oss:20b-cloud -a "GPT-OSS" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision"
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.2.6/readings.csv -m gpt-oss:20b-cloud -a "GPT-OSS" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision"
```
The result was a CSV-formatted list of protocols (`data/synthetic_20251116/readings.csv`, n=411).
The result was a CSV-formatted list of protocols (`data/synthetic_1.2.6/readings.csv`, n=411).
### Further analysis
@@ -126,7 +143,7 @@ The result was a CSV-formatted list of protocols (`data/synthetic_20251116/readi
Per-protocol values are meaningful for the bicorder because, despite varying levels of appropriateness, all of the gradients are structured as ranging from "hardness" to "softness"---with lower values associated with greater rigidity. The average value for a given protocol, therefore, provides a rough sense of the protocol's hardness.
Basic averages appear in `data/synthetic_20251116/readings-analysis.ods`.
Basic averages appear in `data/synthetic_1.2.6/readings-analysis.ods`.
#### Univariate analysis
@@ -136,7 +153,7 @@ First, a plot of average values for each protocol:
This reveals a linear distribution of values among the protocols, aside from exponential curves only at the extremes. Perhaps the most interesting finding is a skew toward the higher end of the scale, associated with softness. Even relatively hard, technical protocols appear to have significant soft characteristics.
The protocol value averages have a mean of 5.45 and a median of 5.48. In comparison to the midpoint of 5, the normalized midpoint deviation is 0.11. In comparison, the Pearson coefficient measures the skew at just -0.07, which means that the relative skew of the data is actually slightly downward. So the distribution of protocol values is very balanced but has a consistent upward deviation from the scale's baseline. (These calculations are in `data/synthetic_20251116/readings-analysis.odt[averages]`.)
The protocol value averages have a mean of 5.45 and a median of 5.48. In comparison to the midpoint of 5, the normalized midpoint deviation is 0.11. In comparison, the Pearson coefficient measures the skew at just -0.07, which means that the relative skew of the data is actually slightly downward. So the distribution of protocol values is very balanced but has a consistent upward deviation from the scale's baseline. (These calculations are in `data/synthetic_1.2.6/readings-analysis.odt[averages]`.)
Second, a plot of average values for each gradient (with gaps to indicate the three groupings of gradients):
@@ -165,21 +182,21 @@ Claude Code created a `multivariate_analysis.py` tool to conduct this analysis.
```bash
# Run all analyses (default)
python3 scripts/multivariate_analysis.py data/synthetic_20251116/readings.csv
python3 scripts/multivariate_analysis.py data/synthetic_1.2.6/readings.csv
# Run specific analyses only
python3 scripts/multivariate_analysis.py data/synthetic_20251116/readings.csv --analyses
python3 scripts/multivariate_analysis.py data/synthetic_1.2.6/readings.csv --analyses
clustering pca
```
Initial manual observations:
* The correlations generally seem predictable; for example, the strongest is between `Design_static_vs_malleable` and `Experience_predictable_vs_emergent`, which is not surprising
* The elite vs. vernacular distinction appears to be the most predictive gradient (`data/synthetic_20251116/analysis/plots/feature_importances.png`)
* The elite vs. vernacular distinction appears to be the most predictive gradient (`data/synthetic_1.2.6/analysis/plots/feature_importances.png`)
![Correlation heatmap](data/synthetic_20251116/analysis/plots/correlation_heatmap_full.png)
![Correlation heatmap](data/synthetic_1.2.6/analysis/plots/correlation_heatmap_full.png)
![Importance ranking](data/synthetic_20251116/analysis/plots/feature_importances.png)
![Importance ranking](data/synthetic_1.2.6/analysis/plots/feature_importances.png)
Claude's interpretation:
analysis/TEST_COMMANDS.md
+6 -6
View File
@@ -7,7 +7,7 @@ Run these tests in order to verify the refactored code works correctly.
Test that prompts are generated correctly with protocol context:
```bash
python3 scripts/bicorder_query.py data/synthetic_20251116/protocols_edited.csv 1 --dry-run | head -80
python3 scripts/bicorder_query.py data/protocols_edited.csv 1 --dry-run | head -80
```
**Expected result:**
@@ -21,7 +21,7 @@ python3 scripts/bicorder_query.py data/synthetic_20251116/protocols_edited.csv 1
Check that the analyze script still creates proper CSV structure:
```bash
python3 scripts/bicorder_analyze.py data/synthetic_20251116/protocols_edited.csv -o test_output.csv
python3 scripts/bicorder_analyze.py data/protocols_edited.csv -o test_output.csv
head -1 test_output.csv | tr ',' '\n' | grep -E "(explicit|precise|elite)" | head -5
```
@@ -76,7 +76,7 @@ llm logs list | grep -i bicorder
Test batch processing on rows 1-3:
```bash
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o test_batch_output.csv --start 1 --end 3 -m gpt-4o-mini
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o test_batch_output.csv --start 1 --end 3 -m gpt-4o-mini
```
**Expected result:**
@@ -106,7 +106,7 @@ with open('test_batch_output.csv') as f:
Test that model parameter works in dry run:
```bash
python3 scripts/bicorder_query.py data/synthetic_20251116/protocols_edited.csv 5 --dry-run -m mistral | head -50
python3 scripts/bicorder_query.py data/protocols_edited.csv 5 --dry-run -m mistral | head -50
```
**Expected result:**
@@ -129,11 +129,11 @@ Compare the new standalone prompts vs old system prompt approach:
```bash
# New approach - protocol context in each prompt
python3 scripts/bicorder_query.py data/synthetic_20251116/protocols_edited.csv 1 --dry-run | grep -A 5 "Analyze this protocol"
python3 scripts/bicorder_query.py data/protocols_edited.csv 1 --dry-run | grep -A 5 "Analyze this protocol"
# Old approach would have had protocol in system prompt only (no longer used)
# Verify that protocol context appears in EVERY gradient prompt
python3 scripts/bicorder_query.py data/synthetic_20251116/protocols_edited.csv 1 --dry-run | grep -c "Analyze this protocol"
python3 scripts/bicorder_query.py data/protocols_edited.csv 1 --dry-run | grep -c "Analyze this protocol"
```
**Expected result:**
analysis/WORKFLOW.md
+6 -6
View File
@@ -68,7 +68,7 @@ python3 scripts/json_to_csv.py data/manual_20260320/json/ \
### Process All Protocols with One Command
```bash
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv
```
This will:
@@ -81,13 +81,13 @@ This will:
```bash
# Process only rows 1-5 (useful for testing)
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv --start 1 --end 5
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv --start 1 --end 5
# Use specific LLM model
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv -m mistral
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv -m mistral
# Add analyst metadata
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv \
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv \
-a "Your Name" -s "Your analytical standpoint"
```
@@ -100,12 +100,12 @@ python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -
Create a CSV with empty gradient columns:
```bash
python3 scripts/bicorder_analyze.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv
python3 scripts/bicorder_analyze.py data/protocols_edited.csv -o analysis_output.csv
```
Optional: Add analyst metadata:
```bash
python3 scripts/bicorder_analyze.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv \
python3 scripts/bicorder_analyze.py data/protocols_edited.csv -o analysis_output.csv \
-a "Your Name" -s "Your analytical standpoint"
```
analysis/data/synthetic_20251116/protocols_edited.csv → analysis/data/protocols_edited.csv
View File
File renamed without changes.
Internal Server Error - Gitea
500 Internal Server Error

An error occurred:

An error occurred

Gitea Version: 28.0.0