refactor: reorganize analysis data to support multiple runs
Strategy: key runs by bicorder version (not date), promote shared inputs
to analysis/data/, and stamp every output with its bicorder_version so
a re-run on edited gradients is self-describing.
Data layout:
- Promote the shared protocol inputs out of the run directory:
analysis/data/protocols_edited.csv (411 cleaned protocols)
analysis/data/protocols_raw.csv (774 un-cleaned entries)
- Rename the v1.2.6 synthetic run:
data/synthetic_20251116/ -> data/synthetic_1.2.6/
so the bicorder version it was scored against is explicit (gradient
structure changes between versions make date-based names ambiguous)
Provenance:
- bicorder_analyze.py now writes a 'bicorder_version' column into every
output readings.csv, recording which gradient structure produced it
Scripts:
- Update the real code defaults that pointed at the old run path
(bicorder_classifier.py, classify_readings.py, sync_readings.sh,
compare_analyses.py) and refresh docstring/help examples
- Remove a stray committed __pycache__/.pyc
Docs: analysis/README.md documents the new layout + how to add a run;
WORKFLOW.md, TEST_COMMANDS.md, INTEGRATION_GUIDE.md paths updated.
This commit is contained in:
Protocolbot committed
1 parent
459015fe17
commit
6ae77a4f9b
474 files changed
+90
-66
No files matched your search
@@ -22,7 +22,7 @@ previously integrated into the Bicorder web application to provide:
|
||||
**Version-based compatibility**: The model included a `bicorder_version` field.
|
||||
The classifier checked that versions matched. When bicorder.json structure changed:
|
||||
1. The version number in bicorder.json was incremented
|
||||
2. The model was retrained with `python3 scripts/export_model_for_js.py data/synthetic_20251116/readings.csv`
|
||||
2. The model was retrained with `python3 scripts/export_model_for_js.py data/synthetic_1.2.6/readings.csv`
|
||||
3. The new model had the updated version
|
||||
|
||||
## Files (research-only now)
|
||||
|
||||
+37
-20
@@ -6,11 +6,28 @@ Scripts were created with the assistance of Claude Code. Data processing was don
|
||||
|
||||
## Datasets
|
||||
|
||||
Readings are organized under `data/<type>_<YYYYMMDD>/`, each self-contained with its own `readings.csv`, `analysis/`, and `json/` subdirectories:
|
||||
Readings are organized under `data/<type>_<version>/`, each self-contained with its own `readings.csv`, `analysis/`, and `json/` subdirectories. The shared protocol inputs (the list of protocols themselves, before any diagnostic is run) live directly under `data/`:
|
||||
|
||||
- **`data/synthetic_20251116/`** — 411 protocols from synthetic LLM-generated readings (see detailed procedure below)
|
||||
- **`data/protocols_edited.csv`** — the 411 cleaned protocol descriptors+descriptions (the input shared by every synthetic run)
|
||||
- **`data/protocols_raw.csv`** — the 774 un-cleaned entries the chunking stage produced
|
||||
|
||||
Runs:
|
||||
|
||||
- **`data/synthetic_1.2.6/`** — 411 synthetic LLM-generated readings scored against bicorder v1.2.6 (see detailed procedure below). Renamed from `synthetic_20251116` to make the bicorder version it was generated against explicit, since the gradient structure changes between versions.
|
||||
- **`data/manual_20260320/`** — manual readings collected at [git.medlab.host/ntnsndr/protocol-bicorder-data](https://git.medlab.host/ntnsndr/protocol-bicorder-data), continuously expanding
|
||||
|
||||
### Adding a new run (e.g. re-running on edited gradients)
|
||||
|
||||
1. Make a new run directory keyed by bicorder version, e.g. `data/synthetic_1.3.0/`.
|
||||
2. Point `bicorder_batch.py` at the shared input and the new output:
|
||||
```bash
|
||||
python3 scripts/bicorder_batch.py data/protocols_edited.csv \
|
||||
-o data/synthetic_1.3.0/readings.csv \
|
||||
-b bicorder.json \
|
||||
-m <model> -a "<analyst>" -s "<standpoint>"
|
||||
```
|
||||
3. The output `readings.csv` now records a `bicorder_version` column, so every run is self-describing about which gradient structure produced it. Run the downstream analysis (`multivariate_analysis.py`, `classify_readings.py`, etc.) against the new `readings.csv` with the new run directory as `--output`.
|
||||
|
||||
### Syncing the manual dataset
|
||||
|
||||
The manual dataset is kept current via a `.sync_source` config file and a one-command sync script:
|
||||
@@ -25,7 +42,7 @@ Options:
|
||||
```bash
|
||||
scripts/sync_readings.sh data/manual_20260320 --min-coverage 0.8 # default
|
||||
scripts/sync_readings.sh data/manual_20260320 --no-analysis # sync JSON only
|
||||
scripts/sync_readings.sh data/manual_20260320 --training data/synthetic_20251116/readings.csv
|
||||
scripts/sync_readings.sh data/manual_20260320 --training data/synthetic_1.2.6/readings.csv
|
||||
```
|
||||
|
||||
### Handling shortform readings
|
||||
@@ -59,7 +76,7 @@ context: "model running on ollama locally, accessed with llm on the command line
|
||||
prompt: "Return csv-formatted data (with no markdown wrapper) that consists of a list of protocols discussed or referred to in the attached text. Protocols are defined extremely broadly as 'patterns of interaction,' and may be of a nontechnical nature. Protocols should be as specific as possible, such as 'Sacrament of Reconciliation' rather than 'Religious Protocols.' The first column should provide a brief descriptor of the protocol, and the second column should describe it in a substantial paragraph of 3-5 sentences, encapsulated in quotation marks to avoid breaking on commas. Be sure to paraphrase rather than quoting directly from the source text."
|
||||
```
|
||||
|
||||
The result was a CSV-formatted list of protocols (`data/synthetic_20251116/protocols_raw.csv`, n=774 total protocols listed).
|
||||
The result was a CSV-formatted list of protocols (`data/protocols_raw.csv`, n=774 total protocols listed).
|
||||
|
||||
### Dataset cleaning
|
||||
|
||||
@@ -74,7 +91,7 @@ The dataset was then manually reviewed. The review involved the following:
|
||||
|
||||
The cleaning process was carried out in a subjective manner, so some entries that meet the above criteria may remain in the dataset. The dataset also appears to include some LLM hallucinations---that is, protocols not in the texts---but the hallucinations are often acceptable examples and so some have been retained. Some degree of noise in the dataset was considered acceptable for the purposes of the study. Some degree of repetition, also, provides the dataset with a kind of control cases for evaluating the diagnostic process.
|
||||
|
||||
The result was a CSV-formatted list of protocols (`data/synthetic_20251116/protocols_edited.csv`, n=411).
|
||||
The result was a CSV-formatted list of protocols (`data/protocols_edited.csv`, n=411).
|
||||
|
||||
|
||||
### Initial diagnostic
|
||||
@@ -83,24 +100,24 @@ This diagnostic used the file now at `bicorder_analyzed.json`, though the script
|
||||
|
||||
For each row in the dataset, and on each gradient, a series of scripts prompts the LLM to apply each gradient to the protocol. The outputs are then added to a CSV output file.
|
||||
|
||||
The result was a CSV-formatted list of protocols (`data/synthetic_20251116/readings.csv`, n=411).
|
||||
The result was a CSV-formatted list of protocols (`data/synthetic_1.2.6/readings.csv`, n=411).
|
||||
|
||||
See detailed documentation of the scripts at `WORKFLOW.md`.
|
||||
|
||||
### Manual and alternate model audit
|
||||
|
||||
To test the output, a manual review of the first 10 protocols in the `data/synthetic_20251116/protocols_edited.csv` dataset was produced in the file `data/synthetic_20251116/readings_manual.csv`. (Alphabetization in this case seems a reasonable proxy for a random sample of protocols. It includes some partially overlapping protocols, as does the dataset as a whole.) Additionally, three models were tested on the same cases:
|
||||
To test the output, a manual review of the first 10 protocols in the `data/protocols_edited.csv` dataset was produced in the file `data/synthetic_1.2.6/readings_manual.csv`. (Alphabetization in this case seems a reasonable proxy for a random sample of protocols. It includes some partially overlapping protocols, as does the dataset as a whole.) Additionally, three models were tested on the same cases:
|
||||
|
||||
```bash
|
||||
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o data/synthetic_20251116/readings_mistral.csv -m mistral -a "Mistral" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
|
||||
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.2.6/readings_mistral.csv -m mistral -a "Mistral" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
|
||||
```
|
||||
|
||||
```bash
|
||||
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o data/synthetic_20251116/readings_gpt-oss.csv -m gpt-oss -a "GPT-OSS" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
|
||||
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.2.6/readings_gpt-oss.csv -m gpt-oss -a "GPT-OSS" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
|
||||
```
|
||||
|
||||
```bash
|
||||
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o data/synthetic_20251116/readings_gemma3-12b.csv -m gemma3:12b -a "Gemma3:12b" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
|
||||
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.2.6/readings_gemma3-12b.csv -m gemma3:12b -a "Gemma3:12b" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision" --start 1 --end 10
|
||||
```
|
||||
|
||||
A Euclidean distance analysis (`python3 scripts/compare_analyses.py`) found that the `gpt-oss` model was closer to the manual example than the others. It was therefore selected to be the model used for conducting the bicorder diagnostic on the dataset.
|
||||
@@ -112,13 +129,13 @@ Average Euclidean Distance:
|
||||
3. readings_mistral.csv - Avg Distance: 13.33
|
||||
```
|
||||
|
||||
Command used to produce `data/synthetic_20251116/readings.csv` (using the Ollama cloud service for the `gpt-oss` model):
|
||||
Command used to produce `data/synthetic_1.2.6/readings.csv` (using the Ollama cloud service for the `gpt-oss` model):
|
||||
|
||||
```bash
|
||||
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o data/synthetic_20251116/readings.csv -m gpt-oss:20b-cloud -a "GPT-OSS" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision"
|
||||
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o data/synthetic_1.2.6/readings.csv -m gpt-oss:20b-cloud -a "GPT-OSS" -s "A careful ethnographer and outsider aspiring to achieve a neutral stance and a high degree of precision"
|
||||
```
|
||||
|
||||
The result was a CSV-formatted list of protocols (`data/synthetic_20251116/readings.csv`, n=411).
|
||||
The result was a CSV-formatted list of protocols (`data/synthetic_1.2.6/readings.csv`, n=411).
|
||||
|
||||
### Further analysis
|
||||
|
||||
@@ -126,7 +143,7 @@ The result was a CSV-formatted list of protocols (`data/synthetic_20251116/readi
|
||||
|
||||
Per-protocol values are meaningful for the bicorder because, despite varying levels of appropriateness, all of the gradients are structured as ranging from "hardness" to "softness"---with lower values associated with greater rigidity. The average value for a given protocol, therefore, provides a rough sense of the protocol's hardness.
|
||||
|
||||
Basic averages appear in `data/synthetic_20251116/readings-analysis.ods`.
|
||||
Basic averages appear in `data/synthetic_1.2.6/readings-analysis.ods`.
|
||||
|
||||
#### Univariate analysis
|
||||
|
||||
@@ -136,7 +153,7 @@ First, a plot of average values for each protocol:
|
||||
|
||||
This reveals a linear distribution of values among the protocols, aside from exponential curves only at the extremes. Perhaps the most interesting finding is a skew toward the higher end of the scale, associated with softness. Even relatively hard, technical protocols appear to have significant soft characteristics.
|
||||
|
||||
The protocol value averages have a mean of 5.45 and a median of 5.48. In comparison to the midpoint of 5, the normalized midpoint deviation is 0.11. In comparison, the Pearson coefficient measures the skew at just -0.07, which means that the relative skew of the data is actually slightly downward. So the distribution of protocol values is very balanced but has a consistent upward deviation from the scale's baseline. (These calculations are in `data/synthetic_20251116/readings-analysis.odt[averages]`.)
|
||||
The protocol value averages have a mean of 5.45 and a median of 5.48. In comparison to the midpoint of 5, the normalized midpoint deviation is 0.11. In comparison, the Pearson coefficient measures the skew at just -0.07, which means that the relative skew of the data is actually slightly downward. So the distribution of protocol values is very balanced but has a consistent upward deviation from the scale's baseline. (These calculations are in `data/synthetic_1.2.6/readings-analysis.odt[averages]`.)
|
||||
|
||||
Second, a plot of average values for each gradient (with gaps to indicate the three groupings of gradients):
|
||||
|
||||
@@ -165,21 +182,21 @@ Claude Code created a `multivariate_analysis.py` tool to conduct this analysis.
|
||||
|
||||
```bash
|
||||
# Run all analyses (default)
|
||||
python3 scripts/multivariate_analysis.py data/synthetic_20251116/readings.csv
|
||||
python3 scripts/multivariate_analysis.py data/synthetic_1.2.6/readings.csv
|
||||
|
||||
# Run specific analyses only
|
||||
python3 scripts/multivariate_analysis.py data/synthetic_20251116/readings.csv --analyses
|
||||
python3 scripts/multivariate_analysis.py data/synthetic_1.2.6/readings.csv --analyses
|
||||
clustering pca
|
||||
```
|
||||
|
||||
Initial manual observations:
|
||||
|
||||
* The correlations generally seem predictable; for example, the strongest is between `Design_static_vs_malleable` and `Experience_predictable_vs_emergent`, which is not surprising
|
||||
* The elite vs. vernacular distinction appears to be the most predictive gradient (`data/synthetic_20251116/analysis/plots/feature_importances.png`)
|
||||
* The elite vs. vernacular distinction appears to be the most predictive gradient (`data/synthetic_1.2.6/analysis/plots/feature_importances.png`)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Claude's interpretation:
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ Run these tests in order to verify the refactored code works correctly.
|
||||
Test that prompts are generated correctly with protocol context:
|
||||
|
||||
```bash
|
||||
python3 scripts/bicorder_query.py data/synthetic_20251116/protocols_edited.csv 1 --dry-run | head -80
|
||||
python3 scripts/bicorder_query.py data/protocols_edited.csv 1 --dry-run | head -80
|
||||
```
|
||||
|
||||
**Expected result:**
|
||||
@@ -21,7 +21,7 @@ python3 scripts/bicorder_query.py data/synthetic_20251116/protocols_edited.csv 1
|
||||
Check that the analyze script still creates proper CSV structure:
|
||||
|
||||
```bash
|
||||
python3 scripts/bicorder_analyze.py data/synthetic_20251116/protocols_edited.csv -o test_output.csv
|
||||
python3 scripts/bicorder_analyze.py data/protocols_edited.csv -o test_output.csv
|
||||
head -1 test_output.csv | tr ',' '\n' | grep -E "(explicit|precise|elite)" | head -5
|
||||
```
|
||||
|
||||
@@ -76,7 +76,7 @@ llm logs list | grep -i bicorder
|
||||
Test batch processing on rows 1-3:
|
||||
|
||||
```bash
|
||||
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o test_batch_output.csv --start 1 --end 3 -m gpt-4o-mini
|
||||
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o test_batch_output.csv --start 1 --end 3 -m gpt-4o-mini
|
||||
```
|
||||
|
||||
**Expected result:**
|
||||
@@ -106,7 +106,7 @@ with open('test_batch_output.csv') as f:
|
||||
Test that model parameter works in dry run:
|
||||
|
||||
```bash
|
||||
python3 scripts/bicorder_query.py data/synthetic_20251116/protocols_edited.csv 5 --dry-run -m mistral | head -50
|
||||
python3 scripts/bicorder_query.py data/protocols_edited.csv 5 --dry-run -m mistral | head -50
|
||||
```
|
||||
|
||||
**Expected result:**
|
||||
@@ -129,11 +129,11 @@ Compare the new standalone prompts vs old system prompt approach:
|
||||
|
||||
```bash
|
||||
# New approach - protocol context in each prompt
|
||||
python3 scripts/bicorder_query.py data/synthetic_20251116/protocols_edited.csv 1 --dry-run | grep -A 5 "Analyze this protocol"
|
||||
python3 scripts/bicorder_query.py data/protocols_edited.csv 1 --dry-run | grep -A 5 "Analyze this protocol"
|
||||
|
||||
# Old approach would have had protocol in system prompt only (no longer used)
|
||||
# Verify that protocol context appears in EVERY gradient prompt
|
||||
python3 scripts/bicorder_query.py data/synthetic_20251116/protocols_edited.csv 1 --dry-run | grep -c "Analyze this protocol"
|
||||
python3 scripts/bicorder_query.py data/protocols_edited.csv 1 --dry-run | grep -c "Analyze this protocol"
|
||||
```
|
||||
|
||||
**Expected result:**
|
||||
|
||||
@@ -68,7 +68,7 @@ python3 scripts/json_to_csv.py data/manual_20260320/json/ \
|
||||
### Process All Protocols with One Command
|
||||
|
||||
```bash
|
||||
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv
|
||||
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv
|
||||
```
|
||||
|
||||
This will:
|
||||
@@ -81,13 +81,13 @@ This will:
|
||||
|
||||
```bash
|
||||
# Process only rows 1-5 (useful for testing)
|
||||
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv --start 1 --end 5
|
||||
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv --start 1 --end 5
|
||||
|
||||
# Use specific LLM model
|
||||
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv -m mistral
|
||||
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv -m mistral
|
||||
|
||||
# Add analyst metadata
|
||||
python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv \
|
||||
python3 scripts/bicorder_batch.py data/protocols_edited.csv -o analysis_output.csv \
|
||||
-a "Your Name" -s "Your analytical standpoint"
|
||||
```
|
||||
|
||||
@@ -100,12 +100,12 @@ python3 scripts/bicorder_batch.py data/synthetic_20251116/protocols_edited.csv -
|
||||
Create a CSV with empty gradient columns:
|
||||
|
||||
```bash
|
||||
python3 scripts/bicorder_analyze.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv
|
||||
python3 scripts/bicorder_analyze.py data/protocols_edited.csv -o analysis_output.csv
|
||||
```
|
||||
|
||||
Optional: Add analyst metadata:
|
||||
```bash
|
||||
python3 scripts/bicorder_analyze.py data/synthetic_20251116/protocols_edited.csv -o analysis_output.csv \
|
||||
python3 scripts/bicorder_analyze.py data/protocols_edited.csv -o analysis_output.csv \
|
||||
-a "Your Name" -s "Your analytical standpoint"
|
||||
```
|
||||
|
||||
|
||||
File renamed without changes.
|
500 Internal Server Error
An error occurred: An error occurred Gitea Version: 28.0.0 |