Reorganize directory, add manual dataset and sync tooling

- Move all scripts to scripts/, web assets to web/, analysis results
  into self-contained data/readings/<type>_<YYYYMMDD>/ directories
- Add data/readings/manual_20260320/ with 32 JSON readings from
  git.medlab.host/ntnsndr/protocol-bicorder-data
- Add scripts/json_to_csv.py to convert bicorder JSON files to CSV
- Add scripts/sync_readings.sh for one-command sync + re-analysis of
  any dataset backed by a .sync_source config file
- Add scripts/classify_readings.py to apply the LDA classifier to all
  readings and save per-reading cluster assignments
- Add --min-coverage flag to multivariate_analysis.py for sparse/shortform
  datasets; also applies in lda_visualization.py
- Fix lda_visualization.py NaN handling and 0-d array annotation bug
- Update README.md and WORKFLOW.md to document datasets, sync workflow,
  shortform handling, and new scripts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Nathan Schneider
2026-03-20 17:35:13 -06:00
parent 0c794dddae
commit 897c30406b
545 changed files with 10715 additions and 718 deletions

View File

@@ -0,0 +1,33 @@
Descriptor,Description,analyst,standpoint,timestamp,shortform,version,Design_explicit_vs_implicit,Design_precise_vs_interpretive,Design_institutional_vs_vernacular,Design_documenting_vs_enabling,Design_static_vs_malleable,Design_technical_vs_social,Design_universal_vs_particular,Design_durable_vs_ephemeral,Entanglement_macro_vs_micro,Entanglement_sovereign_vs_subsidiary,Entanglement_self-enforcing_vs_enforced,Entanglement_abstract_vs_embodied,Entanglement_obligatory_vs_voluntary,Entanglement_flocking_vs_swarming,Entanglement_defensible_vs_exposed,Entanglement_monopolistic_vs_pluralistic,Experience_sufficient_vs_limited,Experience_crystallized_vs_contested,Experience_trust-evading_vs_trust-inducing,Experience_predictable_vs_emergent,Experience_exclusion_vs_inclusion,Experience_restraining_vs_liberating,Experience_dead_vs_alive
Coffee shop as office,,ntnsndr,A person using a coffee shop as an office,2025-11-25T18:07:21.486Z,True,1.1.0,,7,7,,3,,,,,4,3,,8,4,,,,,,,3,,
Street crossing in Buenos Aires,,Venkat,Tourist,2025-12-03T17:16:54.751Z,False,1.2.1,8,6,6,9,6,8,8,2,8,7,3,7,7,2,5,6,,6,6,6,7,4,8
qPCR,,Fotis Tsiroukis,Ethnography of Scientific Practice,2025-12-12T00:00:47.807Z,False,1.2.1,4,8,6,6,4,3,9,2,5,4,4,5,3,6,4,8,,4,5,4,2,7,9
High wind preemptive blackouts,,ntnsndr,Power consumer who lost power at office and home during high winds,2025-12-18T12:21:09.551Z,False,1.2.1,2,3,1,6,7,3,7,8,3,8,9,8,3,1,2,1,,7,7,6,8,5,9
Bedtime routine with kids,,ntnsndr,Parental participant,2026-01-05T04:30:04.916Z,False,1.2.3,2,4,9,9,3,9,7,3,9,8,8,9,1,2,,8,1,6,7,1,,8,9
Quarantine of a sick family member,,ntnsndr,Family member,2026-01-06T04:37:20.251Z,False,1.2.3,8,8,7,9,9,9,5,7,8,3,2,9,8,,,8,2,7,8,2,1,3,9
Email scheduling technique,,ntnsndr,Practioner of the protocol,2026-01-07T00:19:13.592Z,False,1.2.3,1,1,9,9,9,6,,,9,3,9,,7,,9,,2,,3,1,,,
Podcasting (as a social practice),,ntnsndr,"Listener, occasional producer",2026-01-15T19:07:50.435Z,True,1.2.4,,8,7,,8,,,,,3,3,,9,7,,,,,,8,9,,
,,Bob,,2026-01-16T22:58:07.449Z,True,1.2.4,,7,2,,9,,,,,9,9,,1,5,,,,,,9,2,,
Handshake ,,Owocki ,User of it ,2026-01-16T22:56:54.628Z,True,1.2.4,,8,1,,8,,,,,1,8,,8,2,,,,,,8,1,,
Andrew Huberman Sleep,,T,Learner,2026-01-16T22:58:23.015Z,True,1.2.4,,2,8,,5,,,,,9,1,,9,5,,,,,,2,,,
EthBoulder,,ash,contributor,2026-01-16T22:56:10.319Z,True,1.2.4,,7,8,,9,,,,,2,2,,9,8,,,,,,7,7,,
Network Assembly Protocol,,Benjamin Life,"Co-Author, Network Steward",2026-01-16T23:00:21.620Z,True,1.2.4,,3,4,,5,,,,,4,5,,4,2,,,,,,5,5,,
WeChat payment for food,,Jon Bo,User paying for the food,2026-01-16T22:57:20.966Z,True,1.2.4,,9,9,,2,,,,,1,2,,5,2,,,,,,7,9,,
Dating Apps,,Theo,Observer (former subject - now free),2026-01-16T22:57:57.758Z,True,1.2.4,,7,6,,8,,,,,2,3,,9,7,,,,,,2,5,,
On Protocols: Creating Habitat,,Todd Youngblood,RegenHub Boulder - 1515 Walnut,2026-01-16T22:56:12.944Z,True,1.2.4,,8,2,,8,,,,,2,3,,8,4,,,,,,8,8,,
We* Are Agentic,,Neil M Yarnal,Local* is Relative ,2026-01-16T22:56:25.350Z,True,1.2.4,,7,7,,9,,,,,5,5,,5,5,,,,,,6,9,,
The gift,,Tayken,Selflessness,2026-01-16T22:56:14.318Z,True,1.2.4,,7,2,,2,,,,,6,8,,2,6,,,,,,3,1,,
Claude Agent SDK ,,@unforced,Active user,2026-01-16T23:03:14.147Z,True,1.2.4,,7,6,,5,,,,,6,3,,6,6,,,,,,6,6,,
Jon & Savannah Relationship Agreements,,Savannah Kruger & Jonathan Borichevskiy,Co-creators,2026-01-16T22:59:28.744Z,True,1.2.4,,2,9,,9,,,,,1,1,,9,,,,,,,5,1,,
Native protocol,,proofoftom,Co-creator,2026-01-16T22:58:38.756Z,True,1.2.4,,5,8,,7,,,,,7,1,,4,8,,,,,,7,8,,
Free-range kids,,ntnsndr,parent,2026-01-18T23:25:49.803Z,True,1.2.4,,9,9,,7,,,,,1,1,,7,8,,,,,,8,,,
Passing on a sidewalk,,ntnsndr,Pedestrian,2026-01-22T00:34:13.309Z,True,1.2.4,,7,9,,6,,,,,3,2,,7,6,,,,,,6,7,,
Academic job talks,,ntnsndr,an academic,2026-02-04T05:17:44.193Z,True,1.2.4,,7,3,,2,,,,,2,2,,3,3,,,,,,2,2,,
Magic Mountain Talks,,ntnsndr,one of the organizers,2026-02-10T04:24:36.353Z,False,1.2.4,7,3,9,8,3,8,9,3,9,8,1,7,9,2,,7,2,2,7,6,3,5,9
Book draft workshop,,ntnsndr,The designer and victim of the protocol,2026-02-21T20:48:59.352Z,False,1.2.4,2,3,7,7,8,9,6,3,9,7,2,7,7,2,3,7,2,2,7,8,3,8,
Playdate orgabizing,,ntnsndr,A frequent participant,2026-02-21T20:54:34.875Z,True,1.2.4,,8,9,,7,,,,,3,1,,6,8,,,,,,7,3,,
Letters of recommendation,,ntnsndr,A frequent recommender,2026-02-24T01:55:07.661Z,True,1.2.6,,7,3,,4,,,,,2,8,,7,2,,,,,,4,2,,
Deciding how to get the check at a restaurant,,ntnsndr,Person who often feels awkward and wants to be generous,2026-02-25T03:43:22.857Z,True,1.2.6,,8,9,,8,,,,,1,2,,9,7,,,,,,7,,,
Office hours,,ntnsndr,Professor,2026-03-04T22:46:56.138Z,False,1.2.6,7,8,5,9,7,9,3,,8,9,4,9,8,3,,,3,7,8,7,8,9,
Screen time,,ntnsndr,parent,2026-03-09T14:42:00.368Z,False,1.2.6,3,3,6,6,8,7,7,,8,4,5,,3,7,,,,8,3,,,5,
Playing catch with a baseball,,ntnsndr,Baseball coach,2026-03-20T23:13:29.924Z,True,1.2.6,,7,8,,7,,,,,1,1,,7,3,,,,,,3,,,
1 Descriptor Description analyst standpoint timestamp shortform version Design_explicit_vs_implicit Design_precise_vs_interpretive Design_institutional_vs_vernacular Design_documenting_vs_enabling Design_static_vs_malleable Design_technical_vs_social Design_universal_vs_particular Design_durable_vs_ephemeral Entanglement_macro_vs_micro Entanglement_sovereign_vs_subsidiary Entanglement_self-enforcing_vs_enforced Entanglement_abstract_vs_embodied Entanglement_obligatory_vs_voluntary Entanglement_flocking_vs_swarming Entanglement_defensible_vs_exposed Entanglement_monopolistic_vs_pluralistic Experience_sufficient_vs_limited Experience_crystallized_vs_contested Experience_trust-evading_vs_trust-inducing Experience_predictable_vs_emergent Experience_exclusion_vs_inclusion Experience_restraining_vs_liberating Experience_dead_vs_alive
2 Coffee shop as office ntnsndr A person using a coffee shop as an office 2025-11-25T18:07:21.486Z True 1.1.0 7 7 3 4 3 8 4 3
3 Street crossing in Buenos Aires Venkat Tourist 2025-12-03T17:16:54.751Z False 1.2.1 8 6 6 9 6 8 8 2 8 7 3 7 7 2 5 6 6 6 6 7 4 8
4 qPCR Fotis Tsiroukis Ethnography of Scientific Practice 2025-12-12T00:00:47.807Z False 1.2.1 4 8 6 6 4 3 9 2 5 4 4 5 3 6 4 8 4 5 4 2 7 9
5 High wind preemptive blackouts ntnsndr Power consumer who lost power at office and home during high winds 2025-12-18T12:21:09.551Z False 1.2.1 2 3 1 6 7 3 7 8 3 8 9 8 3 1 2 1 7 7 6 8 5 9
6 Bedtime routine with kids ntnsndr Parental participant 2026-01-05T04:30:04.916Z False 1.2.3 2 4 9 9 3 9 7 3 9 8 8 9 1 2 8 1 6 7 1 8 9
7 Quarantine of a sick family member ntnsndr Family member 2026-01-06T04:37:20.251Z False 1.2.3 8 8 7 9 9 9 5 7 8 3 2 9 8 8 2 7 8 2 1 3 9
8 Email scheduling technique ntnsndr Practioner of the protocol 2026-01-07T00:19:13.592Z False 1.2.3 1 1 9 9 9 6 9 3 9 7 9 2 3 1
9 Podcasting (as a social practice) ntnsndr Listener, occasional producer 2026-01-15T19:07:50.435Z True 1.2.4 8 7 8 3 3 9 7 8 9
10 Bob 2026-01-16T22:58:07.449Z True 1.2.4 7 2 9 9 9 1 5 9 2
11 Handshake Owocki User of it 2026-01-16T22:56:54.628Z True 1.2.4 8 1 8 1 8 8 2 8 1
12 Andrew Huberman Sleep T Learner 2026-01-16T22:58:23.015Z True 1.2.4 2 8 5 9 1 9 5 2
13 EthBoulder ash contributor 2026-01-16T22:56:10.319Z True 1.2.4 7 8 9 2 2 9 8 7 7
14 Network Assembly Protocol Benjamin Life Co-Author, Network Steward 2026-01-16T23:00:21.620Z True 1.2.4 3 4 5 4 5 4 2 5 5
15 WeChat payment for food Jon Bo User paying for the food 2026-01-16T22:57:20.966Z True 1.2.4 9 9 2 1 2 5 2 7 9
16 Dating Apps Theo Observer (former subject - now free) 2026-01-16T22:57:57.758Z True 1.2.4 7 6 8 2 3 9 7 2 5
17 On Protocols: Creating Habitat Todd Youngblood RegenHub Boulder - 1515 Walnut 2026-01-16T22:56:12.944Z True 1.2.4 8 2 8 2 3 8 4 8 8
18 We* Are Agentic Neil M Yarnal Local* is Relative 2026-01-16T22:56:25.350Z True 1.2.4 7 7 9 5 5 5 5 6 9
19 The gift Tayken Selflessness 2026-01-16T22:56:14.318Z True 1.2.4 7 2 2 6 8 2 6 3 1
20 Claude Agent SDK @unforced Active user 2026-01-16T23:03:14.147Z True 1.2.4 7 6 5 6 3 6 6 6 6
21 Jon & Savannah Relationship Agreements Savannah Kruger & Jonathan Borichevskiy Co-creators 2026-01-16T22:59:28.744Z True 1.2.4 2 9 9 1 1 9 5 1
22 Native protocol proofoftom Co-creator 2026-01-16T22:58:38.756Z True 1.2.4 5 8 7 7 1 4 8 7 8
23 Free-range kids ntnsndr parent 2026-01-18T23:25:49.803Z True 1.2.4 9 9 7 1 1 7 8 8
24 Passing on a sidewalk ntnsndr Pedestrian 2026-01-22T00:34:13.309Z True 1.2.4 7 9 6 3 2 7 6 6 7
25 Academic job talks ntnsndr an academic 2026-02-04T05:17:44.193Z True 1.2.4 7 3 2 2 2 3 3 2 2
26 Magic Mountain Talks ntnsndr one of the organizers 2026-02-10T04:24:36.353Z False 1.2.4 7 3 9 8 3 8 9 3 9 8 1 7 9 2 7 2 2 7 6 3 5 9
27 Book draft workshop ntnsndr The designer and victim of the protocol 2026-02-21T20:48:59.352Z False 1.2.4 2 3 7 7 8 9 6 3 9 7 2 7 7 2 3 7 2 2 7 8 3 8
28 Playdate orgabizing ntnsndr A frequent participant 2026-02-21T20:54:34.875Z True 1.2.4 8 9 7 3 1 6 8 7 3
29 Letters of recommendation ntnsndr A frequent recommender 2026-02-24T01:55:07.661Z True 1.2.6 7 3 4 2 8 7 2 4 2
30 Deciding how to get the check at a restaurant ntnsndr Person who often feels awkward and wants to be generous 2026-02-25T03:43:22.857Z True 1.2.6 8 9 8 1 2 9 7 7
31 Office hours ntnsndr Professor 2026-03-04T22:46:56.138Z False 1.2.6 7 8 5 9 7 9 3 8 9 4 9 8 3 3 7 8 7 8 9
32 Screen time ntnsndr parent 2026-03-09T14:42:00.368Z False 1.2.6 3 3 6 6 8 7 7 8 4 5 3 7 8 3 5
33 Playing catch with a baseball ntnsndr Baseball coach 2026-03-20T23:13:29.924Z True 1.2.6 7 8 7 1 1 7 3 3