Loading...
Loading...
Contributors:
Dataset URI:
10.5281/zenodo.23102693Variance pilot, no PASS/FAIL (plan committed before any data; see PLAN.md and its sha256 file). Purpose: measure how much the OSIRIS core's held-out loss changes after it trains on conversation, and how that change varies, so that a confirmatory test can be sized. Mentor qwen2.5:7b (digest checked before every exchange); 10 topic blocks x 10 scripted prompts = 100 exchanges, 20 held out; pre = the frozen pilot-0 core (step 6604); post = pre + the training lessons in collection order; 5 replay seeds.Results. Mean improvement on held-out replies: 0.067 bits/byte, positive on 20/20 items and 100/100 item-seed pairs (pilot 0, 3 items: 0.043). Between-item SD of the seed-averaged change 0.0086 (bootstrap 95 % CI 0.0051-0.0112). About 86 % of the variance is the training run: per-seed means range 0.030-0.101 (variance components: run 8.2e-4, item 7.0e-5, residual 6.3e-5). A confirmatory study must therefore replicate training runs - about 4 for a 0.05 bits/byte minimum effect, about 18 for 0.02 - rather than rely on held-out items, for which the plan's item-level table (2-4 items) is overconfident. The trained core (5.56 bits/byte) remains worse than a unigram model of the same replies (4.68) on every item: learning is measurable and small, and the core is far from its speaking gate.Deviation. 79 training lessons, not 80: exchange 8 was flagged not learnable after two empty-reply retries. Collection was interrupted by a host reboot at exchange 10 and resumed from the hash-chained log. The live core checkpoint was unchanged by the replay (live_integrity.json). Prompts were written by Claude and scripted, a known difference from organic conversation.
Keyword(s):
continual learning; errata; language-models; osiris; pilot study.
MeSH Term(s):
Grants and Funding:
No Grants or Funding Data Available
Associated Publication(s):
No Associated Publication Data Available
Supplementary Data:
Related Dataset(s):
No Related Dataset Data Available
The National Library of Medicine (NLM) provides information about biomedical datasets. Inclusion on an NLM Web page does not imply endorsement of, or agreement with, the contents by NLM or the National Institutes of Health. Learn more: NLM Web Policies (nih.gov)