We find that detector performance is highly cohort-dependent: models trained on a large controlled rewrite benchmark achieve AUROC near 0.999 under story-grouped evaluation, and a preregistered hierarchical neural ensemble reaches 0.993 AUROC in-domain, while the same shipped detector calibrated on a small mixed corpus falls to AUROC 0.648 on that benchmark. Holding an entire dataset out of both training and calibration drops every classical tier to 0.52–0.76 AUROC; the frozen neural ensemble transports at 0.802 mean AUROC (worst cohort 0.658) and is the only tier robust to paraphrase (0.98). A dataset-origin probe (0.81 accuracy against 0.33 chance) shows that features encode dataset identity — part of every in-domain number is shortcut. Calibration transfer separates ranking from absolute probability: a frozen scorer re-calibrated across cohorts loses little Brier. The residual frontier is Unicode-level adversarial perturbation — homoglyph and zero-width-space insertion collapse every tier, the neural one most of all. Performance and calibration are therefore cohort properties, not model properties. This demonstrates that detector scores are not portable evidence without population-specific calibration, and that the scientifically stronger question is not whether AI text is universally identifiable, but under what conditions AI participation is statistically identifiable and how the evidence degrades when those conditions change.
We present Panoptes, an open measurement framework for calibrated AI-text attribution under distribution shift. Detector scores are mapped through an independent calibration set to probabilities, converted to prevalence-corrected likelihood ratios, combined with a declared prior in odds form, and wrapped in uncertainty and abstention; expected calibration error is reported as a diagnostic and is not used to rescale the posterior. Three outputs stay distinct — probability of AI participation, probability of AI generation, and exploratory source-family attribution with an explicit unknown state — and watermark tests are a separate subsystem. A license-audited registry partitions cohorts by whether derived weights may be redistributed (pooled training pool: MAGE, RAID clean, DeFactify; evaluation-only: M4GT-Bench, EvoBench, CoAuthor). Code and pinned open weights accompany the paper.
The question “was this text written by a machine?” is now asked daily by educators, editors, reviewers, and platforms. The tools typically answering it reduce a document to a single number, offered without a statement of the evidence that produced it, the sample size behind it, or the regimes in which it should not be trusted. False positives carry real costs for the accused; false negatives carry real costs for institutions. A defensible tool must therefore do more than score: it must show its evidence, keep its uncertainty visible, and refuse to overstate.
We take a stricter position than “show the uncertainty.” AI-text attribution is a measurement problem. The evidential meaning of a detector score depends on the generating population, prevalence, domain, manipulation history, provenance, and calibration cohort. The same likelihood ratio yields different posterior probabilities under a 1% classroom prior and a 50% lab prior; the same calibrated map can be well calibrated on New York Times rewrites and anti-correlated on a mixed prompt corpus. Panoptes is built to measure that evidential strength and its failure modes, not to win a detector leaderboard.
Four research questions are frozen in the measurement protocol (registered 2026-08-13, bench/protocol.json) before further outcome tuning:
RQ1 — Calibration. When Panoptes reports a probability of AI participation, does that probability correspond to empirical frequency under a declared population?
RQ2 — Distribution shift. How much does attribution degrade when the domain, generator, prompt, time period, or editing process changes?
RQ3 — Evidence aggregation. Do explicit calibration, Bayesian aggregation, uncertainty estimation, and abstention improve trustworthiness over conventional classifier scores?
RQ4 — Reproducibility. Can the full experimental claim be independently verified from cryptographically committed datasets, model artifacts, manifests, and results?
Panoptes is built around four design commitments. First, evidence before verdict: the interface leads with a plain-language summary and exposes the full statistical drilldown — formulas, units, assumptions, and limitations — one layer down. Second, calibrated combination: heterogeneous signals are combined in the odds domain with an explicit prior, so that the strength of evidence is separable from base rates. Third, verifiable practice: the project ships a canonical baseline prompt set, human control responses, and a public, hash-chained catalog so that claims about model behavior can be checked, replicated, and extended by anyone without trusting the project with raw outputs. Fourth, the corpus is load-bearing: the reference material is not a gallery but an input — it calibrates the live analyzer through a signed artifact, trains the model zoo under a statistical power gate, and grounds a pre-registered hypothesis battery whose null results we report as faithfully as any positive finding. A companion external validation on the 71,666-record Defactify_Text_Dataset [34] runs the identical protocol where the power gate passes, so the reader can see which conclusions were science and which were sample size (§11). The strongest number in that section is not the near-perfect in-domain score; it is the collapse under transport.
The tasks these questions address are not interchangeable. Table T1 states the labels used throughout.
| Task | Definition |
|---|---|
| Human-only | No AI generation or substantive AI rewriting. |
| AI-generated | The majority or entirety of the content was generated by an AI system. |
| AI-assisted | A human authored the content but AI substantially generated or modified portions. |
| Unknown | Available evidence is insufficient for a supported attribution. |
Three outputs stay conceptually distinct: probability of AI participation (generated or substantially modified), probability of AI generation (majority or entirety generated by an AI system), and source-family attribution with an explicit unknown/open-set score. Watermark detection is a separate subsystem and is not allowed to inflate the passive-attribution claim. Provenance is reported at levels P0–P4 (§4.5) and is never blended with statistical scores.
Watermarking. Kirchenbauer et al. introduced the green-list watermark for large language models, in which a pseudorandom partition of the vocabulary biases token selection and detection reduces to a binomial test [1]. Aaronson and Kirchner developed a related cryptographic scheme [2]. These tests are powerful when the text is unmodified and the scheme is known, and degrade gracefully under editing — behavior we quantify explicitly through effect sizes, power estimates, and a dilution statistic rather than a bare p-value. The family is no longer only academic: Google DeepMind’s SynthID-Text productionized a green-list watermark and reported no measurable quality cost in live traffic [37], and Anthropic announced in August 2026 that Claude’s text watermark is a version of SynthID-Text, applied globally to comply with the EU AI Act [38]. Our reference adapter models exactly this Aaronson/green-list family; because Anthropic’s production key is private, we characterize the family’s robustness rather than any vendor’s specific key (§12). Watermarked outputs are also radioactive: student models trained on them can inherit a detectable green-list bias [46, 47], which makes distillation forensics conceivable but fragile — targeted paraphrase and inference-time neutralization can scrub the inherited signal while preserving distilled knowledge [48]. Separately, closed image watermarks in the SynthID family have been reverse-engineered by spectral analysis [49], underscoring that fixed-structure carriers are extractable; text schemes with per-context pseudorandom green lists are harder but not immune to an arms race. Sampling temperature is a first-order control on embeddability: at temperature 0 (greedy decoding) there is no randomness left to steer (§16).
Detection and its limits. Post-hoc AI-text detectors divide into zero-shot statistics over the generating model — GLTR’s token-rank visualization [32], DetectGPT’s probability-curvature test [5], the Fast-DetectGPT conditional-probability speedup [6], Binoculars’ cross-model perplexity ratio [7] — and trained classifiers over surface or learned features, evaluated on corpora such as HC3 [8], M4 [9], and RAID [10], adversarially hardened variants such as RADAR [33], and hardened against paraphrase attacks such as DIPPER [11] and adaptive attacks generally [12]. Two consistent findings motivate our design: detectors are miscalibrated under domain shift, short inputs, and paraphrase [10, 12]; and small, interpretable feature models are competitive with large neural ones in-distribution while degrading more gracefully under distribution shift. Panoptes therefore treats detection scores as evidence with a stated reliability, never as verdicts, and trains its own models only under a pre-registered statistical protocol with leakage-safe evaluation.
Uncertainty and abstention. Evidential deep learning places a Dirichlet distribution over class probabilities by predicting its concentration parameters, yielding native measures of vacuity (“I don’t know”) and dissonance (“the evidence conflicts”) [13, 14]. Conformal prediction converts any score into prediction sets with distribution-free coverage guarantees [15]. Both ideas are built into the system: the analyzer abstains below a token threshold, and our neural tier outputs Dirichlet evidence wrapped in a split-conformal calibration layer.
Provenance. Content credentials and signed manifests (e.g., C2PA) attest the origin of a file rather than the statistical character of its text [3]. Panoptes treats provenance as an independent evidence channel: when present and verified it dominates statistical inference; when absent it says nothing, and the interface says so.
Verifiable artifacts and documentation practice. Our community catalog draws on the same principle as transparency logs and timestamping services: commit to content with cryptographic hashes, anchor the commitment publicly, and let anyone verify. OpenTimestamps provides a pragmatic anchor to the Bitcoin blockchain without any custom chain [4]. Models and datasets are documented with model cards [16] and datasheets [17], and our dataset statements follow the data-statement discipline of Bender and Friedman [18]; every card and artifact in this paper is machine-checkable against a published schema and carries its own SHA-256.
Input text (or an uploaded document) is segmented into analysis units, and each unit is scored independently before combination. The pipeline maintains an explicit evidence state: when the usable token count falls below a supported threshold (50 tokens), the system reports that evidence is insufficient rather than emitting a number that would invite overconfidence. This refusal path is a first-class result, not an error.
The evidence pipeline is frozen as: raw detector score → probability calibration on an independent calibration set → likelihood-ratio calibration using the declared cohort prevalence π → evidence model → declared prior → posterior → uncertainty → decide or abstain. Given a calibrated cohort-conditional probability p and declared prior odds O0,
LR = (p / (1 − p)) × ((1 − π) / π), O1 = O0 × LR, P = O1 / (1 + O1) (1)Reporting the decomposition — prior odds, likelihood ratio, posterior odds, cohort prevalence — separates “what the tests say” from “what we believed before testing.” Expected calibration error (ECE) on the held-out calibration set is reported alongside as a reliability diagnostic. It is not subtracted from the posterior and does not mathematically discount it; a poorly calibrated map should be replaced or abstained from, not silently shrunk. Figure 1 is the frozen pipeline; Figure 2 shows the posterior as a function of the declared prior for several likelihood ratios; the interface renders the same curve live with the user’s prior marked. The protocol also compares three evidence-aggregation models — naive summation of segment log-likelihood ratios, correlation-aware shrinkage, and a document-level estimate — because a non-rejected Durbin–Watson test is not treated as proof that segments are independent.
For each configured public watermark scheme, eligible tokens are counted and the number falling in the scheme’s green set, G, is compared against the chance expectation under the null. With green-list fraction γ and n eligible tokens,
z = (G − γn) / √(nγ(1 − γ)), p = 1 − Φ(z) (2)Across the family of tests run on one document we control the false discovery rate with the Benjamini–Hochberg procedure, reporting q-values rather than bare p-values [19]. Each test additionally reports the observed green rate with a 95% confidence interval, an effect size (lift over γ), the achieved power of the test at the configured watermark strength and observed n, and a dilution estimate — the fraction of text that would need to carry the watermark to explain the observed lift, which distinguishes “absent” from “edited away.” Tests that cannot run (ineligible text, unknown scheme, insufficient length) report a structured status rather than a silent null.
Two runtime components are fitted on the verified reference corpus (§6) and shipped as a single signed artifact, baseline-calibration.json, which the backend loads at startup and verifies before use. First, the heuristic detector’s raw score is recalibrated with isotonic regression [20] fitted out-of-fold under GroupKFold by prompt, so no record’s calibration is informed by its own prompt’s siblings; the artifact carries the resulting reliability diagram, expected calibration error, Brier score, and a split-conformal threshold [15]. Second, source-family attribution replaces its hand-tuned softmax with corpus-fitted geometry: per-family centroids and covariance in a standardized stylometric feature space, with Mahalanobis distance defining both the conditional family distribution and an explicit unknown score for inputs far from every centroid. The response schema exposes which basis was used (heuristic or corpus-fitted) and the cohort size behind it. If the artifact is absent, corrupted, or fails signature verification, both components fall back to their heuristic forms and the response says so; raw community text is never a runtime input.
When an upload carries a signed provenance manifest, Panoptes verifies the signature chain and reports the outcome as a separate channel with its own status vocabulary (verified, invalid, not present) and a provenance level: P0 no provenance; P1 self-declared model identity; P2 authenticated provider metadata; P3 cryptographically signed generation receipt (verified C2PA); P4 independently verifiable execution (hash-pinned run, Merkle commitment, optional OpenTimestamps). A model’s self-reported identity is P1, not independently verified provenance. Statistical and cryptographic evidence are never blended into a single score; the user sees both, labeled.
The web interface is part of the method. The first view is a plain-language answer with the outcome distribution; the laboratory below exposes the posterior decomposition, posterior-sensitivity and reliability curves, per-scheme watermark statistics, an evidence contribution waterfall, a coverage–abstention curve, the input’s stylometric profile against corpus ranges, and the interpretation limits that apply to the specific input. Every statistic carries an inline explanation of what it measures, so the report can be read by the people it affects.
Claims about model behavior should be checkable by anyone, not just by those with API budgets. Panoptes defines a canonical baseline protocol with four properties:
A canonical prompt set. Sixteen prompts — eight prose, eight code — spanning genres, lengths, and task types, versioned as a machine-readable manifest whose own SHA-256 is embedded in the artifact (2092321d…8b06b for v1.0.0). Every run pins this hash, so results are comparable by construction.
Execution modes and human controls. Runs may be produced manually through a chat interface, programmatically through provider APIs, or agent-assisted inside an editor; the interface used is declared in the manifest, because the channel is part of the evidence. A fourth run type, human, records license-clear human-written responses to the same prompts; eight such controls ship with the release and anchor every human-vs-AI comparison in the paper.
Hash-pinned manifests. A run manifest records the model identity (as reported by the runner), the pinned prompt-set hash, and for each output only its filename, byte count, and SHA-256. Output hashes form a Merkle tree whose root commits to the entire set; the manifest’s own canonical hash identifies the run. Raw outputs never enter the catalog — a validator rejects any manifest that embeds model text, so the public ledger carries evidence, not content.
An append-only public catalog. Submissions append one JSON line to a registry and one manifest file, both validated in continuous integration against JSON Schemas. Optional OpenTimestamps anchoring binds the manifest hash to a Bitcoin block, giving each claim a tamper-evident timestamp [4]. Verification is a single command (baseline.py verify-catalog) and requires no trust in the submitter.
Contribution protocol. A contributor (1) runs the canonical prompt set by any of the execution modes, (2) finalizes the run, which hashes each output with SHA-256, builds the Merkle root, and self-hashes the canonical manifest, (3) optionally anchors the manifest hash with OpenTimestamps, and (4) opens a pull request containing only the registry line and the manifest file. Raw outputs remain on the contributor’s machine, ignored by version control. External corpora participate by reference rather than by inclusion: a signed pointer manifest records the source URL, access conditions, download instructions, and a SHA-256 integrity commitment, so third-party data is never copied into the repository. The training bench (§9) consumes these pointers to validate contributed datasets locally.
Relationship to the analyzer. The catalog itself remains a non-runtime input: no live score ever reads community-submitted text. What the corpus feeds is the offline pipeline of §§6–9 — hypothesis testing, calibration fitting, and model training — whose outputs reach the analyzer only as signed, schema-validated, hash-verified artifacts with stated cohort sizes. Every number the corpus influences is therefore traceable to a specific, re-verifiable input set.
Framing for v2.1. The 104-record verified corpus is Reference Community Corpus v0. It is infrastructure-load-bearing, not inference-load-bearing: its job is to be a worked, end-to-end implementation of the contribution, hashing, manifest, calibration, and power-gating machinery — small enough to inspect by hand, permanent enough to re-verify — rather than to power the primary statistical claims. In v2.0 it also carried the in-domain inference; in v2.1 that inferential load moves to the expanded, license-audited multi-dataset pool (MAGE, RAID, and DeFactify, with CoAuthor, M4GT, and EvoBench held out for evaluation; §13), and this corpus is retained as the canonical demonstration of how a contributor adds a hash-pinned cohort and how power and coverage expand as cohorts accrue — without any claim that eight human controls are representative of human writing. Nothing in this section is used to train or calibrate the production detector.
The release corpus comprises 104 records from 13 verified runs: 96 AI outputs from six frontier model families (claude-opus-5-max, gemini-3.1-pro, glm-5.2-max, gpt-5.6-sol-max, grok-4.6-extra-high, kimi-k3-max; eight text and eight code outputs each) and 8 human control responses to the text prompts. Every record was re-hashed against its run manifest at ingestion; a single mismatch rejects the run. Table 1 lists the registered runs; Table 2 summarizes cohort-level stylometry.
| Model (as reported) | Kind | Run identifier | Merkle root | Manifest SHA-256 |
|---|---|---|---|---|
| claude-opus-5-max | text | claude-opus-5-max_text-20260812t193701z | b918b311bfcf242f… | b1e76e0d530b3d6b… |
| claude-opus-5-max | code | claude-opus-5-max_code-20260812t193712z | ef4da16540dfcf58… | e3c58902e8537258… |
| gemini-3.1-pro | text | gemini-3.1-pro_text-20260812t194331z | e530b6039a9ba684… | 3e7beba818c7cfd2… |
| gemini-3.1-pro | code | gemini-3.1-pro_code-20260812t194331z | 65ff2ec8cd494187… | 22356b6f478deefc… |
| glm-5.2-max | text | glm-5.2-max_text-20260812t194001z | e9bdcfb2a10d16ab… | f28cf49828684e67… |
| glm-5.2-max | code | glm-5.2-max_code-20260812t194011z | 35811aeade2f1b5d… | fa81d4b8842a6cd2… |
| gpt-5.6-sol-max | text | gpt-5.6-sol-max_text-20260812t193039z | 88571429d51bd5b0… | 42987bd8942bd4a6… |
| gpt-5.6-sol-max | code | gpt-5.6-sol-max_code-20260812t193106z | 41394de135815089… | 519925b7b5d17210… |
| grok-4.6-extra-high | text | grok-4.6-extra-high_text-20260812t185911z | ef74161b4e9481cf… | a8d0696096199478… |
| grok-4.6-extra-high | code | grok-4.6-extra-high_code-20260812t185932z | 69a77a793259e1d3… | b605ac384e30cfff… |
| kimi-k3-max | text | kimi-k3-max_text-20260812t185240z | 85068a301bf15f3f… | 942edc2dfe18cbd3… |
| kimi-k3-max | code | kimi-k3-max_code-20260812t185244z | e017bdc5ec3f87b9… | b2eb461859b65e2c… |
| human (control) | text | human-control_text-20260812 | local run | local run |
| Cohort | n | long-word rate | unique ratio | token entropy |
|---|---|---|---|---|
| human (control) | 8 | 0.168 | 0.651 | 6.90 |
| claude-opus-5-max | 8 | 0.199 | 0.629 | 6.91 |
| gemini-3.1-pro | 8 | 0.262 | 0.642 | 6.77 |
| glm-5.2-max | 8 | 0.171 | 0.630 | 6.77 |
| gpt-5.6-sol-max | 8 | 0.252 | 0.696 | 6.85 |
| grok-4.6-extra-high | 8 | 0.189 | 0.646 | 6.90 |
| kimi-k3-max | 8 | 0.204 | 0.664 | 6.83 |
Two observations deserve note. First, the human controls are not stylistically extreme: they sit inside the AI range on unique ratio and token entropy and below it on long-word rate, which is why no single feature separates the classes (§7). Second, the corpus caught a provenance-relevant anomaly: the glm-5.2-max and kimi-k3-max outputs for prompt code-07 (a customer-order SQL aggregation) are byte-identical — the same 202 bytes, SHA-256 committed independently by two runs finalized minutes apart. Whether this reflects shared training data, convergent templating of a canonical query, or a common serving component is not determinable from text alone; what the protocol guarantees is that the observation is permanent, independently re-verifiable, and attributable to a declared time and channel. This is the catalog working as intended: not a leaderboard, but evidence that cannot quietly change.
Before any hypothesis is tested, the feature set passes through an econometric screening layer (bench/methodology.py, pre-registered in bench/hypotheses.json). Seventeen candidate stylometric features are screened for multicollinearity by variance inflation factors, VIFj = 1/(1 − Rj2) from an auxiliary regression of each feature on the rest [21]: values above 5 are investigated and above 10 excluded, with the exclusion and its justification recorded in the signed report. Four features were excluded (log word count, VIF = 405.4; hapax ratio, 39.9; punctuation rate, 18.6; mean word length, 14.9), leaving twelve features with a condition number of 7.6 — comfortably below the conventional concern threshold of 30.
A penalized logistic model (IRLS with a 10−6 ridge, features standardized) fitted on the screened set attains a log-likelihood of −18.1 against a null of −28.2: McFadden pseudo-R2 = 0.358, Tjur’s R2 = 0.236 [22]. Specification tests on this fit are summarized in Table 3. The link test [23], Hosmer–Lemeshow decile test [24], and Ramsey RESET [25] all fail to reject adequacy; Breusch–Pagan finds no heteroscedasticity signal [26]; Jarque–Bera rejects residual normality, as it necessarily does for Pearson residuals of a binary model, and is reported as a diagnostic only [27]. Cook’s distance [28] flags 11 of 104 records above the 4/n screen — expected in a corpus with eight-record cohorts — and the Durbin–Watson statistic on residuals ordered by fitted probability is 2.09, indicating no residual trend along the fit [29].
| Test | Null hypothesis | Statistic | p-value | Verdict |
|---|---|---|---|---|
| Link test | linear logit link adequate | 0.229 | 0.632 | not rejected |
| Hosmer–Lemeshow | deciles calibrated | 4.365 | 0.823 | not rejected |
| Ramsey RESET | no omitted nonlinear structure | 4.667 | 0.097 | not rejected |
| Breusch–Pagan | homoscedastic errors | 14.02 | 0.300 | not rejected |
| Jarque–Bera | normal residuals (diagnostic) | 469.4 | <10−99 | rejected (expected) |
| Durbin–Watson | no residual trend along fit | 2.087 | — | ≈2, no trend |
Six hypotheses about the corpus were registered before testing, each with its null, test, variables, direction, and α = 0.05; q-values are Benjamini–Hochberg adjusted across the family [19]. Table 4 reports the decisions exactly as the signed methodology report records them.
| ID | Statement (abbreviated) | Test | Corpus (n = 104) | Defactify (n = 71,666) | ||||
|---|---|---|---|---|---|---|---|---|
| q | Effect | Decision | q | Effect | Decision | |||
| H1 | connector rate higher in AI text | Welch t | 0.192 | d = 0.11 | null not rejected | <0.001 | d = 0.67 | null rejected |
| H2 | token entropy lower in AI text | Mann–Whitney | 0.051 | r = 0.51 | null not rejected | <0.001 | r = 0.20 | null rejected |
| H3 | unique ratio discriminates beyond length | logistic LR | 0.057 | ΔR2 < 0.001 | null not rejected | <0.001 | OR = 0.018 | null rejected |
| H4 | per-family centroids separated | permutation MANOVA | 0.139 | F = 0.52 | null not rejected | 0.002 | Λ = 0.46 | null rejected |
| H5 | long-word rate higher in AI prose | Welch t | 0.064 | d = 0.55 | null not rejected | <0.001 | d = 1.07 | null rejected |
| H6 | segment residuals uncorrelated | DW permutation | 0.288 | DW = 2.56 | null not rejected | 1.000 | DW = 2.68 | null not rejected |
On the project corpus, no hypothesis survives multiplicity control at q ≤ 0.05. This is a sample-size statement before it is a science statement: with eight human controls, a Welch t-test on a medium effect (d = 0.5) has power near 23%, and the two-proportion power gate used elsewhere in the system (§9) puts the corpus at roughly 7% power for five-point accuracy differences. The Defactify cohort resolves the ambiguity the small corpus cannot: the same six pre-registered tests, re-run unchanged at n = 71,666, reject five of six nulls. The two results that were merely suggestive at n = 104 — token entropy (H2) and long-word rate (H5) — are confirmed with the same sign, as are connector rate (H1), the length-discrimination increment (H3), and per-family centroid separation (H4). H6 again finds no evidence that segment-level residuals are autocorrelated (DW = 2.68, q = 1.0). That non-rejection is not treated as proof of segment independence: the measurement protocol compares naive accumulation, correlation-aware shrinkage, and document-level evidence explicitly (§4.2), because residual diagnostics can look acceptable while summed log-likelihood ratios still overstate document-level evidence. The registry exists precisely so that these numbers cannot be silently re-litigated: the tests, directions, and α were fixed before either cohort was scored. H1–H6 remain supporting stylometry diagnostics; the primary scientific claims of this revision are RQ1–RQ4.
The bench/ package turns the verified corpus — and any schema-valid community dataset — into models under a protocol that refuses to overreach its sample size. Its components: a single-source feature extractor shared with the research layer; a dataset loader that re-verifies every record against its manifest hash; grouped cross-validation by prompt so that no evaluation fold sees a sibling of a training record; a tiered model zoo (scikit-learn [35] logistic and gradient-boosting implementations); and signed model cards [16] and datasheets [17] for everything it trains.
The tier gate. Model complexity is admitted by statistical power, not enthusiasm. Tier 0 (penalized logistic regression) is always admissible. Tier 1 (gradient boosting) requires n ≥ 300. Tier 2 (neural) requires passing a two-proportion worst-case power calculation: to detect a five-point accuracy gain at α = 0.05 with 80% power demands n ≈ 3,140 eligible records. At the current n = 104 the gate fails (Figure 4), and the bench says so on every card it writes; Panoptes-v0 (§10) is therefore reported as an experimental architecture with a comparison battery, never as an improvement.
Reference results. The tier-0 logistic model, evaluated entirely out-of-fold under GroupKFold by prompt, attains AUROC 0.788 (bootstrap 95% CI 0.664–0.891), Brier 0.077, ECE 0.056, and TPR 0.542 at 1% FPR; a split-conformal wrap at α = 0.1 delivers 0.923 empirical coverage with mean set size 1.06 and a 5.8% abstention rate. Figure 5 shows the coverage–accuracy trade the abstention knob offers. For contrast, the shipped heuristic detector’s raw scores on the same held-out folds reach only AUROC 0.589 with ECE 0.127 before isotonic recalibration — a concrete, quantified motivation for the corpus-fitted calibration of §4.4, whose reliability diagram appears in Figure 6.
Community use. Five commands cover the loop: bench train (cross-validate, fit, and card a model), bench evaluate (re-score a saved model on a dataset), bench validate (score your own CSV/JSONL against the shipped model, with fairness slices by length, kind, and family), bench contribute (write a signed hash-pointer manifest for your dataset into datasets/manifests/), and bench predict (score one text). Datasets are validated against a published JSON Schema; cards are canonical-hashed and verifiable offline.
Panoptes-v0 is a small custom neural architecture designed around the system’s ethos: uncertainty should be computed, not painted on. A feature branch (Linear(17,64) → GELU → LayerNorm → Linear(64,64) → GELU) feeds an evidence head (Linear(·,2) → softplus + 1) whose outputs are the concentration parameters of a Dirichlet distribution over the two class probabilities [13, 14]. The prediction is the Dirichlet mean; vacuity (K/S, the fraction of probability mass the evidence cannot allocate) and dissonance (evidence conflict) are native outputs and drive the same SUPPORTED/INSUFFICIENT states the rest of the system uses. Training uses the evidential MSE loss with an annealed KL term [13], AdamW (lr 3×10−4, weight decay 10−2), early stopping on grouped-validation ECE, and three seeds {13, 42, 87} reported as mean ± sd; a split-conformal wrap calibrates the final scores [15]. A character-n-gram sequence branch (128-vocab char embedding, mean-pooled, projected to 64) is admitted by the power gate: off at n = 104 (batch 16, ≤200 epochs, patience 20), on at Defactify scale (batch 512, ≤40 epochs, patience 6). Weights are saved locally under models/panoptes-v0*/ (gitignored, SHA-256 recorded on the card). Panoptes-v0 remains an experimental architecture; the publicly released open-weights instrument is the v2.1 hierarchical neural ensemble (MarketStandard/panoptes-neural-v1).
Results on the project corpus (sub-gate). Out-of-fold under GroupKFold by prompt, the three-seed ensemble attained AUROC 0.754 (bootstrap 95% CI 0.608–0.887), accuracy 0.923, Brier 0.075, ECE 0.084, and conformal coverage 0.923 at α = 0.1. The comparison battery paired Panoptes-v0 against the tier-0 logistic model and the shipped heuristic with McNemar’s exact test [30] and the DeLong AUROC test [31], Benjamini–Hochberg adjusted [19]: no comparison was significant (all q ≥ 0.557), and the gradient-boosting tier was not run (n = 104 is below its 300-record minimum). That iteration is preserved on the signed card as the corpus_trained block.
Results on Defactify (gate passes). Re-trained on the 71,666-record Defactify bench with the sequence branch enabled, the three-seed ensemble attains out-of-fold AUROC 0.9985 (95% CI 0.9983–0.9987; per-seed 0.99837, 0.99839, 0.99827), accuracy 0.989, Brier 0.0088, ECE 0.019, TPR 0.973 at 1% FPR, and conformal coverage 0.900 at α = 0.1. The battery (Table 5) now includes the gradient-boosting tier and is statistically licensed: Panoptes-v0 beats the tier-0 logistic model (McNemar q ≈ 10−123; DeLong ΔAUROC = +0.0095, q ≈ 10−50) and the shipped heuristic (q < 10−300), and ties the gradient booster on exact accuracy (McNemar q = 0.554) while trailing it by a hair on ranking (DeLong ΔAUROC = −0.0002, q = 0.0013). We report that hair-splitting DeLong loss rather than hiding it: at this sample size the tests resolve differences far below any practical significance threshold.
| Pair | Test | Detail | p | q | Sig. @0.05 |
|---|---|---|---|---|---|
| panoptes-v0 vs logistic-tier0 | McNemar | b = 1042, c = 205 | 6.7e−124 | 1.3e−123 | yes |
| panoptes-v0 vs logistic-tier0 | DeLong | ΔAUROC = +0.0095 | 6.0e−50 | 7.4e−50 | yes |
| panoptes-v0 vs heuristic | McNemar | b = 9021, c = 327 | <1e−300 | <1e−300 | yes |
| panoptes-v0 vs heuristic | DeLong | ΔAUROC = +0.1404 | <1e−300 | <1e−300 | yes |
| panoptes-v0 vs gbm-tier1 | McNemar | b = 272, c = 287 | 0.554 | 0.554 | no |
| panoptes-v0 vs gbm-tier1 | DeLong | ΔAUROC = −0.0002 | 0.0012 | 0.0013 | yes |
| logistic-tier0 vs heuristic | McNemar | b = 8420, c = 563 | <1e−300 | <1e−300 | yes |
| logistic-tier0 vs heuristic | DeLong | ΔAUROC = +0.1309 | <1e−300 | <1e−300 | yes |
| gbm-tier1 vs logistic-tier0 | McNemar | b = 1084, c = 232 | 1.1e−121 | 1.8e−121 | yes |
| gbm-tier1 vs logistic-tier0 | DeLong | ΔAUROC = +0.0097 | 6.3e−51 | 9.0e−51 | yes |
Cross-domain honesty. The same Defactify-trained ensemble, scored back against the 104-record project corpus, attains AUROC 0.471 (95% CI 0.282–0.708) — below chance. NYT news prose and the project’s mixed prompt corpus are different worlds, and a detector fitted on one can anti-correlate with the other. Together with the shipped runtime’s 0.648 AUROC on Defactify, this is the paper’s central result, not a footnote: calibration is a cohort property, not a model property, which is why the runtime ships per-cohort calibration bundles with an explicit switch (§11) and why the measurement protocol reports a train-domain × test-domain transport matrix rather than a single leaderboard number. The scientific question is how portable detection evidence is.
Verdict. At n = 104 the honest statement was architectural, not empirical. At n = 71,666 the empirical statement is licensed and it is two-sided: Panoptes-v0 decisively beats the feature-only logistic baseline and the shipped heuristic in-domain, ties gradient boosting on accuracy while trailing it by 0.0002 AUROC, and fails loudly out-of-domain. Evidential uncertainty, conformal calibration, and abstention are native to the detector rather than post-hoc — and the battery now has the power to say so.
Every number above this section comes from the project’s own 104-record corpus, and every null result is bounded by that choice. To find out which conclusions survive contact with a large external dataset — and which were only small-sample artifacts — we evaluate on the Defactify_Text_Dataset of Roy et al. [34]: 73,193 rows pairing New York Times human articles with single-prompt rewrites by six LLM families (Gemma-2-9B, GPT-4o, Llama-8B, Mistral-7B, Qwen-2-72B, Yi-Large). The dataset is deliberately hard: its authors report 53–58% detection accuracy and 5–9% attribution accuracy for their own baselines. It is fetched by python -m bench.fetch_defactify, which verifies each parquet split against a pinned SHA-256, applies documented hygiene filters, and stores the result locally (raw text never enters the repository; a signed pointer manifest records hashes and counts only).
Hygiene. The upstream data contains rows that are API-error strings labeled as GPT-4o output (“Error communicating with OpenAI…”), exact duplicate texts, and texts shorter than the system’s 50-token support threshold. Filtering removes 412 error artifacts, 73 exact duplicates, and 957 short texts, leaving n = 71,666 records (9,483 human, 62,183 AI) — all counts are signed into backend/artifacts/defactify-summary.json.
Story groups and the leakage audit. Each human story has up to six AI near-rewrites, and the upstream documentation does not establish that the official splits are story-disjoint. We therefore reconstruct story groups ourselves: TF-IDF (word 1–2 grams, min-df 2) cosine nearest-neighbor connected components at threshold 0.45, which finds 61,006 groups (mean size 1.17, 58,917 singletons). The audit of the official splits is not reassuring: 1,236 of 10,769 official test rows (11.5%) share a reconstructed story with the official train split, so numbers computed on those splits are optimistically biased. All Panoptes results below instead use group-disjoint evaluation — GroupKFold-5 by reconstructed story — so no fold ever scores a rewrite of a story it trained on.
Detection battery. With n = 71,666 the power gate of §9 passes by a factor of twenty, and the tiered zoo trains at full strength. Table 6 reports out-of-fold results under story-grouped cross-validation. The tier-0 logistic model attains AUROC 0.989 (95% CI 0.988–0.990), the tier-1 gradient booster 0.999 (0.9986–0.9989), and Panoptes-v0 — its character-sequence branch now admitted by the gate — attains 0.998 (0.998–0.999) with ECE 0.019. Against the dataset authors’ 53–58% baseline accuracy these are large numbers, and we make no SOTA claim: the task is detection of single-prompt rewrites of known human stories, and the stylometric feature vector was partly developed on this kind of contrast. The honest external measurement is the fourth row: the shipped runtime — the heuristic detector with corpus-fitted isotonic calibration, which never saw Defactify — attains AUROC 0.648 (0.643–0.652) with ECE 0.029. That is the domain-shift cost of a calibrator fitted on 104 mixed records and applied to NYT prose, and it is exactly the quantity a deployment should want before trusting a score out-of-domain.
| Model | AUROC | 95% CI | ECE | Brier | Accuracy |
|---|---|---|---|---|---|
| logistic tier-0 | 0.989 | 0.988–0.990 | 0.006 | 0.018 | 0.978 |
| GBM tier-1 | 0.999 | 0.9986–0.9989 | 0.005 | 0.008 | 0.989 |
| Panoptes-v0 (sequence branch on) | 0.998 | 0.998–0.999 | 0.019 | 0.009 | 0.988 |
| shipped heuristic (corpus-calibrated, zero-shot) | 0.648 | 0.643–0.652 | 0.029 | 0.118 | 0.868 |
Exploratory attribution. Beyond binary detection, we ask which source produced a text: a seven-class task (human plus six LLM families) evaluated with two contenders — multinomial logistic regression on the stylometric vector and a K=7 Dirichlet variant of Panoptes-v0 — under the same story-grouped folds. The contenders diverge instructively. The multinomial logistic attains macro-F1 0.693 (accuracy 0.694), an order of magnitude above the 5–9% attribution band Roy et al. report for their baselines; per-family F1 ranges from 0.474 (Mistral-7B) to 0.897 (GPT-4o). The K=7 Dirichlet net collapses: under the annealed evidential KL term on an imbalanced seven-class problem it retreats to near-uniform evidence (vacuity ≈ 1) and argmax ties break to the human class, yielding macro-F1 0.033 — the abstention mechanism firing on a task it cannot resolve, which is the designed behavior taken to its degenerate limit. Figure 9 reports per-family F1 for the stronger contender against the Roy et al. band. Attribution is exploratory: family cues in this dataset may be prompt-specific (every AI text answers the same rewrite instruction), the family labels are the generating model names rather than verified provenance, and the result must never be used to accuse an individual of using a specific model.
Cross-domain check. As a final transportability measurement, the Defactify-trained Panoptes-v0 ensemble was scored back against the 104-record project corpus (NYT news prose → mixed prompts and code): the signed card carries this cross_domain block alongside a corpus_trained block preserving the previous sub-gate iteration for comparison. Domain shift runs in both directions, and the card says so in machine-checkable form.
Dual calibration. The runtime ships two signed calibration artifacts: baseline-calibration.json (fitted on the project corpus; the default) and defactify-calibration.json (isotonic map, reliability bins, conformal threshold, and seven-family Mahalanobis geometry fitted on Defactify). The operator selects between them with the PANOPTES_CALIBRATION_BUNDLE environment variable; the selection is allowlist-validated, an unknown or missing bundle falls back to the corpus-fitted default, and every analysis response names the live bundle in calibration.bundle and source_families.basis. The default stays corpus-fitted: the Defactify bundle is opt-in, because a NYT-prose calibrator is the wrong prior for the mixed-content traffic the analyzer actually sees.
The benchmark suite. Three further external benchmarks now follow the same fetch-and-pin pattern (raw text stays local; signed pointer manifests record hashes and counts only): RAID (Dugan et al. 2024 [10]; 5,615,820 raw rows streamed from an 11.8 GB CSV, hygiene-filtered to 4,902,597, group-subsampled to n = 149,977 for the protocol run) contributes unseen generators, domains, decoding strategies, and eleven adversarial attack families; M4GT-Bench (Wang et al. 2024 [9]; n = 151,005 English and n = 205,233 multilingual after hygiene) contributes multi-domain, multilingual, multi-generator text; and EvoBench (Yu et al. 2025 [36]; n = 29,574) contributes generator-generation shift across model versions, with pair groups keyed by a content hash of the human original so no fold ever trains on a text whose rewrite it scores. Table 7 reports the frozen-protocol headline numbers for the shipped heuristic and the logistic tier on every cohort. Two regularities matter more than any single number: the in-cohort logistic tier is strong everywhere (AUROC 0.78–0.99) while the shipped corpus-calibrated heuristic is much weaker everywhere it is off-domain (0.63–0.69 on RAID and M4GT), and the human-class conformal coverage deficit of §12 recurs on every cohort.
| Cohort | n | Detector | AUROC | Brier | ECE | Slope | Conformal (pooled / human slice) |
|---|---|---|---|---|---|---|---|
| Defactify | 71,666 | heuristic | 0.885 | 0.064 | 0.029 | 1.19 | 0.904 / 0.21 |
| logistic | 0.988 | 0.018 | 0.004 | 0.95 | 0.923 / 0.62 | ||
| RAID (attack=none) | 149,977 | heuristic | 0.634 | 0.178 | 0.015 | 0.92 | 0.901 / 0.60 |
| logistic | 0.783 | 0.142 | 0.010 | 0.92 | 0.901 / 0.62 | ||
| M4GT (English) | 151,005 | heuristic | 0.645 | 0.224 | 0.005 | 0.96 | 0.909 / 0.79 |
| logistic | 0.870 | 0.142 | 0.008 | 1.01 | 0.902 / 0.88 | ||
| M4GT (multilingual) | 205,233 | heuristic | 0.689 | 0.216 | 0.005 | 0.98 | 0.951 / 0.90 |
| logistic | 0.808 | 0.178 | 0.004 | 1.01 | 0.907 / 0.89 | ||
| EvoBench | 29,574 | heuristic | 0.771 | 0.169 | 0.020 | 0.87 | 0.915 / 0.76 |
| logistic | 0.970 | 0.054 | 0.007 | 0.99 | 0.905 / 0.86 |
Real adversarial attacks. RAID’s attack families replace the proxy edits of §12 with published adversarial transforms, evaluated as transport cells in both directions (train clean → score attacked, and train attacked → score clean; all cells in cards/measurement-protocol-raid.json). Two attacks break the logistic tier outright: paraphrase collapses clean-trained detection to AUROC 0.495 — exactly chance — and zero-width-space insertion inverts the ranking (AUROC 0.157, calibration slope −0.48), because the surface statistics the tier trusts are destroyed while the text remains human-readable. The remaining nine families (synonym, homoglyph, whitespace, article deletion, alternative spelling, number, upper/lower, perplexity-guided misspelling, paragraph insertion) leave AUROC between 0.75 and 0.88. Detectors that look strong on clean benchmarks can be fragile or even anti-correlated under cheap attacks; the robustness claim a deployment actually needs must name the attack family.
The transport matrix, systematic. The two-point corpus ↔ Defactify result above generalizes to the full train-cohort × test-cohort grid over all six cohorts (logistic tier, source calibration partition only, target untouched; diagonal cells are within-cohort group-disjoint holdouts; large cohorts group-subsampled to 60,000 rows). Table 8 reports AUROC; the signed artifact (backend/artifacts/transport-matrix-external.json) carries Brier, ECE, calibration slope, and selective risk per cell. The diagonal is strong (0.78–0.99); the off-diagonal is where calibration goes to die — cross-cohort AUROC spans 0.23–0.84 with calibration slopes from −0.61 to 1.37 (negative slope means the transported score is anti-correlated with the truth), and the corpus row, fitted on n = 104, transports nowhere (0.41–0.74 off-diagonal). The single worst cell, EvoBench → M4GT, inverts (AUROC 0.23): a detector fitted on model-version shift actively misranks M4GT’s multilingual generations. Transportability is a property of the (train, test) pair, not of the detector.
| train ↓ · test → | corpus | Defactify | EvoBench | M4GT | M4GT-ml | RAID |
|---|---|---|---|---|---|---|
| corpus | 0.67 | 0.54 | 0.74 | 0.40 | 0.46 | 0.50 |
| Defactify | 0.83 | 0.99 | 0.69 | 0.69 | 0.71 | 0.62 |
| EvoBench | 0.56 | 0.61 | 0.97 | 0.23 | 0.42 | 0.57 |
| M4GT | 0.57 | 0.67 | 0.38 | 0.87 | 0.75 | 0.50 |
| M4GT-ml | 0.70 | 0.81 | 0.63 | 0.83 | 0.81 | 0.58 |
| RAID | 0.74 | 0.58 | 0.84 | 0.42 | 0.57 | 0.78 |
The measurement protocol of §2–4 is now executable. On the 104-record verified corpus, python -m bench measure --data corpus produces a signed card (backend/artifacts/cards/measurement-protocol.json) under nested grouped cross-validation (16 prompt groups, 5 outer folds). Train, calibration, and test groups are disjoint; isotonic calibration is never fit on the outer test fold. Table 9 reports the protocol scores. These numbers are not the older GroupKFold-without-a-calibration-partition scores in §9: they are lower, as they should be, because the calibration layer cannot see the test groups.
| Detector | AUROC | AUPRC | Brier | ECE | Slope | TPR@1% FPR | Conformal coverage |
|---|---|---|---|---|---|---|---|
| heuristic | 0.624 | 0.948 | 0.077 | 0.082 | 0.10 | 0.542 | 0.923 |
| logistic tier-0 | 0.686 | 0.955 | 0.078 | 0.059 | 0.12 | 0.615 | 0.923 |
Selective risk. At full coverage the logistic selective risk is 0.077; at 50% coverage it falls to 0.038 (Figure 10). Abstention is doing work: the system can refuse the least confident half of documents and cut error. Coverage is evaluated on the pooled untouched test folds, not on the calibration partition used to fit isotonic regression.
Human–AI mixtures. Pairing each human control with the AI response to the same prompt yields 48 pairs. Token-splice mixtures at 0, 10, 25, 50, 75, 90, and 100% AI contribution produce a high rank correlation (r = 0.99) but a near-flat slope (0.029): estimated participation sits around 0.70 for every mix rate (Figure 11). The heuristic orders the mixes correctly and does not track the controlled rate. That is the scientifically useful result of a coauthoring pilot on eight human documents: ranking is not calibration. AI-suffix and interleaved workflows give the same qualitative picture (signed card cards/mixture-workflows.json). Mean likelihood ratios on this corpus explode when a calibrated probability saturates near 1; we report them as a pathology of small-n isotonic maps, not as transportable evidence.
Unknown-generator rejection. Holding out each of the six AI families in turn, Mahalanobis distance to the remaining family centroids yields mean unknown-rejection AUROC 0.515 — chance (Figure 12). On this corpus, source-family geometry cannot reject an unseen generator. The protocol requires that result to be reported rather than replaced with nearest-family assignment.
Editing robustness (proxy). Truncating documents to 75%, 50%, or 25% of tokens drops heuristic AUROC from 0.673 to approximately 0.31; dropping 20% of tokens or stripping punctuation does the same. Lowercasing and sentence shuffle do not (Table 10). These are not RAID or DIPPER attacks; they are the edits this environment can apply to the hash-verified corpus. They already show that length and surface form move the score more than sentence order. Signed card: cards/robustness-pilot.json.
| Transform | AUROC | Δ AUROC | Notes |
|---|---|---|---|
| identity | 0.673 | 0.000 | unedited |
| truncate 75 / 50 / 25% | 0.312 / 0.305 / 0.306 | −0.36 | length collapse |
| drop 20% tokens | 0.315 | −0.36 | surface deletion |
| lowercase | 0.678 | +0.005 | survives |
| strip punctuation | 0.327 | −0.35 | surface collapse |
| shuffle sentences | 0.672 | −0.001 | order-invariant |
Watermark subsystem. The public KGW-style adapter, run as a separate test on unwatermarked corpus text, rejects 37.5% of the eight human controls at p < 0.05 (n = 8) and 10.9% of AI documents. That is Type I behavior of a public detector on ordinary text, not detection of a known watermark. Truncation and token drop further change reject rates. These numbers must not be added to the passive-attribution claim. Signed card: cards/watermark-degradation.json.
The protocol at scale. Every measurement above was then re-run at external-benchmark scale under the same frozen protocol (signed cards cards/measurement-protocol-{defactify,raid,m4gt,m4gtml,evobench}.json; cohort details in §11). Three findings survive the move from n = 104 to n = 29,574–205,233. First, the unseen-generator null is not a small-n artifact. Leave-one-family-out on Defactify’s six LLM families yields unknown-rejection AUROC 0.39–0.68 (mean 0.53) — the corpus pilot’s 0.515 replicated at 700× the sample size, with the same conclusion on RAID (0.32–0.76 over eleven families) and M4GT (0.35–0.76). Source-family geometry cannot reject a generator it has not seen, at any scale we can measure. Second, pooled conformal coverage hides a class-conditional failure. Every cohort’s pooled 90% conformal coverage lands on target (0.90–0.95), but slicing by class exposes a systematic deficit: the human slice is under-covered everywhere — 0.62 versus 0.96 for AI text under the Defactify logistic tier, 0.62 versus 0.99 on RAID, 0.86–0.90 versus 0.92 on EvoBench and M4GT — while length-bucket and generator-family slices stay near nominal. Marginal coverage is achieved by over-covering the majority (AI) class at the expense of the class whose false accusation matters most. Sliced coverage is now a first-class protocol output for exactly this reason. Third, calibration contrasts are now powered analytically. Each card carries a power block with normal-approximation required-n for Brier and ECE contrasts alongside the accuracy gate: detecting a 0.01 Brier difference at 80% power needs n = 848 under the Defactify logistic tier (n = 3,598 on M4GT), and the same ECE contrast needs roughly 4–8× more — the n = 104 corpus was underpowered for calibration by two orders of magnitude, which is now quantified in the card rather than asserted in prose.
External detector baselines, scored. Binoculars (Falcon-7B observer/performer pair) and Fast-DetectGPT (GPT-J-6B) were run on the frozen protocol splits as zero-shot adapters — never trained on the bench data, isotonic calibration fitted on the calibration partition only, test groups untouched (signed card below). DetectGPT classic is recorded failed with reason (its ~20 T5 perturbation passes per text are prohibitive at Defactify scale); Fast-DetectGPT, the same curvature family without a perturbation model, is scored in its place. The transformer classifier remains registered-unavailable (no fine-tuned weights) and is not replaced by the Panoptes heuristic. Table 11 reports the comparison. Two honest findings emerge. First, at corpus scale the zero-shot detectors win. With only n = 104 there is too little data to train a competitive supervised tier, so Binoculars (AUROC 0.835) and Fast-DetectGPT (0.734) sit well above the logistic tier (0.686). Second, at Defactify scale the relationship inverts. The logistic tier reaches AUROC 0.988 against Binoculars 0.877 and Fast-DetectGPT 0.930, with the best calibration of the four (ECE 0.004, slope 0.95). Neither result is a transportability claim: §11’s matrix shows every one of these detectors collapses off-diagonal, so in-domain superiority — at either scale — says nothing about cross-cohort generalization. Signed card: backend/artifacts/external-baselines.json.
| Cohort | Detector | AUROC | AUPRC | Brier | ECE | Cal. slope |
|---|---|---|---|---|---|---|
| corpus (n = 104) | Panoptes heuristic | 0.624 | 0.948 | 0.077 | 0.082 | 0.099 |
| Panoptes logistic | 0.686 | 0.955 | 0.078 | 0.059 | 0.124 | |
| Binoculars (zero-shot) | 0.835 | 0.985 | 0.066 | 0.047 | 0.356 | |
| Fast-DetectGPT (zero-shot) | 0.734 | 0.974 | 0.071 | 0.033 | 0.321 | |
| Defactify (protocol splits) | Panoptes heuristic | 0.885 | 0.981 | 0.064 | 0.029 | 1.185 |
| Panoptes logistic | 0.988 | 0.998 | 0.018 | 0.004 | 0.952 | |
| Binoculars (zero-shot) | 0.877 | 0.978 | 0.070 | 0.017 | 0.910 | |
| Fast-DetectGPT (zero-shot) | 0.930 | 0.990 | 0.058 | 0.014 | 0.868 |
Watermark-removal robustness. The watermark subsystem above measures Type I behavior on ordinary text. Here we ask the adversarial question: does a removal tool defeat the watermark? Because no watermarked corpus exists, we generated one — 96 passages from a small language model sampled with the KGW green-list logit bias under the demo key (detected at p < 0.05 in 96/96 cases, mean z = 7.1), plus 96 matched unwatermarked controls (empirical FPR 0.083) — then applied a battery of removal attacks and measured how much detection survives. We test two families: the statistical green-list watermark (the Aaronson/SynthID-Text family that Claude now uses [37, 38]) and a Unicode zero-width watermark, the family that “Unicode hygiene” removers target. Table 12 reports retention. Signed cards: cards/watermarked-generations.json, cards/watermark-removal.json.
| Attack | KGW detection | Unicode present | Passive AUROC (Δ) |
|---|---|---|---|
| none (baseline) | 1.000 | 1.000 | 0.673 (—) |
| Unicode scrub (hygiene) | 1.000 | 0.000 | 0.673 (+0.00) |
| synonym substitute 30% / 60% | 1.000 / 1.000 | 1.000 / 1.000 | 0.460 / 0.430 (−0.21 / −0.24) |
| truncate 50% | 0.917 | 0.000 | 0.305 (−0.37) |
| drop 20% tokens | 0.958 | 0.000 | 0.315 (−0.36) |
| shuffle sentences | 1.000 | 0.000 | 0.672 (−0.00) |
| lowercase | 1.000 | 1.000 | 0.678 (+0.01) |
| strip punctuation | 1.000 | 0.000 | 0.327 (−0.35) |
| LLM paraphrase (complete rewrite) | 0.417 | — | — |
| watermarks-remover (external, Unicode hygiene) | 1.000 | 0.000 | — |
Three findings. First, the two watermark families have disjoint attack surfaces. Unicode hygiene destroys the zero-width watermark (100% → 0%) but leaves the statistical watermark fully detectable (100% → 100%); synonym substitution does the reverse, preserving the invisible characters while it perturbs word choice. A remover that handles one family says nothing about the other. Second, the statistical watermark is robust to everything short of a rewrite. Hygiene, truncation, token drop, shuffle, case, and punctuation edits all leave it detectable; only the complete LLM rewrite moves it, cutting detection from 100% to 42% and mean green-rate from 0.82 to 0.56 (near the 0.50 chance floor). This is exactly the behavior Anthropic describes for Claude’s watermark — “light editing probably won’t remove the watermark completely” while “a complete rewrite where every word is replaced will” [38] — reproduced here against a green-list scheme whose key we control. Third, watermark removal and passive evasion are orthogonal. Synonym paraphrase barely dents the statistical watermark yet craters passive AUROC (0.67 → 0.43), while hygiene destroys the Unicode watermark without moving passive detection at all. The two defenses measure different things and are best deployed together, which is why Panoptes keeps the watermark claim separate from the passive-attribution claim.
Evaluating external systems from a git repo. The watermarks-remover row above was produced not by reimplementing the tool but by pointing Panoptes at its repository [39]: python -m bench evaluate-repo <git-url> --kind watermark-remover clones the repo into an isolated directory, resolves a small adapter contract (a panoptes.adapter.json or panoptes_adapter.py exposing a conventional callable — transform(text) for removers, detect(text) for watermark schemes, score(text) for passive detectors), runs it in a subprocess against our fixtures, and signs the result (cards/external-repo-watermark-remover.json). The same harness evaluates any watermark scheme or passive detector directly from source, so third-party claims become reproducible signed cards rather than anecdotes. Because this executes arbitrary cloned code, it is CLI-only, subprocess-isolated, and documented as run-only-repos-you-trust (with an optional network-disabled --docker mode); see docs/testing-external-repos.md.
The v2.0 corpus established the contribution and verification machinery but was too small to power the transportability claim. Version 2.1 expands the evidence base into a pooled, multi-dataset corpus assembled under an explicit license and derived-weight gate, and locks it in two machine-readable artifacts registered before any outcome-bearing run: a dataset registry (backend/artifacts/dataset-registry-v2.1.json) recording each cohort’s role, license, task semantics, group unit, provenance, row count, and content hash, and a split manifest (backend/artifacts/split-manifest-v2.1.json) locking the train/calibration/test group assignments per cohort. The registry is the audit trail for the central data decision of v2.1: a cohort may train the public model only if its license permits redistribution of derived weights.
The license gate partitions the corpus by role. Three cohorts pass the derived-weight gate and form the pooled public-weight training pool: MAGE [40] (Apache-2.0; multi-domain, multi-generator, with publisher train/validation/test/OOD identities preserved), RAID clean [10] (MIT; the attack = none arm only), and DeFactify [34] (CC-BY-4.0; NYT human articles versus six LLM families, re-split by reconstructed story group because the official splits are story-leaky). Cohorts whose licenses are unclear or non-redistributable are evaluation-only and never train the public weights: M4GT-Bench English and multilingual [44] (NOASSERTION), EvoBench [36] (NOASSERTION), and CoAuthor [41] (NOASSERTION, and a mixed human–AI task rather than a binary contrast). RAID’s eleven attack arms and MAGE’s OOD/paraphrase splits are held out as sealed robustness tests. Reference Community Corpus v0 (§6) is retained as the contribution-framework reference, not a powered cohort.
| Cohort | Role | License | Group unit | n |
|---|---|---|---|---|
| MAGE train | training | Apache-2.0 | near-duplicate cluster / aligned source pair | 15,000 |
| RAID clean | training | MIT | source (prompt × generator) | 14,918 |
| DeFactify | training | CC-BY-4.0 | reconstructed NYT story | 5,938 |
| MAGE OOD / OOD-paraphrase | evaluation / robustness | Apache-2.0 | near-duplicate cluster | 1,518 / 2,340 |
| RAID attacks (11 families) | robustness | MIT | source (prompt × generator) | ≈3,957 / family |
| M4GT English / multilingual | evaluation | NOASSERTION | source document | 4,000 / 4,000 |
| EvoBench | evaluation | NOASSERTION | prompt × family | 4,000 |
| CoAuthor (held-out authors) | mixed task | NOASSERTION | author (worker id) | 236 |
| Reference Community Corpus v0 | reference | NOASSERTION | record id | 104 |
Leakage disjointness is enforced on the group unit, not the row. Each cohort declares the unit that must not straddle a split — a near-duplicate cluster or aligned human–machine source pair for MAGE, a prompt×generator source for RAID, a reconstructed story for DeFactify, an author for CoAuthor. The split manifest assigns every group to exactly one of train, calibration, or test, and a global exact-hash and MinHash near-duplicate index is built across all cohorts before assignment so that a near-duplicate of a training document cannot leak into a held-out cell. The calibration partition is group-disjoint from both the pooled training rows and the pilot’s development subsample, so the calibrator never sees a group that informed model selection.
Task semantics are kept distinct per cohort. The pooled training cohorts are binary (human versus fully machine). CoAuthor is a mixed human–AI collaboration task: every session is co-written, so it is evaluated on the participation and contribution-fraction heads against held-out authors and is never treated as a binary fully-AI positive. M4GT multilingual is an external failure/abstention test for the English-scoped model, not training data. This per-cohort task typing is what allows the evidence ledger of §4 to keep participation, generation, and contribution fraction as separate claims rather than collapsing them into one score.
The classical tiers of §10 are interpretable and cheap, but they are not the strongest instrument the bench can build. Version 2.1 adds a preregistered neural detector whose architecture, objective, windowing, aggregation, seeds, and acceptance gates were frozen after a train/development-only pilot, before any final-test access. The pilot compared three encoders — deberta-v3-base [42] (stable primary), ModernBERT-base (long-context challenger), and deberta-v3-small (latency/CPU control) — under three objectives (ordinary ERM, group-balanced, and GroupDRO [43] over dataset×domain×generator×label groups) and two document aggregations (overlap-corrected logit mean versus a small hierarchical summary head). Selection was lexicographic on locked development criteria: maximize worst-cohort AUROC, then minimize worst-cohort Brier, then latency. The frozen winner is deberta-v3-small, group-balanced objective, hierarchical summary head, 512-token windows with 128-token overlap. Every pilot run, including failed candidates, is recorded in the signed selection card (backend/artifacts/cards/neural-pilot.json).
Architecture. A document is split into tokenizer-aligned, overlap-corrected windows of at most 512 tokens (capped at 16 windows); each window is encoded once in a batched call, and a learned hierarchical summary head aggregates the window representations into a single participation logit. This windowed-aggregation design sits between two alternatives in the literature — long-context encoders that attend over the whole document in one pass (Longformer [45], BigBird [46]), and single-window classifiers that truncate to the opening tokens (the standard RoBERTa-MAGE baseline [47]) — preserving long-document evidence without sparse-attention memory cost, and the learned summary head outperformed overlap-corrected logit averaging in the pilot. The output ontology matches the protocol rather than forcing a binary score into a ternary response: a calibrated participation head estimates P(any substantial AI contribution), and the human / ai_refined_or_mixed / ai_generated distribution is constructed monotonically so that generation probability cannot exceed participation probability. Because the pooled training pool is binary, the majority-generation and contribution-fraction heads are external-evaluation-only outputs (§15).
Training under the data firewall. The winner was trained with three seeds {13, 42, 87} on the pooled public-weight training pool (35,856 documents: MAGE 15,000, RAID clean 14,918, DeFactify 5,938), each seed using a group-disjoint inner-development split for early stopping, in mixed precision (fp32 master weights, bf16 autocast) on a single RTX 3090 (≈24–36 min per seed). The isotonic calibrator, split-conformal thresholds, low-FPR operating points, and selective thresholds were then fit on a pooled calibration partition of 14,572 documents that is group-disjoint from both the pooled training rows and the pilot development subsample. No final-test label informed any choice. The three seeds agree closely (calibration AUROC 0.9903–0.9920; worst-cohort AUROC 0.9823–0.9833), so the ensemble is the mean of three well-calibrated, mutually consistent scorers.
| Quantity | Value | 95% group-bootstrap CI |
|---|---|---|
| Ensemble AUROC | 0.9929 | 0.9916–0.9942 |
| Worst-cohort AUROC | 0.9854 | — |
| Brier (calibrated) | 0.0292 | — |
| Worst-cohort Brier (calibrated) | 0.0457 | — |
| Adaptive ECE | 0.0030 | 0.0017–0.0074 |
| Split-conformal coverage (α = 0.1) | 0.900 | by class: 0.900 / 0.900 |
| Threshold at 1% / 0.1% FPR | 0.947 / 0.984 | — |
Runtime and fallbacks. The frozen ensemble ships as three seed checkpoints plus a shared architecture config, tokenizer, and a signed ensemble manifest that records every weight hash; a lazy, singleton, concurrency-safe model manager re-verifies each SHA-256 before loading, rebuilds the encoder offline from the local config (never re-downloading the base model), and batches each document’s windows in one call. Runtime profiles degrade gracefully: the three-seed ensemble on GPU, a single-seed balanced profile, and a CPU/no-model fallback to the calibrated logistic tier and then the heuristic. Code and unsupported languages never route through the English prose model. A separate applicability diagnostic — built from calibration geometry and segment/seed disagreement — may trigger abstention but is reported as a descriptive signal, not as proof that all shifts are detectable. The weights are released publicly at huggingface.co/MarketStandard/panoptes-neural-v1 (immutable revision 76bf26782c783506c9af24d8277be0e1b708d243) only because the preregistered performance, calibration, latency, license, and reproducibility gates passed; had any gate failed, the logistic tier would have remained the default and the negative result would be reported here.
In-domain strength is not the thesis. Table 14 is a strong in-domain calibration result, but it is measured on seen cohorts. The scientific question of this paper is whether that evidence transports to populations the detector never saw — which is exactly what §15 measures by holding entire datasets out of both training and calibration.
Sections 10–14 establish that the detectors are strong in-domain. This section asks the question the bench was rebuilt to answer: does the evidence transport? We preregistered three analyses that are easy to conflate but answer different questions, and we report them separately throughout. Representation transport (leave-one-cohort-out) trains a detector on every cohort except one, then evaluates on the held-out cohort the model never saw — this measures whether the learned representation generalizes across a distribution shift. Calibration transfer freezes the scorer and re-fits only the calibrator on each cohort in turn, measuring whether a probability that was calibrated on one population stays calibrated on another. Pooled generalization trains on the union of the licensed cohorts and evaluates once on a sealed, group-disjoint test partition. Every threshold, calibrator, and conformal level is fit on calibration partitions only; no final-test label sets any operating point. All four detector tiers — heuristic, logistic, GBM, and the frozen neural ensemble of §14 — run the identical protocol through the same harness.
Representation transport is the hard case, and it is where classical detectors fail. Table 15 holds out each licensed dataset in turn. The pattern is consistent: a detector that is strong on its own cohorts loses a third to a half of its discrimination when an entire dataset is removed from training and calibration. The logistic tier, which achieves 0.91 AUROC on seen DeFactify text, drops to 0.60 when RAID is held out and 0.61 when MAGE is held out. GBM is the most robust classical tier but still falls to 0.62 on held-out DeFactify. The heuristic tier is essentially at chance on held-out MAGE (0.52). The neural ensemble — trained on the pooled pool with a group-balanced objective — transports markedly better, as the right column shows.
| Held-out dataset | Heuristic | Logistic | GBM | Neural (§14) |
|---|---|---|---|---|
| MAGE | 0.524 | 0.613 | 0.669 | 0.819 |
| RAID clean | 0.619 | 0.596 | 0.757 | 0.929 |
| DeFactify | 0.762 | 0.847 | 0.620 | 0.658 |
| Mean | 0.635 | 0.685 | 0.682 | 0.802 |
| Worst | 0.524 | 0.596 | 0.620 | 0.658 |
The transport gap is partly a shortcut, and we can measure that directly. A dataset-origin probe — a cross-validated classifier that predicts which dataset a document came from, using only the detector’s input features and never the label — achieves 0.81 accuracy against a 0.33 chance baseline on the three-cohort axis, and 0.46 against 0.10 on the ten-domain MAGE axis. Dataset identity is therefore strongly readable from the feature space, which means part of every detector’s in-domain strength is a dataset shortcut rather than a human-versus-machine signal. This is precisely the confound the leave-one-cohort-out design controls for: by removing an entire dataset from training, the shortcut becomes useless and the measured transport gap isolates the generalizable component. We report the probe alongside the transport numbers so the reader can see how much of the in-domain result is load-bearing.
Within-MAGE leave-one-domain-out is the strict version of the same question. Holding out each of MAGE’s ten domains in turn (Table 16) is harder than holding out a dataset, because the domains share a generator mix. GBM is the only classical tier that stays above 0.80 mean (0.81, worst domain 0.60 on HellaSwag); the logistic tier averages 0.64 and the heuristic tier 0.51, with several domains at or below chance. The hardest domain for every tier is HellaSwag sentence-completion, where the human and machine text are shortest and most stylistically similar. The neural tier is omitted from this table: a within-MAGE fold re-trains the encoder on nine domains, and ten such re-trainings were not preregistered, so we report the neural tier on the cross-dataset and external axes only rather than imply a number we did not measure.
| Analysis | Heuristic | Logistic | GBM | Neural (§14) |
|---|---|---|---|---|
| Within-MAGE LOCO (mean) | 0.508 | 0.636 | 0.809 | n/a |
| Within-MAGE LOCO (worst) | 0.399 | 0.498 | 0.601 | n/a |
| Pooled test — DeFactify | 0.778 | 0.914 | 0.947 | 1.000 |
| Pooled test — MAGE | 0.490 | 0.609 | 0.834 | 0.984 |
| Pooled test — RAID | 0.629 | 0.734 | 0.940 | 0.988 |
Calibration transfers better than representation, but not for free. Freezing each scorer and re-fitting only the isotonic calibrator on each cohort in turn yields the transfer matrix of calibration-transfer.json; Table 17 shows the frozen neural ensemble’s matrix. For GBM the off-diagonal Brier score degrades only from 0.091 (self-calibrated) to 0.096 (cross-calibrated), and the AUROC is unchanged to three decimals — the score ranking is stable across cohorts even when the absolute probabilities shift. The neural matrix makes the same point more sharply: moving the calibrator off-diagonal changes AUROC by at most 0.05 (and usually far less), while the off-diagonal Brier stays within 0.011 of the self-calibrated diagonal. The logistic and heuristic tiers degrade more (off-diagonal Brier 0.17 and 0.22 against on-diagonal 0.14 and 0.16). The practical reading: a detector that ranks well can be re-calibrated to a new population with a modest labelled sample, but the ranking itself must transport first, which is the harder problem of Table 15.
| Calibrator ↓ / Eval → | DeFactify | MAGE | RAID |
|---|---|---|---|
| DeFactify | 0.003 | 0.058 | 0.027 |
| MAGE | 0.003 | 0.047 | 0.023 |
| RAID | 0.002 | 0.058 | 0.021 |
External and adversarial targets are the honest stress test, and they invert the in-domain ranking in both directions. On sealed, never-trained targets the neural ensemble is the strongest tier almost everywhere: EvoBench 0.93, MAGE OOD 0.97, and — where the MAGE-and-RAID-trained logistic tier falls below chance (0.40) — M4GT-English 0.93. The neural tier is also the only one that is robust to RAID’s paraphrase attack (0.98, versus 0.48–0.76 for the classical tiers), the attack that the v2.0 study identified as the classical detectors’ principal weakness; on MAGE’s out-of-distribution paraphrase split it degrades to 0.76, comparable to the classical tiers, so the robustness is specific to paraphrase of seen-distribution text. But the same learned representation that confers paraphrase robustness is more vulnerable to a different attack family: on RAID’s homoglyph substitution the neural ensemble collapses to 0.26 while the feature-based GBM tier is essentially unaffected (0.97), and on zero-width-space insertion every tier, neural included, falls below chance (0.12–0.36). Unicode-level perturbations that survive tokenization are the neural detector’s blind spot; semantic perturbations are the classical detectors’. These are not failures of the harness — they are the preregistered, sealed measurements of where each representation stops working, and they are exactly the populations a deployment would need to abstain on. The full per-target rows are in Table 18.
| External target | Heuristic | Logistic | GBM | Neural (§14) |
|---|---|---|---|---|
| EvoBench | 0.722 | 0.818 | 0.934 | 0.933 |
| M4GT English | 0.631 | 0.404 | 0.805 | 0.927 |
| M4GT multilingual | 0.668 | 0.552 | 0.813 | 0.872 |
| MAGE OOD | 0.725 | 0.739 | 0.672 | 0.970 |
| MAGE OOD paraphrase | 0.699 | 0.718 | 0.631 | 0.762 |
| RAID paraphrase | 0.555 | 0.484 | 0.759 | 0.982 |
| RAID homoglyph | 0.609 | 0.739 | 0.967 | 0.258 |
| RAID zero-width-space | 0.364 | 0.169 | 0.121 | 0.238 |
| Reference Corpus v0 | 0.708 | 0.704 | 0.785 | 0.835 |
CoAuthor is the participation-head stress test, not a binary cell. Every one of the 236 held-out-author sessions is human–GPT-3 collaborative, so the true participation is 1 and there is no human-versus-machine contrast to rank — no AUROC is reported, and CoAuthor never enters the binary transport cells above. The question is whether the participation head fires on lightly assisted text: the mean AI contribution fraction is 0.21, so these are mostly-human documents. All four tiers under-fire relative to the true participation of 1, and the ranking inverts the binary pattern — the classical tiers predict mean participation 0.60–0.63, while the neural ensemble, strongest on every binary external target, is the most conservative at 0.43. This is the expected, honest behavior for detectors trained on a binary human-versus-fully-machine pool: a document that is 21% machine is not the binary positive they were calibrated on, and the pooled pool licenses no mixed-authorship head. CoAuthor therefore measures a different claim — participation under real co-writing — and is reported separately, with the contribution-fraction head remaining external-evaluation-only until a terms-cleared mixed-authorship cohort exists (§21).
What the transport analysis establishes. Three claims survive the move to held-out populations. First, representation transport — not in-domain accuracy — is the binding constraint, and it separates the neural ensemble from every classical tier: the neural mean held-out AUROC is 0.802 against 0.64–0.69 for the classical tiers, and its worst held-out cohort (0.658) is above every classical tier’s mean. Second, calibration is cheaper to move than representation: the neural calibrator’s off-diagonal Brier degrades only 0.024→0.028, so a detector whose ranking transports can be re-calibrated to a new cohort with far less data than it took to train. Third, the dataset-origin probe quantifies how much of any in-domain number is shortcut, which is why this paper reports leave-one-cohort-out — not pooled in-domain — as the headline transport result. The residual frontier is now Unicode-level adversarial perturbation (homoglyph, zero-width-space), where the neural representation is the most vulnerable tier, not paraphrase or held-out generators, on which it is the strongest. Section 21 discusses these as limitations.
Green-list watermarks live in the sampling step: a logits processor biases candidates after top-k/top-p filtering. API temperature therefore controls whether a watermark can embed at all. We sweep temperature (0.0, 0.3, 0.7, 1.0, 1.3) against matched delta (0, 2) generations under the Panoptes demo key and score with KGWReferenceAdapter. The signed card backend/artifacts/cards/watermark-temperature.json reports per-cell detection rate, mean z, and power. At temperature 0 (greedy), sampling randomness vanishes: with delta > 0 the bias can still flip the argmax and embed a deterministic mark; with delta = 0 there is nothing to watermark. At moderate temperatures (0.3–1.3) watermarked cells retain high detection under the demo key while controls stay near the Type I floor. Providers that floor temperature, skip watermarking greedy calls, or apply deterministic logit bias make different quality/embeddability trade-offs; Anthropic has not disclosed which path Claude takes. The card is a family characterization, not a measurement of any private production key.
If a student is trained on watermarked teacher outputs, the green-list bias can transfer — the “radioactivity” property [46, 47]. We evaluate inheritance against our own adapter with a signed card (backend/artifacts/cards/radioactivity.json). The default CPU tier fits a bigram student on synthetic KGW-biased teacher text (CI-friendly); an optional --with-model path runs tiny Hugging Face SFT, and GPU-tier flags accept 7B-class teacher/student pairs. On the synthetic card, students trained on watermarked teachers show elevated detection relative to control-trained students; pre-distillation synonym paraphrase attenuates the signal; post-distillation neutralization (inverse green-list bias at decode with the known demo key) drives mean z strongly negative. These results align with ACL 2025 removal findings [48]: watermarks deter lazy distillation but do not robustly prevent a motivated adversary who paraphrases training data or neutralizes at inference. A positive radioactivity hit is lineage-compatible evidence, not proof of unauthorized distillation — web-scale scraping can contaminate unrelated models with weak radioactivity (§18).
Once providers emit watermarked text into APIs and the public web, calibration corpora absorb a confound that can look like “Claude-like” stylometry or a weak watermark hit. Panoptes therefore records an optional watermark block on baseline-run and dataset manifests (declared-none, declared-active, suspected, unknown). Anthropic runs on or after 2026-08-02 default to suspected unless overridden. The corpus summary aggregates per-cohort status and lists contaminated_cohorts; a known-scheme smoke screen (bench/watermark_screening.py) runs only public adapters and cannot read private vendor keys. On the frozen 104-record corpus the screen flags three of eight human controls at α = 0.05 under the demo key (marginal z ≈ 2.2–2.3) while AI records sit symmetric about zero (mean z = 0.02) — an integrity signal to monitor as the corpus grows, not an authorship claim. The live analyzer appends a calibration limitation when the loaded bundle carries contamination metadata. See docs/watermark-contamination.md. The runtime plugin loader (panoptes.plugins) additionally lets operators attach local watermark detectors under the plugin:<id> namespace without changing the evidence contract.
RQ4 asks whether the claim can be verified from committed artifacts. This version ships a first-party self-check, not an outside reproduction. python -m bench.reproduce re-hashes every signed artifact, re-verifies the baseline catalog, and writes backend/artifacts/reproduction-selfcheck.json. On this checkout, 42 of 42 signed hashes match and the catalog verifies 12 registered runs. The self-check is labeled independent: false because the authors ran it. An outside researcher should follow docs/independent-reproduction.md, record original versus recomputed headline metrics, and report absolute deltas. Until that is done, RQ4 is only partly answered: the artifacts are machine-checkable; they have not yet been checked by someone who did not produce them.
The project’s contribution is infrastructure for evidence that compounds. Concretely: (1) a canonical prompt set and hashing protocol under which anyone’s run becomes permanently comparable to everyone else’s; (2) a public catalog that stores cryptographic commitments instead of content, so contributing evidence never requires disclosing model outputs or personal text; (3) a calibration loop in which contributed hashes, once re-verified, refit the artifact the live analyzer actually loads — the corpus is an input to the system, not an ornament; (4) a training bench with a published power gate, so model complexity grows only when the data can arbitrate it; (5) a pre-registered hypothesis registry that turns “we tested what we felt like” into “these six tests, these decisions, re-run when the data doubles”; and (6) an evidential reference architecture whose vacuity and dissonance outputs make “the model doesn’t know” a first-class, displayable result. Version 2.1 adds three more. (7) A license-audited dataset registry and split manifest (backend/artifacts/dataset-registry-v2.1.json, backend/artifacts/split-manifest-v2.1.json) make the training pool’s legal status and its group-disjoint train/calibration/test partitions machine-checkable, so a contributed dataset slots in with its license, group unit, and provenance declared. (8) A frozen, preregistered neural detector — selected by a train/development-only pilot, hash-verified at load, and released at MarketStandard/panoptes-neural-v1 with a signed model card — provides a strong, reproducible reference instrument rather than a bespoke per-paper model. (9) An evidence-transportability harness runs any detector through leave-one-cohort-out, calibration-transfer, and sealed external-target analyses under the same data firewall, so a new cohort or a new model is measured against the same transport question. Each layer is usable without the next: the analyzer runs offline with heuristic fallbacks, the bench runs on the shipped corpus or your own CSV, and the catalog verifies with one command and no account.
Text-only attribution is fundamentally bounded. Detector likelihood ratios inherit the calibration and bias of the underlying models and degrade under paraphrase, translation, and heavy editing [11, 12]; the reliability error is reported precisely so that this degradation is visible. Watermark tests cover only known, configured schemes and only text that survived editing; a negative result under low power is weak evidence, and the interface says so. Provenance covers only files that carry credentials. In agent-assisted baseline runs, the model identity is self-reported by the runner; the catalog attests what was claimed and what resulted, not the ground truth of the serving stack — the byte-identical code-07 outputs of §6 are exactly the kind of signal this arrangement can and cannot interpret.
The statistical results carry their own bounds. The corpus is small (n = 104), English-heavy, and severely unbalanced at the label level (8 human controls against 96 AI outputs); every null result in §8 is consistent with both “no effect” and “no power,” and we report them as the latter until the corpus decides. The protocol measurement in §12 is honest on the same small set: mixture slope 0.029, unknown-rejection AUROC 0.515, and mean likelihood ratios that explode when isotonic scores saturate. Those are instrument readings, not publication-ready transport claims. The human controls were written by a small number of authors and are not a demographic sample. The v2.1 expansion closes several gaps that v2.0 listed as out of scope — RAID, M4GT, EvoBench, and real CoAuthor coauthoring sessions are now measured — while a dedicated 500–2,000-author human panel, DetectGPT/Binoculars weight release, and an outside reproduction remain open and are listed in docs/v2-updates/. GroupKFold by prompt controls topical leakage but cannot control author leakage within the human cohort. The corpus-fitted geometry describes six model families on sixteen prompts; it is a prior over a small world, and the unknown score exists because the world is larger. Panoptes-v0’s uncertainty estimates are calibrated on the same small corpus that trained it; its conformal coverage guarantee is marginal, not conditional, and says nothing about coverage on text unlike anything in the corpus. We make no leaderboard claims and no SOTA claims.
The Defactify results of §10 carry their own bounds. The dataset is New York Times news prose rewritten under a single prompt; the strong detection numbers describe that contrast and do not transfer automatically to other registers, to human–AI co-writing, or to adversarial paraphrase — the shipped heuristic’s AUROC of 0.648 on the same data is the standing reminder of what domain shift does to a detector fitted elsewhere. The upstream data contained API-error artifacts labeled as model output; we filter 412 of them at fetch time and report the counts, but we cannot exclude subtler labeling noise. Story groups are reconstructed by TF-IDF clustering, not observed, and 11.5% of the official test split leaks into train by that reconstruction — our grouped evaluation avoids the bias, but the reconstruction itself is a model. The Defactify repository declares no license separate from its CC BY 4.0 paper; we redistribute hashes, counts, and fitted parameters only, never raw text, and downstream users should review that posture before mirroring the data. The seven-class attribution experiment is exploratory: family cues may be prompt-specific, and family labels are self-declared generating models, not verified provenance.
The v2.1 neural detector and transport results carry their own bounds. The pooled training pool (MAGE train, RAID clean, and DeFactify train) is binary — human versus fully machine — so the conditional majority-generation and AI-contribution-fraction heads remain external-evaluation-only until a terms-cleared mixed-authorship cohort is available; the participation head is the only outcome the pooled data licenses. The three source datasets differ in provenance and label semantics: MAGE labels are publisher-assigned by generator, RAID clean is a controlled human/machine contrast, and DeFactify is a single-register NYT rewrite task, so the pooled calibrator is a population-conditional map over a heterogeneous mixture, not a universal prior. CoAuthor is GPT-3, English-only, and 63 authors; M4GT carries a NOASSERTION license and weak pairing groups, so it is evaluation-only. The neural detector is scoped to English prose; code and unsupported languages route to separate detectors and never through it. The dataset-origin probe (§15) reports how strongly the encoder can predict which dataset a document came from; where that signal is strong, observed transport gaps may reflect shortcut features rather than robust AI signal, and we report the probe rather than hide it. Calibration and split-conformal coverage are population-conditional and are not guaranteed after distribution shift; the cross-dataset numbers of §15 are the measurement of that degradation, not a refutation of it. Finally, public text detection remains vulnerable to paraphrase and to future generators, and provenance attests a chain of custody rather than metaphysical authorship — the two evidence channels are reported separately precisely so that neither is asked to carry the other.
Watermark intelligence (§§16–18) adds three further bounds. Private vendor keys remain undetectable: Panoptes characterizes the Aaronson/SynthID-Text family with a demo key, never Anthropic’s production key. Radioactivity is lineage evidence under web contamination, not courtroom proof of distillation. Greedy decoding (temperature 0) is a structural dead zone for sampling watermarks unless providers apply a deterministic logit bias. Plugin detectors are operator-local and unsigned unless they ship evaluation cards.
Panoptes is designed against the two failure modes that harm people: accusation on thin evidence, and false reassurance from a negative test. The system surfaces evidence state, sample size, power, and interpretation limits on every report, and its documentation prohibits using a single score as sole grounds for consequential action. Community submissions carry hashes only, so contributing evidence never requires disclosing model outputs or personal text. The power gate and the pre-registered hypothesis registry are ethical as well as statistical devices: they make it structurally difficult for the project — or anyone building on it — to claim more than the data supports.
Panoptes reframes AI-text attribution as a measurement and calibration problem rather than a detector leaderboard. The system reports prevalence-corrected likelihood ratios, declared priors, uncertainty, and abstention; it keeps participation, generation, and source-family attribution distinct; and it treats watermark tests and cryptographic provenance as separate subsystems. The project-corpus numbers remain deliberately unheroic: 104 records, hypotheses that do not clear multiplicity control, a neural architecture that ties its logistic baseline. The Defactify external validation shows what the same machinery concludes when the power gate passes — and, more importantly, what happens when the evidence distribution changes: in-domain AUROC near 0.998 collapses to 0.648 for the shipped detector on that domain and to 0.471 when the Defactify-trained model is transported back to the Panoptes corpus. That transportability gap, not the in-domain score, is the result this paper is built around, and the six-cohort transport matrix now makes it systematic: off-diagonal AUROC spans 0.23–0.84 with calibration slopes from −0.61 to 1.37, so a detector’s reputation is a property of the (train, test) pair. The same protocol-at-scale runs surface two further findings that pooled leaderboards hide: conformal coverage that is marginally correct can be class-conditionally wrong (the human slice is under-covered on every cohort), and published adversarial attacks do not merely degrade a strong tier — paraphrase reduces it to chance and zero-width-space insertion inverts it. The unseen-generator result is a replicated null: leave-one-family-out rejection stays near chance from n = 104 to n = 205,233. The scientifically stronger question is: under what conditions is AI participation statistically identifiable, and how does the evidence degrade when those conditions change?
Version 2.1 sharpens that gap into a systematic, preregistered measurement. A frozen three-seed hierarchical neural ensemble (§14), trained only on the license-clean pooled pool, reaches 0.993 AUROC in-domain; yet the leave-one-dataset-out analysis (§15) shows that this in-domain strength does not transport for free. Every classical tier falls to 0.52–0.76 AUROC on a held-out dataset, and a dataset-origin probe (0.81 accuracy against 0.33 chance) confirms that part of every in-domain number is a dataset shortcut rather than a human-versus-machine signal; the neural ensemble transports at 0.802 mean AUROC across the held-out datasets. Calibration is cheaper to move than representation — the neural calibrator’s off-diagonal Brier degrades only 0.024→0.028 — so a detector whose ranking transports can be re-calibrated to a new cohort with a modest labelled sample. The residual frontier is Unicode-level adversarial perturbation: homoglyph and zero-width-space insertion collapse every tier, and the neural representation — strongest on paraphrase and held-out generators — is here the most vulnerable. Transport, not in-domain accuracy, is the binding constraint, and it is now measurable.
The machinery described here — a frozen protocol, group-disjoint train/calibration/test splits, signed artifacts, power gates, and a hypothesis registry — exists so that the numbers can only get more meaningful from here. We invite replication: run the canonical prompts, hash your outputs, append your evidence to the catalog, and move the marker on the power curve.