ICSAC commited on
Commit
74b5f24
Β·
1 Parent(s): 435e513

Redaction: a vendor tied to a panel seat is fatal in prose too and the RQC's "Claude-position" reads "first-seat"; the boilerplate says human curators, not human editors

Browse files

The RQC on the first external paper called one seat "the Claude-position
reviewers"; the jargon rewrite covered "slot" but not that, and reviewer
prose sits outside the fatal vendor scan by design (a paper's own AI-use
disclosure is legitimate content there). A vendor or model name tied to a
seat is never a disclosure, so it is now fatal wherever it appears, and the
observed phrasing is rewritten before the gate. "Rest with human editors"
and its siblings in the review boilerplate, the RQC status line, the rubric,
the legacy templates and the README now say human curators or the curation
team.

README.md CHANGED
@@ -146,7 +146,7 @@ not a general-purpose academic peer review platform. It is not a service you
146
  can submit to without going through `icsacinstitute.org/submit`. It is not a
147
  replacement for human editorial judgment β€” every panel outcome routes to a
148
  human curator for the final verdict, and any author who disagrees with that
149
- verdict can appeal to a human editor.
150
 
151
  It is also opinionated. The rubrics reflect the institute's editorial scope:
152
  complexity science, information theory, persistence dynamics, and adjacent
@@ -244,7 +244,7 @@ to fork, adapt, and use as the basis for your own institute's review system.
244
 
245
  If your paper was reviewed by this system and you disagree with the panel's
246
  recommendation: write to `help@icsacinstitute.org` with your submission ID in
247
- the subject line. A human editor reads every appeal. The panel is not the last word β€” it is a thorough first pass that the
248
  curator turns into the verdict, and that the editor overrides on appeal.
249
 
250
  If your paper was accepted: the redacted review is published at
 
146
  can submit to without going through `icsacinstitute.org/submit`. It is not a
147
  replacement for human editorial judgment β€” every panel outcome routes to a
148
  human curator for the final verdict, and any author who disagrees with that
149
+ verdict can appeal to the curation team.
150
 
151
  It is also opinionated. The rubrics reflect the institute's editorial scope:
152
  complexity science, information theory, persistence dynamics, and adjacent
 
244
 
245
  If your paper was reviewed by this system and you disagree with the panel's
246
  recommendation: write to `help@icsacinstitute.org` with your submission ID in
247
+ the subject line. The curation team reads every appeal. The panel is not the last word β€” it is a thorough first pass that the
248
  curator turns into the verdict, and that the editor overrides on appeal.
249
 
250
  If your paper was accepted: the redacted review is published at
redaction.py CHANGED
@@ -89,6 +89,14 @@ SOFT_WARN_TOKENS: tuple[str, ...] = (
89
  # Added 2026-04-18 after prompt-injection attack-surface audit. Triggered
90
  # by file paths pointing at our hosts, env-var assignments, and known
91
  # credential prefixes. Match anywhere in the redacted review text.
 
 
 
 
 
 
 
 
92
  FORBIDDEN_EXFIL_PATTERNS: tuple[str, ...] = (
93
  # Absolute filesystem paths likely pointing at our hosts
94
  r"/home/orangepi\b",
@@ -375,7 +383,7 @@ def build_public_markdown(parsed: ParsedReview) -> str:
375
  (
376
  "*Reviews at ICSAC are open and transparent. AI tooling helps "
377
  "the panel draft and structure each review; final acceptance "
378
- "decisions rest with human editors. Reviews are published "
379
  "alongside acceptance for accountability; individual reviewer "
380
  "identities are abstracted to keep focus on the assessment "
381
  "rather than the tooling behind it.*"
@@ -452,6 +460,11 @@ _JARGON_REWRITES = (
452
  (re.compile(r"\bslots\b"), "reviewers"),
453
  (re.compile(r"\bSlot\b"), "Reviewer"),
454
  (re.compile(r"\bslot\b"), "reviewer"),
 
 
 
 
 
455
  )
456
 
457
 
@@ -493,6 +506,7 @@ def scan(text: str) -> ScrubReport:
493
  fatal = _find_substring_hits(structural, FORBIDDEN_VENDOR_TOKENS)
494
  fatal.extend(_find_substring_hits(text, FORBIDDEN_SECRET_PHRASES))
495
  fatal.extend(_find_regex_hits(text, FORBIDDEN_EXFIL_PATTERNS))
 
496
  warn = _find_wordboundary_hits(text, SOFT_WARN_TOKENS)
497
  return ScrubReport(fatal_hits=fatal, warn_hits=warn)
498
 
@@ -742,7 +756,7 @@ def build_public_rqc_markdown(parsed: ParsedRQC) -> str:
742
  enforces this before the redaction writes anything to the site.
743
  """
744
  status_line = (
745
- "Review Quality Control: flagged β€” reviewed by human editors before acceptance."
746
  if parsed.flag
747
  else "Review Quality Control: passed."
748
  )
@@ -775,7 +789,7 @@ def build_public_rqc_markdown(parsed: ParsedRQC) -> str:
775
  lines.extend([
776
  (
777
  "The audit surfaced a concern outside the four scholarly "
778
- "dimensions above. A human editor reviewed the panel output "
779
  "before the acceptance decision was recorded."
780
  ),
781
  "",
 
89
  # Added 2026-04-18 after prompt-injection attack-surface audit. Triggered
90
  # by file paths pointing at our hosts, env-var assignments, and known
91
  # credential prefixes. Match anywhere in the redacted review text.
92
+ # Panel-composition talk in reviewer or RQC prose: a vendor or model name tied
93
+ # to a seat ("Claude-position reviewers", "the GPT slot"). A paper's own AI-use
94
+ # disclosure never takes this shape, so it is fatal wherever it appears, prose
95
+ # included (audit 2026-09-28 item 4).
96
+ FORBIDDEN_PANEL_PATTERNS: tuple[str, ...] = (
97
+ r"(?i)\b(?:claude|anthropic|gpt|chatgpt|openai|gemini|gemma|qwen|deepseek|llama|mistral|nemotron|sonnet|opus)[- ](?:position|seat|slot|reviewer|model)s?\b",
98
+ )
99
+
100
  FORBIDDEN_EXFIL_PATTERNS: tuple[str, ...] = (
101
  # Absolute filesystem paths likely pointing at our hosts
102
  r"/home/orangepi\b",
 
383
  (
384
  "*Reviews at ICSAC are open and transparent. AI tooling helps "
385
  "the panel draft and structure each review; final acceptance "
386
+ "decisions rest with human curators. Reviews are published "
387
  "alongside acceptance for accountability; individual reviewer "
388
  "identities are abstracted to keep focus on the assessment "
389
  "rather than the tooling behind it.*"
 
460
  (re.compile(r"\bslots\b"), "reviewers"),
461
  (re.compile(r"\bSlot\b"), "Reviewer"),
462
  (re.compile(r"\bslot\b"), "reviewer"),
463
+ # The RQC sometimes names a seat by the model that fills it ("the
464
+ # Claude-position reviewers"); the public record names seats, never
465
+ # vendors (2026-09-28, found on the first external paper).
466
+ (re.compile(r"(?i)\bclaude-position\b"), "first-seat"),
467
+ (re.compile(r"(?i)\b(?:gpt|chatgpt|openai|gemini|gemma|qwen|deepseek|llama|mistral|nemotron)-position\b"), "panel-seat"),
468
  )
469
 
470
 
 
506
  fatal = _find_substring_hits(structural, FORBIDDEN_VENDOR_TOKENS)
507
  fatal.extend(_find_substring_hits(text, FORBIDDEN_SECRET_PHRASES))
508
  fatal.extend(_find_regex_hits(text, FORBIDDEN_EXFIL_PATTERNS))
509
+ fatal.extend(_find_regex_hits(text, FORBIDDEN_PANEL_PATTERNS))
510
  warn = _find_wordboundary_hits(text, SOFT_WARN_TOKENS)
511
  return ScrubReport(fatal_hits=fatal, warn_hits=warn)
512
 
 
756
  enforces this before the redaction writes anything to the site.
757
  """
758
  status_line = (
759
+ "Review Quality Control: flagged β€” reviewed by human curators before acceptance."
760
  if parsed.flag
761
  else "Review Quality Control: passed."
762
  )
 
789
  lines.extend([
790
  (
791
  "The audit surfaced a concern outside the four scholarly "
792
+ "dimensions above. Human curators reviewed the panel output "
793
  "before the acceptance decision was recorded."
794
  ),
795
  "",
rubrics/review_quality_control.md CHANGED
@@ -131,7 +131,7 @@ The model emits JSON with this exact shape. The editorial workflow serializes it
131
 
132
  The public markdown/HTML pair at `src/data/public-reviews/<record_id>_review_quality_control.{md,html}` carries:
133
 
134
- - A one-line status: `Review Quality Control: passed.` or `Review Quality Control: flagged β€” reviewed by human editors before acceptance.`
135
  - A short paragraph naming which scholarly dimensions were audited (rubric adherence, internal consistency, specificity, institutional voice) and the audit's purpose.
136
  - A condensed per-slot table showing only the four scholarly dimensions, with positional reviewer labels.
137
  - No `injection_indicators` column. No reference to prompt injection, security, adversarial content, or security architecture.
 
131
 
132
  The public markdown/HTML pair at `src/data/public-reviews/<record_id>_review_quality_control.{md,html}` carries:
133
 
134
+ - A one-line status: `Review Quality Control: passed.` or `Review Quality Control: flagged β€” reviewed by human curators before acceptance.`
135
  - A short paragraph naming which scholarly dimensions were audited (rubric adherence, internal consistency, specificity, institutional voice) and the audit's purpose.
136
  - A condensed per-slot table showing only the four scholarly dimensions, with positional reviewer labels.
137
  - No `injection_indicators` column. No reference to prompt injection, security, adversarial content, or security architecture.
templates/accept-comment.md CHANGED
@@ -10,7 +10,7 @@ The landing page above carries:
10
  - A direct citation reference
11
  - A community signup link
12
 
13
- ICSAC's review process is open and transparent. Reviews are published alongside acceptance for accountability; AI tooling helps the panel draft and structure each review while final acceptance decisions rest with human editors.
14
 
15
  Welcome to the community.
16
 
 
10
  - A direct citation reference
11
  - A community signup link
12
 
13
+ ICSAC's review process is open and transparent. Reviews are published alongside acceptance for accountability; AI tooling helps the panel draft and structure each review while final acceptance decisions rest with human curators.
14
 
15
  Welcome to the community.
16
 
templates/accept.md CHANGED
@@ -27,7 +27,7 @@ Or reference your inclusion in your own words:
27
 
28
  ## About the review
29
 
30
- Your submission was evaluated through ICSAC's open review process β€” a multi-reviewer panel scoring domain fit, methodological transparency, internal consistency, citation integrity, novelty, and AI Provenance Signal β€” with human curator oversight. AI tooling helps the panel draft and structure each review; final acceptance decisions rest with human editors. A copy of your review report is available upon request.
31
 
32
  ## Rights and licensing
33
 
 
27
 
28
  ## About the review
29
 
30
+ Your submission was evaluated through ICSAC's open review process β€” a multi-reviewer panel scoring domain fit, methodological transparency, internal consistency, citation integrity, novelty, and AI Provenance Signal β€” with human curator oversight. AI tooling helps the panel draft and structure each review; final acceptance decisions rest with human curators. A copy of your review report is available upon request.
31
 
32
  ## Rights and licensing
33