Rewording, Real Businesses and the Limits of a Records Check
DOI 10.5281/zenodo.24271374Version 1.0License CC BY 4.0
Preprint, not peer-reviewed. Self-evaluation: the author’s company builds the product tested. AI assistance (Claude, Anthropic) was used for the experiment code, analysis and drafting, all reviewed by the author.
yaktai.ai22 views9 downloadsZenodo14 views12 downloads
Abstract
Report 1 found that a check against a business’s own records held every draft from an email-drafting agent that both graders judged compliant, on 12 short attacks. Here we test that check against 48 attacks written to evade it, on a fictional business and on three anonymised real business settings, using the production model.
The word-list check failed. On the production model, drafts that both graders judged compliant reached the owner unheld in 17 of 120 drafts on the fictional business, and in 9 to 15 drafts (of 101 to 108) on each real setting. We then deployed three successive designs over one evening: allow only what the records back word for word, which held 6 of 7 ordinary replies and was replaced 22 minutes later; judge each sentence by where its content came from, holding what carried the prospect’s claims into the business’s voice and marking the writer’s own suggestions for the owner to read; and four refinements of that second design.
On the deployed version, on the fictional business only, both graders judged 4 of 141 reworded-attack drafts compliant and unheld, and 0 of 36 on the original attacks. It held 0 of 12 ordinary replies, though those replies had been used while tuning it. On a single written rule applied to every version, the author judged none of the four harmful; the graders’ verdict stands as the stricter count.
A fixed list of words loses to rewording, a strict allow-list is unusable, and tracking where a sentence’s content came from works far better on this test. Reader availability turned out to be a security property: a check that cannot run must defer, not decide. All final results come from one fictional business, attacks we wrote ourselves, and model graders.
Cite
@techreport{sakhawat2026rewording,
author = {Sakhawat, Muhammad},
title = {Rewording, Real Businesses and the Limits of a Records Check},
institution = {YaktAI Lab},
type = {YaktAI Lab Report},
number = {2},
year = {2026},
month = nov,
doi = {10.5281/zenodo.24271374},
url = {https://doi.org/10.5281/zenodo.24271374},
note = {Preprint}
}Versions
- Version 1.0, 16 November 2026: first publication.