What a retrieval system copies
A rep pastes a customer’s tender into the assistant: 150 requirements, in the customer’s own words. Within minutes, that text has been copied to places the customer never saw. Each line goes to a model provider with passages from the client’s documents. Search queries go to an embedding provider. Every model call writes an entry to an audit log. If the rep accepted analytics, a session replay records the page around it. The answers end up in a table the rep may send on.
Any retrieval system makes copies like these, and each one can reach more people, travel further or last longer than the original allowed.
The pilot is the sales assistant I built for a manufacturer. It answers technical questions from the company’s product documents, cites every fact, and drafts decks, tender tables and emails. For each copy it makes, I had to decide who can see it, where it goes and how long it stays, and I had to set a cap on what one person can spend.
Privacy is about people: the reps who ask, and the customers whose names and tenders they paste in. Confidentiality is about the client’s material: which documents get indexed, who can retrieve them, and which outside companies see them. Security is about keeping the wrong people and code out, and limiting what they can do if they get in.
Into the index
The index holds the client’s public product documents and 25 support pages that sit behind a password on the client’s site. The client shared those passwords for the pilot. Each of those documents carries a restricted flag, so later steps can treat it differently.
One manual printed the support password in full. Had it been indexed as it was, the assistant could have quoted it to anyone a rep writes to, and the document viewer would have shown it. So the build replaces every password with a placeholder in every document before writing the index. Afterwards I searched the whole corpus for each password and found none.
The index is a file, built offline and committed with the code. Nothing is fetched from the web while the app runs, so a page someone tampers with reaches the assistant only at the next rebuild, which I run by hand. Scripts, styles, forms and iframes are stripped first. A test fails if a rebuilt index brings images from a site the content security policy doesn’t list.
Who can retrieve what
In the pilot, every invited rep can retrieve every passage. That’s acceptable here, because the client already hands these documents to customers. It would be wrong for a company’s own files.
An index flattens permissions: a passage loses the access rules of the folder it came from unless search puts them back. The fix is to drop the passages a person can’t open before ranking, so the model never sees them. I’ve prototyped that separately. For this app it’s a written design, to build with the first internal document: one check, applied before ranking, on every path a passage can take to the model or the screen. That means search, re-ranking, earlier turns in a conversation, the document viewer, and answers a rep copies.
Retrieval also sends text out. Each search query goes to an embedding provider, and when a rep builds a deck, that query includes the customer context they typed.
Reps paste customers’ emails to ask the question inside them, and a pasted email brings the customer’s name, address and phone number with it. No product question needs any of that. Since 1 November, one detector masks names, email addresses and phone numbers in the browser, as the rep pastes and again when they send. The server masks them a second time on every route, before the log, the router, a model or an email sees the text. The rep sees what was removed and can put back a name taken by mistake, but never an email address or a phone number. Company names are never masked, and the log records how many details were removed, not what they were.
Before it shipped, a read-only scan of production found customer names in 3 of 148 logged questions, each a “Hi” followed by a name at the top of a pasted email. It also read a serial number as a phone number, which is fixed. On 20 synthetic emails in three languages, every email address and phone number was masked. On 33 ordinary product questions, nothing was.
Every model and embedding call goes through one gateway, so the app holds no key for those providers. Every call also asks for zero data retention and no training on prompts. The gateway refuses a provider that can’t honour that rather than falling back to one that keeps data. I checked it with one live call per model.
In the version under test, every model call is also pinned to EU data centres. A call that can’t run in the EU fails instead of running elsewhere, and the log records where each one ran. The search embeddings can’t have both rules, because the only embedding model offered in the EU keeps data. I chose zero retention over the region for them, so search queries, already masked, run with no region set. While calls are pinned, there’s no fallback model, and two services with no EU option are switched off: an outside re-ranker the app can use, and the faster routing model from another vendor. That router is called directly, outside the gateway, and now receives masked text only.
Each rep sees their own conversation history. Admins see everyone’s, and a read-only role can review the log without changing anything.
Treating passages as data
Every passage comes from a website, and websites get hacked. A passage could carry instructions aimed at the model: quote a price, drop the citations, add a link. The prompts tell each model that sources are data. A prompt is only a request, so code backs it up:
- The answer model has one tool, and it only reads the index. An injected instruction has no action to trigger.
- Answers don’t render images the model writes. An image’s address could carry the conversation to another site, and the browser would fetch it without a click. The content security policy also blocks images and connections to sites it doesn’t list.
- Citations are matched word for word against the index, so an edited conversation can’t add evidence.
- Before any model sees a request, code checks its shape and size: only user and assistant messages, at most 100 messages, 20,000 characters per question and 600,000 per conversation. I set those limits by judgment, well above normal use; they aren’t calibrated on real conversations.
None of this makes the answer true. The check that grades each answer can’t catch a poisoned passage, because the passage supports its own claims.
A 21-case injection set in three languages tests the prompts. Its first run, seven runs across three models, gave 147 answers. None followed an injection, but 31 quoted the attacker’s text while warning the rep about it, a false price and planted codes included. A rep who copies that answer sends the warning on, figure and all. In the version under test, the rule now tells the model not to repeat what a source tells it to say, and the same seven runs passed 146 of 147. The set calls the model directly with fixed passages, so it doesn’t test the router, the check, the masking or the log, and it has no image, tool or multi-turn attacks.
What leaves with the answer
A passage has two audiences: the rep reading the answer, and the customer the rep writes to. The assistant drafts emails, decks and tender tables, so some passages leave the company.
Emails link only public pages. A password-protected page would open on a password form for the customer. The email can still use facts from those pages; only the link is left out. Decks and tender tables still list every source, because the decision so far covers emails only. The names, addresses and phone numbers a rep types for the customer are masked before the draft is written. The company name stays.
Exports end up in spreadsheets, and they hold rep questions and model-written text. A cell that starts with “=” or “+” could run as a formula in Excel, so the export turns it into plain text.
What the system remembers
Every model call writes one entry to an audit log in an EU region, after the response, so a logging failure can’t affect the answer. A database trigger refuses any edit or deletion. That guards against application bugs. Someone holding the database credentials can still get round it.
Append-only and a retention period pull against each other. In the version under test, the trigger lets a delete through only inside a transaction that sets a cutoff, and only for rows older than it; edits are still refused. A nightly job deletes log entries past the log’s period, and feedback screenshots past their own period, which can’t be longer. Entries a team note was written from are kept, and every purge is recorded in a table that is append-only too. The periods are the client’s to set, and until it does, nothing is deleted. My proposal is 12 months for the log and 90 days for screenshots. What the log held before masking began is unchanged, and clearing it belongs with that decision.
Analytics starts only after a person accepts it, and it runs in an EU region. Click events carry no element text. The first browser test found question text inside them, because the example questions are buttons.
Session replays were the bigger risk. As configured, the recorder captured every page once someone consented: sign-in forms, admin screens and whatever a rep pasted. I excluded the sign-in cards, the admin pages and the customer fields. A review the next day found I’d missed the tender requirements box, which holds the customer’s own text, and it’s excluded now. The questions reps type are still recorded, because replays show where a rep got stuck. Since 1 November every recorded input goes through the same masking first, and passwords record as stars. One detail came from the recorder’s code: its masking hook runs only when every input is set to be masked, so the setting that looked like it should be off had to be on. I haven’t yet seen a real replay with it. Replays are kept for 30 days.
Smaller copies need the same care:
- Invitation links don’t stay in the address bar. The token moves into a short-lived cookie before analytics, replays or the next site can see it.
- Feedback can include a screenshot. It’s taken only when someone clicks, shown to them before sending, and removable. Only super admins can open it, and the email that announces it carries the start of the message and a link, never the image. The message itself is masked, with no exceptions.
- The in-app version history named colleagues and model vendors, and every user could read it. It now says what people can do, without names, and a test fails if a vendor’s name comes back.
A spending cap per person
Cost is a security question because it decides availability. One answer can mean a routing call, up to seven searches and a check, and a 150-line tender is 150 requests in a few minutes. A stolen session or a script stuck in a loop could empty the shared AI budget, and then nobody gets answers.
So each person gets $5 a day. Spend is counted from the billed cost of every answer, email, deck and tender line, and checked after the shared budget on all four AI routes. A spent cap refuses the request, and the rep sees, in their language, that the limit resets at midnight UTC. Admins are exempt. A request that starts under the cap can finish over it, by one answer’s cost. The cap went live in production on 1 November 2026, so there’s no usage data on it yet.
Why $5? A light answer costs about $0.004, a standard one a few cents, and a 150-line tender $2–3, so $5 covers a heavy day with a tender in it. I set it by judgment. My coding agent isn’t allowed to read per-person spend from production, so there was no distribution to set it from. In the version under test, the super admin sees each person’s spend per day against the cap: the highest day and what it paid for, the 95th-percentile day, and the days over half the cap. I’ll review the cap against it on 15 November.
There’s no per-minute limit. The tender screen sends three requests at a time, 18 to 36 a minute on a long tender, so a suggested 30-a-minute limit would throttle real work. The daily cap already bounds the spend. Tests cover the rule, and removing the timeout or the admin exemption makes one fail.
Around the retrieval system
Some of the work isn’t specific to retrieval, but it still decides where copies end up.
- Previews used to share the production database, so any pushed branch ran with write access to real accounts and the log. Previews and local runs now use a database branch with production’s tables and none of its rows.
- Accounts are by invitation, and a deployment with no sign-in configured refuses every request. Microsoft sign-in checks each ID token’s audience, issuer and lifetime, even though the token comes straight from Microsoft. In the version under test, it also refuses a login whose email address isn’t in a domain its directory has verified, so one company’s directory can’t claim an account invited from another. One sign-in feature also opened endpoints I hadn’t asked for, one of which revealed whether a username exists. They’re off, and tests check that they return “not found”.
- CI runs three scanners on every pull request: for secrets, for workflow files, and for known vulnerabilities in the dependencies. They report without blocking, and the four accepted advisories each have a reason and an expiry date. Actions are pinned to exact commits, and new package releases wait seven days before they can be installed. Since 28 October no job has started, because of a billing limit on the GitHub account, so the scanners aren’t running.
- Coding agents wrote much of this code, and an agent’s transcript is one more copy. Invitation links were kept out of it, in a private file. Database addresses were set through an API, so no value appeared on a command line. The secret scanner redacts what it finds. One slip got through: during an audit, an agent printed a deployment-protection bypass secret into its transcript. It had never been used, and I revoked it the same day.
How it’s checked, and what’s open
Before the pilot widened, I wrote a threat model: 14 threats across eight trust boundaries, each with its mitigation and the risk left over. Two days and 15 merged changes later, I audited the app against it again. The biggest risks were still open, and the new findings all came from housekeeping. On 1 November I worked through the five items the first draft of this note listed as open. Each fix has a note listing what it didn’t test.
Still open:
- The EU pinning, the retention rule, the spend table, the injection rule and the directory check are in the version under test, not yet in production.
- Search embeddings run with zero retention but no region, and the retention periods wait for the client.
- Search doesn’t filter by permission. The design is written; the code comes with the first internal document.
- Masking misses a lone surname, and Microsoft sign-in hasn’t met a real directory.
- The injection set doesn’t go through the whole app.
- The $5 cap hasn’t yet been checked against real per-person spend.
- CI isn’t running until the billing limit is lifted.
Some findings are still being worked through, and their details stay off this page.
Lessons
Whatever protected a document stops at the index unless retrieval puts it back, so that’s the first thing to ask before indexing a company’s own files.
The rep can read things the customer shouldn’t receive, so what leaves with an answer needs its own rules.
Check what actually gets copied. The question text in click events and the tender box in replays were both found by looking at the recorded data, not at the configuration.
Make the rule fail closed where it’s set. A region or a retention promise that quietly falls back to another provider when the first can’t serve the call isn’t a rule.
Facts from the app’s implementation notes, decision records and threat model, October and November 2026. The client, its products, documents and conversations are withheld.