Security & compliance engineering
Provenance and egress control for health data
Most applications can tell you what their data is. Far fewer can tell you where it has been, who moved it, what left the building, and what came back. That gap is the whole subject here.
Scope note. This describes architecture and engineering practice. It is not a claim of certification, attestation, or regulatory approval, and it is not legal advice. Compliance is a program — agreements, policies, risk analysis, and independent review — of which software architecture is one part. What follows is that part.
The question an audit actually asks
An AI application in healthcare sends data to third parties by design. Embeddings go to one vendor, document extraction to another, generation to a third. Each is a disclosure, and if any of them has an incident, the question you will be asked is not “are you encrypted.” It is: exactly what did you send them, for which patients, on whose behalf, and how do you know.
Answering that from application logs is guesswork dressed as evidence. Logs are mutable, incomplete at exactly the moments that matter, and prove nothing about their own integrity. The design goal was that the answer be a query, not a reconstruction.
Four things that had to be true
The audit trail proves its own integrity
Events are sealed into a hash chain, each one committing to the one before it, on an append-only table. Editing history means recomputing every subsequent link, and a verification pass finds the break. Writes are fail-closed: if the event cannot be recorded, the operation does not proceed. An audit log that silently drops writes under load is worse than none, because it looks complete.
A detail worth stating because it is the kind of thing that gets missed: during pre-deployment testing, the row-level trigger enforcing append-only did not fire on TRUNCATE, which is a statement-level operation. The entire ledger could have been erased in one command without tripping the guard. Found during a self-audit, closed with a statement-level trigger, and verified by trying it.
The boundary cannot be forgotten
A rule that says “record an event before calling a vendor” is a rule someone eventually forgets under deadline. So the vendor connector’s write method is only reachable through a port that evaluates policy first. There is no code path that reaches the network without passing the guard — not because everyone remembers, but because the alternative does not compile.
The guard is fail-closed. An unclassified destination is blocked rather than allowed pending review, which is the correct default when the failure mode is an unrecorded disclosure.
Encryption that does not break the product
Sensitive free-text columns are encrypted at rest under a per-tenant data key, itself wrapped by a customer-managed key. The registry of encrypted fields is declarative and validated at boot, and encryption is applied by a walk over the query structure rather than by remembering at each call site — so a nested read three levels deep is covered by construction.
It fails loudly on purpose. A missed field surfaces as visible ciphertext rather than silent plaintext, because the failure you can see is the one that gets fixed.
The hard part is that encrypted columns are not searchable, and search was a core feature. Solved with a per-tenant keyed blind index for exact-token narrowing, then ranking over the decrypted candidate set. The index uses its own key, independent of the data key, so rotating one does not invalidate the other.
Leaving actually means leaving
Deleting a tenant’s rows does not delete their data, because backup snapshots retained for years still contain it. Offboarding therefore destroys the tenant’s encryption keys: after that, every encrypted value for them — live and in every retained snapshot — is permanently unreadable. Stored file bytes are swept first, since they are not covered by field encryption and the shred cannot reach them. The audit event recording the purge is written before the shred, so a deletion is never unaudited, and it deliberately survives the purge.
Recording the return leg
The first version modeled egress as fire-and-forget: something left, we wrote a receipt. But every one of these calls is a round trip. Embeddings come back. Extracted document text comes back. Generated output comes back. All of it re-enters the trust boundary, and nothing was recording that it had.
A receipt saying data left is half an answer. It does not say what returned, from where, or whether anything downstream was entitled to trust it. So the return leg gets its own record, with its own policy and its own classification of the source as trusted-internal or untrusted-external — the fact the retrieval pipeline then enforces structurally with its nonce envelope. Each return is linked to the outbound event that caused it, so a single user action reads back as one connected trace rather than scattered rows.
This also fixed a misattribution risk. The system spans several processes, and one of them reshapes payloads substantially before anything reaches a vendor. A ledger recording only at the outer edge would produce receipts saying “this content went to vendor X” when the content was actually produced by our own code in between. False attribution is worse than missing attribution, because it is evidence that actively misleads.
The sharp edge nobody warns you about
Hash-chained records are only verifiable if the value you hash today is byte-identical to what you hashed originally. Canonical serialisation omits keys that are undefined — so a column that was never set must read back as absent, not as a default. Map one nullable column to a fallback value on read and you silently recompute every historical hash. Verification then reports a break at the first row of every chain, indistinguishable from tampering, and unrecoverable without rewriting an append-only table.
That failure is one keystroke wide, and no test catches it at write time. It is written down in the codebase as an invariant, because knowing about it is the only defense.
What we rejected, and why it is written down
“We never considered it” and “we considered it and here is why the answer is this” are different answers to the same audit question. The rejections are recorded as carefully as the controls.
Pattern-based redaction before sending to vendors
Scoped in detail and declined. Pattern matching does not reliably catch names, and free-entered demographic detail arrives abbreviated or misspelled. It would break correct behavior more often than it removed anything. The deeper objection is about honesty: a partial redactor creates a claim you cannot back. Your posture becomes “we minimize what we send,” and every miss makes that false. Better to send what you send, record it exactly, and describe it accurately.
Sending less context to reduce exposure
Context is what makes a regulatory answer correct. Degrading it to reduce disclosure to a processor that is already contractually covered trades a real harm for a nominal one: a wrong answer about a regulation can contribute to an incident in a facility. That is a deliberate weighing of harms, and it is on the record as one.
Filtering sensitive data out of the vector database
A filter is a control that can regress. Instead the destination was removed: sensitive vectors live in our own database, and the external vector service only ever receives public regulatory text. Nothing has to keep working correctly forever for that to hold.
Built to outlive one product
The provenance and egress machinery is developed under one rule: it imports nothing from the application it currently lives in. The acceptance test for any file is whether it could be handed to a different product unchanged. Anything encoding a decision specific to this application — which vendors are covered, what a given tag means, how sensitive a payload is — lives on the application side of the line and is injected.
We are developing that layer as Noctra, a standalone provenance and egress-control component. It is not generally available yet and we will write it up properly when it is. Mentioned here only because the boundary discipline is the reason the rest of this architecture stayed clean.
Where this generalises
None of this is unique to health data. Any system that hands regulated or contractually restricted data to third parties has the same problem, and most discover it during diligence or after an incident, when retrofitting is most expensive. Financial services, legal technology, and anything under GDPR data-subject requests land in the same design space.
If you are building something where the data is sensitive and third parties are unavoidable, the cheapest time to design the boundary is before the first vendor integration ships.
Sensitive data, third parties, real consequences
If you cannot currently answer what you sent, to whom, and on whose behalf, that is worth fixing before someone asks.