Audit Logging: Tamper-Evident Records, What to Log, and Retention

What you'll learn: By the end of this lesson you will be able to separate an audit log from an application log, choose what to record and what never to record, explain why a hash chain alone misses tail truncation and full rewrites, add an external anchor and a verifier, and set retention with NIST SP 800-92 and the OWASP Logging Cheat Sheet in hand.

A bank vault keeps a visitor book with numbered pages, filled in ink. If someone tears out a page, the missing number shows. If someone tears out the last three pages and the book simply ends, nothing shows, unless the manager photographed the last page each evening and keeps the photo at home. An audit log has the same two weaknesses, and the same fix.

ShopStream Stage 6 shows a JSON audit record and a hash chain, and Security for Distributed Systems has an audit paragraph. This lesson covers what they leave open: completeness, anchoring, what to leave out, and retention.

Verified against the OWASP Logging Cheat Sheet, NIST SP 800-92 (2006), the AWS CloudTrail log file integrity documentation and RFC 9162, on 2026-11-08.

The core idea

An application log helps you debug. An audit log has to survive an argument. It records who did what to which resource, with what result, so an investigator or auditor can rely on it. That needs three properties an ordinary log lacks: complete (important actions cannot happen without a record), attributable (the actor comes from the authenticated session, never the request body) and tamper-evident (changes and deletions can be detected). NIST SP 800-92 notes that many rootkits are designed to alter logs, and that keeping original logs unaltered supports their evidentiary use.

Three examples run through the lesson: a minimal example (a Java hash chain with an anchor and verifier), a realistic example (ShopStream's fee-change audit) and a failure example (an insider deletes the last records).

What to log, and what never to log

The OWASP Logging Cheat Sheet says each event must record "when, where, who and what": a timestamp and an interaction identifier linking related events, the application and host, the source address and user identity, and the event type, severity and description. Among the events it says to log where possible: authentication successes and failures, authorisation failures, use of higher-risk functions (user administration, administrative privilege, default, shared or "break-glass" accounts), access to sensitive data, and data import or export. For ShopStream: refunds, fee changes, role grants, bulk exports, support access to payment history, and denied attempts at these.

What stays out: the cheat sheet says to remove, mask, hash or encrypt session identifiers, access tokens, passwords, connection strings, encryption keys, payment cardholder data and sensitive personal data. Store ids, not names. Sanitise event data against log injection, prefer structured formats such as JSON, and limit the size of user-supplied values; in Java, parameterised logging alone does not prevent CR/LF injection, so use an encoder that escapes values.

{
  "seq": 4812,
  "occurred_at": "2026-11-08T09:41:12Z",
  "actor": { "type": "staff", "id": "agent_204" },
  "action": "refund.issue",
  "resource": { "type": "payment", "id": "pay_7731" },
  "details": { "amount_minor": 1000, "currency": "USD" },
  "outcome": "success",
  "request_id": "req_5c1e"
}

Identifiers are examples. The gap-free seq counter is new compared with the Stage 6 record.

Making it tamper-evident

In a hash chain each entry's hash covers the previous hash, a sequence number and the payload, so editing an entry breaks every later hash. NIST SP 800-92 describes the general technique, log file integrity checking: compute a message digest per file, store it securely, compare later, and protect the original digests (for example on read-only media). Its 2006 text names MD5 and SHA-1 as the most common algorithms; use SHA-256, as the ledger lesson and CloudTrail do.

Figure 1. A hash chain plus an anchor. How to read this: solid arrows are the chain. The dashed arrow is the anchor, a copy of the latest sequence number and hash kept where the log's operators cannot change it. The chain proves consistency; the anchor proves it is the same chain as before.

The code is short. The verifier re-derives every link, checks the sequence is gap-free, and compares against the anchor:

static String link(long seq, String payload, String prev) throws Exception {
    return sha256(prev + "|" + seq + "|" + payload);
}

static String verifyAgainst(List<Entry> log, Anchor a) throws Exception {
    String chain = verifyChain(log);                 // re-derives every link, checks seq is gap-free
    if (!chain.equals("chain ok")) return chain;
    Entry last = log.isEmpty() ? null : log.get(log.size() - 1);
    if (last == null || last.seq() < a.seq()) return "chain ok but TRUNCATED: anchor says seq " + a.seq();
    Entry at = log.get((int) a.seq() - 1);
    return at.hash().equals(a.hash()) ? "chain ok and matches anchor" : "chain ok but REWRITTEN: differs from anchor at seq " + a.seq();
}

Run on JDK 26.0.1 against a four-entry log whose head was anchored, five attacks print:

1 intact                     : chain ok and matches anchor
2 edit one record            : broken at seq 2
3 delete the last two        : chain alone says 'chain ok'; with anchor: chain ok but TRUNCATED: anchor says seq 4
4 edit and recompute chain   : chain alone says 'chain ok'; with anchor: chain ok but REWRITTEN: differs from anchor at seq 4
5 remove a middle record     : gap before seq 4

Rows 3 and 4 are the lesson. Deleting the tail leaves a shorter, valid chain, and someone with write access to the whole store can edit a record and recompute every later hash. Neither is visible to the chain; both are visible to a copy of the head kept elsewhere. In row 5 the message names the first entry seen after the gap.

Real systems use the same idea. AWS CloudTrail's log file integrity validation hashes every delivered log file; every hour it delivers a digest file referencing the last hour's files with a hash of each, signed with a private key, including the previous digest's signature, in a separate folder so security policies can differ (SHA-256 for hashing, SHA-256 with RSA for signing). Transparency logs go further: RFC 9162 defines a log as a single append-only Merkle tree, with inclusion proofs that an entry is in the tree and consistency proofs that an older tree head is a prefix of a newer one, so a verifier can check that the log only ever grew.

Completeness and retention

Tamper-evidence covers what was written. Completeness covers what should have been. If the audit write is a best-effort call after the business action, a crash leaves an action with no record. The stronger design inserts the audit event in the same database transaction as the change (an outbox, as in the order pipeline) and ships it onward asynchronously; the sequence number then exposes any gap. Where an action cannot be wrapped in a transaction, document the window.

Figure 2. No action without a record. How to read this: the refund and its audit event commit together; the shipper moves events to a store the application cannot edit.

Retention is a policy decision. The OWASP cheat sheet says logs must not be destroyed before the required period or kept beyond it, and that legal, regulatory and contractual obligations may set the periods. NIST SP 800-92 distinguishes log retention (regular archiving as normal operations) from log preservation (keeping logs that would normally be discarded because they record activity of particular interest, typically for investigations). Keep periods as configuration, store ids rather than personal details so erasure requests do not force a chain rewrite, restrict and review read access, and record all access to the logs. Synchronise time across servers, or record a confidence level where you cannot.

Trade-offs

In practice: "who changed the fee rate?"

This is the realistic example, an educational scenario on ShopStream that describes no real company.

This is a realistic scenario, not a case history.

The situation

A store disputes a fee. Finance asks who changed the fee rate for that store's category, and when. The service logs show the deploy, not the person.

Starting architecture

Figure 3. The audit path. How to read this: events reach the insert-only store only through the shipper. The head goes hourly to an account the store's operators cannot write, and a verifier checks the chain against it.

Options considered

The decision

Option C. The fee service writes the event with its change; the actor comes from the authenticated session; the shipper appends to the audit store; a verifier runs hourly and pages on broken, gap or TRUNCATED.

Rollout and verification

Ship the transactional write first and compare event counts with change counts for a week; add the shipper and store; enable the anchor and verifier last. Verify with a drill: edit a copy, truncate a copy, and confirm the verifier flags each.

The 2 a.m. failure

The verifier pages: chain ok but TRUNCATED: anchor says seq 4812, and the log's last entry is 4810.

Figure 4. Truncation is invisible to the chain and visible to the anchor. How to read this: the insider changed the store but could not change the anchor.

First three checks: confirm the anchor is intact, check who had delete rights on the store, and compare with the service database's unshipped events for 4811 and 4812. If the source still holds them, re-ship; if not, record the lost-evidence window. Revoke the delete path and preserve the logs (NIST's term) for the investigation.

Architect's notes

• Separate duties: the people who run the audit store must not control the anchor.
• Test the verifier with a tamper drill.
• Log denials, not only successes; repeated denied admin attempts are the early signal.

Common mistakes

• Letting the request choose the actor. Take identity from the authenticated session.
• Logging secrets or full personal data because "it is only the audit log".
• A chain with no anchor and no sequence numbers, which misses truncation, rewrites and gaps.
• Keeping everything forever. Retention must not run beyond what is required.

Key takeaways

• An audit log is evidence: complete, attributable and tamper-evident.
• Use the OWASP "when, where, who, what" fields; log privileged and break-glass use, exports and denials; exclude secrets, tokens and card data.
• A hash chain detects edits and gaps, not truncation or a full recompute; an external anchor detects both.
• Write the audit event in the same transaction as the action, and ship it to an insert-only store.
• Retention is a policy: not shorter than required, not longer than allowed, with ids instead of personal data.

Practice exercises

Exercise 1 (30 minutes): List ten ShopStream actions that need an audit record and the fields of each entry.

Exercise 2 (2 hours): Extend the chain demo to sign the anchor with a key and verify it, and add a sixth attack: replacing the anchor with a forged one.

Exercise 3 (half a day): Design the audit pipeline for three ShopStream services: transactional write, shipping, store permissions, anchoring cadence, verifier alerts and retention configuration.

Related reading

Official documentation

• OWASP Logging Cheat Sheet - events to log, data to exclude, log injection, protection, time and retention.
• NIST SP 800-92: Guide to Computer Security Log Management - log protection, integrity checking, retention and preservation (2006).
• AWS CloudTrail: Validating log file integrity - hashed files, signed hourly digests.
• RFC 9162: Certificate Transparency Version 2.0 - append-only Merkle trees, inclusion and consistency proofs.