Our team reviews 70–100 pull requests a week. Before we automated the first pass, a senior engineer was spending 4–6 hours a day on reviews that ranged from genuinely interesting architectural decisions to "you forgot to close the stream." We weren't going to hire our way out of that — the solution was changing what humans review, not adding more humans.
Six months ago we shipped an AI first-pass reviewer into our GitHub Actions pipeline. It runs on every PR, posts inline comments on the diff within 90 seconds of the push, and consistently catches 30–40% of real issues before a human reads a single line. Here's the complete architecture, prompt, workflow, and the numbers after 4,200 pull requests.
What This Is Not
AI code review is not a replacement for human review. It's a filter and an amplifier.
The AI catches: null pointer patterns, missing null checks, obvious resource leaks, SQL injection risks, hardcoded secrets, synchronization problems, missing error handling, type safety issues, test coverage gaps in the changed code.
Humans catch: architectural tradeoffs, business logic correctness, team convention alignment, long-term maintainability decisions, subtle concurrency bugs that require understanding the full system.
A well-configured AI reviewer handles the first category reliably enough that human reviewers spend their time almost entirely on the second. That's the value — not replacing judgment, but protecting it.
The Architecture
Two design decisions worth explaining:
Chunking by file, not by line count. When a PR is large, splitting at an arbitrary line boundary produces incoherent context. Splitting by file means each chunk is a complete, reviewable unit — the AI sees the full before/after for each file.
Inline comments, not a summary. A summary paragraph at the bottom of the PR thread gets ignored. An inline comment on line 47 of PaymentService.java is impossible to miss. The GitHub PR Review API lets you attach comments to specific lines in the diff — that's what makes the feedback actionable.
The GitHub Actions Workflow
# .github/workflows/ai-review.yml
name: AI Code Review
on:
pull_request:
types: [opened, synchronize]
branches: [main, develop]
jobs:
review:
runs-on: ubuntu-latest
# Skip dependabot, release branches, and draft PRs
if: |
github.actor != 'dependabot[bot]' &&
!startsWith(github.head_ref, 'release/') &&
github.event.pull_request.draft == false
permissions:
pull-requests: write
contents: read
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Fetch PR diff
id: diff
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr diff ${{ github.event.pull_request.number }} \
--repo ${{ github.repository }} > pr.diff
lines=$(wc -l < pr.diff)
echo "lines=$lines" >> $GITHUB_OUTPUT
echo "PR diff: $lines lines"
- name: AI Review
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_NUMBER: ${{ github.event.pull_request.number }}
REPO: ${{ github.repository }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
run: |
node .github/scripts/ai-review.mjsThe workflow is intentionally minimal — all the logic lives in ai-review.mjs so it can be tested locally without triggering Actions.
The Review Script
// .github/scripts/ai-review.mjs
import { readFileSync } from "node:fs";
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const GH_TOKEN = process.env.GH_TOKEN;
const PR_NUMBER = process.env.PR_NUMBER;
const REPO = process.env.REPO;
const HEAD_SHA = process.env.HEAD_SHA;
const REVIEW_PROMPT = `You are a senior Java/Spring Boot engineer reviewing a pull request diff.
Review the diff below and identify ONLY real, high-confidence issues. Do not flag:
- Style preferences or opinions
- Minor naming issues that don't affect correctness
- Missing Javadoc on non-public methods
- Issues already present in unchanged lines (context lines starting with space)
Flag these categories when you find them:
- NULL_DEREF: Potential NullPointerException or missing null check on added lines
- RESOURCE_LEAK: Stream, connection, or IO resource opened without try-with-resources
- SECURITY: SQL injection, hardcoded secret, XSS vector, unsafe deserialization
- CONCURRENCY: Unsynchronized shared mutable state, missing volatile, race condition
- ERROR_HANDLING: Swallowed exception (empty catch block), exception converted to void
- LOGIC_BUG: Off-by-one, incorrect boolean logic, unreachable code
- TEST_GAP: New public method with no corresponding test in the diff
Respond with a JSON array. Each object has:
{
"file": "path/to/File.java",
"line": <line number in the NEW file where the issue appears>,
"category": "<one of the categories above>",
"severity": "high" | "medium" | "low",
"comment": "<specific, actionable comment referencing the exact code — max 3 sentences>"
}
If you find no real issues, respond with: []
Do not add preamble or explanation outside the JSON array.`;
async function reviewChunk(diffChunk) {
const message = await client.messages.create({
model: "claude-opus-5", // Use strongest model for code review
max_tokens: 4096,
messages: [{
role: "user",
content: `${REVIEW_PROMPT}\n\n\`\`\`diff\n${diffChunk}\n\`\`\``
}]
});
const text = message.content[0].text.trim();
try {
return JSON.parse(text);
} catch {
// Model returned non-JSON (rare) — skip this chunk
console.warn("Non-JSON response from model, skipping chunk");
return [];
}
}
function splitDiffByFile(diff) {
const chunks = [];
const lines = diff.split("\n");
let current = [];
let inFile = false;
for (const line of lines) {
if (line.startsWith("diff --git")) {
if (current.length > 0 && inFile) chunks.push(current.join("\n"));
current = [line];
inFile = true;
} else {
current.push(line);
}
}
if (current.length > 0 && inFile) chunks.push(current.join("\n"));
return chunks;
}
async function postReviewComments(findings) {
if (findings.length === 0) {
// Post a clean LGTM review
await ghApi(`/repos/${REPO}/pulls/${PR_NUMBER}/reviews`, {
commit_id: HEAD_SHA,
event: "COMMENT",
body: "**AI Reviewer** — No issues found in this diff. Passing to human review.",
comments: []
});
return;
}
// Group by severity for the summary body
const high = findings.filter(f => f.severity === "high");
const medium = findings.filter(f => f.severity === "medium");
const low = findings.filter(f => f.severity === "low");
const summary = [
`**AI Reviewer** — Found ${findings.length} potential issue(s):`,
high.length ? `🔴 ${high.length} high severity` : "",
medium.length ? `🟡 ${medium.length} medium severity` : "",
low.length ? `🟢 ${low.length} low severity` : "",
"",
"_These are automated findings. Use judgment — the AI can be wrong._"
].filter(Boolean).join("\n");
// Map findings to inline PR review comments
const comments = findings
.filter(f => f.file && f.line)
.map(f => ({
path: f.file,
position: f.line, // GitHub API: position in the diff, not file line number
body: `**[${f.category}]** ${severityEmoji(f.severity)} ${f.comment}`
}));
await ghApi(`/repos/${REPO}/pulls/${PR_NUMBER}/reviews`, {
commit_id: HEAD_SHA,
event: "COMMENT",
body: summary,
comments
});
}
function severityEmoji(s) {
return { high: "🔴", medium: "🟡", low: "🟢" }[s] ?? "⚪";
}
async function ghApi(path, body) {
const res = await fetch(`https://api.github.com${path}`, {
method: "POST",
headers: {
Authorization: `Bearer ${GH_TOKEN}`,
"Content-Type": "application/json",
"X-GitHub-Api-Version": "2022-11-28"
},
body: JSON.stringify(body)
});
if (!res.ok) {
const err = await res.text();
throw new Error(`GitHub API ${path} failed (${res.status}): ${err}`);
}
}
// ── Main ──────────────────────────────────────────────────────────────────────
const diff = readFileSync("pr.diff", "utf-8");
// Skip if no meaningful diff (binary files, lock files only, etc.)
if (diff.trim().length < 50) {
console.log("Diff too small — skipping AI review");
process.exit(0);
}
const chunks = splitDiffByFile(diff);
const MAX_CHUNK_CHARS = 12_000; // ~3K tokens — leaves room for prompt + response
// Large file diffs get truncated to avoid exceeding model context
const safeChunks = chunks.map(c =>
c.length > MAX_CHUNK_CHARS ? c.slice(0, MAX_CHUNK_CHARS) + "\n... (truncated)" : c
);
console.log(`Reviewing ${safeChunks.length} file chunk(s)...`);
const allFindings = (
await Promise.all(safeChunks.map(reviewChunk))
).flat();
// Deduplicate by file+line (same issue caught in multiple chunks)
const seen = new Set();
const findings = allFindings.filter(f => {
const key = `${f.file}:${f.line}:${f.category}`;
if (seen.has(key)) return false;
seen.add(key);
return true;
});
console.log(`Found ${findings.length} issue(s) after deduplication`);
findings.forEach(f => console.log(` [${f.severity}] ${f.file}:${f.line} — ${f.category}`));
await postReviewComments(findings);
console.log("Review posted.");The Prompt Engineering That Actually Works
The prompt is the hardest part. A poorly calibrated prompt produces one of two failures:
Too noisy: The AI flags style issues, subjective naming, and things that are obviously fine. Engineers start ignoring the comments within a week.
Too quiet: The AI reports nothing on every PR. Zero trust, zero value.
Three decisions made our prompt work: