LLM Code Reviews Are Not What You Think - An Opinionated Take on AI-Assisted Quality

We introduced LLM reviewer to our Java/Spring Boot PR process several months ago.

It has flagged many good patterns as errors, ignored an important serialization error which was found by a junior developer, and even raised issues related to logic problems which were approved by several senior reviewers.

This is an opinionated breakdown of what works, what doesn’t, and how we’ve evolved in the process.

TL;DR

What we did: Embedded Claude (via Bedrock) as an automated PR reviewer for a Java microservices codebase

What it does well: Finds dead code, inconsistent error handling, missing null checks, log-level inconsistencies

What it doesn’t get right: Patterns unknown to it because of domain specifics or deliberate deviations from best practices

The takeaway: LLM code reviews are not an alternative to human reviewers - they are a filtering mechanism revealing another set of problems

Outcome: Bugs found that human eyes overlooked, and the list of false positives to suppress via prompt tuning continues to grow

Why We Did This

It wasn’t about replacing reviewers. There was another problem we faced.

Our codebase consists of 80+ microservices. Each week 15-20 PRs go in across different services. Human reviewers are great at finding design flaws and business logic bugs. But they are bad at spotting consistent issues across 80+ services maintained by different teams.

Things that get overlooked during human review again and again:

• A catch clause logging at INFO instead of ERROR in case of downstream timeout
• Missing @Transactional annotation on a method that changes two tables
• A REST controller returning entities rather than DTOs
• Zombie code that remained after the feature flags cleanup

This isn’t rocket science, but it sure is tiring to keep track of. This is precisely the area where LLM review comes in handy.

The Architecture

flowchart LR
    A[PR Opened on GitHub] --> B[GitHub Action Triggered]
    B --> C[Diff Extraction: changed files + context]
    C --> D[Prompt Assembly: diff + review rules + file metadata]
    D --> E[Claude via AWS Bedrock]
    E --> F{Findings?}
    F -- Yes --> G[Post Inline Comments on PR]
    F -- No --> H[Approve: No issues found]
    G --> I[Human Reviewer: Accept / Dismiss]

The main design choices:

1. Diff only, not the whole file. We send the modified lines along with 30 lines of context before and after from each file. Using the whole files consumed the context window and brought noise in form of unmodified code.

2. File information is valuable. The prompt has the file path, the module name, and the type of the file (controller, service, repository or config). It helped to reduce false positives by 40%, since the LLM stopped applying the controller patterns to batch jobs.

3. Review criteria are clear. We don’t ask LLM to "review this code". We present the specific list of criteria for code review. Details below.

The Prompt Structure

It was the trickiest part of the solution. Our initial prompt was:

Review this Java code diff for bugs, security issues, and best practices.

It returned the wall of generic suggestions. "Consider using Optional here." "This method can be shortened." Noise.

The final version of the prompt after 6 iterations:

You are a senior Java code reviewer for a Spring Boot microservices codebase.

Review ONLY the changed lines in the diff below. Do not comment on 
unchanged code.

## Review Checklist (flag ONLY if violated):
1. Error handling: catch blocks must log at ERROR for downstream failures,
   WARN for recoverable issues. Never swallow exceptions silently.
2. Transaction boundaries: methods modifying multiple database tables must
   have @Transactional. Read-only queries should use 
   @Transactional(readOnly = true).
3. DTO enforcement: REST controllers must not return JPA entities directly.
4. Null safety: parameters from external inputs (request params, message
   payloads, DB results) must have null checks or use Optional.
5. Dead code: unused imports, unreachable branches, commented-out blocks
   older than the current PR.
6. Logging hygiene: no PII in log statements. Account numbers, SSNs,
   emails must be masked.

## File Context:
- File: {file_path}
- Module: {module_name}
- Type: {file_type} (controller / service / repository / config / test)

## Diff:
{diff_content}

## Response Format:
For each finding, provide:
- Line number in the diff
- Rule number violated (1-6)
- One-sentence explanation
- Suggested fix (code snippet)

If no rules are violated, respond with: "No issues found."

Why the checklist model succeeds: It limits the LLM to a limited number of rules. No philosophical guidelines. No stylistic preferences. Only one question: did this diff break any of these six rules? Yes or no.

What the LLM Finds That Human Reviewers Don’t

With time, certain patterns became evident where the LLM outperformed human reviewers consistently:

1. Log Level Issues

This is the most valuable category by itself. People don't always verify that the log level matches. The LLM verifies every time.

// Flagged by LLM - Rule 1 violation
try {
    paymentGateway.processRefund(refundRequest);
} catch (GatewayTimeoutException e) {
    log.info("Refund processing timed out for request: {}", 
             refundRequest.getId());
    // ^^^ Should be log.error - downstream timeout is not informational
}

This was seen in multiple services. All were human-approved. All were detected by the LLM. There were no false positives here.

2. Absence of @Transactional Boundaries

// Flagged by LLM - Rule 2 violation
public void transferBalance(String fromAccount, String toAccount, 
                            BigDecimal amount) {
    accountRepository.debit(fromAccount, amount);
    accountRepository.credit(toAccount, amount);
    // ^^^ Two table modifications without @Transactional
    // A failure after debit but before credit = inconsistent state
}

However, most of these were true misses. There were some false positives in cases where the transaction boundary is controlled by an enclosing method.

3. PII in Log Statements

// Flagged by LLM - Rule 6 violation
log.debug("Processing application for customer: {}, email: {}", 
          customer.getId(), customer.getEmail());
// ^^^ customer.getEmail() is PII - must be masked

One false positive came from a test class containing artificial test data. The solution here was to add a suppression rule for all files matching the pattern *Test.java.

What the LLM Gets Wrong

False Positive Category 1: Domain-Specific Patterns

There is a custom @RetryableTransaction annotation in the codebase, which combines @Transactional functionality with retry mechanism. As the LLM has no idea about this annotation, each occurrence of @RetryableTransaction is reported as a @Transactional missing.

The fix: We have added the "Known Patterns" part to the prompt:

## Known Patterns (do NOT flag these):
- @RetryableTransaction includes @Transactional behavior -
  do not flag as missing transaction boundary
- Methods in classes extending AbstractBatchProcessor manage
  their own transaction
- Repository methods annotated with @Query manage read-only
  transactions through Spring Data

This alone removed most of our false positives.

False Positive Category 2: Intentional Null Returns

There are a number of API endpoints in our system where we intentionally return null to denote "no content" (mapped to HTTP 204 by a response wrapper). The LLM picked this up as missing null checks.

The fix: Added to the Known Patterns section:

- Controllers returning null delegate to ResponseEntityWrapper
  for 204 handling - do not flag as missing null safety

The Bug the LLM Missed

In particular, one PR introduced a change to the way the Jackson serializer mapped JSON fields. One developer decided to change from @JsonProperty("transaction_id") to @JsonProperty("txnId"). This resulted in a bug in a downstream consumer who was expecting "transaction_id" in the JSON.

LLM didn't flag this problem since the syntax was valid and none of the review rules addressed API contract issues. This bug was caught manually by a junior engineer on our team since she had worked on the downstream consumer previously.

The lesson: LLM will find structural problems. Semantic contract issues are not going to be caught unless explicitly programmed into the rules.

The Distribution of Problems

Here is the general distribution of problems surfaced by the LLM review, based on our experience:

pie title Findings Distribution by Category
    "Log-level mismatches" : 31
    "PII in logs" : 14
    "Missing @Transactional" : 8
    "Dead code" : 11
    "DTO violations" : 9
    "Null safety" : 14

A few takeaways from all of this:

That’s a minimal cost. A few pennies per PR is less than the time one engineer could spend reviewing each PR to check for structural consistency.

The Development of Trust

The adoption curve was a predictable one throughout the group:

flowchart TD
    A["Week 1-2: Skepticism\nDevelopers dismiss most LLM comments"] 
    --> B["Week 3-4: Calibration\nTeam tunes prompt, false positives drop"]
    B --> C["Week 5-8: Selective Trust\nDevelopers check LLM comments for\nlog-level and PII categories only"]
    C --> D["Week 9+: Integrated Workflow\nLLM review is part of the PR checklist\nHumans focus on design and business logic"]

It became clear in Week 5 because an incident related to a log-level problem that the LLM identified on the PR but which the developer had dismissed. From there, the team stopped dismissing the LLM comments without even looking at them.

What I Would Do Differently

1. Begin with 3 rules, not 6.
We started with 6 rules and ended up drowning ourselves in false positives for the first month. If I had a chance to do it again, I would start with 3 rules that provide the highest value signals and add them one by one.

2. Create the "Known Patterns" part in advance of the first review.
Each code base has its own set of coding patterns that break the best practice rules. Identify them in advance. We learned the lesson the hard way with @RetryableTransaction comment.

3. Measure the false positive rate as a primary metric.
The false positive rate has not been measured until Week 4. At that point, developer trust was already severely damaged. The false positive rate should be kept under 15%. Otherwise, developers will dismiss all comments.

4. Never allow the LLM to approve PRs.
The LLM makes comments. It never approves or rejects PRs. It only gives the recommendation to a human who actually approves the PR. This is not a limitation but a separation of concerns.

Lessons learned

LLM review and human review capture different types of issues. The LLM works well for consistency in a large codebase. Human review works best for intent, domain knowledge, and contract changes. Each complements the other.

The prompt is the product. More effort was put into refining the prompt than in the GitHub Actions workflow itself. The quality of the prompt is what makes or breaks the results. The rest is plumbing.

False positives are more expensive than false negatives. One missed issue can be fixed manually. But many false positives will train your whole team to disregard the tool. Precision > recall.

The LLM must be explicitly informed about domain specifics. The LLM does not know about your custom annotations, architectural idiosyncrasies, or intentional deviations from coding style. If you don’t teach it that, it will flag it.

Where We Are Headed

The next logical step would be closing the feedback loop. At present, the LLM makes the comment and a human takes action. But the logical progression would be an agentic workflow where the LLM reviewer spots a problem, another agent comes up with a solution to it and suggests it as a commit to the PR without a human taking a look at it.

That future is not quite here yet. But what we have already developed – the rules for reviewing and the patterns we have identified – is precisely the governance system that the agentic pipeline for fixing and suggesting changes needs.

Until then, the trade-off of paying a few dollars a month for bugs that senior engineers repeatedly overlook seems rather fair.

Are you running LLM code reviews on your Pull Requests? Please let me know what rules apply to your stack. Comment down below or connect with me at beaconoftech.com.

Engineering Leader | 20 years in financial services | AWS Certified Solutions Architect | IEEE Senior Member | beaconoftech.com