Bonyasoftالعربية
Back to home

An AI reviewer on every merge request: what it catches, and what it can't

Code review at our company used to wait for one person: me. We built a small AI reviewer that reads every merge request within minutes. Here is how it works, what it is good at, and the rules that keep it useful.

Amr Abdelfatah · Team Lead / CTOAI · 5 October 2026 · 8 min read

For a long time, code review at Bonyasoft had one bottleneck, and it was me.

A developer would open a merge request in the morning. I would be in a client call, or deep in a design, or on another project's deadline. The review came in the evening. The developer had moved on to something else, and now had to switch back. Some days, small changes were merged with only a quick look, because waiting felt worse.

This is a very normal problem in a small team. We decided to fix it with an AI reviewer. Not to replace human review, but so that no merge request waits for a first answer.

It has been running for a few weeks now. This post is what we built, what it is good at, what it is bad at, and the rules we learned the hard way.

What it does

When a developer opens a merge request, a review appears in our team chat within about ten minutes. It tags the author and attaches the review as a file. A short note goes on the merge request too.

When the developer has fixed things, they write /review as a comment on the merge request. The bot reads the new code and posts a recheck that says what was fixed and what is still open.

sequenceDiagram
    participant Dev as Developer
    participant Git as Git server
    participant Bot as Review bot
    participant Chat as Team chat
    Dev->>Git: opens merge request
    loop every 2 minutes
        Bot->>Git: anything new?
    end
    Bot->>Bot: read the diff + the project rules
    Bot->>Chat: review file, tags the author
    Bot->>Git: short note on the merge request
    Dev->>Git: pushes fixes, comments /review
    Bot->>Chat: recheck: fixed / still open

That is the whole thing. It is small on purpose.

How it is built

Three parts, all running on our own small server:

  1. A watcher. Every 2 minutes it asks the Git server for new merge requests and new /review comments.
  2. A reviewer. For each one, it gets the diff and runs a language model with our review instructions. The instructions are the same checklist I use myself, written down.
  3. A poster. It sends the result to the team chat and adds a note on the merge request, so the bot knows next time that this one is done.

Why a watcher that asks every 2 minutes, and not a webhook that is pushed instantly? Because it is simpler to run and nothing is lost if the bot restarts. A 2-minute delay does not matter to anyone. A lost review does.

The rules that made it useful

The first version worked, but people did not like it. These rules changed that.

Rule 1: It advises. It never blocks.

The bot cannot approve, reject or block a merge. It gives an opinion. A human decides.

This sounds like a small detail. It is the most important rule. When a tool can block your work, you start arguing with it. When it only advises, you read it like a note from a colleague, and you take what is useful.

Rule 2: Same format as a human review

Every review has the same shape, the one I always used:

  • 🔴 Must fix before merge
  • 🟠 Should fix, or explain why not
  • 🟢 Fine, small notes only

Developers already knew how to read this. They did not have to learn a new tool, only a new reviewer.

Rule 3: Give it the project's rules, not just the code

A model that only sees the diff gives generic advice: "consider adding error handling". That helps nobody.

So the reviewer also reads the project's own written rules: how we name things, how permissions are checked, what every endpoint must have. Now it says things like "this new endpoint has no permission check, and every other booking endpoint has one". That is a real review.

Rule 4: Put it where people already look

In the first week, a developer replied to the bot in the chat, asking it to look again. Nothing happened, because the bot only reads comments on the merge request.

That was our mistake, not his. Now every review ends with one line: "After fixing, comment /review on the merge request." When a tool confuses people, change the tool, not the people.

Rule 5: Keep the keys out of the pipeline

It was tempting to run the reviewer as a step in our build pipeline. We did not, because pipeline jobs can be changed by anyone who can push a branch. Any secret the job uses can be printed by that branch. The bot runs as its own small service instead, and the keys never leave it.

What it is good at

After a few weeks, the pattern is clear. The AI reviewer is good at things that are written down somewhere:

  • A missing permission check, compared with the endpoints next to it.
  • An error that is caught and then quietly ignored.
  • A new feature with no test, or a test that does not test the new part.
  • A rule from the project docs that the change breaks.
  • Small things: a leftover debug log, a magic number, an unclear name.

And it is fast. The first time the full loop ran, with nobody touching it, the merge request was opened at 7:56. The review arrived at 8:06. The developer fixed it, commented /review, and the recheck came back before 9:10.

What it can't do

It is bad at anything that is not written down:

  • Is this the right feature? It cannot know what the client meant in a call last week.
  • Will this design hurt us in six months? It sees one change, not the whole direction.
  • Across projects. If the mobile app expects a field the backend just renamed, it will not see it from one side.
flowchart LR
    subgraph Bot["AI reviewer"]
        A[Written rules]
        B[Patterns in the code]
        C[Missing tests]
    end
    subgraph Human["Human reviewer"]
        D[What the client meant]
        E[Long-term design]
        F[Changes across projects]
    end

So we did not remove human review. We changed what the human looks at. The bot clears the checklist. The human spends their time on the questions only a human can answer.

What changed for the team

  • Nobody waits for a first review. Most of the small problems are fixed before a human even opens the merge request.
  • Human reviews got shorter and better. I no longer write "you forgot the permission check". I write about design.
  • The rules got better. Every time the bot gave bad advice, it was usually because a rule was missing or unclear in our docs. Fixing the docs fixed the bot, and helped new people too.

That last point surprised me most. An AI reviewer is only as good as the rules you give it. Building it forced us to write down what "good" means here. That was worth doing even without the bot.

If you want to try this

Start smaller than you think:

  1. Write your review checklist down. One page.
  2. Run the model by hand on a few old merge requests. Compare with what you said in the real review.
  3. Only then automate it, and make it advise, never block.

Read more