Better tools made Copilot code review worse. Here's how we actually improved it.

You’d think better tools always mean better results. GitHub’s experience with Copilot code review proved otherwise. When they upgraded their tooling, they initially made the problem worse—the AI agent became less effective at catching issues and reviewing pull requests. What sounds like a paradox actually reveals something important about how AI agents work: the tools available shape not just performance, but the entire decision-making process. GitHub’s solution involved stepping back and rebuilding their approach around Unix-style code exploration tools, ultimately reducing costs while improving quality.

Here’s what happened technically. When GitHub enhanced Copilot’s code review capabilities with more sophisticated tools, the agent had more options available at each step. But more options created a problem: the AI spent computational resources exploring tangents instead of following the signal toward actual issues. Think of it like giving a debugger access to every possible logging statement—without discipline, you end up investigating noise. GitHub’s fix was counter-intuitive: they constrained the toolset to Unix-style utilities (grep, diff, file operations) that were already deeply optimized for code exploration. These aren’t fancy—they’re proven. By reshaping the agent’s workflow to gather “pull request evidence” systematically using these basic tools, the agent became more focused. It could answer “what changed?” efficiently before answering “is this a problem?” The result was measurable: reduced computational cost and faster, more accurate reviews.

Why this matters for your infrastructure and automation work is straightforward. If you’re building AI agents—whether for AWS resource analysis, infrastructure reviews, or deployment validation—this lesson applies directly. Better tools don’t automatically mean better agents. Sometimes the constraint forces better reasoning. A well-designed agent using basic, reliable tools (AWS CLI with specific filters, structured API responses, simple text parsing) often outperforms one overwhelmed with sophisticated options. When you’re designing automation around Copilot or building your own agents, think about the evidence they need to gather first, then pick tools that gather that evidence cleanly. It’s the opposite of “move fast and break things”—it’s “move deliberately and understand what you’re building.”

The practical takeaway: if you’re implementing code review automation, security scanning, or any deployment validation with AI agents, start with constraints. Use focused toolsets. Structure your agent’s workflow to gather evidence systematically before making decisions. GitHub’s approach shows that sometimes the path to better performance isn’t piling on more capabilities—it’s understanding what your agent actually needs to succeed, then building exactly that.