An agent in continuous integration is convenient, and it costs more than it looks. The cost hides because spend follows your team's activity, not your decisions.
Here is how to keep the value and cut the surplus bill.
The main mistake: running on every commit
A developer pushes ten times a day while polishing a branch. If the agent starts on every push, nine of those ten runs were paid for in vain: nobody read the result.
Attach the agent to pull requests, not to pushes. Better still, run it on the opened and ready-for-review events instead of on every branch update.
Narrow the scope
An agent that reads the whole repository on each run pays for that context again and again. Hand it the diff only.
Most CI work is reviewing a change, not auditing a project. The difference in spend is an order of magnitude.
Pick a model for each check
CI usually holds checks of different weight, and they need different models.
- style, leftover debug statements, commit message formatting: a smaller model
- reviewing the logic of a change and hunting for defects: a stronger one
- architecture audits: the flagship, and certainly not on every PR
Do not pay twice for the same thing
The pipeline reran because a linter failed? The agent has no reason to walk the same diff again. Cache its result against a hash of the change.
Build matrices work the same way: the agent runs once, not in every cell.
A separate ceiling for automation
Give CI its own key so automation spend does not blend with manual work. Bound the workload in CI itself.
Create a named key for CI so its usage shows up separately, choose Premium or Cheap, and set an expiry if needed. For Cheap you can allow a Premium fallback. The key API cannot set per-key budgets, rate limits or model allowlists, so bound job retries and workload in your own CI configuration.
What to measure
Three numbers worth keeping in view:
- spend per pull request
- share of runs whose output nobody opened
- cost per defect the agent actually found
The third one is unpleasant, and it is exactly the number that tells you whether the automation pays for itself.
In short
Attach the agent to pull requests, give it a narrow diff, match the model to the task and set up a named automation key. Cap the workload on the CI side and watch the usage statistics.