# How to Choose an AI Agent Provider: Criteria You Can Verify

> How to choose an AI agent provider on criteria you can verify: evaluations, traceability, permissions, portability and a clean exit, plus the red flags.

- Canonical: https://arkatai.com/en/choosing-ai-agent-provider/
- Site: Arkatai (https://arkatai.com) — agentic operations as a service
- Language: en
- Published: 2026-07-19

---


To choose an AI agent provider, stop scoring the demo and start scoring what you can verify: how they measure the agent's work, what trace it leaves, what permissions it asks for, whether you keep your operating architecture when you leave, and whether they can swap the model without rebuilding the system. Everyone passes the demo; the real work is where they separate. This is the checklist I would use from your seat, with the questions to put in the RFP and the signals that would make me leave the meeting.

## Judge by what you can verify, not by what they show

A demo runs on chosen cases, clean data and no exceptions. It proves the technology can work, not that it will work with your operation. So the criteria that count are the ones that leave evidence: something the provider can show you today, or commit to by contract, and that you can check afterwards. Everything else is narrative.

The second idea that organizes this decision: you are not buying a pilot, you are buying a relationship that has to survive model changes, changes to your process and changes to your team. The question is not "does this work in the demo?" but "who answers when it stops working a year from now?".

## The six criteria that actually separate providers

| Criterion | What to ask for | What it reveals |
|---|---|---|
| Evaluations | How they measure agent quality and with which cases | Whether there is engineering or just faith in the model |
| Traceability | A record of what it received, which rule it applied, what it did | Whether you can audit decisions or get a black box |
| Permissions and controls | What it may read, write and must never touch | Whether risk is governed or left open |
| Portability | What you take when you leave, and in what format | Whether you are locked in or own your operation |
| Model agnosticism | How they switch models when another is better | Whether you are tied to one lab |
| Clean exit | What the day you leave looks like | Whether the contract protects you or retains you |

**Evaluations you can see.** A serious provider measures how the agent performs before giving it real work: test cases with known answers, quality thresholds, failure review. Ask to see the method. If the answer is "the model is very good" or "we supervise it manually", there is no evaluation, there is faith. Without measurement there is no serious way to say something works, let alone to catch when it stops.

**Traceability of every decision.** Every agent action should be recorded: what input it received, which rule it applied, what it did and why it escalated. Without a trace there is no audit, and without audit a board should not approve the deployment. It is the same requirement I develop in [AI agent governance](/en/ai-agent-governance/): the trace is not a technical extra, it is the condition for being able to answer to a customer, a regulator or your own board.

**Explicit permissions and controls.** The agent works inside your systems, so the provider must be able to state precisely what it may read, what it may write and what it must never touch. An agent with broad, unbounded permissions is an incident waiting for its turn. Concrete permissions, per system and per action, are a buying criterion, not an implementation detail.

**Portability of your operating architecture.** This is where ownership divides. The platform, the code and the runtime belong to the provider and you will not take them. But the rules, the exceptions, the controls, the data contracts, the integration specs and the evaluation criteria describe *your* operation, and that should be yours. Ask what you export at the end and in what format. That portability does not remove the cost of reimplementing elsewhere, but it removes the cost of rediscovering your own process from scratch.

**Model agnosticism.** AI models are already a utility: contracted through an API and swapped when another is better, cheaper or faster. A provider who encodes your process inside one lab's primitives ties you to that lab. Ask how they select the model per task and what happens the day they want to change it. A forward-deployed engineer working for the model maker is mandated to deploy the stack their company sells. The one working for you should optimize your operation and preserve the ability to switch models. It is a conflict of incentives and architecture, not of good or bad intent.

**A clean exit.** The best moment to negotiate how you leave is before you enter. What you receive, on what timeline, with which data, results and traces. A provider confident in their value does not need to retain you with exit friction. They retain you with the outcome.

## Questions to put in the RFP

- How do you evaluate an agent's quality before and during production? Show me a real example of test cases and thresholds.
- What is recorded about each decision, and how long do I keep it?
- What concrete permissions does the agent need in each system, and how are they revoked?
- If I end the contract, what exactly do I take, and in what format?
- How do you choose the model per task, and what does changing it in six months involve?
- Who operates and maintains the system when my process or the model changes?

That last question separates the most. Buying a platform leaves maintenance to you. Contracting the operation leaves the outcome and its maintenance with the provider. It is a decision about structure before it is a decision about a provider, and I analyze it in [buy vs build for enterprise AI](/en/buy-vs-build-enterprise-ai/) and [in-house, consultancy or boutique](/en/inhouse-consultancy-or-boutique/).

## Red flags in a demo

Be wary when the demo only shows the happy path and dodges exceptions, because real operations live on exceptions and that is where pilots die. Be wary when there is no way to see the trace of a decision, when "security" is settled with a sentence instead of concrete permissions, when the word "evaluation" never appears, and when you ask about the exit and the answer turns vague. A concession about what their system does not yet do well is worth more than a demo without a single seam.

## Where this fits in your decision

Choosing a provider is the last step, not the first. Before it comes deciding whether you buy, build or contract the operation, and with whom. The comparable cost of each option is covered in [how much it costs to implement AI agents](/en/ai-agents-implementation-cost/). The return that sustains the investment in front of a board belongs to [AI operations ROI](/en/ai-operations-roi/), and I do not repeat it here. And if you doubt whether your own house is ready for any provider, that prior review is in [prepare your company for agents](/en/prepare-your-company-for-agents/). For the full frame of the topic, the pillar is [AI agents for business](/en/ai-agents-for-business/).

## Frequently Asked Questions

### What matters most when choosing an AI agent provider?

That you can verify their criteria: evaluations they show you, traceability of every decision, concrete permissions per system, portability of your operating architecture, and a clean exit by contract. The demo proves the technology can work. These criteria prove it will work with your operation and that you are not locked in.

### How do I avoid being locked into one provider or model?

Demand model agnosticism and portability from the contract. The provider should be able to change the model per task without rebuilding the system, and you should take your rules, exceptions, integrations and traces when you leave. That does not remove the cost of reimplementing, but it avoids starting from zero.

### Which questions reveal most in an agent RFP?

The ones that ask for evidence, not promises: show me a real evaluation, show me the trace of a decision, tell me the exact permissions per system, and describe the day I leave. A provider with judgment answers with examples. One without it answers with adjectives.

### Should I choose a provider before deciding the structure?

No. First decide whether you buy a platform, build an internal capability or contract a managed operation, because that determines who maintains the system. Choosing a provider before settling the structure usually ends up buying what demoed best, not what fits best.