决策智库
评估 AI 供应商：demo 之外应当测试什么
从五个 demo 无法展示的维度评分：可逆性（迁移到其他供应商所需的时间与返工，以及提示词、评估集、微调产物与向量的可移植性）；评估透明度（能否用自有测试集查看逐条结果）；数据条款（保留、是否用于训练、删除）；真实运行成本的波动；以及故障发生时的责任边界。
Decision intelligenceWritten by NirjiX AI AdvisoryPublished January 2026Last reviewed February 20268 min read
Direct answer
直接回答
从五个 demo 无法展示的维度评分：可逆性（迁移到其他供应商所需的时间与返工，以及提示词、评估集、微调产物与向量的可移植性）；评估透明度（能否用自有测试集查看逐条结果）；数据条款（保留、是否用于训练、删除）；真实运行成本的波动；以及故障发生时的责任边界。
以下深度分析保留英文原文。 查看英文完整指南
Evaluation criteria that separate vendors
| Criterion | What to ask for | Red flag |
|---|---|---|
| Reversibility | Export of prompts, evaluation sets, tuned artefacts and embeddings in a usable format | Portability described as possible but never demonstrated |
| Evaluation transparency | Run your own test set; see per-case outputs and failures | Only aggregate scores, or benchmark results you cannot reproduce |
| Data terms | Written retention, training-use, processing-location and deletion terms | Terms that can change unilaterally with notice |
| Roadmap dependency | Which committed capabilities are shipped today versus promised | The business case depends on a feature in preview |
| Operational fit | Latency and rate limits under your peak, plus incident history | No named support path and no status history |
Running a bake-off that produces a decision
- 01
Write the acceptance threshold first
Define what performance is good enough for this use case before you see any vendor's output.
- 02
Build one evaluation set you own
Real cases, including the hard and the ambiguous ones. It becomes a durable asset across future vendor changes.
- 03
Test the same set with every candidate
Same prompts, same data, same reviewers. Score per case, not by impression.
- 04
Price the second decision
Model the cost of moving away in eighteen months. That figure is the real difference between finalists.
Contract terms worth negotiating before signature
- Export rights for your data, prompts, evaluation sets and tuned artefacts, in a documented format.
- Notice and price-protection terms for pricing model changes.
- Commitment that your data is not used for model training without explicit opt-in.
- Defined support response times and a named escalation path.
- The right to run independent evaluation and publish internal results.
NirjiX view
The NirjiX view
The right question is not which vendor is best today. Model capability moves fast enough that the ranking will change; what will not change quickly is how expensive you have made it to switch.
Buy the layers that commoditize, own the layers that encode your judgement — evaluation sets, prompts and process design are the assets that survive a provider change.
Frequently asked executive questions
- Should we standardize on a single AI vendor?
- Standardize the platform layer for operability, but keep the model layer swappable behind an internal interface. Single-vendor simplicity is worth real money; single-vendor lock-in at the model layer rarely is.
- How do we compare vendors fairly?
- One evaluation set you own, run identically against every candidate, scored per case by the same reviewers against a threshold written before testing. Anything else compares sales effort.
- Are open models a serious option?
- For some workloads, yes — particularly where data residency, cost at volume or control over behaviour dominates. They shift cost from licence to engineering and operations, so the comparison must include the people to run them.
- How long should a vendor evaluation take?
- Weeks. A longer process usually means the acceptance threshold was never written, so no result can end it.
延伸阅读
相关指南
更多人工智能决策指南
- 人工智能就绪度评估
AI 就绪度衡量的不是是否拥有模型，而是数据、业务流程、治理控制、人才与预算能否支撑生产级运行。评估应基于真实的决策与运营数据，而非自我评分。
- 人工智能自建与采购对比
应当自有的是与竞争优势直接相关的数据、业务逻辑与评估标准。模型基础设施与通用工具适合外部采购，只将差异化能力内建。
- AI 用例优先级：先投资什么
优先级由价值规模、数据可得性、嵌入业务的难度与风险容忍度四个维度决定。应从能够真正融入现有流程的用例入手，而非技术上最新奇的项目。
Transparency
Sources and methodology
This page reflects NirjiX advisory practice rather than a survey or a vendor benchmark. The structure of the assessment — the dimensions, the maturity language and the sequencing logic — is the same framework used inside the NirjiX AI readiness assessment and the AI plan builder.
Where we describe patterns ("most organizations discover…"), we are describing what we observe across client engagements, not a measured statistic. We deliberately avoid quoting market numbers we cannot verify, because an AI investment case built on borrowed statistics collapses the first time a CFO tests it.
Any figure that ends up in your own plan should come from your own data: your cost base, your cycle times, your error rates, your volumes. The assessment and plan builder are designed to force that discipline.
Turn the judgement into a plan you can fund
The AI readiness assessment scores where the organization actually stands; the AI plan turns that into a sequenced, costed set of moves.
Outputs are preliminary and intended for advisor validation before funding decisions.