AI Vendor Evaluation and Procurement

Vendor and procurement decisions for AI tools are consequential and hard to reverse. A rigorous evaluation process protects against expensive mistakes.

Defining requirements before talking to vendors

Vendor conversations should follow, not precede, a clear internal definition of requirements: what tasks need to be automated, what quality levels are required, what integration points are needed, what data will be processed, and what compliance requirements apply. Without this, vendor conversations will be shaped by the vendor capabilities on offer rather than your actual needs.

Key evaluation dimensions

• Task performance. Evaluate on your own representative data, not vendor benchmarks. Results can vary dramatically.
• Data handling. Where is data processed and stored? Who has access? What is the data retention policy? This is critical for organisations handling personal, health, or commercially sensitive information.
• Security posture. SOC 2 Type II, ISO 27001, and equivalents. Review penetration testing practices and incident response history.
• Integration. What APIs and connectors exist? What are the rate limits? How is latency managed at scale?
• Pricing transparency. Token-based pricing scales non-linearly with use. Model vendor pricing carefully before committing to usage-based contracts.
• Lock-in risk. How hard is it to switch to a different model or provider? Data portability and API compatibility matter.

NZ-specific considerations

Data residency matters for health, government, and some financial data. Confirm whether NZ or Australian data centres are available. Check whether the vendor has a NZ or AU entity, which affects legal recourse.

The most common procurement mistake is evaluating AI tools on demo performance against ideal inputs. Real-world performance on your messy, edge-case, real data is all that matters.