An AI vendor risk assessment should answer a business question: can this product support the proposed workflow at an acceptable level of risk? A long security questionnaire is not enough. The review must connect vendor claims, contract terms, technical controls, and operating evidence to the way your organization will actually use the system.
Start with the use case, not the logo. The same product can be low risk when it summarizes public research and high risk when it handles health information, recommends employment decisions, writes production code, or lets an agent change a system of record.
Define the proposed use before reviewing the vendor
Document the business owner, intended users, input data, connected systems, output audience, decisions affected, and degree of autonomy. State what the product must never do. If the use case is still vague, the review can only produce vague assurances.
Classify the workflow before deciding the depth of diligence. A low-consequence writing assistant may need a focused privacy and access review. A system that affects people, money, regulated data, external communications, or production infrastructure needs deeper legal, security, model, and operational evidence.
Review eight evidence areas
1. Product and model boundaries
Identify the exact product tier, model providers, hosting regions, subprocessors, optional features, connectors, and administrative controls. Do not transfer a claim from one product edition to another.
2. Data use and retention
Ask what enters prompts, files, retrieval stores, logs, support systems, feedback channels, and model-improvement processes. Confirm retention periods, deletion behavior, data-location commitments, and whether administrators can configure them.
3. Identity, access, and permissions
Review single sign-on, provisioning, role-based access, service accounts, API keys, session controls, and audit logs. For agents, verify which tools and records the system can reach and whether permissions can be narrowed by action.
4. Security and resilience
Request current evidence for the controls that matter to the workflow. Review encryption, vulnerability management, incident response, availability, recovery, isolation, change control, and the vendor's process for handling model or dependency failures.
5. Model behavior and evaluation
Ask how the vendor evaluates accuracy, safety, bias, prompt injection, data leakage, tool misuse, and known failure modes. Require results that resemble your use case. A general benchmark does not prove reliable performance in your workflow.
6. Human oversight and accountability
Confirm how users review outputs, interrupt actions, correct records, appeal decisions, and escalate incidents. The organization deploying the tool still needs an accountable business owner even when the vendor provides strong controls.
7. Contract and exit terms
Review confidentiality, data rights, breach notification, audit rights, service changes, indemnities, deletion, portability, suspension, and termination support with qualified counsel. Confirm what happens to stored prompts, files, configurations, and logs when the relationship ends.
8. Ongoing monitoring
Define which vendor changes require notice or reapproval: a new model, subprocessor, connector, data-use term, hosting location, permission, or autonomous capability. Assign an owner and review cadence before approval.
Use a decision record, not a pass-fail score
A weighted score can help triage, but it can hide a critical gap. Record the decision as approved, approved with conditions, pilot only, needs remediation, or rejected. Include the approved use, prohibited use, required controls, evidence reviewed, exceptions, owners, expiration date, and reapproval triggers.
Conditions should be testable. “Use responsibly” is not a control. “Only approved Workspace accounts may be used; customer records are prohibited; external outputs require manager review; activity is reviewed quarterly” is actionable.
Questions to include in an AI vendor questionnaire
- Which customer data is used for model improvement, and under which product terms or settings?
- Which subprocessors, model providers, regions, and connected services are involved?
- Can retention, deletion, sharing, and administrator access be configured and verified?
- What evaluations cover the proposed use case and its most important failure modes?
- How are material model, control, or contract changes communicated?
- What logs, exports, and evidence can the customer retain?
- How can an administrator suspend access, revoke credentials, or stop an agent?
- What happens to customer data and configurations at termination?
A proportionate review sequence
First, screen the use case and reject obviously prohibited workflows. Second, collect vendor evidence and identify unresolved questions. Third, test the product with representative but controlled data. Fourth, document conditions and owners. Fifth, approve a narrow pilot with monitoring. Expand only when evidence supports the broader scope.
NIST's AI Risk Management Framework treats third-party AI risk as a lifecycle responsibility. That is the right operating principle: procurement approval is the start of oversight, not its end.