Start with intended use
A vendor can be suitable for one use and unacceptable for another. Define the data, people affected, decisions influenced, tools connected, consequence of error or service loss, jurisdictions, operators, and alternatives before requesting evidence.
Map where data actually goes
Terms such as private, enterprise, no training, and bring your own cloud need architectural and contractual definition. Ask what telemetry or prompts leave the claimed boundary and which party controls keys, deployment, support access, deletion, and backup retention.
- Data collected, logged, cached, embedded, or retained
- Model providers, subprocessors, support personnel, and locations
- Training, evaluation, abuse monitoring, and product-improvement use
- Tenant isolation, identity, privilege, encryption, export, and deletion
Request proof behind the claims
An accuracy percentage without the evaluation population, task definition, error distribution, threshold, comparator, and limitations is not decision-ready evidence. The same is true of broad compliance claims without the underlying scope.
- Architecture and data-flow diagrams for the offered deployment
- Independent test scope, exceptions, remediation status, and dates
- Secure development, dependency, release, and change practices
- Evaluations relevant to your population, data, task, and failure consequence
Turn gaps into decision conditions
Translate gaps into conditions: restricted data, limited permissions, additional testing, human approval, stronger contract duties, shorter review cadence, a tested fallback, an exit requirement, or a decision not to proceed. State who can accept the residual risk.
Primary reference
Guidance should be checked for the version applicable to the decision date.
CISA Secure AI System Development