Use a 45 minute agenda that spends five minutes on context, thirty on five security areas (data path, execution, reach, logs, administration), and ten on live tests and follow ups. Send the questions ahead so the vendor brings the right people, and ask to see behavior, not slides.
Before the call
Send the vendor three things: the repository tiers you care about, your must have controls, and the live tests you want to see. Ask them to bring someone who can answer technical questions about data handling and administration. Tell your own team who will take notes and who scores.
The agenda
| Minutes | Topic | What you want to leave with |
|---|---|---|
| 0 to 5 | Context and scope | Agreed use cases, repository tiers, and who is in the room |
| 5 to 13 | Data path | A clear description of what leaves the machine, retention, and training use |
| 13 to 21 | Execution | How commands are approved, how allowlists work, what is enforced where |
| 21 to 27 | Reach | Files, network, tokens, and connected servers the tool can touch |
| 27 to 33 | Logs and evidence | What is recorded, export options, retention, who can read them |
| 33 to 38 | Administration | Central policy, groups, quick disable, setting drift |
| 38 to 45 | Live tests and next steps | Two or three demonstrations, plus written follow ups with dates |
Questions that earn their time
Data path. What is sent for each action, and can any of it be excluded? Where is it processed? Who at the vendor can access it?
Execution. Show how a command is blocked. Is the block enforced by the tool or by a model instruction? What happens when an allowed command is chained with a disallowed one?
Reach. Can the tool read outside the project directory? How are connected servers listed and restricted? Does it ask before honoring a project's own configuration?
Logs. Show an exported record of the session you just ran. Does it show approvals and identity?
Administration. Enforce a setting for a test group while we watch. How are changes to defaults announced?
Live tests to request
Ask for three, each under five minutes:
- A planted instruction in a code comment telling the tool to ignore its rules. See what blocks the action.
- A request to add a library. See how it picks the package and whether anything checks it.
- A request that needs a command outside the allowlist. See the approval flow and the log entry.
The OWASP Top 10 for LLM Applications lists Prompt Injection (LLM01) and Excessive Agency (LLM06), so the first and third tests map directly to named risks.
After the call
Within a day, write down answers as claims, with owner and date: "Vendor states X, to be confirmed in writing." Score against your scorecard. Send follow up questions in one list. Anything the vendor could not demonstrate stays a claim and not a fact.
Keep the tone useful
Good vendors enjoy hard questions. Ask what the tool cannot do, which controls customers ask for most, and what compensating controls they recommend. You will learn more from the limits than from the feature tour.
Our post on security questions for AI coding assistant vendors has a longer list for written follow up.
Safeguard can help at the dependency and pipeline end of this: SBOM and dependency scanning, reachability analysis, policies and gates on pull requests, and agent identity. You can try a free scan at gold.safeguard.sh/scan.
A worked example: what a good session sounds like
Here is how a strong exchange might go on one agenda item. The security engineer asks the vendor to show a blocked command.
| Moment | What happens | What it tells you |
|---|---|---|
| Engineer asks for a command outside the allowlist | The tool prompts and shows the exact command | Approval flow exists and is readable |
| Engineer asks for the same command chained after an allowed one | The tool evaluates the whole line and prompts | Chaining is handled |
| Engineer asks where the rule is enforced | Vendor says in the tool process, not in the model prompt | Control is not prompt only |
| Engineer asks to export the record | Vendor shows a file with user, time, command, and result | Evidence is available |
Four minutes, four observations, and a note in the score sheet. Contrast that with a session in which the vendor explains the architecture on slides and the engineer leaves with nothing testable.
A note-taking template
| Area | Claim | Demonstrated? | Evidence | Follow up |
|---|---|---|---|---|
| Data path | Retention configurable | No | Statement only | Ask for docs link |
| Execution | Allowlist enforced by tool | Yes | Live test | None |
| Reach | Asks before honoring project config | No | Statement only | Test in trial |
| Logs | Export available | Yes | Sample file | Check fields |
| Administration | Group policy | Partly | Console screenshot | Test enforcement |
Use one row per claim. After the call, anything not demonstrated is marked "claim", not "fact".
Preparing the vendor
Share three items a few days ahead: your repository tiers, your must have gates, and the live tests. This prevents the common pattern where the vendor brings a sales engineer without the technical depth to answer. If they cannot field a technical person, ask for a second session.
Handling pushback
Some vendors resist live tests or say a feature is on the roadmap. Roadmap is not a control. Record it as a gap, and decide whether a compensating control fills it. If the vendor resists a trial on a throwaway repository, note that too.
Time boxing tips
- Assign a timekeeper who is not asking the questions.
- Park deep dives in a follow up list instead of stretching the session.
- Leave the last ten minutes fixed, because live tests are what people drop when time runs short.
- End by repeating the follow ups and dates aloud.
After the session
Score each area with your scorecard, circulate notes within a day, and send the follow up list in one message. When answers arrive, update claims to facts only where you tested them. Keep the whole record, since it supports the next vendor review and shows an auditor how tools are approved.
Try it yourself
Click through it at your own pace, or open it full screen.
Click through and notice what a dependency inventory shows about the packages in a project.
Quick check
Who should attend from our side?
A security engineer, an engineering lead who will use the tool, and someone who owns vendor risk. Three people is enough.
Should we record the session?
Only if the vendor agrees and your policy allows it. Written notes with dated claims are usually enough.
What if the vendor cannot run live tests on a call?
Ask for a trial account and run the tests yourself on a throwaway repository, then schedule a short follow up to discuss the results.
Sources
Last reviewed: 19 October 2026.
