Safeguard
Industry Analysis•

AI agent demo versus production: questions that expose the gap

A smooth AI agent demo hides defaults, limits and failure modes. Questions to ask about scheduled tasks, files, desktop apps and sandboxes before you sign.

Safeguard Security Team
6 min read

A demo shows the happy path with ideal data and an admin account. Production adds limits, failures, many users and untrusted input. Ask which plan and settings the demo used, what happens on failure, what an ordinary user sees, and ask to repeat key steps in your own trial tenant.

What demos leave out

Demos are rehearsed. They use clean data, a single user, a fast network and the most capable plan. They rarely show a failed run, a revoked credential, a sleeping laptop or a hostile document. They also tend to start from a configured account, so you never see default settings.

None of this is dishonest. It is how demos work. It means your questions should aim at what the demo did not show.

Questions by feature

Scheduled tasks

  • Which identity ran this task, and what would a normal user's task run as?
  • What happens if the device is off, or the run fails halfway?
  • Where do I see the run history?

File generation

  • Is that link public or private by default?
  • What does the file contain if the source text starts with a formula character?
  • Can an admin delete this file and prove it is gone?

Desktop app

  • What did the app ask the operating system for at install?
  • Which of these settings can IT lock?
  • How does the next update arrive?

Remote sandbox

  • Is this environment fresh, and what network could it reach?
  • Where was the secret shown in this run stored?
  • Who at your company can see inside it?

The plan and settings question

Ask what plan, region and settings the demo account used. Features, limits and retention can differ by plan. If the demo ran on a higher tier than the one you will buy, your tests should run on the lower one.

Ask for a repeat, in your hands

The most useful request is simple: let us run these three steps ourselves. A vendor confident in the product will agree to a trial tenant. Prepare a short script, perhaps drawn from the one week bake-off plan, and run it with dummy data.

Watch for evasion patterns

PatternWhat to do
"That is on the roadmap"Score it as unknown until it ships and you test it
"Most customers do not need that"Ask whether it is available and how to turn it on
"We follow industry standards"Ask which practices, and for the document
"We can show that offline"Ask for it in writing
Long answers that skip the questionRe-ask the exact question in an email

A roadmap item is not a control. Treat it the way you would treat an audit still in progress: planned, not done.

Put answers on paper

After the demo, send a written list of the questions and ask for written answers by a date. Compare them with what the sales team said live. Differences are worth a follow up call with a technical contact.

Record and compare

Add the answers to your scorecard with their evidence level. If you want a template, see the vendor-neutral scorecard. Safeguard's compliance and evidence automation can keep the exchange with your vendor review records.

A worked example: one question, three answers

Suppose you ask: "Can an admin delete a file generated by a user, and prove it is gone?" Here are three responses a vendor might give.

  1. "Yes, our admins can do that." This is a claim with no detail. Ask where the setting is and whether deletion is logged.
  2. "Yes. In the admin console under files, search by user and date, then delete. The audit log records the deletion, and links stop working immediately." This is documented, specific and checkable. Ask for a screenshot or a trial tenant check.
  3. "Users can delete their own files, and we are working on admin delete." This is a roadmap statement. Score it as unknown today.

The same question produced three very different pieces of information. Without a precise question, all three might have been heard as "yes".

Step by step: run a demo review

  1. Before the call, write your questions for each of the four feature areas.
  2. Ask for the plan, settings and account type used in the demo.
  3. During the call, note each answer as confirmed on screen, claimed or deferred.
  4. After the call, email the unanswered questions with a date.
  5. In the trial, repeat three demonstrated steps yourself with a standard user.
  6. Compare results with the demo, and ask about any difference.
  7. Update the scorecard with evidence levels.
Demo momentQuestion to add
Task runs on a scheduleWhat identity, and what if it fails?
File appears instantlyWhere is it stored and who can open the link?
Desktop app does something cleverWhat permission lets it do that?
Sandbox runs codeWhat can it reach, and does it keep state?

Reading the room

Notice who answers. If a salesperson answers every technical question, ask for a technical contact on the next call. If answers change when you rephrase, note it. A vendor comfortable with its design will answer plain questions plainly.

Keep the tone constructive

You are not trying to catch anyone out. You are trying to learn the real behavior so you can approve the right settings. Most vendors respond well to specific, reasonable questions, and the exchange often improves your own plan.

Turn answers into controls

Each answer should become either a setting you will apply, a rule for staff, a test you will repeat, or a gap to track. If an answer does not lead to one of these, the question may not have been the right one.

Try it yourself

Click through it at your own pace, or open it full screen. Watch an assistant work with a connected tool, and note which actions you would want logged and approved.

Quick check

Should we trust a recorded demo?

It shows a possible behavior at one time. It does not prove defaults, limits or current versions. Test in your own tenant.

What is the single most useful question?

Ask to repeat the key step yourself with your own data and an ordinary user account.

Is it fair to ask for a trial instead of a demo?

Yes. A trial tenant is a normal request for security reviews, and vendors expect it.

Last reviewed: 19 October 2026.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.