Ask every IAST or RASP vendor the same questions on languages, attach method, performance testing, failure behavior, data handling, update path, and evidence. Score the answers in a table and verify the important ones in a trial, because written answers describe intent, not your environment.
Coverage and fit
- Which languages, frameworks, and versions are supported? Ask for the exact versions you run, not a marketing list.
- How is the agent attached? On the JVM, Oracle documents startup attach and later attach with different failure behavior. Ask which the product uses.
- Does it work in containers and on serverless? Ask for a reference deployment.
- Does it handle microservices? Ask how findings from many services roll up.
Behavior and safety
- What happens if the agent fails to load or crashes? Fail open and fail closed lead to very different incidents.
- How was overhead measured? Ask for the workload, hardware, and versions, then repeat it yourself.
- Is there a monitor mode before blocking? Essential for RASP. See the rollout plan.
- How are false positives handled? Look for suppression with an audit trail.
Data and trust
- What data leaves the host? Ask for specifics: request bodies, parameters, stack traces, secrets that may appear in traffic.
- Where is it stored, for how long, and who can read it?
- How is the agent built, signed, and updated? An agent is privileged code. Oracle notes that whoever deploys one is responsible for verifying it is trustworthy.
Evidence and operations
- What reports can we hand to an auditor or a customer? Ask for a sample, and ask how findings map to the control language your reviewers use.
A scoring table you can copy
| Question area | Weight | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|
| Language fit | ||||
| Attach and failure behavior | ||||
| Overhead evidence | ||||
| Data handling | ||||
| Update and signing | ||||
| Evidence and reporting |
Agree the weights before you read any answers, so enthusiasm for a demo does not set the numbers.
What to verify yourself
Anything that affects uptime or data exposure should be tested, not only read. Failure behavior, overhead, and data egress all fit that rule. Put them in your proof of concept plan.
Running the evaluation fairly
Give every vendor the same environment, same sample application, and same time box. Ask each to document setup steps so you can compare effort. Hold a short internal review after each trial while memories are fresh, and score against the table you agreed on before. Keep notes on surprises, since surprises during a trial usually show up again in production. Finally, ask for references from customers with a similar stack, and ask them about support and upgrades rather than features.
A worked example: scoring two imaginary answers
Take question five, about what happens when the agent fails to load. The first vendor answers that the agent logs a warning and the application continues unprotected, and offers a heartbeat alert you can configure. The second answers that failure behavior depends on configuration and does not say more. Both answers are useful, but not equally. The first states a behavior and a mitigation you can test. The second signals a conversation you still need to have. Score them accordingly, and plan a test of the first vendor's claim during the trial.
Questions people forget
Add these to your list.
- How do you handle a runtime version we use that is newer than your supported list?
- What is the process when your agent has a vulnerability?
- Can we export all findings and history if we leave?
- What does support look like at night when the agent blocks a production request?
- Which of your answers are documented publicly, and which are verbal?
Weighting by your situation
| Situation | Weight more heavily |
|---|---|
| Many languages | Language coverage and rollout tooling |
| Regulated data | Data handling and retention |
| Small team | Triage experience and support |
| High availability service | Failure behavior and overhead |
Agree weights before reading answers.
Turning answers into trial tests
Every claim that affects uptime or data exposure should become a test in your proof of concept. Make a two column list of claims and tests, and tick them off. Claims you cannot test should be written into the contract as commitments, with your counsel reviewing the language.
Reference calls that tell you something
When you speak to a customer reference, skip the product tour and ask about the hard parts. How long did rollout take across their services? What broke during the first month? How do upgrades go? How responsive is support when something fails at an awkward hour? Would they run it again knowing what they know now? Two honest answers on these topics are worth more than a long list of features.
Try it yourself
Click through it at your own pace, or open it full screen.
Click through to see how findings and evidence connect to control language. Your RFP question about reports for auditors is easier to judge once you have seen a mapping like this.
Quick check
Should the RFP ask for a feature list?
Ask for the capabilities you need, tied to your stack. A long feature list invites box ticking and hides the questions that decide outages.
Do I need legal review of the answers?
Contract terms about data and liability are for your counsel. This list covers the technical side only.
How many vendors should be in the trial?
Two or three is manageable. More than that stretches your team and makes comparisons shallow.
See also a neutral shortlist method for a related category. Safeguard's free scan is at gold.safeguard.sh/scan if you want a dependency baseline for your own services.
Sources
- Oracle: java.lang.instrument package
- OWASP DevSecOps Guideline: Interactive Application Security Testing
Last reviewed: 19 October 2026.
