Every time Google ships a new frontier model, security leaders get the same set of questions from their boards. Is this model going to replace our AppSec team? Can we use it to reason through our entire supply chain risk? The short answer is that Gemini Ultra is excellent at many things, but security reasoning has properties that general reasoning benchmarks do not capture.
This post walks through the specific properties of security reasoning and compares how Gemini Ultra and Griffin AI handle them. The goal is not to dismiss Gemini Ultra, which is genuinely impressive. The goal is to help teams understand where it fits and where a security-specific system earns its keep.
What Security Reasoning Actually Requires
A chess-style reasoning benchmark tests whether a model can plan many steps ahead from a complete and known state. A security decision looks almost nothing like that. The state is incomplete, the evidence is contradictory, the source of truth is spread across a dozen systems, and the cost of a wrong answer is paid by whoever is on call at two in the morning.
Security reasoning requires four things that benchmarks rarely test:
- Grounded evidence. Every conclusion must be traceable to a specific artifact: a CVE record, a commit, an SBOM component, a VEX statement, a policy rule.
- Consistency over time. The same question asked twice must produce the same answer unless the underlying evidence changed.
- Coverage of absence. "There is no evidence of exploitation" is as important as "there is evidence of exploitation," and they must not be confused.
- Multi-source correlation. An advisory, an SCA finding, a runtime trace, and a ticket history often have to be reconciled before a decision is made.
Gemini Ultra is a foundation model. It can do impressive reasoning on structured prompts, but it is not intrinsically grounded, consistent, or aware of absence. Wrapping it in RAG, tools, and prompts can close some of those gaps. Griffin AI closes them at the engine layer, before the LLM ever sees the question.
A Concrete Example: Reasoning About Reachability
Consider a realistic question: "Given the CVEs in our dependency graph, which ones actually expose our production authentication service?"
Gemini Ultra, handed the list of CVEs and the dependency tree, will produce a plausible prioritization. It will describe likely attack paths, reference common CWE categories, and offer a reasonable-looking ranked list. It will not have run the reachability analysis, parsed the call graph, or validated that the vulnerable function is actually reached from the authentication entry point. It is guessing, eloquently.
Griffin AI treats the same question as a compound query against the engine. It resolves the dependency graph, runs the reachability analyzer against the compiled artifact, joins reachable components to the CVE feed, filters by exploitation evidence, and returns a result with per-item provenance. The LLM summarizes the output; it does not generate the output. If a CVE is not reachable, Griffin says so with evidence. If the evidence is incomplete, Griffin flags the gap rather than papering over it.
The first approach is reasoning by analogy. The second is reasoning by evidence. For security, only the second is defensible when the incident happens.
Chain-of-Thought on Unstable Ground
Gemini Ultra's chain-of-thought capabilities are strong. Give it a complex multi-step problem and it will decompose the problem, explore branches, and often converge on the right answer. The catch is that each step is a generation, which means each step can introduce error. Over long chains, errors compound.
In security, that compounding is expensive. A chain-of-thought that concludes "therefore this vulnerability is not exploitable because the function is not called in production" based on three intermediate inferences that were each 90 percent likely to be right is ultimately about 73 percent likely to be right. For a critical security decision, 73 percent is not acceptable.
Griffin AI decomposes the same reasoning into a pipeline where each step is either a deterministic engine call or a small, well-scoped LLM task. The engine calls have known accuracy. The LLM tasks are bounded to translation and explanation, which are tasks models are reliable at. The end-to-end accuracy is the product of a small number of high-reliability operations, not a long chain of medium-reliability generations.
Hallucinating CVE Identifiers
One of the most frustrating error modes in Gemini-based security workflows is fabricated CVE identifiers. A model asked about vulnerabilities in a specific library will sometimes produce CVE numbers that do not exist, CVSS scores for those non-existent CVEs, and detailed narratives about their exploitation. Every element is fluent. No element is true.
This is not a Gemini-specific problem; it is a foundation model problem. But the contrast with Griffin AI is instructive. Griffin AI cannot return a CVE identifier that does not exist, because the identifier comes from the CVE feed, not from the model. If the feed has no matching record, the result is empty. The LLM can explain that the result is empty, but it cannot invent content.
For any team considering an AI-assisted security program, the difference between "can be wrong in subtle ways" and "cannot be wrong about basic facts" is the line between a production tool and a toy.
Consistency and Audit
A regulator, an auditor, or an incident responder will eventually ask: "why did the system classify this finding as low-priority on March 3rd, and then high-priority on March 4th?" If the answer is "the model's output is non-deterministic," the organization has a problem.
Gemini Ultra's reasoning can vary between runs, sometimes in small ways and sometimes in larger ways. For a chat product, that variability is tolerable. For an audit surface, it is not.
Griffin AI records the inputs, the engine evaluations, the policy rules applied, and the outputs for every decision. Variation in the output is traceable to variation in the inputs. If an auditor asks why a finding's classification changed, the answer is a diff of the evidence, not a shrug at the model's sampling temperature.
Where Gemini Ultra Earns Its Keep
Gemini Ultra is not a poor choice for security adjacent work. It excels at drafting threat models, exploring hypothetical attack paths, summarizing long incident timelines, and explaining complex vulnerabilities to non-experts. These are real uses, and we recommend them.
What Gemini Ultra should not be is the source of truth for your security decisions. A security program needs deterministic evaluation of real artifacts. An LLM, no matter how capable, cannot provide that alone.
Layering Strategy
Teams that get the most from both tools use them in layers. Gemini Ultra handles the ideation, the writing, and the stakeholder communication. Griffin AI handles the evaluation, the evidence, and the workflow. The LLM does what LLMs are good at. The security engine does what only a security engine can do.
That is the practical architecture. It is less exciting than "one model to rule them all," but it is the architecture that survives incidents.
The Bottom Line
Gemini Ultra is a remarkable reasoning system. It is not a security reasoning system. Security reasoning requires grounded evidence, consistent evaluation, awareness of absence, and multi-source correlation, all of which are structural properties of a purpose-built engine rather than a foundation model.
Griffin AI is built around those properties. It uses an LLM where an LLM helps and keeps the LLM out of the places where only evidence matters. That division of labor is what makes the output trustworthy, auditable, and useful under pressure.
For teams serious about operationalizing AI in their security program, the question is not which model is strongest. It is which architecture produces the right answer when it counts.