CWE-1434 covers a component that relies on a generative AI or ML model configured with inference parameters that produce an unacceptably high rate of erroneous or unexpected outputs.
It is relevant to teams that deploy models and use their output for anything with consequences.
How it happens
Text and image models use inference parameters that steer their behavior, such as temperature, Top P and Top K. They affect the probability distributions the model samples from. Wrong settings can produce text hallucinations, unrealistic images, or failure to converge in training.
On its own that is a quality problem. It turns into a security problem when results feed security critical operations or decisions, because then the application can violate its intended security policy.
The record lists no CVE examples.
What an attacker gains
MITRE lists integrity impacts. The product can generate inaccurate, misleading or nonsensical information, and if outputs drive critical decisions, errors can spread to other systems or components.
How to find it
MITRE lists automated and manual dynamic analysis: vary the inference parameters and compare results to assess the effect of chosen values. Build a suite of systems with targeted tools for problems such as prompt injection (CWE-1427), and consider statistically measuring token distributions.
How to fix it
- Tune with a process. Develop and follow parameter tuning with extensive testing and validation, at implementation, configuration and operation.
- Keep measuring. Implement feedback mechanisms that assess and adjust model performance continuously.
- Document it. Provide comprehensive guidance on parameter settings so behavior stays consistent.
Treat the settings as reviewed configuration, kept in version control and changed through the same review as code.
Quick check
When do inference parameters become a security issue?
When model output is used in security critical operations or decisions, since erroneous output can then violate the security policy.
