For reported phishing, let ordinary code extract and check indicators, let a model draft a plain summary of what the message tries to do, and keep the verdict and any deletion or blocking with a person. The email itself is untrusted text, so assume it may try to instruct your model.
Why this is a good first workflow
Reported phishing is frequent, text heavy and mostly repetitive, which suits summarizing. Mistakes are fairly cheap to catch if a person reads the message too. Pair it with a clear employee reporting path; see a phishing reporting process employees will use.
The workflow
Step 1: intake. Reports arrive in a mailbox or form. Code collects the original message with headers, not a forwarded copy that loses information.
Step 2: deterministic extraction. Plain code parses the sender, reply to address, links, attachments and authentication results. It does the lookups: is the domain newly registered, does a link redirect, is the attachment type risky. Facts come from systems and not from model memory.
Step 3: redact. Remove personal details the task does not need, as in what log data to redact.
Step 4: model summary. Send the message body in a clearly labeled block marked as untrusted data, along with the extracted facts. Ask for a short structured output: what the message asks the reader to do, urgency tactics used and a suggested category. Validate the format in code.
Step 5: analyst decision. The analyst reads the message, the facts and the summary together, and decides: benign, spam or malicious.
Step 6: action with approval. Actions such as removing the message from other mailboxes or blocking a domain go through a person's approval, since a mistake affects many people.
Step 7: reply to the reporter. Tell them the result. A reporter who hears nothing stops reporting. A model can draft the reply for editing.
Guard against injection
A phishing email is written by an adversary, and some will include text aimed at AI readers, such as a line telling an assistant the message is safe. OWASP says there is no clearly foolproof prevention, and recommends constraining the task, validating outputs, least privilege, and human approval for high-risk actions. Give this assistant no tools beyond reading. See prompt injection in SOC AI assistants.
Test before relying on it
Collect a set of past reports with known outcomes, including tricky benign ones like real vendor invoices. Compare the model's category with outcomes and review every miss. Run the plan from the two week pilot.
Keep records
Store the message ID, extracted facts, model output and the human decision, as outlined in what to log for AI analyst actions.
Quick check
Should the model open links?
No. Let a controlled sandbox or reputation service fetch them and give the model the results, so the model never touches attacker infrastructure.
Can we auto delete messages the model rates malicious?
Not at the start. Require approval until you have measured both missed and wrongly flagged messages over a meaningful sample.
Sources
Last reviewed: 19 October 2026.
