Carnegie Mellon University's CyLab reported that its free annual picoCTF competition drew more than 18,000 participants in 2025, making it one of the largest capture-the-flag events aimed at students anywhere in the world. That number is often cited as proof that gamified security training works — put points, badges, and a leaderboard in front of developers and engagement follows. The engagement part is true and well documented. What's less settled is whether that engagement translates into developers who actually write more secure code or spot more phishing attempts months later. Academic literature on Security Education, Training and Awareness (SETA) programs repeatedly names gamification as a commonly cited success factor, yet direct evidence linking game mechanics to measurable behavior change remains thin. This piece separates what the evidence actually supports — CTF-style exercises building transferable skill through applied problem-solving — from what it doesn't: the assumption that a leaderboard alone changes how anyone codes. Along the way we'll look at why picoCTF's format differs meaningfully from a points-for-completion LMS module, what design choices researchers say matter, and where security teams should be skeptical of vendor claims that gamification "improved security posture" without defining what that means.
What does the evidence actually show gamification improves?
The evidence most consistently shows gamification improves engagement metrics — time spent on a platform, module completion rates, and how often people voluntarily return — not downstream security behavior. That distinction matters because vendors and internal programs alike tend to report the metric that's easiest to measure. Completion rates and login frequency are trivial to pull from an LMS dashboard; a reduction in real-world phishing click-through or a rise in secure coding habits requires a controlled before/after comparison that most organizations never run. The research gap here is genuine rather than a minor caveat: multiple sources reviewing SETA literature note that studies linking specific game-design elements — points, badges, leaderboards — to sustained behavior change are limited in number and rigor. That doesn't mean gamification is worthless. It means a security team should treat "engagement went up" and "our people are more secure" as two separate claims requiring two separate kinds of evidence, and should be wary of any pitch that conflates them.
Why do CTFs outperform leaderboard-and-badge schemes for skill transfer?
CTF-style exercises are more consistently associated with actual skill transfer than passive points-and-badges schemes because solving a CTF challenge requires applying a technique, not just accumulating credit for showing up. A leaderboard bolted onto a video-based compliance module rewards watching the video; a CTF flag rewards finding the SQL injection, exploiting the deserialization bug, or correctly parsing the malformed input, which means the participant had to actually exercise the skill being taught. Research on learning cybersecurity through gamified formats points to a specific set of psychological drivers behind why this works: immediate feedback (you know within seconds whether your exploit worked), progressive difficulty (challenges escalate so participants stay in a productive struggle zone rather than getting bored or stuck), and social or competitive framing that keeps motivation high across a multi-hour or multi-day event. picoCTF's format reflects this directly — challenges are organized into categories like binary exploitation, web security, and cryptography, each requiring hands-on problem solving rather than passive consumption, which is a structural reason it scales to tens of thousands of participants without becoming rote.
Where does leaderboard-chasing actively work against learning?
Leaderboard-chasing works against learning when the fastest path to points diverges from the path that builds understanding, and competitive platforms make that divergence easy to find. A participant racing to climb a leaderboard has every incentive to search for a shortcut, copy a hint from a teammate, or brute-force a guess rather than work through the underlying vulnerability class — the score goes up identically either way. This is the same substitution effect long observed in gamified education generally: the visible proxy (points, rank, streak) becomes the target instead of the actual competency it was meant to represent, a dynamic sometimes called Goodhart's law in miniature. In a corporate training context this shows up as employees speed-clicking through phishing-simulation modules to log a completion badge before a deadline, without absorbing a single indicator taught in the module. The fix isn't removing competition — it's tightening the coupling between what earns points and what constitutes genuine skill demonstration, discussed below.
What design principle separates gamification that works from gamification that doesn't?
The design principle that separates effective programs from decorative ones is whether the game mechanic maps directly onto the target security behavior, rather than being layered on top of unrelated content as generic motivation. A CTF flag tied to actually finding and exploiting a real vulnerability class — an insecure deserialization bug, a broken access control check, a path traversal flaw — measures the behavior a security team actually wants developers to internalize. A points system that awards the same ten points for watching a video, clicking "complete," and passing a CTF challenge measures none of those things distinctly; it rewards participation, full stop. Vendor case studies describing internal CTF programs, such as JFrog's account of running its own security CTF for engineering staff, consistently frame the exercise around applied challenges mirroring real vulnerability classes the company cares about, not abstract trivia. The practical takeaway for a security team building a program: before adding a leaderboard, define the specific behavior you're trying to change, then check whether earning points actually requires demonstrating that behavior — if it doesn't, the leaderboard is measuring enthusiasm, not security improvement.
How should a security team measure whether its gamified training is working?
A security team should measure gamified training against behavioral outcomes it can actually observe in production, not against engagement metrics the platform reports by default. Useful signals include the rate of secure-coding-pattern violations caught in code review before versus after a CTF-style training rollout, the proportion of internally reported vulnerabilities that originate from developers who completed hands-on exercises versus those who didn't, and repeat-offense rates on the same vulnerability class in pull requests over time. These require deliberately instrumenting your own pipeline rather than trusting a training vendor's dashboard, because the platform's own numbers — completions, streaks, badges earned — measure exactly the thing gamification is good at improving and say nothing about the thing most security leaders actually care about. Given how limited direct causal evidence still is in the academic literature, treating any single gamified program as a proven fix rather than one input alongside code review, static analysis, and reachability-based triage is the more defensible position for a team accountable to a board or an auditor.