> ## Content Index > Fetch the complete content index at: https://www.notatechguy.com/llms.txt > Use this file to discover other available public pages before exploring further. # Tencent: AI coding agents see security safeguards in 27-36% of runs - URL: https://www.notatechguy.com/tencent-ai-coding-agents-see-security-safeguards-in-27-36-of-runs/ - Published: 2026-11-05T14:47:30.000Z - Updated: 2026-11-05T14:47:30.000Z - Description: Tencent's Wukong team built CASE-Bench from 145 real CVEs to measure how repository context shapes the security of AI-generated code. - Author: Marcello Babbili - Tags: Technology & AI, OpenAI, AI Models Tencent's Wukong Code Security Team found that AI coding agents notice the relevant security safeguard in only 27 to 36 percent of their attempts [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com). The context those agents assemble before writing code is tuned almost entirely for functional correctness, so the model may never see the authorisation check or input validation it should be imitating [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com). **My read:** The 27 to 36 percent figure is the number that should worry any team shipping AI-generated code into production. What I am skeptical of is the "up to" framing on the percentage-point effects: these are maxima across 13 methods and six models, not averages, so the typical impact on a single workflow could be smaller. The finding I would watch is the 6.3-point jump from a single insecure exemplar, because real repositories are full of legacy code that would qualify. ### How CASE-Bench works The team, led by Jiongchi Yu and colleagues, built CASE-Bench: 145 tasks drawn from real CVEs across 51 open-source projects [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com)[P²](https://arxiv.org/abs/2611.06294?ref=notatechguy.com). Each task runs in a Docker container with functional tests and a proof-of-concept exploit validated in both directions, meaning the benchmark confirms both that a fix works and that the original vulnerability was real [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com). Every task is annotated with the specific repository code that enforces or undermines its security, so the researchers can control exactly what the model sees [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com). This matters because existing secure-code benchmarks treat the context as fixed [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com). They hand the model a prompt and check the output. Real coding agents do not work that way: they crawl the repository, grab files they think are relevant, and build a context window before generating anything. That process, designed to make code functionally correct, can surface insecure patterns for the model to copy or bury the safeguards the model needs to see, a dynamic that echoes how [a new LLM backdoor hid attacks in logical reasoning](https://www.notatechguy.com/new-llm-backdoor-hides-attacks-in-logical-reasoning/) rather than in explicit instructions. The team compared 13 context construction methods against six controls across six LLMs, then edited individual context elements and traced coding-agent trajectories to see what the models actually looked at [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com). The specific models are not named in the paper. ### The gap between working code and safe code Functionality-oriented context lifts the functional pass rate by up to 8.2 percentage points but lifts the secure pass rate by at most 3.7 points [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com). The same context-building step that helps the model write code that works does less than half as much to help it write code that is safe. Remove a safeguard from the context and the secure pass rate drops by up to 8.1 points [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com). Insert a single insecure code example and correct-but-exploitable output rises by 6.3 points [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com). The model does not need to be told to copy a bad pattern. It just needs to see one. Coding agents observed the relevant safeguard in only 27 to 36 percent of trajectories, meaning the security-critical code was either not retrieved or not attended to the rest of the time, even when it existed in the repository [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com). ### What holds and what doesn't The paper is a preprint on arXiv and has not been peer-reviewed [S¹](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com). The percentage-point figures are maxima, not averages, so they represent the strongest observed effect across all tested conditions rather than a typical result. The six LLMs are unnamed, which makes it impossible to judge whether the findings apply to the specific model a given team uses. This work lands alongside a wider push toward context-aware code security: [Qodo added company context to OpenAI Codex code review](https://www.notatechguy.com/qodo-adds-company-context-to-openai-codex-code-review/) on 2 November, and [OpenAI hardened its own defences with AI in a new security sprint](https://www.notatechguy.com/openai-hardens-defences-with-ai-in-new-security-sprint/) the following day. A team running a coding agent in production should check what their context builder retrieves before the model generates code. If the retrieval step surfaces insecure legacy code or skips authorisation checks, the model will write code that passes tests and fails security review. The CASE-Bench code and the A.S.E v2.0.0 release are both on GitHub, so any team can run the 145 tasks against their own model today [P⁴](https://github.com/Tencent/AICGSecEval/releases/tag/v2.0.0?ref=notatechguy.com)[P⁵](https://github.com/BriansIDP/CASEBench?ref=notatechguy.com). --- *Sources: [S1 — A.S.E 2.0: Understanding How Repository Context Impacts the Security o](https://arxiv.org/abs/2611.06294v1?ref=notatechguy.com) · [P2 — A.S.E 2.0: Understanding How Repository ContextImpacts the Security of](https://arxiv.org/abs/2611.06294?ref=notatechguy.com) · [P3 — RepoFusion: Training Code Models to Understand Your Repository](https://arxiv.org/html/2306.10998?ref=notatechguy.com) · [P4 — A.S.E v2.0.0](https://github.com/Tencent/AICGSecEval/releases/tag/v2.0.0?ref=notatechguy.com) · [P5 — BriansIDP/CASEBench](https://github.com/BriansIDP/CASEBench?ref=notatechguy.com)* ## Related reading - [Qodo adds company context to OpenAI Codex code review](https://www.notatechguy.com/qodo-adds-company-context-to-openai-codex-code-review/) — our technology desk, 2026-11-02 - [New LLM backdoor hides attacks in logical reasoning](https://www.notatechguy.com/new-llm-backdoor-hides-attacks-in-logical-reasoning/) — our technology desk, 2026-09-22 - [OpenAI hardens defences with AI in new security sprint](https://www.notatechguy.com/openai-hardens-defences-with-ai-in-new-security-sprint/) — our technology desk, 2026-11-05 --- *Written from 5 sourced items, 4 of them primary.*