Benchmarks Find ‘DeepSeek-V3-0324 Is More Vulnerable Than Qwen2.5-Max’

With the latest stable release dated January 28, 2025, Qwen2.5-Max is classified as a Mixture-of-Experts (MoE) language model developed by Alibaba. Like other language models, Qwen2.5-Max is capable of generating text, understanding different languages, and performing advanced logic. According to recent benchmarks, it is also more secure than DeepSeek-V3-0324.

Using Recon to scan for vulnerabilities

A team of analysts with Protect AI, the company behind a red teaming and security vulnerability scanning tool known as Recon, recently used their platform to compare the security of Qwen2.5-Max against that of DeepSeek-V3.

The team’s assessment reads, in part: “We observed that DeepSeek-V3-0324 is more vulnerable than Qwen2.5-Max, with Recon achieving an almost 25% higher attack success rate (ASR).”

While it may be more secure than its competition, Qwen2.5-Max isn’t exactly perfect. According to their tests, the AI model is most susceptible to prompt injection attacks, as these represented almost 48% of all successful cyberattacks against Qwen2.5-Max. Evasion and jailbreak attacks proved to be less successful with an approximate ASR of 40% for both.

Exposing vulnerabilities in DeepSeek-V3

Recon utilizes a comprehensive Attack Library to scan current-gen AI models and identify vulnerabilities across six specific categories:

• Evasion techniques
• System prompt leaks
• Prompt injection attacks
• AI jailbreak attempts
• General safety controls
• Adversarial suffix resistance

In addition to simulated cyberattacks, Recon also assesses the AI models’ resistance to generating potentially harmful or illegal content. For example, during adversarial suffix resistance tests, Recon attempts to manipulate the AI model into generating harmful or illegal content.

The Protect AI team ran Recon against both Qwen2.5-Max and DeepSeek-V3, with the former boasting a lower attack success rate (ASR) across a variety of attacks; including jailbreaks, prompt injection, and evasion techniques.

Whereas Qwen2.5-Max had a 47% ASR against prompt injection attacks, compared to DeepSeek-V3’s notably higher 77%. Against evasion techniques, Qwen2.5-Max scored a 39.4% ASR against evasion techniques, while DeepSeek-V3 scored 69.2%. Both AI models displayed similar results across other simulated cyberattacks.

Analyzing DeepSeek-V3’s strengths

Despite its security weaknesses, DeepSeek-V3-0324 still outperforms Qwen2.5-Max in several different benchmarks. Unlike the ASR, a higher score in these tests actually indicates better performance.

According to these benchmarks, DeepSeek-V3-0324’s strengths include general language understanding (MMLU-Pro), advanced topics such as biology, physics, and chemistry (GPQA Diamond), mathematics (MATH-500, AI in medicine (AIME 2024), and coding (LiveCodeBench).