SAST (Static Application Security Testing) tools have become indispensable for modern security teams yet face fundamental limitations. While these tools excel at identifying certain technical vulnerabilities, their dependence on static rule sets, limited ability to track data transformations across complex execution paths, and lack of holistic understanding of application context and behavior often results in three critical shortcomings: incomplete detection of conventional technical vulnerabilities, overwhelming number of false positives, and an inability to detect business logic/broken authentication vulnerabilities.

ZeroPath is a SAST tool designed to address fundamental limitations in conventional static analysis approaches. To validate its effectiveness and compare it with existing solutions, we required a more comprehensive benchmarking framework than currently available. Existing evaluation methodologies lack the precision needed to accurately measure true and false positive rates across vulnerability classes, particularly for complex issues like business logic flaws and authentication vulnerabilities.

To address these challenges, we have forked the XBOW benchmark, enhancing it to overcome many of the shortfalls associated with traditional benchmarking approaches. Our adapted version now serves as a reliable SAST benchmark, providing accurate measurements of both true positive and false positive rates for conventional classes of issues along side business logic and broken authentication vulnerabilities. All changes can be found on our GitHub XBOW fork, alongside the scripts required to reproduce our findings.

Note: For more detailed information on how and why we forked the XBOW benchmark, along with a discussion of its limitations, please refer to our blog post.

Results: Breaking Down Detection Capabilities

We evaluated three leading SAST tools alongside ZeroPath: Bearer (by Cycode), Semgrep, and Snyk. Our results demonstrate the limitations of current scanning approaches across traditional and complex vulnerability types.

Technical Vulnerabilities

Scanner	Detection Rate	False Positive Rate
ZeroPath	80.0%	25.0%
Snyk	40.0%	30.0%
Semgrep	57.1%	45.0%
Bearer	5.7%	0.0%

Based on 31 benchmarks with conventional technical vulnerabilities such as XSS, SQLI, and SSTI (alongside many more classes of issues)

Business Logic & Authentication Vulnerabilities

Scanner	Detection Rate	False Positive Rate
ZeroPath	87.5%	0.0%
Snyk	0.0%	0.0%
Semgrep	12.5%	0.0%
Bearer	0.0%	0.0%

Based on 8 business logic benchmarks, including broken authentication, missing authorization, and complex data validation issues

Key Findings

Our analysis of leading SAST tools revealed significant gaps in detection capabilities across the industry. While traditional tools excelled in specific scenarios, they consistently struggled with complex vulnerabilities and produced high false positive rates. ZeroPath demonstrated notable improvements in reducing false positives while maintaining strong detection rates, particularly in complex scenarios.

Business logic and authentication vulnerability detection remains a fundamental challenge for most SAST solutions. Through advanced analysis techniques, ZeroPath showed more robust capabilities in identifying these complex vulnerability patterns, though there remains room for improvement across all tools in this category.

Comparative analysis highlighted how newer approaches to static analysis can overcome traditional limitations. While no tool achieved perfect results, ZeroPath's enhanced detection mechanisms for business logic vulnerabilities and authentication flows represent meaningful progress in addressing long-standing SAST challenges.

Tool Comparison and Features

The feature comparison reflects recent innovations in SAST technology. While established tools provide solid foundational security scanning, ZeroPath's integration of advanced capabilities like natural language patch modification and business logic analysis demonstrates the potential for more sophisticated vulnerability detection.

Language support varies significantly across SAST solutions. While our benchmark focused specifically on Python codebases, where ZeroPath showed strong detection capabilities, further testing across other languages would be needed for a comprehensive comparison. Most tools claim broad language support, but real-world effectiveness can vary substantially between languages.

Feature Comparison

Feature	ZeroPath	Snyk	Semgrep	Bearer
Secret Detection	✅	✅	✅	✅
SAST (Static Analysis)	✅	✅	✅	✅
SCA (Software Composition Analysis)	✅	✅	✅	❌
Broken Authentication Detection	✅	❌	❌	❌
Business Logic Analysis	✅	❌	❌	❌
IaC Scanning	✅	✅	✅	❌
Auto-Patch Creation	✅	✅	✅	❌
PR Code Reviews	✅	✅	✅	✅
Natural Language Patch Modification	✅	❌	❌	❌
Q&A / Code Indexing	✅	❌	❌	❌

Language Support

Language	ZeroPath	Bearer CLI	Semgrep	Snyk Code
JavaScript/TypeScript	✅	✅	✅	✅
Python	✅	❌	✅	✅
PHP	✅	❌	✅	✅
Golang	✅	❌	✅	✅
Java	✅	✅	✅	✅
Ruby	✅	✅	✅	✅
C#	✅	❌	✅	✅

Reproducing Our Results

We believe in transparency and reproducibility. All our testing tools and scripts are available on GitHub:

• CLI setup: ZeroPath CLI
• Validation benchmarks: ZeroPath Benchmarks

Follow our setup guide in the repository to run the benchmarks yourself and validate our findings.