| Title |
Beyond accuracy: a cross-benchmark investigation of label consistency and reliability in AI-based software vulnerability Detection |
| Authors |
Iavich, Maksim ; Kuchukhidze, Tamari ; Lopata, Audrius |
| DOI |
10.3390/sym18111768 |
| Full Text |
|
| Is Part of |
Symmetry.. Basel : MDPI. 2026, vol. 18, iss. 11, art. no. 1768, p. 1-32.. ISSN 2073-8994 |
| Keywords [eng] |
vulnerability detection ; benchmark datasets ; label agreement ; symmetric relations ; data quality ; asymmetry ; data leakage ; empirical software engineering |
| Abstract [eng] |
Machine-learning research on software vulnerability detection measures progress against a small set of public benchmark datasets, treated as a shared yardstick for comparing models. This approach rests on the untested premise that benchmarks agree: if two datasets have the same function, they give it the same label. We test this assumption by comparing four popular C and C++ benchmarks against one another on their complete data. Label agreement is a symmetric relation, but its strength is uneven. The two benchmarks that derive labels from CVE-linked fixing commits disagree on only 0.68% of 75,019 shared functions, whereas a third disagrees with them on 51.21% and 41.28%. Leakage, in contrast, is directed: we identify 1083 instances in which training data from one benchmark reappears in test data from another, with different counts in each direction. A manual check of 30 contradictory functions traces the disagreements to commit-level labeling and to treating functions left unchanged by a fix as safe. The costs are asymmetric: contradictory labels impair training only slightly, while up to 3.12% of a test set carries a contradicted label, shifting measured accuracy by up to two points. Therefore, reported accuracy reflects both model capabilities and benchmark selection. |
| Published |
Basel : MDPI |
| Type |
Journal article |
| Language |
English |
| Publication date |
2026 |
| CC license |
|
