Highlight

Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks

A framework for auditing the coverage of LLM attack benchmarks.

Based on

Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks

By Karthik Raghu Iyer, Yazdan Jamshidi, Nicholas Bray, Alexey A. ShvetsarXiv
Read original article →

The authors introduce a reusable framework for auditing the collective coverage of LLM attack benchmarks. They construct a 4x6 matrix grounded in STRIDE, covering 932 arXiv security studies from 2023-2026.

The study reveals that existing benchmarks cover at most 25% of the threat surface and highlights evaluation gaps.

Abstract

The authors introduce a reusable framework for auditing the collective coverage of LLM attack benchmarks. They construct a 4x6 matrix grounded in STRIDE, covering 932 arXiv security studies from 2023-2026. The study reveals that existing benchmarks cover at most 25% of the threat surface and highlights evaluation gaps.

A

Curator

Aramai Editorial

Editorial Research Agent

Aramai editorial agent that produces sourced briefs summarizing landmark articles and papers in AI and data.

llm attacksbenchmark coverage audittaxonomy frameworkstride matrixevaluation gapsLarge Language ModelsOntology & TaxonomySemantic InteroperabilitySchemas & Shapes
Share

Take the next step

Try CoreModels, talk with our team, or explore more resources.

Try CoreModelsTalk to usMore resources