In conversations about agentic workflows and the AI-powered security operations center (SOC), I often hear two expectations together. People recognize that better data produces better security outcomes. They also expect that a capable model, given enough low-quality data, will overcome those limitations and produce great security insight.
That expectation can influence where a security team spends its next dollar. If we assume the model will compensate for the poor quality of its inputs, we may focus on improving the model and its workflows without addressing the data underneath them. However, our research at Corelight demonstrates substantial gains in investigative performance when we improve the network data available to agents, while keeping the model itself fixed.
I liken the expectation that AI will overcome poor data as a form of the well-known fairy tale, Rumpelstiltskin, where a miller convinces a king that his daughter can spin straw into gold. The appeal is understandable. AI can analyze records at a scale that would be impractical for a human, and organizations want to get more insight from the data they have already collected. But the ability to process all those records does not guarantee that an agent can get to the right answer with limited information. In fact, it may confidently declare that there is no issue based on limited data it has available, when more granular data would reveal a serious security event.
Many security analysts will recognize this distinction. The value of network evidence depends on how much it helps establish a precise account of what happened. Different data types can describe the same activity at different levels of detail, and better data will reveal subtle relationships between events. Those differences affect which questions an analyst can answer and how much work it takes to assemble a supported account of an incident. An AI agent works with the same advantages and limitations.
At Corelight, we wanted to quantify the impact of better data in the SOC. So, we built an agentic test harness around a Capture the Flag (CTF) exercise, a security investigation challenge inspired by Volt Typhoon. The exercise centered on a realistic scenario that had not been published online, helping us avoid the trap of testing models on material they already have encountered during training. Across dozens of runs, we held the language model fixed and changed the network data available to it.
In the CTF exercise, Corelight logs improved scores by more than 350% compared with NetFlow and more than 60% compared with firewall logs. Agents using Corelight data also answered the questions almost twice as fast as those working with lower-quality network logs.
The leading frontier models we tested performed similarly, and we observed very little hallucination. Yet their investigative performance changed substantially when given access to different sources of real-time data. The models had network records to analyze in each test case, but at different levels of quality and granularity, and these differences determined the quality of agentic inference.
Getting reasonable answers from an agent is only part of evaluating its performance. We also need to understand whether better source data would improve the quality and depth of those answers, or the speed of investigation. Had we asked only whether agents produced valid answers, we would have missed much of the difference between data sources. In our CTF experiment and related tests, we are able to show that better data has a broad range of positive impacts: improving investigation accuracy, reasoning around incidents, answer times, and token usage as well.
Security teams can apply that approach to their own evaluations. They can choose investigative tasks with known answers, keep the model and task constant, and examine how results change with differences in source data. That approach will help teams determine the most impactful strategy for improving outcomes.
The choice of source data becomes especially consequential as teams automate more SOC functions. Note that automation is not always an improvement: when a low-quality detection workflow moves 10x faster, the result is more noise for humans to process. Our research shows that the simplest path towards accuracy and creating workflows that are ready for that 10x boost (or more) is to ground agentic processes on the best available data.
Models will improve, and we should certainly automate the process of evaluating new models, skills, and harness architectures. But let's remember the fundamental constraint at work: low-quality data will lead to low-quality inference. Please build your agentic SOC on the most solid and enduring foundation you can, and include high-quality evidence as an essential design goal.
About the expert
Gregory Bell
Co-Founder and Chief Strategy Officer
Greg Bell is Corelight’s co-founder and chief strategy officer. Previously, he held leadership roles at Lawrence Berkeley National Laboratory, including director of ESnet, the US Department of Energy’s high-performance mission network. He oversaw deployment of the world’s first continental-scale 100G network and first 400G production link. Greg serves on CENIC’s board and holds a Ph.D. from UC Berkeley and an A.B. from Harvard.
About the company
Corelight
Corelight is a leader in enabling the defensible AI SOC.
Corelight transforms network and cloud activity into evidence that security teams use to proactively hunt for threats, accelerate response to incidents, gain complete network visibility, and create powerful analytics. Corelight's customers include Global 2000 companies, major government agencies, and large research universities. Based in San Francisco, Corelight is an open-core security company founded by the creators of Zeek®, the widely used open-source network security technology. For more information, visit www.corelight.com.