Over the past year, conversations with analysts, customers, and prospective buyers have revealed a clear trend: security teams are rethinking how security data flows through their environments.
What was once considered a subcomponent of observability tooling is now emerging as a distinct architectural layer. Increasingly, analysts and industry practitioners are referring to this layer as Security Data Pipelines, also known as security telemetry pipelines.
This shift reflects a broader realization: modern security operations are fundamentally a data problem, not just a tooling problem.
Below, we explore the key trends driving this emerging category and what they mean for security architects and SOC leaders.
The Emergence of Security Data Pipelines
Until recently, telemetry pipelines were generally viewed as part of the broader observability stack, alongside metrics and traces. However, security teams are increasingly recognizing that security telemetry comes with its own distinct requirements.
The volume of security data continues to grow rapidly, driven by:
- Endpoint telemetry
- Identity logs
- Cloud infrastructure logs
- Network monitoring
- SaaS audit data
- Threat intelligence enrichment
As data volumes increase, SIEM costs rise accordingly. Many organizations are discovering that inadequate data management, rather than their detection platforms, is putting significant pressure on their security budgets.
Without effective pipeline controls, organizations may encounter:
- Uncontrolled ingestion costs
- Noisy and low-quality telemetry
- Reduced detection accuracy
- Delays in investigations
Security leaders are increasingly recognizing that data quality and control need to be addressed upstream of the SIEM. We explore this further in our article on how high-quality security data dramatically improves detection and cost efficiency.
This growing awareness is one reason analysts are beginning to recognize Security Data Pipelines as a distinct product category, providing a dedicated control plane for managing security data.
Data Quality Is Becoming a Strategic Security Capability
Many organizations initially approach pipeline management as a way to reduce costs. However, its real strategic value lies in improving data quality.
High-quality data supports:
- More reliable threat detection
- Better enrichment and correlation
- Faster investigations
- Improved AI model performance
- Fewer false positives
Conversely, poor-quality data not only increases SIEM costs but also undermines overall security outcomes.
As a result, security teams are placing greater emphasis on pipeline capabilities such as:
- Data classification
- Enrichment
- Normalization
- Schema standardization
- Deduplication
- Filtering
These capabilities transform raw telemetry into structured, meaningful security signals before the data reaches downstream tools.
For a closer look at why data quality is becoming a core security capability, read why data quality makes security pipelines strategic.
Reducing Complexity to Enable Architectural Flexibility
Although cost reduction remains a major driver, infrastructure complexity is one of the most pressing challenges facing security architects.
Large organizations rarely rely on a single security analytics platform. Instead, they typically operate a combination of solutions, including:
- Splunk
- Microsoft Sentinel
- Elastic
At the same time, organizations often introduce additional technologies and services, such as:
- MDR providers
- UEBA systems
- Data lakes
- AI-driven detection platforms
Managing these different tools can create a fragile architecture. A change to one downstream platform may require significant modifications to the data collection layer.
To address this challenge, security architects are looking for future-proof collection architectures that can:
- Centralize data ingestion
- Classify data within the pipeline
- Normalize data once
- Route the right data to the appropriate destination, such as sending low-value log messages to cold storage and security-relevant data to the SIEM
- Simplify the replacement of individual tools
This approach is commonly known as pipeline unification, where a single pipeline layer manages telemetry across both observability and security workloads.
Learn more about this architecture in our overview of unified telemetry pipelines.
A practical example can be found in this case study, which explores how a government organization carried out a Google SecOps migration using a unified pipeline architecture.
The Connection to Detection Engineering
Another emerging trend is the growing relationship between telemetry pipelines and detection engineering.
Modern detection engineering teams depend on structured, well-understood data to develop reliable detection logic. Without high-quality telemetry, even sophisticated detection rules may fail to deliver the expected results.
As one detection engineer explained:
If you want to do detection engineering, you have to have the right data—and understand what it means.
As a result, advanced security organizations are increasingly dividing responsibilities between specialized teams:
- Detection Engineering: Focuses on detection logic and rule development.
- Security Data Engineering: Manages telemetry pipelines, normalization, and enrichment.
This separation allows detection engineers to concentrate on threat logic, while pipeline engineers ensure that the right data reaches the right tools.
For example, a large healthcare organization significantly improved its observability performance by implementing pipeline-based data filtering and routing.
Processing on the Edge: Shifting Left in the Pipeline
Another important development is the shift-left approach to data processing and detection capabilities.
Traditionally, most detection logic has been handled within the SIEM. However, SIEM platforms are expensive environments for performing extensive data processing.
Security teams are increasingly looking to perform streaming analysis directly within the telemetry pipeline.
By moving classification, filtering, normalization, enrichment, and labeling into the pipeline, organizations can also begin incorporating capabilities such as:
- Sigma rule matching
- Threat intelligence matching
- Pattern and string detection
Performing these operations before data reaches the SIEM allows organizations to:
- Remove noisy or low-value data at an earlier stage
- Reduce ingestion costs
- Accelerate detection workflows
- Improve SIEM performance
This approach effectively transforms the telemetry pipeline into a real-time security data processing layer.
Read more about this development in our article on why SOCs are rethinking the role of the security data pipeline.
Optimizing SIEM Performance Through Smarter Data Design
Another challenge facing security teams is inefficient SIEM indexing strategies.
Platforms such as Splunk depend heavily on indexed fields to support fast searches and data correlation. However, poorly designed indexing strategies can significantly increase operational costs.
Security telemetry pipelines can help address this challenge through:
- Pre-normalizing fields
- Standardizing key attributes
- Controlling the creation of indexed fields
- Reducing unnecessary field cardinality
By preparing and structuring data before ingestion, pipelines help make SIEM platforms faster, more cost-efficient, and easier to manage.
For example, this guide explains how organizations can optimize Splunk indexed fields using upstream pipeline processing.
The Rise of the Autonomous Data Layer
As data volumes continue to expand, organizations are increasingly moving towards automated governance of security data flows.
Rather than manually adjusting ingestion rules across multiple tools, security teams want centralized control over:
- Data routing
- Enrichment policies
- Filtering logic
- Cost controls
- Compliance requirements
This approach is often described as an autonomous data layer, where the pipeline automatically handles:
- Data curation
- Data volume and cost reduction
- Tiered storage
- Pipeline visibility and health monitoring
Explore this concept further in our discussion of the autonomous data layer for controlling security data cost and cyber risk.
Enabling AI Security Agents
One of the fastest-growing areas in security operations is the adoption of AI technologies.
Organizations are increasingly experimenting with:
- AI SOC assistants
- Automated investigation agents
- Threat hunting copilots
- LLM-based detection systems
However, these technologies depend on large volumes of clean, structured data.
Raw telemetry is often inconsistent across different sources, making it difficult for AI systems to interpret information reliably. This is where security data pipelines play an important role.
By normalizing data into standard schemas, such as OCSF, the pipeline layer can deliver clean, structured telemetry directly to AI-driven security tools.
This approach also enables decoupled security architectures, allowing AI systems, data lakes, and SIEM platforms to consume the same normalized telemetry.
Learn more about this architectural approach in our article on the rise of decoupled SIEM architectures.
Security Data Pipeline Management Is Becoming a Core Discipline
Taken together, these developments highlight the emergence of a new operational discipline: security data pipeline management.
Modern security teams increasingly require the ability to:
- Collect telemetry at scale
- Normalize and enrich data
- Reduce noise and costs
- Route data across multiple tools
- Support detection engineering
- Prepare data for AI systems
In essence, the pipeline layer is becoming the control plane for security data.
Discover more about this emerging operational model in our overview of security data pipeline management.
The Bottom Line
Security telemetry pipelines are evolving beyond their traditional role as behind-the-scenes infrastructure components. They are becoming a strategic layer within modern security architectures.
Organizations that recognize security data as a valuable asset and establish dedicated pipelines to manage it can achieve several benefits:
- Lower SIEM costs
- Better detection outcomes
- Simpler architectures
- Faster adoption of AI security tools
In an environment where security effectiveness increasingly depends on data, the telemetry pipeline is rapidly becoming one of the most important layers of the entire security stack.
About DT Asia
DT Asia began in 2007 with a clear mission to build the market entry for various pioneering IT security solutions from the US, Europe and Israel.
Today, DT Asia is a regional, value-added distributor of cybersecurity solutions providing cutting-edge technologies to key government organisations and top private sector clients including global banks and Fortune 500 companies. We have offices and partners around the Asia Pacific to better understand the markets and deliver localised solutions.
How we help
If you need to know more about Why Your AI SOC Is Only as Good as the Data Feeding It, you’re in the right place, we’re here to help! DTA is Quest Software’s distributor, especially in Singapore and Asia, our technicians have deep experience on the product and relevant technologies you can always trust, we provide this product’s turnkey solutions, including consultation, deployment, and maintenance service.
Click here and here and here to know more: https://dtasiagroup.com/axoflow/
