The volume of data generated continues to grow, driving the need for ongoing investment and expansion in network monitoring, security, and analytics systems. As data volumes increase, organisations face growing demands on storage capacity, system performance, and operational resources.
Cubro offers solutions that help reduce the amount of data generated and stored for network monitoring, security, and analytics while retaining all the data required for these use cases. By minimising unnecessary data, organisations can reduce the capacity and footprint required by network tools and associated data lakes, while improving performance and reducing capital expenditure (CAPEX), operational expenditure (OPEX), power consumption, and space requirements.
This blog explores the key challenges of data retention in large-scale environments and how Cubro solutions can help address them.
Key Challenges in Managing Data Retention
Managing data retention across large-scale systems presents several challenges. As data continues to accumulate, organisations must balance accessibility, storage requirements, and operational costs.
1. Storage Capacity
Large-scale systems generate and accumulate vast amounts of data over time. Maintaining sufficient storage capacity to retain this information requires a scalable and cost-effective infrastructure capable of handling the volume, velocity, and variety of data generated.
2. Data Lifecycle Management
Data retention involves managing the entire data lifecycle, from creation and storage to access and eventual disposal. At scale, organisations need clearly defined policies and processes for data classification, archiving, and deletion. Implementing and maintaining these practices becomes increasingly complex as data volumes grow.
3. Data Accessibility and Retrieval
Retrieving specific information from large volumes of stored data can be challenging. Efficient indexing, search capabilities, and metadata management are essential for ensuring that relevant information can be located quickly and accurately. Developing and maintaining effective retrieval mechanisms can also be complex and resource-intensive.
4. Cost Management
Data retention comes with ongoing storage and operational expenses. As data volumes increase, so do the costs associated with storage infrastructure, including hardware, software, and maintenance. Organisations must therefore balance their data retention requirements with cost efficiency to avoid unnecessary expenditure.
Addressing these challenges requires effective technical solutions, well-defined policies and procedures, and a clear understanding of organisational data retention needs.
Efficient Data Retention Management with Cubro Solutions
Cubro provides solutions designed to help organisations manage data volumes more efficiently, reduce unnecessary storage requirements, and ensure that relevant information remains available for network monitoring, security, and analytics.
Filtering Non-Relevant Traffic
Effective data retention starts with filtering out traffic that does not contribute meaningful value to monitoring, security, and analytics activities. Cubro's Content Filtering solution uses advanced Deep Packet Inspection (DPI) techniques to filter data based on its content or metadata.
Did you know that more than 70% of internet traffic consists of video and audio content from trusted platforms such as Netflix and YouTube? Although this traffic may not be relevant to certain data retention and LinkedIn (LI) use cases, it can still consume valuable storage capacity and system resources.
To address this challenge, Cubro has developed an advanced application filtering solution specifically designed for large-scale, high-volume networks. The solution enables organisations to efficiently filter out non-relevant traffic, helping ensure that only meaningful data is retained.
By reducing unnecessary data at the source, organisations can optimise storage utilisation, minimise the demands placed on network tools and data lakes, and improve the overall efficiency of their monitoring, security, and analytics infrastructure.
With this solution, the traffic bandwidth can be reduced by 60% to 70% compared to a complete capture solution. This means much lower cost and longer retention time.
To further reduce the footprint, we propose utilizing a more efficient analytic approach as the second solution – Time Window-Based Monitoring.
In the context of data retention, it is unnecessary to examine each individual TCP/IP session associated with a service session. In today’s WEB 2.0 traffic landscape, a service session, such as streaming a Netflix video or browsing on Amazon, does not rely on a single user-to-server connection. Rather, it encompasses multiple TCP/IP connections.
For instance, when streaming a Netflix video, 50 or more TCP/IP connections may be established and distributed across various servers and different data centres, potentially even spanning other locations worldwide. The information obtained from retaining data related to these multiple TCP/IP sessions is minimal since the traffic is SSL cyphered, making it inaccessible to anyone other than the service provider. Additionally, the messages and protocols employed within these services frequently change. However, traditional data retention solutions continue to operate based on the retention of these TCP/IP sessions.
For each 5-tuple connection, one NetFlow Call Detail Record (CDR) is generated. However, many of these CDRs cannot be detected as Amazon related because the external domain cannot be resolved.
After the flow is generated, it is forwarded to the Flow Cache. A Flow Cache can accommodate a substantial number of entries, ranging from hundreds of thousands to millions. It’s worth considering that this process consumes memory resources.
When the flows expire, they’re exported to the NetFlow Collector, which will constantly analyse and archive them for future reference.
The Cubro way uses DPI signatures to combine all the flows in one single flow.
For each client, there is a bucket for each application, if needed, and we collect/count all packets for a specific time window (configurable). When the window is closed, an XDR is produced/enriched and sent out. The advantage is that the traffic is reduced by 90% and avoids constraints and performance issues in the workflow along to the Data Lake.
This new Cubro solution makes a huge difference to the cost of storing the data, but retrieving the data is much faster. See the table below.
In the below table, you see a TCP/IP flow-based storage calculation. For one TB/sec, traffic input and 90-day retention would require 29 PB storage. This is a small data centre.
The Cubro solution now needs only 2,5 PB which is easily achievable with a data lake.
An optimal solution is when both approaches are combined, reducing the data by filtering first and then using the Cubro CDR.
This would result in 750 TB per 1 Tbit/s traffic input.
Use cases supported by Cubro CDR
Bandwidth Monitoring – Cubro CDR enables monitoring of bandwidth information per subscriber per application with a remarkable 1-second resolution. This high-resolution granularity is crucial for capturing real-time insights into network traffic patterns and ensuring accurate analysis of bandwidth consumption.
Subscriber Traffic Analysis: The accompanying image illustrates the efficiency of Cubro CDR. For instance, consider the WhatsApp traffic of a specific subscriber. The red graph represents incoming traffic, indicating a bandwidth of 29 kbps, while the green graph displays outgoing traffic with a mere 500 bytes. This indicates that the subscriber primarily receives messages (red) while the green traffic represents the handshake with the server where the message is received. This is just one example of how Cubro CDR provides efficient and detailed insights into subscriber traffic behaviour.
This graph shows the application one subscriber uses in a certain time frame.
Additional options to reduce the footprint:
Cubro understands that filtering may not always be an option, especially when customers want to observe all applications. To address this, we have developed a solution that aggregates services into groups. This approach significantly reduces the output of CDR and lowers the overall cost of data storage in the data lake.
Alternatively, we can reduce the output to events over time for specific applications, providing information on which applications were used during specific time frames without including bandwidth details. This flexible approach caters to diverse data retention requirements while optimizing storage efficiency.
Innovative Hardware Design
Cubro’s solution stands apart by leveraging ARM-based smart NIC cards, which are developed in-house. By moving away from traditional Intel Server Hardware, we significantly reduce the footprint in terms of cost, size, and energy consumption. This innovative hardware design ensures a compact solution that occupies only 19U of rack space while handling input rates of up to 2 Tbps.
Summary
As the quantity of data increases then the challenges associated with its retention, access performance and costs continue to increase. Cubro’s solutions deliver new, innovative ways of addressing these challenges to improve performance, reduce CAPEX and OPEX, and reduce power and space consumption.
About DT Asia
DT Asia began in 2007 with a clear mission to build the market entry for various pioneering IT security solutions from the US, Europe and Israel.
Today, DT Asia is a regional, value-added distributor of cybersecurity solutions providing cutting-edge technologies to key government organisations and top private sector clients including global banks and Fortune 500 companies. We have offices and partners around the Asia Pacific to better understand the markets and deliver localised solutions.
How we help
If you need to know more about ways to cost-effectively manage data retention in large-scale systems, you’re in the right place, we’re here to help! DTA is Cubro’s distributor, especially in Singapore and Asia, our technicians have deep experience on the product and relevant technologies you can always trust, we provide this product’s turnkey solutions, including consultation, deployment, and maintenance service.
Click here and here and here to know more: https://dtasiagroup.com/cubro/
