{"id":17444,"date":"2026-10-12T14:23:52","date_gmt":"2026-10-12T06:23:52","guid":{"rendered":"https:\/\/dtasiagroup.com\/?p=17444"},"modified":"2026-10-12T14:23:53","modified_gmt":"2026-10-12T06:23:53","slug":"how-to-cost-effectively-manage-data-retention-in-large-scale-systems","status":"publish","type":"post","link":"https:\/\/dtasiagroup.com\/vi\/how-to-cost-effectively-manage-data-retention-in-large-scale-systems\/","title":{"rendered":"How to cost-effectively manage data retention in large-scale systems"},"content":{"rendered":"
\"\"<\/figure>\n\n\n\n

The volume of data generated continues to grow, driving the need for ongoing investment and expansion in network monitoring, security, and analytics systems. As data volumes increase, organisations face growing demands on storage capacity, system performance, and operational resources.<\/p>\n\n\n\n

Cubro offers solutions that help reduce the amount of data generated and stored for network monitoring, security, and analytics while retaining all the data required for these use cases. By minimising unnecessary data, organisations can reduce the capacity and footprint required by network tools and associated data lakes, while improving performance and reducing capital expenditure (CAPEX), operational expenditure (OPEX), power consumption, and space requirements.<\/p>\n\n\n\n

This blog explores the key challenges of data retention in large-scale environments and how Cubro solutions can help address them.<\/p>\n\n\n\n

Key Challenges in Managing Data Retention<\/h2>\n\n\n\n

Managing data retention across large-scale systems presents several challenges. As data continues to accumulate, organisations must balance accessibility, storage requirements, and operational costs.<\/p>\n\n\n\n

1. Storage Capacity<\/h3>\n\n\n\n

Large-scale systems generate and accumulate vast amounts of data over time. Maintaining sufficient storage capacity to retain this information requires a scalable and cost-effective infrastructure capable of handling the volume, velocity, and variety of data generated.<\/p>\n\n\n\n

2. Data Lifecycle Management<\/h3>\n\n\n\n

Data retention involves managing the entire data lifecycle, from creation and storage to access and eventual disposal. At scale, organisations need clearly defined policies and processes for data classification, archiving, and deletion. Implementing and maintaining these practices becomes increasingly complex as data volumes grow.<\/p>\n\n\n\n

3. Data Accessibility and Retrieval<\/h3>\n\n\n\n

Retrieving specific information from large volumes of stored data can be challenging. Efficient indexing, search capabilities, and metadata management are essential for ensuring that relevant information can be located quickly and accurately. Developing and maintaining effective retrieval mechanisms can also be complex and resource-intensive.<\/p>\n\n\n\n

4. Cost Management<\/h3>\n\n\n\n

Data retention comes with ongoing storage and operational expenses. As data volumes increase, so do the costs associated with storage infrastructure, including hardware, software, and maintenance. Organisations must therefore balance their data retention requirements with cost efficiency to avoid unnecessary expenditure.<\/p>\n\n\n\n

Addressing these challenges requires effective technical solutions, well-defined policies and procedures, and a clear understanding of organisational data retention needs.<\/p>\n\n\n\n

Efficient Data Retention Management with Cubro Solutions<\/h2>\n\n\n\n

Cubro provides solutions designed to help organisations manage data volumes more efficiently, reduce unnecessary storage requirements, and ensure that relevant information remains available for network monitoring, security, and analytics.<\/p>\n\n\n\n

Filtering Non-Relevant Traffic<\/h3>\n\n\n\n

Effective data retention starts with filtering out traffic that does not contribute meaningful value to monitoring, security, and analytics activities. Cubro's Content Filtering solution uses advanced Deep Packet Inspection (DPI) techniques to filter data based on its content or metadata.<\/p>\n\n\n\n

Did you know that more than 70% of internet traffic consists of video and audio content from trusted platforms such as Netflix and YouTube? Although this traffic may not be relevant to certain data retention and LinkedIn (LI) use cases, it can still consume valuable storage capacity and system resources.<\/p>\n\n\n\n

To address this challenge, Cubro has developed an advanced application filtering solution specifically designed for large-scale, high-volume networks. The solution enables organisations to efficiently filter out non-relevant traffic, helping ensure that only meaningful data is retained.<\/p>\n\n\n\n

By reducing unnecessary data at the source, organisations can optimise storage utilisation, minimise the demands placed on network tools and data lakes, and improve the overall efficiency of their monitoring, security, and analytics infrastructure.<\/p>\n\n\n\n

\"technical<\/figure>\n\n\n\n

With this solution, the traffic bandwidth can be reduced by 60% to 70% compared to a complete capture solution. This means much lower cost and longer retention time.<\/p>\n\n\n\n

To further reduce the footprint, we propose utilizing a more efficient analytic approach as the second solution \u2013 Time Window-Based Monitoring. <\/p>\n\n\n\n

In the context of data retention, it is unnecessary to examine each individual TCP\/IP session associated with a service session. In today\u2019s WEB 2.0 traffic landscape, a service session, such as streaming a Netflix video or browsing on Amazon, does not rely on a single user-to-server connection. Rather, it encompasses multiple TCP\/IP connections. <\/p>\n\n\n\n

For instance, when streaming a Netflix video, 50 or more TCP\/IP connections may be established and distributed across various servers and different data centres, potentially even spanning other locations worldwide. The information obtained from retaining data related to these multiple TCP\/IP sessions is minimal since the traffic is SSL cyphered, making it inaccessible to anyone other than the service provider. Additionally, the messages and protocols employed within these services frequently change. However, traditional data retention solutions continue to operate based on the retention of these TCP\/IP sessions.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

For each 5-tuple connection, one NetFlow Call Detail Record (CDR) is generated. However, many of these CDRs cannot be detected as Amazon related because the external domain cannot be resolved.<\/p>\n\n\n\n

After the flow is generated, it is forwarded to the Flow Cache.  A Flow Cache can accommodate a substantial number of entries, ranging from hundreds of thousands to millions. It\u2019s worth considering that this process consumes memory resources. <\/p>\n\n\n\n

When the flows expire, they\u2019re exported to the NetFlow Collector, which will constantly analyse and archive them for future reference.<\/p>\n\n\n\n

The Cubro way uses DPI signatures to combine all the flows in one single flow.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

For each client, there is a bucket for each application, if needed, and we collect\/count all packets for a specific time window (configurable). When the window is closed, an XDR is produced\/enriched and sent out. The advantage is that the traffic is reduced by 90% and avoids constraints and performance issues in the workflow along to the  Data Lake.<\/p>\n\n\n\n

This new Cubro solution makes a huge difference to the cost of storing the data, but retrieving the data is much faster. See the table below.<\/p>\n\n\n\n

In the below table, you see a TCP\/IP flow-based storage calculation. For one TB\/sec, traffic input and 90-day retention would require 29 PB storage. This is a small data centre.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The Cubro solution now needs only 2,5 PB which is easily achievable with a data lake.<\/p>\n\n\n\n

An optimal solution is when both approaches are combined, reducing the data by filtering first and then using the Cubro CDR. <\/p>\n\n\n\n

This would result in 750 TB per 1 Tbit\/s traffic input. <\/p>\n\n\n\n

Use cases supported by Cubro CDR<\/h2>\n\n\n\n

Bandwidth Monitoring<\/strong> \u2013 Cubro CDR enables monitoring of bandwidth information per subscriber per application with a remarkable 1-second resolution. This high-resolution granularity is crucial for capturing real-time insights into network traffic patterns and ensuring accurate analysis of bandwidth consumption. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Subscriber Traffic Analysis<\/strong>: The accompanying image illustrates the efficiency of Cubro CDR. For instance, consider the WhatsApp traffic of a specific subscriber. The red graph represents incoming traffic, indicating a bandwidth of 29 kbps, while the green graph displays outgoing traffic with a mere 500 bytes. This indicates that the subscriber primarily receives messages (red) while the green traffic represents the handshake with the server where the message is received. This is just one example of how Cubro CDR provides efficient and detailed insights into subscriber traffic behaviour.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

This graph shows the application one subscriber uses in a certain time frame.<\/p>\n\n\n\n

Additional options to reduce the footprint:<\/strong><\/p>\n\n\n\n

Cubro understands that filtering may not always be an option, especially when customers want to observe all applications. To address this, we have developed a solution that aggregates services into groups. This approach significantly reduces the output of CDR and lowers the overall cost of data storage in the data lake.<\/p>\n\n\n\n

Alternatively, we can reduce the output to events over time for specific applications, providing information on which applications were used during specific time frames without including bandwidth details. This flexible approach caters to diverse data retention requirements while optimizing storage efficiency.<\/p>\n\n\n\n

Innovative Hardware Design<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Cubro\u2019s solution stands apart by leveraging ARM-based smart NIC cards, which are developed in-house. By moving away from traditional Intel Server Hardware, we significantly reduce the footprint in terms of cost, size, and energy consumption. This innovative hardware design ensures a compact solution that occupies only 19U of rack space while handling input rates of up to 2 Tbps.<\/p>\n\n\n\n

Summary<\/h2>\n\n\n\n

As the quantity of data increases then the challenges associated with its retention, access performance and costs continue to increase. Cubro\u2019s solutions deliver new, innovative ways of addressing these challenges to improve performance, reduce CAPEX and OPEX, and reduce power and space consumption.<\/p>\n\n\n\n

<\/p>\n\n\n\n

Gi\u1edbi thi\u1ec7u v\u1ec1 DT Asia<\/strong><\/p>\n\n\n\n

DT Asia \u0111\u01b0\u1ee3c th\u00e0nh l\u1eadp v\u00e0o n\u0103m 2007 v\u1edbi s\u1ee9 m\u1ec7nh r\u00f5 r\u00e0ng l\u00e0 x\u00e2y d\u1ef1ng b\u01b0\u1edbc th\u00e2m nh\u1eadp th\u1ecb tr\u01b0\u1eddng cho c\u00e1c gi\u1ea3i ph\u00e1p b\u1ea3o m\u1eadt CNTT ti\u00ean phong kh\u00e1c nhau t\u1eeb M\u1ef9, Ch\u00e2u \u00c2u v\u00e0 Israel.<\/p>\n\n\n\n

Ng\u00e0y nay, DT Asia l\u00e0 nh\u00e0 ph\u00e2n ph\u1ed1i gi\u00e1 tr\u1ecb gia t\u0103ng khu v\u1ef1c v\u1ec1 c\u00e1c gi\u1ea3i ph\u00e1p an ninh m\u1ea1ng, cung c\u1ea5p c\u00e1c c\u00f4ng ngh\u1ec7 ti\u00ean ti\u1ebfn cho c\u00e1c c\u01a1 quan ch\u00ednh ph\u1ee7 tr\u1ecdng \u0111i\u1ec3m v\u00e0 c\u00e1c kh\u00e1ch h\u00e0ng h\u00e0ng \u0111\u1ea7u thu\u1ed9c khu v\u1ef1c t\u01b0 nh\u00e2n, bao g\u1ed3m c\u00e1c ng\u00e2n h\u00e0ng to\u00e0n c\u1ea7u v\u00e0 c\u00e1c c\u00f4ng ty thu\u1ed9c danh s\u00e1ch Fortune 500. Ch\u00fang t\u00f4i c\u00f3 c\u00e1c v\u0103n ph\u00f2ng v\u00e0 \u0111\u1ed1i t\u00e1c kh\u1eafp khu v\u1ef1c Ch\u00e2u \u00c1 - Th\u00e1i B\u00ecnh D\u01b0\u01a1ng nh\u1eb1m th\u1ea5u hi\u1ec3u r\u00f5 h\u01a1n c\u00e1c th\u1ecb tr\u01b0\u1eddng v\u00e0 cung c\u1ea5p c\u00e1c gi\u1ea3i ph\u00e1p mang t\u00ednh b\u1ea3n \u0111\u1ecba h\u00f3a.<\/p>\n\n\n\n

Ch\u00fang t\u00f4i h\u1ed7 tr\u1ee3 nh\u01b0 th\u1ebf n\u00e0o<\/strong><\/p>\n\n\n\n

If you need to know more about ways to cost-effectively manage data retention in large-scale systems, you\u2019re in the right place, we\u2019re here to help! DTA is Cubro\u2019s distributor, especially in Singapore and Asia, our technicians have deep experience on the product and relevant technologies you can always trust, we provide this product\u2019s turnkey solutions, including consultation, deployment, and maintenance service.<\/p>\n\n\n\n

Nh\u1ea5n v\u00e0o \u0111\u00e2y \u0111\u1ec3 t\u00ecm hi\u1ec3u th\u00eam:\u00a0https:\/\/dtasiagroup.com\/cubro\/<\/a><\/p>","protected":false},"excerpt":{"rendered":"

The volume of data generated continues to grow, driving the need for ongoing investment and expansion in network monitoring, security, and analytics systems. As data volumes increase, organisations face growing demands on storage capacity, system performance, and operational resources.<\/p>","protected":false},"author":11,"featured_media":17445,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[56],"tags":[],"class_list":["post-17444","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-articles"],"_links":{"self":[{"href":"https:\/\/dtasiagroup.com\/vi\/wp-json\/wp\/v2\/posts\/17444","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dtasiagroup.com\/vi\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dtasiagroup.com\/vi\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dtasiagroup.com\/vi\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/dtasiagroup.com\/vi\/wp-json\/wp\/v2\/comments?post=17444"}],"version-history":[{"count":1,"href":"https:\/\/dtasiagroup.com\/vi\/wp-json\/wp\/v2\/posts\/17444\/revisions"}],"predecessor-version":[{"id":17446,"href":"https:\/\/dtasiagroup.com\/vi\/wp-json\/wp\/v2\/posts\/17444\/revisions\/17446"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dtasiagroup.com\/vi\/wp-json\/wp\/v2\/media\/17445"}],"wp:attachment":[{"href":"https:\/\/dtasiagroup.com\/vi\/wp-json\/wp\/v2\/media?parent=17444"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dtasiagroup.com\/vi\/wp-json\/wp\/v2\/categories?post=17444"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dtasiagroup.com\/vi\/wp-json\/wp\/v2\/tags?post=17444"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}