Your application is highly data intensive.
You scored a [INSERT SCORE] out of 100. You score high in [CATEGORIES], and medium in [CATEGORIES]. Organizations with this level of data intensity often struggle to meet their Service Level Agreements (SLAs), and have to deploy multiple data storage technologies to compensate for weaknesses in the core database. Peak times are a significant challenge, and can lead to missed SLAs that negatively impact your business.
|
|
Data Size
Modern databases must be able to effortlessly manage terabytes to petabytes of data generated by increasingly data-intensive applications.
Data sizes of less than 1 TB are lightweight analytics, or small-to-medium operational workloads.
Data sizes 1 - 10 TB are very large for operational workloads, but medium-sized for analytics workloads.
Data sizes of tens of TBs or larger are typically big data analytics.
|
|
|
Data Ingest Speed
Data-intensive applications must be able to load - or ingest - data at very high rates, including hundreds of thousands or millions of rows per second.
Many databases can handle a few hundred to a few thousand rows per second. Workloads with thousands to millions of rows per second or higher are typical in data-intensive applications. This is where many data infrastructure systems struggle.
|
|
|
Query Complexity
Data-intensive applications must be able to handle both simple and complex queries equally, effectively and fast.
Complex data queries join multiple data sets with multiple filters and often include aggregation and other calculations. The more joins and subqueries, the greater the complexity.
|
|
|
Low
|
Most operational databases (and some data warehouses) can handle low complexity queries in fractions of a second.
|
|
Med
|
There are several databases that can manage medium complexity with simple joins, distributed databases perform scale better.
|
|
High
|
Highly complex queries on a large data set require the use of distributed systems to run.
|
|
|
Query Latency
Query latency is the amount of time it takes a database to execute a query and receive a result. Data-intensive applications often have strict SLAs, or requirements around query latency.
Query latency requirements can vary dramatically depending on the use case.
Minutes/1-10 Sec Daily reports can take minutes or hours to complete. Single-node databases or big data open-source databases are sufficient.
100 ms - 1 Sec Applications delivering interactive experiences need queries to complete in less than half a second.
10 - 100/0-10ms For operationalized Machine Learning use cases, queries need to be < 100 ms.
|
|
|
Concurrency
Data-intensive applications often need to support a large number of users or concurrent queries while managing low-latency queries simultaneously.
Data infrastructures need the capacity to meet demand; however, most legacy data warehouses were built for generating reports, not handling concurrency. While many operational databases have improved their ability to handle concurrency, they do so only for simple queries.
-
Applications with a limited set of users (e.g., departmental applications, low traffic web applications) generate few queries.
-
Fast-growing SaaS applications with high numbers of concurrent users generate tens to hundreds or thousands of queries at peak times.
|
|
|
|
