Ananth Tiru

Streaming Analytics - FAQ

Blog Post created by Ananth Tiru Employee on Aug 7, 2016

What value does streaming processing and analysis  provide ?


Ability to respond within the window of opportunity

For example insights such as trends, anomalies, burst detection, outlier occurrences, causal dependencies, congestion detection when obtained on streaming data can aid in decision making at the right window of opportunity.

Insights can be broadly categories into perishable and non-perishable insights.   Perishable insights are those whose value deteriorates or vanishes within a time window. Perishable  insights are useful for tactical decision making and  non-perishable insights are useful for strategical decision making.

The duration of the window depends on the vertical specific requirement.  For example in finance application the window of opportunity could be in split second, in manufacturing the window of opportunity to do something could be in seconds and in some other vertical the opportunity can be minutes.

Stream processing and analysis is the solution for capturing perishable insights and enabling the ability to take actions within the window of opportunity.


Designed for continuous computation

Stream processing  is designed from ground up for continuous processing and  computation.  A well designed streaming platform  such as HSDP is  designed for building high performance custom processing and computation applications.

Let us say I want to understand the trend in average utilization of energy in homes which is sending data 10 seconds for every 30 minutes and make a decision based on the trend then  computing the trend using streaming platform is extremely efficient.

If the same type of computation is carried out after storing the data then the complexity of computation, resource required, performance and more importantly the value will be negatively impacted.


Decision making is based on aggregated trends

Typically in any application the decision making is based on aggregated trends. For example  if I need to      alter a control in  an  industrial system (e.g. HVAC), or make a decision to place a trade or respond to an anomalous event I will do it based on aggregated trends and not a discrete  data point. Stream processing and analysis is designed to enable decision making based on aggregated trends.


Smart data lakes

     The cost of moving volumes of data, storing it and then processing and analyzing it is highly inefficient.   It is lot more efficient to process, analyze,      aggregate data prior to landing it in a data lake.  Whether you are doing descriptive or predictive analytics the quality of the data on which you are doing the      analysis  is important.  The better the data is organized the better will be the results.



What are some the key additional differentiating factors of HSDP in stream processing and analytics ?


Insights anywhere  and anytime.

HSDP offers  capability of providing insights and take actions  anywhere on the global solution network. By providing insights anywhere on the solution network HSDP enables decentralized decision making and actions.  Today's solution network is not constrained in a single data center it is highly distributed it is not practical to enforce centralized decision making.  It is not practical because the  insights and actions vary based on the location therefore it is efficient and valuable to take localized actions. Also, hauling data from the edges to the core, storing it and performing computations can get expensive.


Hierarchical computation and cascading

HSDP offers capabilities to perform computations on a geo-distributed solution network and enable cascading of insights from the edges to the cores. While it may not practical to haul raw data,  it may be useful to cascade insights (or summaries) to the next stage or to the core to address strategical needs. For example, it may be practical to cascade average energy usage from homes in each city to a core, this insight can be useful for future planning for balancing energy distribution among cities. 


Ingest  and Publish and Act continuously

HSDP offers the capability to continuously ingest data  from the sources, compute, publish insights and act on those insights.  One of the characteristics of a well designed streaming platform like HSDP is that it does not store data, compute, publish and updates the stored data with newer data and repeats the process, it simply ingests data, performs computations and publishes actionable insights.  This characteristic enables high performance (low latency and high throughput),reduced dependencies on external technologies, and simpler architecture for releasing a geo-distributed application.


What are some of the use cases I can use stream processing and analysis?

Many of the use cases we discuss below are not new, however, by enabling streaming processing and analysis  into these use cases will offer newer perspectives on reducing  costs, improving efficiencies and developing newer business model which would otherwise not be possible.  The following uses cases specified are generic and can apply to any vertical domain.


  1. Decision support systems.
  2. Monitoring and control systems.
  3. Compliance and Fraud systems
  4. Flow management systems.


Where can I learn more about HSDP ?

Please refer to HSDP FAQ Blog Post.