Every big company has hundreds of event streams constantly flowing through its corporate networks. For many business purposes, it is practical to process the data in these event streams with conventional application programs or business intelligence (BI) tools. In such cases, the application developer or business analyst typically doesn’t even think of the input data as an event stream, it is “just data.”

However, sometimes conventional applications or BI tools lack needed features or can’t calculate the desired results efficiently enough because the data is arriving too fast and the processing logic is too complicated to execute in the available time. In these cases, the application developer or business analyst should use an event stream processing (ESP) platform or other specialized software that is specifically designed to handle event data. Some examples of ESP platform products are listed at the end of this article.

How do you know if you need specialized software? It depends on the nature of the business problem. The purpose of this article is to help architects and developers identify the usage scenarios in which ESP platforms or other specialized software are appropriate.

What are event streams?

An event is anything that happens. An event object (or “event,” event message, or event tuple) is an object that represents, encodes, or records an event, generally for the purpose of computer processing. Event objects usually include data about the type of activity, when the activity happened (e.g., a time and date stamp), and sometimes the location of the activity, its cause, and other information. An event stream is a sequence of event objects, typically in order by time of arrival.

Large companies have three kinds of event streams:

• The first is a copy of business transactions, such as customer orders, insurance claims, bank deposits or withdrawals, customer address changes, call data records (in telecommunication companies), advance shipping notices, airline seat reservations, or invoices. These are generated mostly internally, and reflect the operational activities of the company.
• The second are information reports, such as tweets, news feed articles, market data, weather reports, and social media updates, including Facebook and Linkedin posts. Most of these sources are external to the company, but may contain information that is relevant to a decision within the company.
• The third, and fastest growing, kind of event stream contains sensor data coming from physical assets. This is generally characterized as Internet of Things (IoT) data, and it includes GPS-based location data from vehicles or smart phones, temperature or accelerometer data from sensors, RFID tag readings, heart beats from patient monitors, and signals from supervisory control and data access (SCADA) systems on machines.

The purpose of performing analytics on one or more event streams is to extract information value from the data. A stream analytics application converts the raw input data (base events), into a form, derived events, that is better suited for making decisions. The derived events are complex events, which means that they are events that are abstracted from one or more other events (it is technically possible for a complex event to arise without being computed from other event objects but that it rare and besides the point of this article).

Event Stream Analytics

Stream analytics are implemented in either push-based, continuous intelligence systems or in pull-based, on-demand analytics systems. Push systems recalculate as new data arrives without being asked, in contrast to pull-based systems that run when a person enters a request or a timer sends a signal to produce a batch report (e.g., hourly, daily or weekly). ESP platforms are mostly relevant in highly demanding, push-based systems, but they are occasionally used for pull-based analytics on historical data.

Push-Based Continuous Intelligence

When people think of ESP, they usually think of push-based continuous intelligence systems. Continuous intelligence systems ingest ongoing flows of event data. They provide situation awareness and support near-real-time, sense-and-respond business processes. Continuous intelligence systems typically refresh dashboards every second or minute, send alerts, or implement hands-free decision automation scenarios. They may be used to monitor a data source, such as Twitter, or a business operation, such as a customer contact center, supply chain, water utility, telecommunication network, truck fleet, or payment process.

Low-to-moderate volume continuous intelligence systems may be implemented in vertical or horizontal packaged applications, custom written applications, business activity monitoring (BAM) tools, operational intelligence platforms, or stream-enabled data discovery products. For example, a customer contact center monitoring system dashboard may show the number of calls and the average call duration in a half hour moving window, updated every five minutes. It could also send alerts to mobile devices when the average customer on-hold wait time during a 10 minute interval exceeds one minute. The processing overhead for this application is modest because the volume of data is relatively low (under 50 events per second), new results need to be calculated only every few minutes, and the calculations are relatively simple.

If a continuous intelligence scenario has even lighter volume and latency requirements, a conventional data discovery or BI reporting tool can be run a frequent intervals, e.g., every 10 minutes, to show the state of the operation. In effect, this kind of solution is an on-demand, pull-based application emulating a push-based continuous intelligence system by running regularly.

Using Event Stream Processing Platforms for Continuous Intelligence

Event stream processing platforms are software subsystems that process data in motion, as each event arrives. The query is pre-loaded, so the data comes to the query rather than the query coming to the data. ESP platforms retain a relatively small working set of stream data in memory for the duration of a limited time window, typically seconds to hours – just long enough to detect patterns or compute queries. The platforms are more flexible than hardwired applications because the query can be adjusted to handle different kinds of input data, different time windows (e.g., one minute or one hour instead of ten minutes) and different search terms.

Continuous intelligence applications are best implemented on ESP platforms if

• the volume of data is high (thousands or millions of events per second),
• results must be recalculated frequently (every millisecond or every few seconds), or
• multiple simultaneous queries are applied to the same input event stream.

For example, ESP platforms, including Twitter’s own Storm and Heron, are used to monitor Twitter, which averages about 6,000 tweets per second. A simple query might report the number of tweets that included the word “inflation” in the past ten minutes. However, at any one time, there may be thousands of simultaneous queries in effect against Twitter, each looking for different key words or different time windows.

In high volume scenarios, ESP platform applications can scale out vertically (multiple engines working in parallel on the same step in a processing flow) and/or horizontally (split the work up in a sequence or pipeline where work is handed from one engine to the next while working on the same multistep event processing query (i.e., an event processing network).

Some sophisticated ESP platforms provide native support for incremental computation to reduce processing overhead and latency. Consider an ESP application that reports the moving average price of Amazon’s stock in the most-recent ten minute window, updated every second.  Stock exchanges may report thousands of price quotes and trades per second for any given instrument. For queries that involve moving windows on event data, the fastest and most efficient way to re-calculate a metric such as average price is to subtract out event data that has aged out of the window and add in the most recent data. This uses far fewer instructions than using brute force to sum all of the events in the time window and dividing by the count every time the window advances.

Some business problems require detecting patterns that involve relationships between individual pairs of events or among many pairs of events. This logic is more sophisticated than simple aggregations such as count, sum, average, maximum and minimum, because the relative order and timing of the events matter (in temporal patterns), or the relative location matters (geospatial patterns).

For example:

If the price of Amazon stock drops more than 5% within a 30 minute window, then sell our holdings.

This stop-loss order can be implemented by a continuous intelligence application that looks for the occurrence of a particular pattern. It constantly calculates the difference between individual event records that report the current price and the thirty minute high price (maximum value). The order of events matters (the lower price must occur after the higher price), the price difference (5%) between the events matters, and the relative timing matters (comparisons only occur within the 30 minute window). Systems such as this retain data on individual events in memory so relationships between event pairs can be tracked.

Business problems that requiring tracking many different people or things that are moving or otherwise changing also benefit from ESP platforms. For example:

Send a text message to me if anyone from my family and friends affinity group comes within ½ mile of my location.

This is effectively a large number of queries in operation at once. A mobile phone company that wants to provide this service to its subscribers could need to track the location of 100,000s or more people simultaneously. The system would receive tens of thousands of updates (location events) per second. For each, it must correlate the person’s location and the location of all others in their affinity group. Although the logic to calculate distance is simple, the number of instructions required to compute all of the possible combinations of people and locations would be prohibitive using conventional application design practices.

Pull-Based Applications

On-demand analytics are pull-based applications that support ad hoc data exploration, visualization and analysis of data. On-demand analytics is used with historical event data to build analytical models. In this context, historical means stored event streams that are hours, weeks or years old. The analytical models can be used for either of two purposes:

• To design rules and algorithms to be used in real-time continuous intelligence applications (see above), or
• To make one-time, strategic, tactical and long-term operational decisions.

For example, engineers use records of sensor event streams to find patterns that predict when machines will break; security analysts study network intrusion patterns; financial risk managers simulate the effect of interest rates and economic disruptions on portfolios; and marketing departments analyze customer behavior over time.

The most common tool for on-demand analytics with historical data is a data discovery product or data science platform. However, companies occasionally use ESP platforms to run analytics on historical event streams by re-streaming the old event data through the ESP engine. This is particularly relevant when developing models for subsequent use in real-time, continuous intelligence ESP applications. For example, quantitative analysts in financial markets back-test new trading algorithms by replaying months’ worth of market data to see how the new algorithms would have performed. A variation of re-streaming event data for analytical purposes has been described as the Kappa architecture (see https://www.oreilly.com/ideas/questioning-the-lambda-architecture).

ESP platforms are not the only kind of software that is optimized for high performance analytics on event stream data. Some stream analytics products combine analytics and longer term data storage in one product. . Examples of these products, include First Derivatives KDB+, Interana Platform, Logtrust Platform, One Market Data OneTick, Quartet ActivePivot, Splunk Enterprise, and others. These products typically provide on-demand, pull-based analytics, but some are also used for continuous, push-based continuous intelligence. They ingest and store high volume event streams very quickly, making the “at rest” data immediately available for interactive queries, exploration and visualization. Their high performance is enabled by their respective proprietary data models. Some are optimized for partly structured log data, others for structured sensor or market data. They may store many TBs of data for months or years, depending on the application.

Another technology that has been used to perform on-demand analytics on historical event data at rest is the Rapide language developed at Stamford University in the 1990s. Rapide is an event-oriented modeling language that is capable of supporting very sophisticated models, including concepts such as horizontal and vertical causality. Rapide can support forensic analyses to debug the design of chips or to investigate the cause of power blackouts. A detailed description is in “The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems.” by David Luckham (Addison-Wesley Professional, 2002).

ESP Platform examples

1. Amazon Kinesis Analytics
2. Apache Flink (data Artisans)
3. Apache Samza (LinkedIn)
4. Apache Spark Streaming (Databricks)
5. Apache Storm (Twitter)
6. Axiros Axtract
7. Concord.io Concord
8. EsperTech Esper, EsperTech NEsper
9. EVAM (Event and Action Manager)
10. Fujitsu Software Interstage Big Data Complex Event Processing Server
11. Hitachi uCosminexus Stream Data Platform
12. IBM Streams, Operational Decision Manager (ODM)
13. Impetus StreamAnalytix
14. LG CNS EventPro
15. Microsoft Azure Stream Analytics, Stream Insight
16. Oracle Stream Analytics and Stream Explorer
17. RapidMiner Streams
18. Red Hat Drools Fusion/JBoss Enterprise BRMS
19. SAP Event Stream Processor
20. SAS Event Stream Processing Engine
21. SQLstream s-Server
22. Software AG Apama Streaming Analytics
23. TIBCO BusinessEvents, StreamBase CEP
24. Twitter Heron
25. Vitria Advanced Analytics Platform
26. WSO2 Complex Event Processing Server