The cloud-native platform for centralized log management
Analytics, visualizations, and workflows purpose built for practitioners
Leading firms gaining more value from their machine data
Any source, any velocity – centralize logs, metrics, and traces for full visibility.
Close the gap between detection and response with an analyst-focused, cloud-native approach.
Understand complex environments with visual analysis and KPIs that matter most.
The most recent articles & research from Devo
When do you need an Event Stream Processing platform?
Every big company has hundreds of event streams constantly flowing through its corporate networks. For many business purposes, it is practical to process the data in these event streams with conventional application programs or business intelligence (BI) tools. In such cases, the application developer or business analyst typically doesn’t even think of the input data as an event stream, it is “just data.”
However, sometimes conventional applications or BI tools lack needed features or can’t calculate the desired results efficiently enough because the data is arriving too fast and the processing logic is too complicated to execute in the available time. In these cases, the application developer or business analyst should use an event stream processing (ESP) platform or other specialized software that is specifically designed to handle event data. Some examples of ESP platform products are listed at the end of this article.
How do you know if you need specialized software? It depends on the nature of the business problem. The purpose of this article is to help architects and developers identify the usage scenarios in which ESP platforms or other specialized software are appropriate.
An event is anything that happens. An event object (or “event,” event message, or event tuple) is an object that represents, encodes, or records an event, generally for the purpose of computer processing. Event objects usually include data about the type of activity, when the activity happened (e.g., a time and date stamp), and sometimes the location of the activity, its cause, and other information. An event stream is a sequence of event objects, typically in order by time of arrival.
Large companies have three kinds of event streams.
The purpose of performing analytics on one or more event streams is to extract information value from the data. A stream analytics application converts the raw input data (base events), into a form, derived events, that is better suited for making decisions. The derived events are complex events, which means that they are events that are abstracted from one or more other events (it is technically possible for a complex event to arise without being computed from other event objects but that it rare and besides the point of this article).
Stream analytics are implemented in either push-based, continuous intelligence systems or in pull-based, on-demand analytics systems. Push systems recalculate as new data arrives without being asked, in contrast to pull-based systems that run when a person enters a request or a timer sends a signal to produce a batch report (e.g., hourly, daily or weekly). ESP platforms are mostly relevant in highly demanding, push-based systems, but they are occasionally used for pull-based analytics on historical data.
When people think of ESP, they usually think of push-based continuous intelligence systems. Continuous intelligence systems ingest ongoing flows of event data. They provide situation awareness and support near-real-time, sense-and-respond business processes. Continuous intelligence systems typically refresh dashboards every second or minute, send alerts, or implement hands-free decision automation scenarios. They may be used to monitor a data source, such as Twitter, or a business operation, such as a customer contact center, supply chain, water utility, telecommunication network, truck fleet, or payment process.
Low-to-moderate volume continuous intelligence systems may be implemented in vertical or horizontal packaged applications, custom written applications, business activity monitoring (BAM) tools, operational intelligence platforms, or stream-enabled data discovery products. For example, a customer contact center monitoring system dashboard may show the number of calls and the average call duration in a half hour moving window, updated every five minutes. It could also send alerts to mobile devices when the average customer on-hold wait time during a 10 minute interval exceeds one minute. The processing overhead for this application is modest because the volume of data is relatively low (under 50 events per second), new results need to be calculated only every few minutes, and the calculations are relatively simple.
If a continuous intelligence scenario has even lighter volume and latency requirements, a conventional data discovery or BI reporting tool can be run a frequent intervals, e.g., every 10 minutes, to show the state of the operation. In effect, this kind of solution is an on-demand, pull-based application emulating a push-based continuous intelligence system by running regularly.
ESP platforms are software subsystems that process data in motion, as each event arrives. The query is pre-loaded, so the data comes to the query rather than the query coming to the data. ESP platforms retain a relatively small working set of stream data in memory for the duration of a limited time window, typically seconds to hours – just long enough to detect patterns or compute queries. The platforms are more flexible than hardwired applications because the query can be adjusted to handle different kinds of input data, different time windows (e.g., one minute or one hour instead of ten minutes) and different search terms.
For example, ESP platforms, including Twitter’s own Storm and Heron, are used to monitor Twitter, which averages about 6,000 tweets per second. A simple query might report the number of tweets that included the word “inflation” in the past ten minutes. However, at any one time, there may be thousands of simultaneous queries in effect against Twitter, each looking for different key words or different time windows.
In high volume scenarios, ESP platform applications can scale out vertically (multiple engines working in parallel on the same step in a processing flow) and/or horizontally (split the work up in a sequence or pipeline where work is handed from one engine to the next while working on the same multistep event processing query (i.e., an event processing network).
Some sophisticated ESP platforms provide native support for incremental computation to reduce processing overhead and latency. Consider an ESP application that reports the moving average price of Amazon’s stock in the most-recent ten minute window, updated every second. Stock exchanges may report thousands of price quotes and trades per second for any given instrument. For queries that involve moving windows on event data, the fastest and most efficient way to re-calculate a metric such as average price is to subtract out event data that has aged out of the window and add in the most recent data. This uses far fewer instructions than using brute force to sum all of the events in the time window and dividing by the count every time the window advances.
Some business problems require detecting patterns that involve relationships between individual pairs of events or among many pairs of events. This logic is more sophisticated than simple aggregations such as count, sum, average, maximum and minimum, because the relative order and timing of the events matter (in temporal patterns), or the relative location matters (geospatial patterns).
If the price of Amazon stock drops more than 5% within a 30 minute window, then sell our holdings.
This stop-loss order can be implemented by a continuous intelligence application that looks for the occurrence of a particular pattern. It constantly calculates the difference between individual event records that report the current price and the thirty minute high price (maximum value). The order of events matters (the lower price must occur after the higher price), the price difference (5%) between the events matters, and the relative timing matters (comparisons only occur within the 30 minute window). Systems such as this retain data on individual events in memory so relationships between event pairs can be tracked.
Business problems that requiring tracking many different people or things that are moving or otherwise changing also benefit from ESP platforms. For example:
Send a text message to me if anyone from my family and friends affinity group comes within ½ mile of my location.
This is effectively a large number of queries in operation at once. A mobile phone company that wants to provide this service to its subscribers could need to track the location of 100,000s or more people simultaneously. The system would receive tens of thousands of updates (location events) per second. For each, it must correlate the person’s location and the location of all others in their affinity group. Although the logic to calculate distance is simple, the number of instructions required to compute all of the possible combinations of people and locations would be prohibitive using conventional application design practices.
On-demand analytics are pull-based applications that support ad hoc data exploration, visualization and analysis of data. On-demand analytics is used with historical event data to build analytical models. In this context, historical means stored event streams that are hours, weeks or years old. The analytical models can be used for either of two purposes:
For example, engineers use records of sensor event streams to find patterns that predict when machines will break; security analysts study network intrusion patterns; financial risk managers simulate the effect of interest rates and economic disruptions on portfolios; and marketing departments analyze customer behavior over time.
The most common tool for on-demand analytics with historical data is a data discovery product or data science platform. However, companies occasionally use ESP platforms to run analytics on historical event streams by re-streaming the old event data through the ESP engine. This is particularly relevant when developing models for subsequent use in real-time, continuous intelligence ESP applications. For example, quantitative analysts in financial markets back-test new trading algorithms by replaying months’ worth of market data to see how the new algorithms would have performed. A variation of re-streaming event data for analytical purposes has been described as the Kappa architecture (see https://www.oreilly.com/ideas/questioning-the-lambda-architecture).
ESP platforms are not the only kind of software that is optimized for high performance analytics on event stream data. Some stream analytics products combine analytics and longer term data storage in one product. . Examples of these products, include First Derivatives KDB+, Interana Platform, Logtrust Platform, One Market Data OneTick, Quartet ActivePivot, Splunk Enterprise, and others. These products typically provide on-demand, pull-based analytics, but some are also used for continuous, push-based continuous intelligence. They ingest and store high volume event streams very quickly, making the “at rest” data immediately available for interactive queries, exploration and visualization. Their high performance is enabled by their respective proprietary data models. Some are optimized for partly structured log data, others for structured sensor or market data. They may store many TBs of data for months or years, depending on the application.
Another technology that has been used to perform on-demand analytics on historical event data at rest is the Rapide language developed at Stamford University in the 1990s. Rapide is an event-oriented modeling language that is capable of supporting very sophisticated models, including concepts such as horizontal and vertical causality. Rapide can support forensic analyses to debug the design of chips or to investigate the cause of power blackouts. A detailed description is in “The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems.” by David Luckham (Addison-Wesley Professional, 2002).
ESP Platform examples
Sign up to stay informed with the latest updates from Devo.