IDC estimates we will reach 175 zettabytes of data by 2025, a 61 percent increase from today’s data volumes.

Business leaders and IT executives overwhelmingly agree that they can do more to harness this data, but today’s enterprises are stuck in the land of silos and replication, and too much data wrangling that consumes an already oversubscribed budget. Data should provide high impact, continuing value, and leaders can achieve this by empowering IT to extract insight from all data and share its value with all business units. 

At Devo we’ve observed an evolution of enterprise log management adoption across clients’ businesses, meant to help leaders address where they fall in their journey towards operationalizing machine data and seeing its value. The model, below, demonstrates the level of technical and organizational maturity and reveal the kinds of architectural choices and scale needed for different use-cases. 

Consider the IDC data at the beginning of this blog and the staggering volumes of data expected to be mostly housed in the cloud. When confronted by the fact that organizations tend to harness so little of their own machine data, we must explore how to bridge this gap by establishing some of the common obstacles towards becoming fully “instrumented”, and how some companies have achieved instrumentation via the maturity model. 

Obstacles on the road to accelerated 

The road to a mature, accelerated business can be paved with obstacles. Most organizations have a blend of the following challenges that can inhibit progress: 

• There is an always-on tension between the desire to get it now vs. get it right. Line-of-business leaders looking for a new application, workflow, or business insight wanted it yesterday, and operations and business leaders must work together to strike a balance.
• When handling different data types and sources, the technology obstacles can be overwhelming. Migration challenges and data transfer are real considerations even when better and more cost effective alternatives are available.
• Log management, and overall observability, is an untold and often under-appreciated task for DevOps and cloud adoption. DevOps practices and cloud adoption alone can’t guarantee success of digital transformation initiatives, if those teams can’t harness log data from the entire environment. Unfortunately, these teams are mired in complex and distributed apps, too many point tools, and a noisy environment that makes getting to the root cause of issues even more challenging than it needs to be. 
• In addition to teams facing challenges with the tools and workloads, there’s a skills shortage placing greater pressure on DevOps teams to build and manage new infrastructure that takes advantage of machine data. 
• Many orgs try to offset costs by using open source log management; however, the cost factors associated with it can quickly stretch a budget thin.

The full promise of machine data has stalled out because infrastructures based on first-generation log management are too limited and costly. Today’s breakthrough projects require more – faster performance, the power of machine data, ML capability, all data available all the time. But that doesn’t mean there should be a constant flow of money into things like adding more servers or more developers, nor should companies cut corners by collecting less data. Instead of wrestling with tradeoffs and inefficiencies, companies need to shift to a new architecture and overcome challenges by taking steps following the maturity model. 

Enterprise Log Management Maturity Model

In countless engagements, Devo has observed the evolution that can take place when organizations harness more value from their machine data. This is the Enterprise Log Management Maturity Model – there’s a distinct, IT-centric transition between basic and traditional log management, followed by later stages of augmented, unified and accelerated, in which businesses progressively tap new data types and analytics for business model invention and transformation. Let’s discuss the stages of maturity and what they look like in each organization. 

Basic

In a basic organization, log management is deployed within a single domain in an enterprise, sometimes starting with a predefined set of related metrics and KPIs, such as network monitoring. In other words, this is bread-and-butter IT monitoring focused on a single domain or vector with known outcomes or implications, allowing organizations to stay ahead of outages, fix issues faster, manage change, report on SLAs, and plan or justify future upgrades. 

Traditional

In this phase of the model, we start seeing the first indicators of growing pains with basic log management.  Essentially the monitoring surface starts growing and enterprises realize that there is an incomplete picture in the monitoring reporting they are getting. For example, the network and enterprise periphery has moved farther out, and newer interaction and business models to support new demographics – think mobile, applications like Zelle or Venmo, and subscription-style trends like AirBnb – have come to the fore. In these organizations, IT is beginning to realize monitoring traditional data vectors and sources such as security and network aren’t enough to accommodate the volume, velocity and variety of data moving through the organization and they are missing out on key insights and business context that matters.  

Augmented

The augmented stage is the first step towards modern enterprise log management. In this stage, machine data begins to surpass previous transactional volumes and velocity and enterprise leaders realize the opportunities to tap newer varieties of data sources – often from edge devices, partner, and customer ecosystems – in order to pursue digital transformation goals. 

This stage is a graceful and more evolutionary approach to get to modern ELM, as opposed to a ‘big bang’ transition. The risk is manageable because traditional ELM and SIEM remain the workhorses for basic use-cases, at the same time organizations can embrace a cost-effective and much more scalable data platform for additional data and more sophisticated use cases such as data science and threat hunting. These parallel workflows let organizations quickly test different use cases while enabling ongoing operations to continue without disruption. 

However, while this can be a best of breed approach, these deployments to require more coordination across different operations and incident management teams – just more to manage both process and tooling.

Unified

In a unified-stage organization, all data silos have been eliminated. Multiple operations teams can make use of all machine, log, and metric data in raw form for application delivery, DevOps, service operations, security, and data science, for a variety of analytical initiatives and visualizations. In this stage, there is often a central log management team with logging as a service delivered to multiple units as a shared service. 

While a transition to a truly unified ELM can be time-consuming – 12-18 months, given the organizational and technology shifts – the benefits are worth the time investment. For example, organizations can get a cost reduction, gain agility and speed once in production, become more adept at driving common incident management and orchestration from a single ELM analytics platform, and gain holistic visibility into applications as well as improving security posture and resilience. 

Accelerated 

Enterprises become fully instrumented when they reach the accelerated phase of enterprise log management. They’ve reached technical maturity on practices such as DevOps or DevSecOps, have data governance and centralized data management teams in place with clear alignment to corporate initiatives, and most importantly in these organizations, data is viewed and treated as a primary source of competitive advantage.

In this state, traditional log and machine data is enriched, augmented and extended with business and enterprise sources of data – CRM, support, sales, marketing, even customer-provided content – ultimately compounding the value of raw data to deliver new business models and revenue generating innovations for the business.

What’s Next

The stages of maturity are simple enough to understand, although in practice, these concepts can be difficult to drive. In the next blog post of this series, we will provide examples of how Devo’s customers operated at each stage, and how they matured through the use of modern centralized log management.