Data Analytics / By Chris O'Brien It’s a trap! If you’re going to learn how to escape the log management total cost of ownership (TCO) trap, the first place to start is understanding what the cost factors are and how they are interrelated. Then, understand what the traps are and how you can avoid them. Follow along to learn more. Data management cost factors: the reality you don’t want to hear The cost factors – which are people costs, hardware and storage costs, and licensing costs – have remained relatively constant, but we’re seeing a shift in how they balance each other out as companies generate higher volumes of machine data (logs and metrics) and look to the cloud to address performance and shift to OpEx spend. What I mean by that is companies can try to reduce one of the factors, but the other two factors may increase alongside it. For example, if you use an open source enterprise log management tool, you’ll certainly spend less on licensing, but people costs may go up. If you move to the cloud your hardware spend will decrease, but licensing costs will increase since they incorporate those hardware costs; there’s also the potential for even higher costs in an OpEx model where data retention can be costly. In general, as the economy grows, so too will costs for talent and licensing. Some businesses today try to skirt the cost factors, but in a world where machine data and business data must intersect to deliver context and insights, this is difficult to do and is what we’d call a band-aid fix. Some enterprise logging platforms will store less data in an effort to reduce OpEx costs on behalf of users, such as storing just summaries of six months’ worth of data. This method may cut costs, but it’s insufficient and doesn’t let you search all of your historical data; this is particularly notable if you have a data breach and need all your machine data to carry out an investigation. Other platforms in an effort to reduce costs might store data in Amazon S3 or another cold data storage service, but this significantly cuts performance when you inevitably need to search your data quickly. What are the data management TCO traps? There are two main traps companies fall into when they approach data management: Open source is cheap. Unfortunately, this is rarely the case. If you’re a small company starting out with a single use case and generating about 50 gigabytes of data per day, your people cost will be low, since you’ll likely just need one person to manage data ingestion, analytics, and storage, and you’ll likely see high performance. But what we’ve seen in the last five years is the increasing dependency and appreciation for an operational data platform and typically the more data you put into it, the more insights you get out. Great! But as you scale to TBs of data per day and are still using exclusively open source tools, costs will balloon due to capital expenditures like hardware, and operational expenditures like increased headcount and cloud licensing (if deployed there). SaaS solutions are the panacea. When you shift to an OpEx and cloud model, your woes will be solved, right? Probably not. What we at Devo hear regularly from large customers is that ingress and egress costs can exceed $1M per month for a competing enterprise log management platform environment – that cost doesn’t even factor in licensing spend! Additionally, when on-prem solutions are migrated (lift and shift) for use in the cloud the data management requirements migrate as well, which is not so great. Best Practices for Scoping Log Management With these two TCO traps in mind, how can you circumvent them? What are the best practices? Know your requirements. Before you take any action on your log management needs, first understand what’s required for your day to day management; track events per second, data ingestion, etc. and set your benchmarks, then extrapolate those on an annual basis. What’s easy now might not be so easy in the future. Get ahead of the talent shortage. It’s no surprise that the skills gap and overall talent scarcity has not narrowed, but this should be top of mind. Chances are you will only be able to find and afford the people you have now. Make sure your log management platform can scale with the headcount you already have. Avoid vendor lock-in. It can be expensive to switch log management platforms once you’ve chosen one, and if costs to switch are high you can also expect to see licensing costs increase accordingly. Keep your options open with a balanced solution that can handle your data volumes during the decision-making process. Cloud is here to stay, so be wary of solutions created for on-premises applications – for a cloud business, this means you may pay for the data management costs of forklifting an enterprise on-prem solution into the cloud. Having a log management solution that’s truly cloud-native with clear licensing and highly efficient usage of hardware and storage means you’ll realize the lower expenditures you’ve always needed in order to collect and analyze all your data and turn it into action with the insights you need.