In the business world, particularly when it comes to data software, a company’s observability is defined as the ability to collect data about program execution, internal states of modules, and the effectivity of communication between components. It’s essentially a measure of “how well internal states of a system can be inferred from knowledge of its external outputs” (Wikipedia). According to new research from observability data firm LogDNA, 74 percent of companies are still struggling to achieve true observability despite substantial investments in tools.
The firm’s new study, The Observability Challenge, shows that two-thirds of organizations are currently spending $100,000 or more annually on observability tools, with 38 percent spending $300,000 or more annually. While many organizations have four or more tools in their arsenal, they’re often dissatisfied—in fact, more than half are unable to implement the tools they want because of vendor lock-in.
The report findings were based on a survey by The Harris Poll of more than 200 senior engineering professionals who are responsible for observability and log data management at companies across the United States. The goal was to better understand the most pervasive pain points around observability.
“It’s clear that organizations need to drive more value from their observability data, but they are struggling to find tools that make it easy for multiple stakeholders to extract actionable insights,” said Tucker Callaway, CEO of LogDNA, in a news release.
The report highlights the critical importance of log data to observability use cases, such as tracking performance of applications and capacity resources, informing product improvements, and identifying threats and anomalous activity. However, fewer than half of respondents are very satisfied with their ability to use log data for any one of these purposes. And when outcomes fall short of expectations, respondents said frustration stems from three key areas—tools are not easy to use (66 percent), collaboration with colleagues on multiple teams is difficult (67 percent), and they have challenges with routing security events (58 percent). In each situation, teams report spending more time than they should trying to resolve issues.
“The Harris Poll found that the issues with cross-team collaboration appear to exist across organizations, with all levels of seniority, tenure, and company size consistently agreeing on this as a challenge,” said Callaway. “Development teams, IT operations, SRE, and security teams all need access to the same data, but they need it for different reasons, and routing that data to the right people at the right time is often cumbersome and expensive.”
Still, results show that sentiment toward observability is generally positive, with 85 percent of participants responding they believe true observability is possible. This supports a need for new innovation that will improve ease of use and facilitate stronger cross-team collaboration. Results also support increasing demand for observability data pipeline solutions that enable enterprises to ingest all of their data to a single platform, normalize it, and seamlessly route it to the appropriate teams, so they can take meaningful action quickly within their workflows.
