A report this month called "Observability: The New Technical Debt Crisis." The number that stopped me: enterprises now spend as much as ten million dollars a year watching their own systems, and still, according to Forrester, most meaningfully observe only about five percent of their applications. Millions spent, and almost nothing seen.
It reminded me of a decision I made years ago, long before anyone was spending those sums. Our managed services platform was running out of capacity, and we had a budget approved to fix it. At the same time, our service manager was spending days before every customer review pulling figures together by hand. I asked to redirect part of that budget so we could fix both problems at once. That was not the comfortable request, and it was the right one.
Two problems, only one of them visible
The capacity problem was easy to see. Everyone agreed the platform was straining, and nobody argued about spending to relieve it. The second problem was harder to see, which is exactly why it had survived so long.
We had almost no usable visibility into what the platform was doing. Logging was the worst of it. The data existed somewhere, scattered and unreachable, but no one could actually look at it in a way that helped. For a team delivering managed services to external customers, that is not a small gap. It is the difference between running a modern service and running a black box and hoping.
The cost showed up every month in the same place. Before each customer service review, the service manager spent days assembling the material by hand, pulling numbers from different corners, stitching them into something presentable, checking it, and reformatting it. Skilled effort, every cycle, to produce a static snapshot that was already stale by the time anyone read it. The work was necessary. It was also pure waste, not because the person was slow, but because the platform could not tell us anything about itself.
The harder request
So instead of spending the whole budget on capacity, I asked to redirect part of it. We would bring the logs we actually needed, the ones for incident investigation and service reporting, into one searchable place and build dashboards that updated themselves from live data instead of from someone’s weekend.
That was the uncomfortable part. Capacity was in the plan. This was not. Funding it meant taking money from something that looked more obviously justifiable to pay for something whose value was hard to point at before it existed. Visibility is always a harder sell than capacity, right up until the moment you need it and do not have it.
I pushed for it because I had spent enough years close to systems to know that invisibility is not a neutral state. It is a risk you are already carrying, whether or not it has shown up on a report. You just do not get to choose when it finally does.
What changed?
The manual reporting that used to take days collapsed into dashboards that generated themselves from live data. The reviews stopped being about assembling numbers and became about what the numbers meant. And the shift that mattered most operationally: we moved from learning about problems after a customer felt them to seeing them form while there was still time to act. For a managed service, where your product is reliability, that is the whole game.
I kept the scope deliberately narrow, and that matters. The temptation with logging is to collect everything and sort it out later. We did the opposite. We brought in the data we needed to investigate incidents and report to customers, and no more. That discipline is worth stating, because the industry has since gone the other way. Research this year found organizations now run an average of seven separate logging tools and still discard most of what they collect, with a majority saying the cost of those tools now outweighs their value. Collecting more was never the point. Being able to answer a question was.
The decision a leader can use
Before deferring an investment like this, I would ask two questions.
What is the team already spending to work around the gap? In our case, days of skilled time per review cycle indefinitely. That cost was real, it was recurring, and it was invisible because it never appeared as a line item. It just lived inside someone’s job.
And what customer problem would we detect or resolve sooner if we fixed it? That question reframes the spend from a cost into protection of the thing customers actually pay for.
The hardest investments to justify are the ones whose absence is invisible until it becomes expensive. Capacity, speed, and the things people feel get funded easily. Visibility gets deferred because you cannot photograph a risk that has not landed yet. That deferral is a decision, even when no one makes it on purpose. The job is to see the cost that is not on the spreadsheet and to fund it before the bill arrives in a form you cannot choose.
What recurring manual work has made the case for an infrastructure investment in your team?