Everything’s Green. So Why Is The App Still Slow? Investigating An Application Performance Incident With Owen Murphy

The checkout page is taking seconds to respond. The APM view points to POST /checkout and shows time spent in a database call. But error rates have not changed. Traffic is steady. The database appears healthy by the usual measures. CPU is 38%. Disk latency is 4 milliseconds.

The app is slow. The dashboards are green. And the application owner is expected to coordinate the answer across teams that each see only part of the incident.

It’s a common scenario. When infrastructure metrics look healthy, the assumption is often that the database is operating normally. In our recent webinar, Everything Was Green. The App Was Still Slow: A Guided Investigation Into an Application Performance Incident, we challenged that assumption through a simulated incident and demonstrated how application owners can drive a more effective investigation. While the retailer, people, and data points in this story are fictional, the troubleshooting approach reflects a practical process that application teams can apply immediately.

The Incident Was Hurting Customers, Even Though Nothing Looked Broken

Tailwind Traders is a fictional retailer with 340 stores. E-commerce accounts for 38% of its revenue. Its checkout service typically processes around 1,100 orders per hour.

During a weekday morning, checkout p95 latency increases from 430 milliseconds to 2.4 seconds, peaking at 3.1 seconds. Error rates remain at 0.2%. Throughput barely changes. Over approximately 25 minutes, checkout completion drops from 71% to 58%, resulting in roughly 110 abandoned orders in this scenario.

The application owner has a clear business problem.

They do not yet have a clear technical explanation.

That distinction matters. A healthy database instance does not prove that every query is healthy. Resource metrics show whether the system is busy. They do not always reveal whether a specific request is spending time waiting.

The Obvious Suspect Was Not the Strongest One

A release had gone live shortly before the slowdown. It was the natural place to start.

But checkout performance remained normal immediately after deployment, and the change affected the promotions module rather than the order lookup query involved in the transaction path. The release was worth checking. The timing alone was not enough to blame it.

Meanwhile, the application team already had useful information. Their APM platform identified the affected endpoint, highlighted the increase in latency, and showed a database call within the request flow. Infrastructure dashboards showed normal database and application resource utilization. The database team saw no instance-wide warnings.

None of that meant the monitoring tools had failed. In fact, they had already narrowed the problem significantly.

The next step was turning those clues into a useful handoff.

The Handoff Changed the Conversation

Instead of sending a message that simply said: “Checkout is slow. Can you check the database?”  the application owner shared a concise incident record:

• Affected transaction: POST /checkout
• Business impact: Checkout completion declining while throughput and error rates remain steady
• Incident window: Time range plus known-good comparison periods from earlier that day and the previous week
• Database platform: SQL Server 2022 running OrdersDB
• Database operation: Order lookup involving dbo.Orders and dbo.OrderLines
• Items already verified: Instance resources, connection pool behavior, and release scope

Then they asked three specific questions:

1. Did query behavior change?
2. Was the query actively executing or primarily waiting?
3. Was the pattern isolated to this workload?

That creates a much stronger starting point for the database team. The application owner has not diagnosed the database. They’ve simply supplied the evidence needed to investigate it.

The Database Was Not Busy. One Query Was Waiting.

Database-side evidence revealed a sharp increase in wait time for one query family.

Its average wait time increased from approximately 40 milliseconds to 1.9 seconds. The increase was anomalous when compared with the established baseline. The predominant wait type was lock waits. The team identified a blocking session caused by a nightly inventory reconciliation job that had continued running well into the business day. Other query families showed no comparable behavior.

Connection pool metrics weakened an alternative theory. Wait time remained just 2 milliseconds, with only 18 of 50 connections active. This became the strongest explanation to test. It was still not proof. The team ended the blocking session. Checkout p95 latency returned to approximately 440 milliseconds. Checkout completion recovered.

In this fictional scenario, that provided evidence that lock waits affecting the targeted query family were connected to the slowdown and that removing the blocker restored application responsiveness. One question remained unanswered. Had inventory table growth caused the reconciliation job to run longer than expected? The incident evidence could not prove that.

That required a separate investigation. Maintaining that distinction is important. It prevents a plausible theory from becoming an unsupported conclusion.

Recovery Is Not the Same as a Durable Fix

Terminating the blocking session solved the immediate problem.

It did not guarantee the problem would not return.

The follow-up actions included:

• Rescheduling the reconciliation job
• Reducing its locking impact
• Creating alerts for wait-time anomalies affecting the checkout query family

The team also verified recovery through application performance data, where customers actually experienced the issue, rather than relying exclusively on improved database metrics.

The practical sequence is straightforward:

1. Act on the strongest explanation.
2. Verify that the application recovered.
3. Separate the immediate remediation from the long-term fix.

A Handoff Checklist for Your Next Slowdown

Before escalating an application performance issue, make sure the next team receives:

1. The business or user impact
2. The affected endpoint or transaction
3. The incident window and a known-good comparison
4. The database platform and instance
5. The query or database operation, if known
6. What has already been ruled out, with supporting evidence
7. The specific questions that need answers:
   • Did it change?
   • Is it waiting?
   • Is it isolated?
8. The application metric that will be used to verify recovery
9. A clear distinction between confirmed findings and working theories

You do not need distributed tracing, a shared monitoring platform, or a trace link to begin this process. A transaction name and a time window are often enough to make an escalation significantly more actionable.

Database Performance Tools

Frequently Asked Questions

Why do database dashboards look healthy when an application is slow?

Traditional infrastructure monitoring focuses on aggregate resource utilization such as CPU, memory, and disk latency. A database instance can operate well within acceptable utilization thresholds while individual queries experience contention, blocking, or excessive wait times.

What is the most useful information an application owner can provide to the database team?

The most valuable handoff includes the affected endpoint, business impact, incident window, comparison baseline, known database operations involved, and evidence showing what has already been investigated or eliminated.

How does wait-based analysis differ from resource monitoring?

Resource monitoring shows how busy a system is. Wait-based analysis shows where query execution time is being lost. It helps identify waits caused by locking, memory contention, storage, or other database-level bottlenecks that may not appear in infrastructure metrics.

Do teams need distributed tracing or APM-to-database linking to use this process?

No. While integrated APM and database observability workflows can accelerate investigations, the process described here starts with information most application teams already have available through standard monitoring and incident data.