Skip to content
Nerdy beta

Uncovering a $1M Developer Experience Gap

Engineering Developer Experience Metrics Case Study

Background

A multi-brand e-commerce group of roughly 150 engineers was heading into fiscal-year planning with five investment areas sketched out with little evidence to back any of them. The portfolio consisted of eight acquired platforms, each with its own stack, pipelines, and architectural history. The CTO wanted real numbers, defensible enough for the board.

We ran our Improving Developer Experience program: fourteen drivers were selected with their engineering leaders, an organization-wide survey was conducted through our unique partnership with leading measurement tool maker DX, and several interview sessions were held with a core sample of respondents to understand the context behind the numbers. It’s worth noting that 99% of contributors responded.

Their results were great, but there’s always room for improvement. The Developer Experience Index (an average across all sampled drivers) returned a DXI score of 75, placing the group firmly in the top 25% (P75).

So we benchmarked this group against the 90th percentile (P90), which means attaining a DXI score of 82. Every point below represents an opportunity for improvement and time/cost savings.

In short, a higher score means more free developer time to work on modernizations, migrations, new features, and new capabilities, rather than losing time to flaky test suites, slow builds, frequent interruptions, and the like.

Interpreting the Data

To keep this short, we’re presenting a small sample of the 14 drivers measured in the DX survey.

DriverScorevs P90VotesComments
Customer focus58-13#1013
Build and test63-16#322
Documentation65+3#216
Deep work68-10#123
Dev environment69-10#613
Change confidence92-1#125

“vs P90” is the gap to the top 10% of the industry benchmark. “Votes” is where engineers ranked that driver when asked what to fix first. Though it seems counterintuitive, it’s common to see something score fairly high but be prioritized by the developer community. These discrepancies warrant further investigation.

Build and test carried the largest gap from P90 at -16. Eight teams named the same third-party static analysis gate for false positives and blocked merges. One team measured it: CI steps disrupted delivery approximately 15-20% of the time. Another described four review mechanisms — static analysis, linters, persona reviews, and AI bot reviews — that contradicted each other to the point where a single-commit pull request became nearly impossible.

Documentation scored above the benchmark yet still drew the second-most votes. +3 against P90, but engineers wanted it fixed anyway. Upon further investigation, we discovered the problem was around discoverability and consistency, not necessarily quality. With central platform groups supporting various properties acquired through M&A, this inefficiency is real and worth solving.

Deep work drew the most votes and comments while scoring mid-range, and the fix was a mix of better meeting design, jettisoning a status-oriented “Scrum of Scrums” for an aggressive blocker-report-and-remove session, and tightened calendar discipline.

Change confidence at 92 sat within a point of the top decile. We like to call out the bright spots, and this was certainly one of them.

Another feature of the DX survey is allocation: where are teams spending their time? Are they delivering new product features to satisfy unmet user needs, or are they keeping the system running?

New capabilities accounted for 56% of engineering effort, compared with the top-decile benchmark of 63.8%. Closing that gap shifts about twelve developers’ worth of work from maintenance to building things each year, providing a noticeable boost to modernization. The goal is clear: increase the proportion of new development to toilsome or repetitive KTLO-BAU.

Time is Money

DX researchers publish a statistically significant correlation quantifying gains: one point on the Developer Experience Index equates to about 10 hours saved per contributor per year. We employed this strong, research-backed correlation to model the ROI of investable areas.

At 150 contributors and a blended rate of $100 an hour, each additional point of DXI yields 1,500 hours of saved time, valued at $150,000.

Closing the seven-point gap between their score and the P90 threshold frees up 1,300 engineer-days valued at just over $1,000,000 — a significant amount of time and money that could be put toward work with a bigger payoff.

Our Recommendations

We counted eight opportunities, ranked by 1) contributor sentiment, 2) how far the score sat below benchmark, and 3) how cheap it was to move.

The first three cost almost nothing: convert the cross-team status meeting into a tracked dependency-resolution meeting, restore a no-meeting day, audit the four overlapping review mechanisms down to a coherent set, and fix the static analysis configuration in the design system repository. These motions fit neatly into the existing continuous improvement budget established by leadership.

The fourth opportunity related to internal platform resourcing. Three engineers and two architects on a nascent platform meant to consolidate ten properties are simply not enough to move the needle on modernization in any meaningful way. We recommended increasing staffing while shifting the platform group from building in anticipation of consumer demand to building with them, letting real onboarding evolve the platform capabilities. This is a platform as a product play rather than a speculative development play, and it is the investment tied most directly to the allocation gap.

For documentation needs, we recommended defining a handful of useful documents (OpenAPI specs, runbooks, READMEs, high-level C4 diagrams, etc.) that each product team should maintain. We suggested building a simple doc drift agent that compares document freshness against the code commits to be implemented in an AI PR review tool.

Socializing the Results

A table with fourteen columns, each showing every team’s scores, will be read as a scoreboard, so we cautioned against this instinct at every opportunity. These numbers describe systems and perceptions, which differ team-to-team.

For example, a team in a decades-old monolith and a team on a five-month-old platform should not be expected to have comparable build times; different values can be healthy given a team’s context. Engineers in two sessions raised this unprompted, along with the observation that any metric framed as a ranking eventually gets gamed: even if they don’t know Goodhart’s Law, people will aim for a single metric if it becomes a target, with unintended and harmful consequences.

DX’s survey works team by team, each reading its own column against its own previous snapshot. This forms a sort of personal record or personal best metric, with each team taking agency to improve what’s relevant to them. That takes a manager who can tell which low score is a real constraint versus which numbers come with the territory of a given system or product. Without that reading, a snapshot is a static list of numbers. Our Using Metrics Responsibly guide says more about a POV and is worth a look.

Summary

Our program ran for six weeks and defined five investment areas, each with a ranked list, a dollar cost, an ROI, and a baseline to test against in the next quarter. This takes the same shape as our ROI Calculator — blended rate multiplied by time multiplied by frequency.

The organization did not need to be told it had problems. It needed them priced, ranked, and separated from the things it was already good at. By combining, we were able to tie investment opportunities to dollars and a defensible ROI.

The best part of this approach is that it invites everyone into a transparent dialogue about what makes sense for them to tackle in the next quarter. As we’re fond of saying, people will support a world they help to create.

Resources