The XY-problem and senior engineering

Posted: 2022-10-23

The XY problem is a common communication antipattern in software development. Someone asks for help implementing their chosen solution rather than solving their underlying problem. Many of the posts on the topic are about public support contexts, but it’s just as important within a team or engineering organization.

Solving problems

Companies have engineering organizations to solve business problems.1 An engineering team of a certain size (SWAG of ~50) will no longer have any one person with context on all the problems the system needs to solve.2 Someone may have context on customer and cost requirements, but the internal problems of each component can be obscure. For example, query planning and index formats in a database might be tightly coupled, and if there is an installed base with lots of assumptions on query performance or large indexes of a certain format, the solutions to performance problems might be constrained.

One of the benefits of experience is that senior engineers might intuit when a question falls into the XY problem, even without context on the specific problem. They can redirect questions away from chosen solutions towards a broader discussion of the problem. There is an archetype of senior engineer who contributes mostly by knowing who is solving which problems and connecting people with related issues.3

Education

Part of the senior engineers’ job is to educate and develop junior engineers into reliable delegates. It’s therefore less important for senior engineers to detect individual XY problems than it is for them to turn the XY problem into an education opportunity. While the specifics vary by context, there are two critical elements.

1. Remove solution blinders

Very junior engineers, especially those who have only done school projects, sometimes think they will only be given concrete tasks, and that if they perform those tasks they will be rewarded and the team will succeed and everything will be great.4 A junior engineer with such solution blinders on will fall into the trap of the XY problem, because they think that the senior engineers already chose all the solutions!

A senior engineer on the project can probably answer both X- and Y-style questions, but should focus on discussing the problem as much as possible. Teaching junior engineers to think about the reason they’re doing work in the first place has high leverage for future productivity.

Senior engineers should also teach junior engineers to provide background and context as quickly as possible. This makes the junior engineer more immediately productive, since they’ll ask better questions. It can also help junior engineers ask questions sooner, maybe as soon as they realize they don’t know how to solve a problem. That’s a larger cultural effort, but XY problems are opportunities to reinforce the question-asking culture that you want.

Senior engineers should lead by example. When they work on tasks and get stuck, even if they could figure it out on their own, they should ask questions with the XY problem in mind. For example, if an engineer is building a reporting pipeline and has a mild preference for email delivery, they should ask:

  1. How the team uses reports to make decisions
  2. How the team would like to consume those reports
  3. How to send the report via email

This puts the focus on the real goal of making better decisions, demonstrates that no solution is a decree, and makes it possible to move forward very quickly if email is good enough.

2. Increase problem detection

Removing solution blinders can help junior engineers grow individually more productive. Senior engineers can have even higher leverage on their skills by teaching junior engineers to be problem detectors.

All senior engineers used to be junior engineers, and a formative experience was probably becoming the person who knew the most about a system component. XY problems are a great opportunity to teach junior engineers to detect problems, either the ones the system is meant to solve or new ones that require a larger rethink. Strong engineers will eventually see problems to solve even before anyone asks a question. When those folks design components or services, they’ll have learned to keep a tight focus on the goals, and as a consequence have a more robust system.

Thoughts

The XY problem can be frustrating. It can feel like a waste of time to the question answerer, who dove into details that turned out to be unnecessary. It can make the question asker feel dumb, since they missed an easier solution to their problem. By focusing on education and leading by example, senior engineers can direct this frustration towards a higher performing engineering team rather than angst and disappointment.

--Chris

  1. This should be noncontroversial, but sometimes it makes people feel icky if it’s interpreted as a mandating quick hacks and downplaying maintainability. Businesses generally strive to have sustainable growth in the long term. Unmaintainable software is a problem for such a business, to be prioritized and solved by engineering. Regardless of my aesthetic preferences, maintenance work is not always valuable. ↩︎

  2. This might happen even earlier, especially when the system’s users are very different from the engineers writing the system. On the other hand if most people on a team of 50 have full context, there might not be enough decomposable work for everyone to contribute productively. Whether this is a problem or not depends on team culture, and you may not be able to choose how it works out. Bored clever people are at least as good at creating problems as they are at solving them, and problem creation is often incentivized in large organizations. ↩︎

  3. See e.g. Tanya Reilly’s excellent talk on being glue. While my post uses the terms “senior” and “junior” engineer, I’m talking about role rather than layers on a corporate job ladder. In some jobs the ladder can matter a lot, and her talk goes into detail about managing your career in those environments. ↩︎

  4. Such beliefs won’t survive contact with reality for long. Another job of the senior engineer is to make that experience positive. Learning that everyone is still figuring out what to do should be exciting, not depressing. ↩︎

Home | Feedback | RSS