I Like Legacy Systems

Most of the decade or so I’ve spent working professionally in software has been spent in codebases written by others. Almost all of them existed before I started working on them. Some have predated almost all of the engineers who worked on them. Their tech stacks and design philosophy were typically a couple generations behind the technological zeitgeist. You could pretty easily find ugly code in them. Legacy systems, in other words.

Judging by recruiter emails, forum posts, conference talks, etc, people don’t like legacy technology. It’s a dead end, right? Good engineers like to build the future with the latest exciting technology. If you do have to touch a legacy system, it should be toward the end of splitting the tangled legacy mess into modern services, achieving digital transformation and so on. A legacy system, by comparison, is at best dull. You work on it because you’re an engineer who’s not passionate about the latest technology and you don’t want to leave your comfort zone. If you don’t want to become that, you should aim to transfer onto a different project, lest you become outdated by not knowing the latest framework. ¹

I like legacy systems. I owe much of what makes me a good engineer to years spent working with them. I think they’re a fine way to learn basic engineering principles, and I think they’re are uniquely good at teaching some things. I know I won’t convince many people to share my affinity for them, but I hope I can outline some of the positive aspects of working with legacy systems.

I think a few types of reader might benefit from this article:

• People entering the industry who despair that they’re starting off on the wrong foot by not working on brand new projects with the latest technology at their first job.
• People who operate at a high level (director, principal, architect, etc) who make decisions about the shape of technology in a company at a high level, without necessarily delving into implementation details; in particular, people in this category who did now grow into these roles from within the company.
• Senior people who’ve spent their career in mature codebases and want a reminder that their hard work wasn’t a dead end, and of the maybe non-obvious but valuable things their hard work helped them to learn.

Preface

For the sake of this article, I’ll define a legacy system as a project for which the following are true:

• is user- or customer-facing (for whatever notion of customer/user is applicable).
• spent a significant amount of time (months or more) under active development while being used, usually by a variety of developers over time.
• performs some critical function for a business in its current form (customer-facing feature, some sort of internal reporting, etc).
• is still under active development (possibly at a reduced pace compared to when it was new), where active development includes new features as well as bug fixes and other maintenance.
• exists alongside a different/alternative system or pattern that (for whatever reason) is preferred by developers and the business for new features.

A consequence of the last point is that legacy vs. non-legacy is relative, and depends on more than the tech stack/architecture/design choices of a project. I could write more on that but I won’t; suffice it to say that it agrees with my experiences in the industry. ²

I should also note that my perspective is shaped by a career spent predominantly in small startup or startup-adjacent companies, working on B2B or B2C products on the web. Readers with different experiences (more mature companies, non-web engineering, etc) may not relate to what I’m discussing here.

It’s fairly obvious from the following, but I have a strong bias towards not throwing away a working system, and I think a lot of systems qualify as working systems.

Things legacy systems are particularly good at teaching

I’ll say up front that you can (and should!) learn any of the skills below on any project, legacy or no. I think legacy systems are especially good at teaching them, because:

• Their original authors are often gone or otherwise unavailable to answer questions, so you have to work harder for answers than you would otherwise.
• They’ve generally been around long enough for people to have asked them to do things that their initial creators didn’t think of, increasing the likelihood that product or business requests will differ from what is easily possible.
• Similarly, they’ve been around long enough for business and product stakeholders to depend on them. Due to roadmaps, business needs, etc, this introduces time pressures that an initial green field project might not have, leading to implementation compromises one wouldn’t see in a fresh green project.
• They’ve often grown to a level of internal complexity above and beyond many fresh projects.
• They’ve existed long enough for multiple people other than the original authors to contribute designs and code, which may introduce subtle differences in design philosophy, code style, etc.

In other words, the reasons many people dislike legacy systems can make them good teachers.

Reading code

Code reading is an essential skill for my work. I use it to learn my way around when I switch teams or companies. I use it to give code reviews and other feedback. I use it when responding to a late-night production outage involving a feature I’m unfamiliar with, to feel confident that my own code isn’t going to have a negative impact elsewhere in a project, even to look at third-party library code when investigating bugs. Legacy systems are excellent places to get good at code reading. Their age, the fact that they reflect a product roadmap over months and years, and the fact that many hands have touched them make them more challenging to grok at first glance. I found this intimidating as a less experienced engineer. I slowly got better over time, and as a more senior person have a grab bag of tricks and shortcuts to help make the process easier. I ramp up a lot quicker as a new hire now. The shortcuts and tricks apply on a smaller scale, too. I can use them to quickly vet an idea for a PM, or dive deep into a pull request and give insightful feedback. I think I would be a lot worse at this if I’d spend my career avoiding legacy projects.

Working with constraints

It can be limiting to work in legacy systems. Large, risky refactors may be inappropriate or may require special care on a critical, customer-facing system. We may not be able to change a core component of a tech stack in a years-old system with a packed product roadmap and few developers. A poor data modeling choice years ago may be effectively untouchable in the present due to the effort involved in migrating away from it. Green field projects, at least at inception and during initial buildout, can seem free of these constraints. Engineers have a brief moment of freedom to make everything just right, setting the project up for success years later. ³

I find constraints challenging and interesting. Thinking of a way to build a brand new system in isolation to solve a particular product requirement is not usually that challenging. Taking that same product requirement, breaking it down into concepts that make sense within a large existing system, implementing it in a way that’s consistent with that system & rolling it out in a way that causes no disruption for existing system users can be very challenging. It’s fun to meet and solve these challenges, and, in doing so, I build a deeper and richer collection of techniques and design patterns than I would if I was able to work with fewer constraints.

Product and business needs

The constraints of a legacy system don’t just affect engineers. I often have to communicate those constraints to non-engineers who want the system to do something that it doesn’t and (due to constraints) can’t. My job as a senior engineer isn’t to simply say no; it’s to propose alternatives that do something like what a non-engineer originally wanted, but maybe look a little different than what they were thinking of. I need a good intuitive sense of what non-engineers care about to do this well, along with the communication skills to convey constraints in an understandable way (i.e., without a bunch of engineering jargon). This is a good skill to have in general; I don’t think I’ve ever wished that I was less able to understand what someone from the business was asking for. I’m pretty good at this now (after 10 years), and I think working in legacy systems (where I have to say no more often than I might otherwise) has helped me become good at this.

Empathy and tolerance

I have a tendency to be more critical of code style than is sometimes necessary or productive. Working with legacy systems has helped me notice and correct this instinct.

First, it’s helped me to see how much code style actually contributes to the long-term maintainability of a project. A project that has giant, ugly function definitions but gets the core concepts of its problem domain right, has good automated tests, and some coherence/consistency about what-goes-where is often pretty nice to work in. A project can have beautiful, idiomatic code but be absolutely awful to work in due to incorrect or inappropriate abstraction or domain modeling. Code style — the thing I most often look at for a knee-jerk assessment of the quality of project, and the thing called out in so many engineering blog posts and books — doesn’t turn out to tell me a whole lot about the big picture. Even for the original authors of a system, the correctness of the big picture may not be obvious until months or years after a system launches. Pretty code is nice, and ugly code is a speed bump, but I care more that the big picture is reasonable, and I’m grateful when I realize that a developer who I might have judged at first for long functions got the big things right. ⁴

There’s also usually a why behind ugly code, and it can be interesting to learn. In some cases the ugly code is a reflection of a subtle business requirement (often one that I didn’t know about), and some quirk of it is actually really important. In other cases it is just ugly, but maybe it’s ugly because a company co-founder wrote it years ago for an important demo, or it was the best last minute solution to a prod emergency, or something else. Giving a past developer the benefit of the doubt and assuming that they had a good reason for what they wrote makes me more likely to explore a commit log or ticket tracker and actually discover these things.

Seeing how design decisions pan out

There’s a quote from “Teach Yourself Programming in Ten Years” by Peter Norvig ⁵ that I like:

Work on projects after other programmers. Understand a program written by someone else. See what it takes to understand and fix it when the original programmers are not around. Think about how to design your programs to make it easier for those who will maintain them after you.

I’ve been in plenty of earnest whiteboarding sessions, design reviews and code feedback threads over the years where future maintainability or enhancement was (justifiably) a point of discussion. I have my own biases about what is and isn’t important for future maintainability (some of which I’ve alluded to here), as do other engineers. It can be hard to refine and develop these instincts over time, especially if I move between jobs every year or two. I might not be able to say with confidence that a foundational design decision I made was correct until a year or more after I’ve made it (taking into account the time required to implement the design, and the time required for others to work with and extend the design). ⁶ Working in a project that’s years old exposes me to many time tested design decisions that I can evaluate and learn from. This isn’t quite the same as owning my own decisions after years ⁷, but it has helped me build a mental library of interesting design choices and the ways they worked out in different situations. If I sit in a discussion today and hear something similar to one of these choices, I can give informed, specific feedback that I wouldn’t otherwise be able to.

Incremental change

As mentioned above, there can be a significant learning curve with legacy systems for a variety of reasons. This can be discouraging, especially if my teammates regard it as a waste of time, a mess, a dead end, etc. I’ve found that it’s worthwhile to push past this learning curve.

I’ve never regretted knowing how to do things inside of a legacy project. It can be like a superpower if most people on the team actively avoid the legacy project. Maybe a problem that other engineers are planning to solve with a new system (involving multiple people and weeks of work) can be solved by one person tinkering with a single function in the legacy codebase in an afternoon. Some features will necessarily touch a legacy system for various reasons; people are generally thankful to have someone who can help them figure out how to do this, provide time estimates, etc.

Even given the constraints of a legacy system, it’s quite often possible to address some of the downsides without throwing the whole thing away and starting over. Specific modules or subsystems can be rewritten, entangled concepts broken apart, etc, resulting in a system that’s easier to work in and change. Small-scale, incremental, thoughtful change can be less satisfying than a full rewrite (it can be hard to feel like you’re doing anything by just taking small steps), but it has advantages in terms of overall risk, time investment/resourcing, and a shorter feedback loop as compared to more ambitious projects (which require more engineers and take more time to deliver). I don’t think it’s possible to do this type of incremental change well or responsibly without a deep understanding of the code being changed, and I think working with that code over time is one of the best ways to get that understanding.

Rewriting successfully

There are obviously cases where replacing a legacy system is the right move. There’s an Internet full of posts talking about this, so I won’t go into why here. I will say that I feel a lot more confident in embarking on a rewrite if I’ve worked on the system I’m being asked to replace. If there is a way to achieve what is desired without a rewrite, I’m far more likely to be able to speak to that. If a rewrite is in fact necessary, I’m able to guide the discussion towards specific problems that come up in day to day work on the system, and propose solutions that address those. All this increases the likelihood of success: the project being completed on time and having a generally positive impact on the company.

As a concrete example, I worked for a few months on one of the oldest parts of a legacy system. People wanted to rewrite it before I started working on it, and had various pitches for how to do so; some involving new tech stacks, new services, etc. I spent time with the system, understood its history, the features it currently offered, the type and cadence of the asks made of it by product and business stakeholders, the outcome of bugs when they happened, and what people saw happening in this area of the company in the next year or so. Using that knowledge, I could identify high priority issues that often blocked work on new initiatives (missing concepts in the code, tightly coupled concepts in the code, implicit assumptions, etc) and note which issues weren’t really important (tech stack, whether it was a separate service). I came up with a modest series of enhancements, operating within the same codebase and using the same data model, which addressed the high priority issues in a way that minimized risk and (compared to a more ambitious rewrite) scope/effort/time investment. I think this was a successful rewrite, after launching it and watching it evolve in production for more than a year. I think a rewrite was certainly justified in this case, but I’m confident that the rewrite would not have been successful if I hadn’t spent a few months getting to know the old code.

Reasonable expectations

In the introduction to “Practical Object-Oriented Design”, Sandi Metz writes:

Unfortunately, something will change. It always does. The customers didn’t know what they wanted, they didn’t say what they meant. You didn’t understand their needs, you’ve learned how to do something better. Even applications that are perfect in every way are not stable. The application was a huge success, now everyone wants more. Change is unavoidable. It is ubiquitous, omnipresent, and inevitable.

I often think of this quote when I see a piece of code, a design, or a system that rubs me the wrong way.

As engineers, we perceive the complexity and heft (mental and LOC) of a legacy system as a roadblock, at least relative to some ideal world where we spend half an hour reading documentation and understand all aspects of a large codebase ⁸. We then think of technological solutions to this problem. If we use a modern language, if we use a modern pattern, if we adopt a specific design philosophy then everything will be simple. I think this is a mistake. If we work in environments where our software does something useful, then the forces of change identified above will act on it over time. Our software will eventually accumulate cruft, debt, sad corners of hacks where our original assumptions break down. Luck, the nature of our product, and our skill at design can influence how quickly and how often cruft forms, but not (I argue) ever stop it. I think this isn’t obvious to a good number of developers. Books, blogs, education, etc push us towards perfection as software designers. A system with cruft is defective, and can be avoided if only the engineers working on it are prescient enough to forecast future needs, use the right technologies and design patterns, etc.

Put another way, I think nearly all software systems will eventually (and often quickly) accumulate characteristics of a legacy system. Tech stack, architectural pattern, DDD/MVC/EBI are no guarantee against this ⁹. A newer project may have less crufty complexity than a years-old codebase; a greenfield project may, before it sees users, briefly resemble the sort of ideal we see in the literature, but they are very unlikely to stay that way. So, if I start a job, even a job with a trendy tech stack, even a job where the recruiter assures me that I’m building the future of the company, I shouldn’t expect to find an unimpeachably perfect codebase because I’m vanishingly unlikely to do so. Complexity and cruft are the common case. Complexity is not a priori justification to replace a project, and avoiding “legacy” projects will not free me from having to deal with it.

Conclusion

It goes without saying, but every situation, project, and company is different. I think all projects, even dead-end projects, have something to teach us as engineers. That said, there is a difference between a legacy system that has good bones and can be productively enhanced despite being a little behind the times and an irredeemable pile of mud that gets changed only when absolutely necessary. Working exclusively on the latter for years is unlikely to be a good career move, and I’m not encouraging anyone to do that. I’m only arguing that legacy system does not imply pile of mud.

I’m also not advocating that you should spend your entire career working on existing systems. Green field projects do have a lot of valuable things to teach you (and these tend to be well and numerously documented on the web). They’re also fun, and there’s something to be said for that. Rather, I’m saying that legacy projects can also be a valuable way to learn and grow as an engineer.