AskAppSec - Dependency Updates

When one of my former managers commented on my blog post on Painless Usable Security, asking about our approach of keeping dependencies up to date, I realized that there's more to the topic and I should write a separate post about it. So here it is!

Keeping dependencies up to date has been a big topic in lots of the teams I've been part of. When I created our AppSec strategy for my current team beginning of the year, this topic once again stood out to me as the first one to tackle. This is based on our context. We have a whole bunch of services we own and most of them are around for quite a long time (and still valuable). We inherited a system that had degraded over time, and knowledge had been lost. Over the last two years, we learned to understand this system a lot better, as a whole team. We invested in several endeavors to get it in shape again, to preserve its value, and also ease extendibility.

Before going into details, let's first take a few steps back.

Why even have dependencies in the first place? Nowadays, most software is not built in an echo-chamber. It'll be based on lots and lots of other software that lots and lots of other people provided, so that we all can achieve more with less. There are specific contexts where dependencies will indeed be very constrained, usually in situations where stakes are very high. Yet even then you probably don't invent your own programming language, operating system or infrastructure. For the kinds of products that I've worked on so far, the system will have lots of dependencies to third-party frameworks, libraries, and more.

So, we need dependencies. Yet why should we update them and keep them updated? Can't we just keep everything as it is?

• Even if we don't change anything on our side, the world around us does not stop. Tech evolves every day. If we don't change anything, our system will naturally degrade from both business value and security perspective. More vulnerabilities in the dependency versions used will be found, and the system will be more and more at risk. Until suddenly it's very time-critical to have that risk mitigated. Remember Log4Shell, anyone?
• Besides security fixes, you'll also want to use new capabilities that dependencies offer in their newer versions. These can help preserve or even increase the value of your system and keeping it relevant to users, business and other stakeholders.

Fine. Yet why do people struggle so much with updates? Just do it, right?

• Dependency updates oftentimes come with the need to adapt your system to the new version. Sometimes that means completely re-architecting your solution; very painful. Sometimes dependencies had removed or replaced features in their newer versions, and you'll need to cater for that. Even if there's no obvious need to adapt the system, updates might break functionality in lots of surprising ways - so you better have suitable measures in place to mitigate that risk and detect regressions early.
• Updates can come in very fast, sometimes multiple times a day. Think about the JavaScript ecosystem for example. It can be overwhelming and feeling like a Sisyphean task: as soon as you got your system in an up-to-date state, there's a new version of at least one dependency released again!
• Dependencies have dependencies themselves. Oh my. Updating one dependency often means it's not compatible anymore with the rest, so you need to update a whole bunch of others as well; especially when updating frameworks. This also means, by the way, that dependencies have to keep on top of their own dependency updates. The struggle is real.
• Dependencies might be discontinued and not get any updates any longer due to a variety of reasons. They might get moved to other places, integrated into other projects, you name it. Sometimes you can migrate, sometimes you have to find similar tools from scratch again to meet your needs. For any newly introduced dependency, it needs evaluation and validation again (better do it regularly for existing ones as well). Licenses are crucial to consider, especially when including open source dependencies. Checking its usage, how many people contribute to the project, how many issues had been already identified or fixed, when the dependency was last updated, how much community support it has, and so on. It's good to have a guideline in your context what's considered suitable to choose and what not, and what's the reasoning behind it.
• You have more dependencies than you might think. The number of libraries directly used by your services alone is probably high. Yet there's also the infrastructure you're running on and its dependencies, like container images with their operating systems and - surprise - their respective dependencies. There are dependencies to data storage solutions and so on. It's a lot to keep up to date.
• Probably the biggest hurdle I've seen is getting buy-in to do this kind of maintenance "keep the light on" work as part of normal everyday business. It's an investment and there's opportunity cost - if you invest in one thing, you can't do another at the same time. This often leads to not investing in maintenance at all. A strategy which comes around to you soon enough, presenting you with even bigger investment needs to get back on track again, while already being hindered to follow other opportunities that you wanted to prioritize for business reasons. It can slow you down to a complete halt. Yet as that's usually only a future problem of a potential risk, it's really tempting for people to ignore it (guilty of that myself). It requires lots of experience, discipline, and good practices to still keep your system tidy and in shape, continuously. And keeping each other accountable, we're fallible human beings.
• Last but not least, context is crucial. In industrial cybersecurity settings, updates might have a hugely different impact, not only financially yet also when it comes to safety. Lesley Carhart describes it well in her CyberWork podcast episode. Context really matters, risks can differ heavily.

What about tools to help us keep dependencies in shape?

• There are Software Composition Analysis (SCA) tools to help you detect outdated dependencies, including their known vulnerabilities. They usually compare your dependencies' versions against a list of available versions and reported issues for each. Integrating tools into your source code hosting platform and delivery pipeline can help as well. You might have heard of OWASP Dependency-Check, Dependabot, Renovate, Snyk Open Source, and the like.
• Package managers often come with integrated checkers. Like NPMs dependency-check which also offers to update potentially straightforward ones automatically for you.
• Modern IDEs support you by indicating outdated or vulnerable dependencies, like IntelliJ IDEA Ultimate does. Check out Marit van Dijk's awesome talk Keep your dependencies in check to see it demonstrated along with lots of useful advice.
• There are repositories like Maven Central to get dependencies from in all available versions, which often also help if dependencies moved their artifacts, got renamed, deprecated and more.
• Tools like OpenCVE allow you to subscribe to updates for your technology stack to get alerts on potentially relevant security issues.
• Sometimes frameworks offer migration tools to help with bigger updates.
• More and more people talk about software bill of materials (SBOM) to keep inventory of all kinds of software including dependencies in use.

Lots of aspects come into play when updating dependencies, and there are probably a lot more factors to consider than listed here. The big question to answer is what's most helpful in your context to actually get the job done, get the dependencies updated and keep them updated.

Let's talk about strategies. How can we keep dependencies updated?

In a previous team, we worked with major updates once per quarter, going through everything. This worked okayish for the given context of an internal product with limited usage. In my current team's context, however, we have a customer-facing product with a hugely different attack surface. So far, the following worked for us to update our services' dependencies.

1. Establish, encourage and ensure 20% time for every team member and use it to drive tech initiatives. Like getting dependencies of our services in shape. Having dedicated time to improve certain areas like this was a massive cultural foundation for lots of good stuff happening.
2. Use tooling to support easier updates where feasible. Automated scanners to indicate outdated dependencies, utility tools to adapt required related documentation for compliance reasons, and automated checks to discover potential regressions. Tools like these were great in combination with our system knowledge, so we could quickly unveil more surprises where automation reached its limits.
3. Do the easiest, most straightforward, quick win updates first and get them out of the way. It'll reduce cognitive load and clear up headspace for the bigger challenges, like the ones requiring updates of several dependencies or even frameworks. The advice to "solve the smallest problems first" helped me massively with legacy systems like ours; as far as I remember the credit goes to Nat Bennett, yet I can't find the source anymore (if anyone does, let me know). Having said that, it doesn't mean you shouldn't prioritize updating your most critically vulnerable dependencies first. 
4. Small changes done frequently compound. The system will get better step by step, you'll get better at updating the system step by step. Always a bit better. It might not look like much today, yet a month from here it's already painting a different picture. We'll get there.
5. Build on existing energies and practices. This is one of my tools that helped me with lots of culture change initiatives. We also used this to keep dependencies in shape. We have regular tasks needed for each release, and updating dependencies simply became one of them.

All this, however, likely only worked due to the team culture we fostered where people are sharing everything; knowledge, skills, load, a common goal, and more. This made it clear from the start that keeping dependencies up to date is a team task as well and we're all responsible for it, together. I hope to share more once we've lived this approach for a longer time. I'm myself curious if we can manage to keep our system in shape this way or what other approaches will turn out to be more successful in the end.

A word of caution when it comes to tooling. Scanners are only as good as the team can respond and act on their results. The cry for more tools to make a problem go away, or the desire to just finding the one right tool to save us all is not going to fix the underlying issue. Just throwing more tools on a problem won't move anything towards better - most likely, the opposite will occur. All of this becomes noise. It's overwhelming. There's so much other work to do as well. People start shutting themselves off and ignoring alerts just in order to be able to deliver anything (most likely the thing that others put them under pressure for). Yes, alert fatigue is very real. We see the same overwhelm with observability tools, monitoring alerts, test findings, static analysis feedback, and more. Having yet another class of alerts you don't get time for to understand and fix just does not help get into a better place. Not to forget false positives! Alerts that are simply not alerting you on anything real or actionable or relevant for your context are like poison.

At BSides Munich 2023, Jasmin Mair talked about "My CI/CD pipeline contains all security tools available! Now what...?" (check out the recording, too). I loved the strategy she presented to add one tool at a time, train developers, set a baseline, and manage findings. Like eliminating one class of vulnerabilities at a time, and allowing people to follow. For the case of dependency updates, we decided to go service by service, starting with the most critical ones first, and only afterwards include more tools where needed. First make alerts be seen and worth responding to again. Once we live that, we can always improve further. By the way, that was also our approach to clean up other alerts, errors, log spam, and similar noise - enabling ourselves to see what's actually important again when it occurs.

Here's another side note. Getting things back in shape wasn't done with updating dependencies. It also included other topics like removing unused code. It's always been a struggle to clean up functionality that had been superseded by something better, was just not invested in anymore, or had never been used at all while being dragged along. In one of my previous teams, we decided to get rid of all that under one big theme that also could be sold to business easier: reducing complexity. It's also part of opportunity cost to keep maintaining things that are simply not worth it any more (if they ever were). We can't foretell the future and hence need space to try things out in our product to learn what actually solves the problems our users and business have. Yet if we want to make progress towards a shared goal, we do need to make clear decisions what to keep and what not, and follow through. When I joined my current team with our big legacy system, I was stunned how much unused code was still around from many years ago. This year, we finally had buy-in to clean up - oh how much I loved it! Not only did it indeed reduce complexity and increase maintainability, made things a lot clearer for new teammates, and so on - it also reduced our attack surface! A win on all sides.

A similar case can be made for services and features we want to keep, yet that are complex, hard to understand, inconvenient to modify, or people are afraid to touch them. Not a great starting point when things go awry and a security incident comes in. Actually, if any incident occurs. Even if nothing bad happens in that area, this part will tend to stay untouched until no one in the team knows about it anymore and it's even costlier and scarier to touch it.

All in all, when I came across the following statement in Tanya Janca's book "Alice and Bob Learn Application Security", it made so much sense to me: "technical debt is security debt". Big aha moment, this resonated heavily with me. Good for maintenance and extendibility can be really good for security, enabling us to adapt fast to an ever-changing world.

It's time to bring that original question back to the community: What do you do to make keeping dependencies up to date work?