Reading:
Dependency Roulette: How One Tiny Update Took Down Production

Image

Dependency Roulette: How One Tiny Update Took Down Production

Third-party dependencies are a double-edged sword—convenient until they suddenly break everything. Here’s a real-world incident from our collection that perfectly illustrates this risk.

The Setup: How We Managed Python Dependencies

We use the pretty standard and simple PIP for installing Python modules, along with a requirements.txt file to list packages and define their versions.

With npm or yarn, you get a lock file out of the box, which is great. But pip doesn’t have that (at least not by default). In requirements.txt, just like in any similar package manager config file, you can specify strict version constraints or ranges. We usually lock most dependencies to exact versions, but for some reason, boto3 (the AWS library with looooooads of everything useful for AWS) was defined as >=1.25.0.

How_One_Tiny_Update_Took_Down_Production

When Everything Broke: A Boto3 Update Disaster

Now, boto3 gets new releases almost daily for Python. And for a loooong time, everything was running smoothly—until one day it wasn’t.

A new boto3 release introduced a bug affecting file uploads to S3. As is often the case, we successfully passed:

Local testing
✅ Development
✅ Staging
✅ PRC (Production Release Candidate) testing

Everything seemed fine.

But in production… things went south.

The Hard Lesson: Pip Alone Isn’t Enough

The lesson from this whole fiasco? Forget pip, time to switch to something better—like uv.

Now, you might say: Why such a drastic change? Why not just lock versions with ==X.Y.Z in requirements.txt?

Simple. requirements.txt only controls what you explicitly list. But installing a third-party package is like trying to pull out a single spaghetti strand from a pot that’s been sitting in the fridge overnight.

Dependency Hell: The Hidden Risk

If some sub-dependency somewhere was defined as >=x.y.z (or had no version constraint at all), every pip install becomes a lottery. And only Python itself knows what might get installed under the hood.

Then there’s another fun scenario:

1️⃣ Imagine you have lib-1, which depends on lib-2.
2️⃣ At some point, you copy-paste a piece of code that also uses lib-2. You realise you need to import it, hit Alt+Enter, and see—oh, it’s already installed, great!
3️⃣ Fast forward a month—lib-1 drops lib-2 from its dependencies.
4️⃣ Next thing you know? ImportError.

Lock Files: The Only Real Fix

That’s where lock files save the day. They capture the entire dependency tree down to the last sub-dependency, recursively, ensuring stability across the board.

So yeah—pip, R.I.P. 🪦

Related Stories

Frameworks Choose the Right Tools and Use Them Wisely
June 4, 2024

Frameworks: Choose the Right Tools and Use Them Wisely

When selecting a technology stack for a new project, we all look for frameworks and libraries that can speed up delivery and simplify our lives.

Cover Image for Legacy Systems: Reviving an old Flash app
April 29, 2022

Legacy Systems: Reviving an Old Flash App

Common Issues and Robust Solutions_Enhancing Security in Fintech Projects
May 21, 2024

Common Issues and Robust Solutions: Enhancing Security in Fintech Projects

Security is paramount in RegTech and FinTech projects. Even when designed with security in mind, software solutions become vulnerable and less secure over time.