Reading:
Dependency Roulette: How One Tiny Update Took Down Production

Image

Dependency Roulette: How One Tiny Update Took Down Production

Third-party dependencies are a double-edged sword—convenient until they suddenly break everything. Here’s a real-world incident from our collection that perfectly illustrates this risk.

The Setup: How We Managed Python Dependencies

We use the pretty standard and simple PIP for installing Python modules, along with a requirements.txt file to list packages and define their versions.

With npm or yarn, you get a lock file out of the box, which is great. But pip doesn’t have that (at least not by default). In requirements.txt, just like in any similar package manager config file, you can specify strict version constraints or ranges. We usually lock most dependencies to exact versions, but for some reason, boto3 (the AWS library with looooooads of everything useful for AWS) was defined as >=1.25.0.

How_One_Tiny_Update_Took_Down_Production

When Everything Broke: A Boto3 Update Disaster

Now, boto3 gets new releases almost daily for Python. And for a loooong time, everything was running smoothly—until one day it wasn’t.

A new boto3 release introduced a bug affecting file uploads to S3. As is often the case, we successfully passed:

Local testing
✅ Development
✅ Staging
✅ PRC (Production Release Candidate) testing

Everything seemed fine.

But in production… things went south.

The Hard Lesson: Pip Alone Isn’t Enough

The lesson from this whole fiasco? Forget pip, time to switch to something better—like uv.

Now, you might say: Why such a drastic change? Why not just lock versions with ==X.Y.Z in requirements.txt?

Simple. requirements.txt only controls what you explicitly list. But installing a third-party package is like trying to pull out a single spaghetti strand from a pot that’s been sitting in the fridge overnight.

Dependency Hell: The Hidden Risk

If some sub-dependency somewhere was defined as >=x.y.z (or had no version constraint at all), every pip install becomes a lottery. And only Python itself knows what might get installed under the hood.

Then there’s another fun scenario:

1️⃣ Imagine you have lib-1, which depends on lib-2.
2️⃣ At some point, you copy-paste a piece of code that also uses lib-2. You realise you need to import it, hit Alt+Enter, and see—oh, it’s already installed, great!
3️⃣ Fast forward a month—lib-1 drops lib-2 from its dependencies.
4️⃣ Next thing you know? ImportError.

Lock Files: The Only Real Fix

That’s where lock files save the day. They capture the entire dependency tree down to the last sub-dependency, recursively, ensuring stability across the board.

So yeah—pip, R.I.P. 🪦

Related Stories

It-works-on-my-machine_1024x533
Featured Image AWS EKS Kubernetes access trap
February 3, 2021

The Amazon EKS Access Trap

When it comes to Amazon EKS, your cluster creators matter. Let's look at Kubernetes clusters and the importance of keeping control over your infrastructure.

Connection Timeout: Communication is key
March 4, 2024

Communication Timeout: A Horror Story

We cannot stress enough the importance of communication on software development projects, especially when it comes to distributed or remote teams.