If you never make mistakes, you never learn. I had applied firewall rules to our render license box. Monitored, it looked okay and went for lunch. Came back to an angry mob of animators and a pissed Head of IT.

I hadn't allowed an undocumented port. This caused the animation software to exit without save as the software couldn't communicate with the license server.

Coincidentally, when I went to lunch, picked up my food; I had accidentally dropped it in a puddle around the same time the licenses were revoked. Spookily felt like I was somehow spiritually connected.

Interesting anecdote. However hard to generalize that small teams avoid writing code with unit tests & safety checks. I worked in a adtech before - small team of 2 Data scientist, 1 backend guy & a manager. All our code were type checked & ran through a strict code-review QC, before deploying. Current place has us down to a team of 3 & we do strict version control, QC & robust orchestration because we have to cater to very demanding customers. Although I can imagine the "move fast, disrupt thing" mantra generally works - but it doesn't mean moving fast without safety

Good article.

It reminds me that the so-called "observer effect" prevents a debugging and testing environment from reflecting production perfectly. However, there's certainly a balance which must dictate the degree to which the QA process is relied upon.

Given that OK-Cupid is a dating app meant for "fun", I'm not surprised that the testing environment was somewhat loose and informal. However, as the author alludes, when the consequences of failure ramp up... testing and QA become absolutely indispensable.

IMHO, the trick is being able to mirror as much as possible between the QA and production environments... and then determining when to deploy. This needs a fair bit of experience, subject matter expertise, and of course luck. You can never discount luck.

> If she forgot the square root of -1 was undefined, she’d never write a test for it.

Uh...