Why you should test critical parts of your app

Last Monday, I sent the usual newsletter to the users of CloudNews. But instead of receiving  one unique email, they received dozens of them, some received more than a hundred in a few seconds.

As I don’t want this to happen again. It’s time for a bit of thinking.

Why did it happen?

Simply put, I didn’t have any automated tests. I launched CloudNews in November after building it in 3 weeks. I preferred development speed and actual value for users over reliability and code quality.

Few years ago, I built a product that didn’t get any traction. It had a staging environment, automated tests, and a complicated CI. It checked all the boxes of good practices. But it had no active users. 

As an engineer, I was delighted with the result, it was working perfectly and it probably could have scaled. If only people were using it. 

I now have a full time job, I can’t spend months on a project only to see it failed miserably. My time is as limited as ever. Which is why I chose to skip the good practices this time.

What caused the issue?

I edited a critical part of my app, which I don’t update often. 

The module that sends emails. I updated a SQL request that fetches the users based on some attributes and a misconceived JOIN duplicated all email addresses.

I didn’t realise it, because the local database was only populated with my own email address.

What could have been worse?

I could have been on the free tier of my email provider. I could have reached the daily limit on email sent in seconds. It then could have sent the remaining emails placed in a waiting queue when came the next day.

Well it happened too.

What action did I take to solve this?

A few things could have prevented it. I now have implemented most of them.

  • My local database has the same data as the production database.
  • I added automated tests on the critical parts of the app.
  • I added safeguards triggered on abnormal situations.‌
    ‌For example, if there are more emails to be sent than total users. This triggers an exception.
  • I used to code on master (I know I know), now I code on a dev branch and review each pull requests.
  • Finally I added GitHub Actions.  

Conclusion

There is a fine line between being too scrappy and being too perfectionist. I now know the trade offs better and it taught me a valuable lesson.

I’ll end this post with a borrowed conclusion from Francesco Di Lorenzo:

For me, the biggest learning in shipping and iterating on many projects has been the ability to recognize lagom — a Swedish word that means “not too much, not too little, just right. As I don’t want this to happen again. It’s time for a bit of thinking.