Frey Thorvaldsson

Needless complexity

Tearfully pleaded by Frey Thorvaldsson

Most of my time is spent working alone.

This has some benefits. I get to impose my creative vision, without needing to justify it to anyone, but it also means that when I'm not working nothing is getting done.

I also end up having to make all the decisions around tech.

Consequently I spend a lot of time sussing out which piece of 'best practice' coming from the internet actually applies to me and what advice works best in the context of a larger company.

Here are some thoughts I've had kicking around in my head about not getting sucked into the latest and greatest.

How simple can you make things?

Pieter Levels runs RemoteOK, a business doing $30,000 MMR which, behind the scenes, is a single index.php file.

It doesn't have version control and his 'deploy pipeline' consists of FTP-ing files into his server.

The automated tests are an assortment of UptimeRobot monitors, sending requests to pages with various parameters [1]. If those monitors fail to catch mistakes his users end up making the unhappy discovery.

This 100% works as a business. It also 'good' software in that the site is nice to use. The site does exactly what it needs to do and not much more. The main compromise seems to be that he's fine with shipping bugs.

So if this works, how did web development get to be so complex?

There are at few pressures which could push a project like RemoteOK to grow into something you'd see on a typical engineering team.

1. The team grows

If you add another member to your team you'd want to have version control. Unless you want to email code between each other. With Git you'll usually also add a more mature looking deploy pipeline instead of FTP-ing into the server as in the days of old.

This also gives you the opportunity to add automated tests which are a bit more complex than smoke tests.

2. The codebase grows

RemoteOK is a simple app so a single file might just be enough. Most software businesses aren't so easy.

Making sense of an application composed of a 10,000 line index.php is the stuff of nightmares. So, as the codebase grows you usually need to be more disciplined about how you organise your files and start using some sort of architecture.

3. The traffic grows

Once you've reached a certain size you might have trouble scaling your application. Before you do anything drastic you can usually ease pressure off of your server with aggressive caching and upgrade your server. [2]

Assuming this isn't enough, you'd probably want to wrap the application in a docker image, put a load balancer in front of it and scale it out that way.

If that's not working you might want to split out your application and deploy it as seperate microservices. This adds so much complexity (messaging, debugging, deploy pipelines for every service, etc) that you better have a good reason to do this.

4. You want the benefits of JS libraries / TypeScript / CSS pre-processing

As you grow your project you might have difficulty managing state on your frontend. So it might make sense to bring in a proper framework like React, Vue or Angular. Or you'd like to start using Typescript or CSS pre-processors.

Whatever the case is, to get these benefits you'll need to introduce a build step.

You get something conrete (easier to manage css, better state managment, etc) in exchange for weekends spent in the warm embrace of your Webpack config.

5. Managing tooling complexity from the above

As the site has grown (in traffic, team and features) we've added loads of complexity! We've tried to manage it with more tools (like Git, Webpack, Docker) which in turn have tools to manage it (Github, CI, loads of config to scale containers).

What to make of this

If RemoteOK is on the "is this even engineering?" end of this spectrum, then small teams using microservices and Kuberenetes is on the grossly overenegineered end. (unless you're Monzo)

My view is that most businesses with small engineering teams should lean towards emulating Pieter instead of emulating large engineering organisations.

That doesn't mean you should be copying Pieter, you just need to think about trade offs instead of blindly following 'best practices'.

I'd consider nothing to be holy and to try to work things out from first principles. Waiting for the pain to hit is a good way to know what's actually essential.

  • If you're having issues with QA then introduce testing.
  • If you're struggling with scaling then start looking at microservices.

If you try to solve everything up front you can end up spending all your time ticking arbitrary boxes.

You can only take on so much complexity before you're spending your entire day managing your projects instead of building your product.

The Folly of Man: My story of wanting to be a Real Developer™

A few months ago I started a side project with some friends. It was an application where you rated music albums. I found a Node template online which had some cool features out of the box. Schema migrations, nice architecture, JWT auth and postgres. This was wrapped in two Docker containers.

I thought "Nice. This is how Real Developers™ do things"

I wound up spending some 20 hours wrangling with docker, building a deploy pipeline, making sure we can roll back schema changes and writing tests. This way our nonexistent users wouldn't be inconvenienced by unhandled errors, you see.

Basically, I was doing all the clever stuff that people point to when asked about software engineering excellence.

But when I ran out of steam and got bored of the project we hadn't actually made a proper feature yet. You could only log in and list fake albums from a database.

Oops.

Running schema migrations on production databases is a pain. Installing dependencies for a project is a pain. Having bugs in production is a pain.

But I wasn't feeling any of this pain.

I was solving theoretical future problems, for what was supposed to be a fun side project, because I wanted to feel smart and do things The Real Developer™ way

I think that's what drives a lot of developers. They don't want to be the person with a massive index.php file, even though it might be the perfect fit for the project.

What complexity is actually worth it?

Which tool (and consequently which additional complexity) is worth taking on depends highly your skill set. If you're a Docker wiz it might be easier for you to develop with Docker from the start than otherwise.

If you know Typescript already then it's most likely better for you to start with Typescript than not. At that point it's worth embracing the tools.

However, since no one is an expert in everything it's worth interrogating the tools we use.

What I haven't found so useful are things like:

  • Docker
  • Microservices
  • Writing documentation
  • TDD (or any variation of 100% test coverage)
  • Complex branch workflows
  • Managing multiple staging enviroments
  • Building complex data pipelines / data warehouses (query prod until it hurts!)
  • MongoDB

Tools that I've found really useful despite the added complexity:

  • Git. Not much of a debate, but it was a small hurdle when I stared out [3]
  • Writing comments about intent
  • Making deploys as simple as possible
  • Having smoke tests
  • End to End Testing (cypress FTW)
  • TypeScript
  • Linting / Prettier
  • Uptime monitoring and bug tracking (not complex, but worth mentioning)

In the end what we're talking about here is pragmatism and not conforming for the sake of it.

Be pragmatic, do your own damn thinking and don't let people make you feel bad for not doing things the way big companies do them.


[1] An example would be example.com/countries/?s='India' should return India on the page in the search results. Solid sanity check.
[2] Worth checking this out for some inspiration for how far caching can take you, given the right circumstances.
[3] If you're a developer who's been working in tech for a while you might not see tools like Git as "added complexity". Absolutely, but think back to when you learned it first. Same applies to testing. Or anything really. Everythings easy when you know it...
← Back to articles

Let me spam you!

Say goodbye to inbox zero! E-stalk me via email! Follow along with 5+ other subscribers including my mom and my girlfriend as I struggle to keep up a consistent writing habit.