An ignorant noob’s journey to microservices system management and core lessons learned
I would like to start immediately with saying while ignorance is great for short-term personal mental well-being, looking at the ugly truth straight on is much better to avoid massive overtimes when the inevitable shit, hits the fan.
We currently have a complex microservice architecture and being the person that was here since the humble beginnings, seeing the system blossom from a single service to this, maybe not gigantic but reasonably large system with many environments running on massive system resources is an interesting feeling. As production dates near, the interesting feeling is more akin to staring at a km high tsunami approaching at too many miles per hour.
So, how did we come to this?
I have always worked on monolith-macroservice development, with macro being a generous term to put it, and there wasn’t much docker usage where I came from back then. Hence when we started here, while my other skills were reasonably up-to date, the designing of a system architecture part where everything will be run was not so. Thankfully, people who knew their stuff set a course and guided me through some new hellscapes and voila, we had our first dockerized system. Moving on, we proceeded to a k8s architecture with the why being, “lets be adventurous and learn on the way”. We were happily skipping along thinking “we da bomb”.
We did learn a lot, that’s for sure. Most of the bells and whistles of a k8s system was known (or so I thought). However, while we did many things, we forgot a critical tenet of coding which was: Think twice, code(do) once. Mistake 1.
We never had the time to stop and think and something new always got slapped on the team, there were 10–20 work items waiting at all times. Mistake 2, I should have stopped and created the space to think, to consolidate.
This is probably the critical problem with learning on the way. While you can slap on whatever you learned to a system and make it run, it may feel like a several decades old windows computer which never got formatted. Most developers have felt this feeling that when you look at your old code, you feel a repulsion and feel the immediate urge to fix it. Bit by bit you improve everything on par with your updated knowledge. Visit it again in a year or two, and exactly the same feeling. Eventually, you will need to rebuild something core and have to rip out most of what you did.
So coming back to the topic, I decided to delve deeper than reading surface level stuff supported by my outdated knowledge and take an in-depth course. While at it, maybe I could get myself a certificate to go along.
However to my growing dismay, at every chapter, I saw something new that may be implemented, something excessive we used, something core we missed. I missed. Since I was the driving force for a long while, my outdated knowledge was plastered all over the system and now we have to rip out chunks and merge a lot of things to fix them. Have we missed anything else?
To know for sure, lots more in-depth training and poc’s need to happen and the team has a tougher captain at the head of the system now, so there is still hope.
An important lesson out of all this is even if we are at a managing position, if we are deciding on anything technical, even with other peoples’ support we must at the very least, decently know whatever topic we are talking about. Knowledge ages, it ages bad and must be updated all the time. Cutting losses and diving quickly to refresh reduces the stress and amount of work that must be put in. Technical debt is accumulated as long as we look away, every day.
Relying on people alone is not enough hence we may not know the extent and effect of their decisions and if they are right at all. People make mistakes often.
