Our infrastructure ages with the company, and even though we do our best to keep it current, we need to be thinking long term. There are times in a company’s growth when you need to take a long, hard look at what you’re doing and think about ways to do it better, cheaper, and faster. That’s where our Research and Development (R&D) team comes in. Their goal is to set us up for success with next generation infrastructure, best practices, and standards.
Why I joined R&D team
As a member of the CDN team, I recently had the pleasure of joining our R&D team for about 6 weeks. The reason for my joining is that I have prior experience with monitoring and Kubernetes — a bit of a sore point for a team on deadline. We needed to be production-ready soon; we were supposed to launch a critical component in the new environment so I was brought over to focus on getting monitoring and alerting on par with the rest of the infrastructure.
R&D is a part of Showmax, and a majority of the team members were recruited internally. The culture stays the same — we always strive for the best possible solutions, we encourage each other to do better, we provide fair feedback during code reviews, and we use the same tools as the rest of the company (Phabricator, Slack, Opsgenie).
One of the main differences in R&D is that the usual standups, which happen before 10am in most other teams, are written notes instead of in-person meetings every morning. This has multiple benefits, including but not limited to allowing me to sleep in if I need or want to. There’s also the added value of having everything written down, so note taking is unnecessary. The notes are then collected and automatically published after committing them into git.
There are also downsides, including but not limited to my not being forced to get out of bed and tackle the day (a constant struggle).
Another (huge) difference is that the team uses monorepo. I experienced first-hand the woes when multiple people were updating things right under my hands. It’s something you need to get used to in a dynamic environment like this one.
I also felt the good parts, like when I needed to rename all of the labels and annotations we use for alerting in Prometheus. This was super easy because it consisted of a single pull request. Phabricator also helps to alleviate some of the downsides by squashing all commits into a single one when you merge your feature branch. I still haven’t decided whether I like it more than the traditional multi-repo approach - I need more time and exposure to it to weigh the pros and cons.
All-in-all, it was an interesting experience. I got very intimate with Terraform, and, honestly, I do feel like it’s a bit overhyped for what it’s doing. That being said, it’s definitely a step up over some of the older methods that I’m used to; we currently use a rather old Puppet to take care of our infrastructure hosted on physical hardware in Hetzner.
The R&D team achieved some huge leaps forward by being able to set up a new deployment from scratch in a matter of minutes, and I’m very happy that I got to be a part of the process.