Microservices and Tech Stack - Lessons Learned
In this blog, I want to talk about how our technology stack has evolved over time while transitioning from Enterprise to adopting Microservice architecture as a main stream development practice and share some of the lessons learned.
We have hundreds of different ยต-services written in different languages. .netcore, golang, node’s and python are the most popular choices and everything gets deployed as docker containers. Services talk to each through nginx or rabbitmq. Each service stack has its own CI/CD pipeline. I dont have every component here but i hope this gives a good idea of where we are today. We also have the monolithic stack which is ok as long as there is no active development on it. Anytime we find a need to touch monolithic, we try to chip it away into a nice service offering. Today we are deploying multiple times in a day (with zero downtime) and we can iterate over features must faster! At backend we are using SqlServer, MySql, DynamoDB and redis cache depending on which problem we are trying to solve.
Some Lessons Learned
Docker is a must
Tech stack for us has evolved from being a Monolithic Enterprise to Cloud to now Microservices. During Enterprise days, it looked something liked this:
At the app layer, we had a desktop client written in Windows form and WPF. The app would talk to the Service Layer which was SOA architecture written completely in C#. It was the only language allowed for us to use. These were monolithic and stateful services communicating to each other over WCF. We used SQL server as backend storage. All this get deployed on premise meaning our clients buy our software and host it on their own hardware.
I work for Finance industry (Stock Market to be precise) and they were sort of late adopter of public cloud. They absolutely hated the idea of putting their data on public cloud. I have seen clients who would lock down any sort of possible communication with the outside world from their infrastructure to prevent leakage of data. But overtime, this mindset has changed drastically and companies started appreciating public cloud and the security around it. Around the same time, our stakeholders also decided to put a product on the cloud. One good decision they took was to not make it a migration - meaning they didn’t ask us to migrate the original product into the cloud but rather gave us the opportunity to build a product from scratch. This was like a dream come true! We as developers always want that opportunity to re-write our software but it rarely happens. At that time, no one in the company really knew how to do cloud development and we ended up tech stack like this:
Our app became an Angular SPA app but on the service side, the only major difference was that we moved from monolithic stateful C# service to monolithic stateless C# services communicating to each other over rabbitMQ. We continued to use Sql server as the backend storage. All this gets deployed to AWS. As you can tell, we were deep into the enterprise mindset and basically ended up building a monolithic cloud.
At some point, we figured what we are doing is not working. Our cycle time on bugfixes and features was slow, deployments were painful and required lots of coordination. Around the same time, microservices were picking up. We heard amazing talks from companies like Netflix/Google/EBay on how adopting micro services has transformed their business. Following these threads, It probably started as an experiment for some team and today its the mainstream development practice.
Our app became an Angular SPA app but on the service side, the only major difference was that we moved from monolithic stateful C# service to monolithic stateless C# services communicating to each other over rabbitMQ. We continued to use Sql server as the backend storage. All this gets deployed to AWS. As you can tell, we were deep into the enterprise mindset and basically ended up building a monolithic cloud.
At some point, we figured what we are doing is not working. Our cycle time on bugfixes and features was slow, deployments were painful and required lots of coordination. Around the same time, microservices were picking up. We heard amazing talks from companies like Netflix/Google/EBay on how adopting micro services has transformed their business. Following these threads, It probably started as an experiment for some team and today its the mainstream development practice.
Today the architecture looks something like this:
We have hundreds of different ยต-services written in different languages. .netcore, golang, node’s and python are the most popular choices and everything gets deployed as docker containers. Services talk to each through nginx or rabbitmq. Each service stack has its own CI/CD pipeline. I dont have every component here but i hope this gives a good idea of where we are today. We also have the monolithic stack which is ok as long as there is no active development on it. Anytime we find a need to touch monolithic, we try to chip it away into a nice service offering. Today we are deploying multiple times in a day (with zero downtime) and we can iterate over features must faster! At backend we are using SqlServer, MySql, DynamoDB and redis cache depending on which problem we are trying to solve.
Some Lessons Learned
Docker is a must
This may be very obvious now but it was an interesting one for us since we were heavy on .net C# stack and both of them were incompatible for a very long time. The amount of overhead we had to deal with in absence of docker was too much and the community around docker is highly devoted to take all that pain away. This year's dockercon they announced full support for windows so there so bo no reason why not to use it. It simplifies your delivery, testing and developer experience. Just use docker!
Allow flexibility for teams to pick language/technology stack:
Why? They are the closest to the problem and probably have much better idea on how to solve it. Not just from business logic point of view but also the tools and technology that they want to use to solve the problem. If complete flexibility is not possible at least have the options to pick from an approved list of stack (and have many of them). Gone are the days when everything is either C# or Java. We as leaders/managers can guide them to take this decision but don’t make this decision for them. We want team to take complete ownership of the things they are putting in production not just the business logic but the complete service offering.
Don’t create service framework and force teams to use it.
This one is my favorite. When we started cloud development we approached it with the same enterprise mindset. We created a framework team which was responsible for writing a framework to solve all frameworkish problem so the other product teams dont have to worry about it and they can focus on the business logic. I was also part of this team. We wrote layer and layers of abstraction and at the end of the day lets just say it was not a good framework. But this was not the only problem. The bigger problem was that we created this binary dependency problem for everyone. Product teams required framework binaries in order to run their services. And they followed the same pattern meaning when they needed something from the other product teams, they did it by sharing binaries. And this created a big binary dependency problem which ultimately caused complex deployments and reduced our cycle time on bug-fixes/features.
There is a great talk from Ben Christensen@Netflix on this particular problem and he goes way in more details why this is a bad idea(https://www.youtube.com/watch?v=-czp0Y4Z36Y)
We had to learn this the hard way. So whats happening now? Better emerging pattern is that teams creates service-templates and share it with other teams. Usually a team has expertise on 2/3 different languages and as they are continuously building new services, they make service-templates to make their life easy. With these templates, they can spin up a new service in almost no time and then they share it with other teams. Now whenever i am exploring a new language, I reach out to teams who are already champions in it and see what template they have.
Design for Delivery
This is also a very interesting one. Usually the thought process in the Enterprise goes like this - Team has a problem statement --> They design the solution --> Build it --> And finally somehow deliver it.
Don’t create service framework and force teams to use it.
This one is my favorite. When we started cloud development we approached it with the same enterprise mindset. We created a framework team which was responsible for writing a framework to solve all frameworkish problem so the other product teams dont have to worry about it and they can focus on the business logic. I was also part of this team. We wrote layer and layers of abstraction and at the end of the day lets just say it was not a good framework. But this was not the only problem. The bigger problem was that we created this binary dependency problem for everyone. Product teams required framework binaries in order to run their services. And they followed the same pattern meaning when they needed something from the other product teams, they did it by sharing binaries. And this created a big binary dependency problem which ultimately caused complex deployments and reduced our cycle time on bug-fixes/features.
There is a great talk from Ben Christensen@Netflix on this particular problem and he goes way in more details why this is a bad idea(https://www.youtube.com/watch?v=-czp0Y4Z36Y)
We had to learn this the hard way. So whats happening now? Better emerging pattern is that teams creates service-templates and share it with other teams. Usually a team has expertise on 2/3 different languages and as they are continuously building new services, they make service-templates to make their life easy. With these templates, they can spin up a new service in almost no time and then they share it with other teams. Now whenever i am exploring a new language, I reach out to teams who are already champions in it and see what template they have.
Design for Delivery
This is also a very interesting one. Usually the thought process in the Enterprise goes like this - Team has a problem statement --> They design the solution --> Build it --> And finally somehow deliver it.
In microservice world this thought process is different. Team has a problem statement-->They first think about delivery. Why? because Delivery has lot of impact on how you design it. It helps you better decompose you solution meaning lets say in the problem statement - if there are some very active and some not so active areas then you may want to decompose your design in this manner and deliver and scale them independently. You would not do things like writing too much logic in the database or take reference on any dependency that could potentially slow down your own delivery.
There will always be production bugs and performance problems to deal with and the only way a team can react to it faster if they have designed their solution for Delivery over anything else.
Next time i will share my experience of how adoption of microservices has impacted Developer Experience!
