Microservices and Delivery - Lessons Learned

One of the most important selling point of moving towards microservice architecture is the speed of delivery! Today customers can research products, compare reviews and ultimately switch to better products online with unprecedented ease. To survive in this high customer-focused and fast moving industry, we need our teams to deliver software at an accelerated rate and possibly multiple times in a day.

For us over the time, our delivery pipeline has also evolved:











During the Enterprise days, the release cycle was 6 weeks long. If you missed anything - wait for another 6 weeks. We used home grown scripts to coordinate all the deployments and overall experience was just awful. During the early days of Cloud, we moved to a daily deployments but these were at the end of the day with significant downtime and required lots of coordination with different teams. Today we have deployments on demand. Teams are deploying multiple time in a day with zero downtime!

We love Slack at work! This is one of the bots that shows #number-of-deployments in last 14 days to production env. 














Lessons Learned

Empower Development Teams to push code to production:

If there is any hand-off process b/w the path from developer machine to prod - it will never scale. Invest into tools and infrastructure to enable zero downtime deployments and empower the development teams to push code to prod. Sounds Crazy? Totally works!

Improve Development Practices: 

Educate, Implement and REWARD teams that are adopting continuous-delivery friendly practices in their development cycle. Master shipping code using Dark-Launches, Feature Toggles and Backward/Forward compatibility. Bring Performance and Chaos Engineering practices early in development.

Monitoring

Monitoring becomes an important aspect of the of the daily job. Teams should invest into learning monitoring tools (spunk/datadog/grafana …or which ever you use). On-call/pager-duty is a must! Mature teams adopt scrum-ops - an extended version of scrum in which after the regular scrum, the entire team looks at the health/performance of their stacks for last 24 hours and identifies good/bad/normal operating scenarios.

Blame free Post-Mortems

When things do go wrong, provide a space for teams to perform blame free post-morterms. Only focus on Timeline/what contributed/what could we have done better/Action items!

Popular posts from this blog

Break functional and orchestration responsibilities for better testability

Image
Microservice architecture paradigm fits well within the Unix’s first principle/philosophy: “Make one program do one thing well”.  We love creating small services which has identity of its own (within a bounded context). These services have their own CI/CD pipelines and provides loose coupling b/w components which significantly improves our ability to deliver code to production. In reality, a workflow can span multiple services. Its very common to find scenarios like below: In the above example, a workflow spans 3 services (Service-1, 2 and 3). Service1 receives the original request. In order to process this request, it GET(s) some data from its dependent services(d1/d2/d3). It further runs some business logic and produces an intermediate response #A. It then makes a state mutable call (POST/PUT/DELETE/PATCH) to Service-B using #A as input and waits for its response. Service-B follows the same paradigm and the workflow ends when all services returns.  The scenario above
Read more

Microservices and Tech Stack - Lessons Learned

Image
In this blog, I want to talk about how our technology stack has evolved over time while transitioning from Enterprise to adopting Microservice architecture as a main stream development practice and share some of the lessons learned. Tech stack for us has evolved from being a Monolithic Enterprise to Cloud to now Microservices. During Enterprise days, it looked something liked this: At the app layer, we had a desktop client written in Windows form and WPF. The app would talk to the Service Layer which was SOA architecture written completely in C#. It was the only language allowed for us to use. These were monolithic and stateful services communicating to each other over WCF. We used SQL server as backend storage. All this get deployed on premise meaning our clients buy our software and host it on their own hardware. I work for Finance industry (Stock Market to be precise) and they were sort of late adopter of public cloud. They absolutely hated the idea of putting their data on p
Read more

An Effective TTL-Caching Pattern

Image
Microservice architecture comes with a high cost of service dependencies which impacts the overall performance of the system. In a typical setup, when a request hits the service, it first needs to grab data from its dependent apis and after which it will perform more operations and return the final result back to the caller. Dependent apis may further call other apis or talk to a DB in order the return the requested data and hence creating a complex graph of dependencies.  Caching is an effective way to improve the performance of such system. In many cases, you might be able to cache the response of the dependent apis locally and avoid making those extra calls specially if the data is somewhat static in nature. TTL is a great feature which allows you to cache a data for certain duration after which it expires. The pattern looks something like this: First check if the data is cached. If not, get the data from the original source and cache it for future calls. Recently, I worke
Read more