Microservice Practices - Dark Launches and Feature Toggles

Dark-launches and Feature toggles are two great practices to deploy code faster to production and get early feedback. Combined with microservices architecture, it becomes a very powerful tool and once adopted, there is no turning back! You will be thrilled by the speed with which you can build and ship software in a safe manner! In this blog, I will try to explain what these terms means.

Dark Launch
Dark launch is a practice where you can test/validate your new functionality in production without having to worrying about disturbing the original workflow. This technique is very useful when writing a new version of the functionality or even adding more functionality that can be triggered based of any existing workflow. The steps to achieve it are very simple:
  1. Deploy the new functionality in prod. 
  2. Incase if it is new version of an existing functionality and you want to test correctness: 
    • clone the request going into v1-logic —> then call the v1-logic —> clone the v1-result. 
    • Now in the background (fire and forget) call the new logic with cloned v1 request and compare v2-result with v1-result. Since this happens in background, there is no impact on the original workflow 
  3. In case if you want to test new functionality invoked of an existing workflow 
    • During the execution of the workflow, assemble the information you need in order to call the new functionality 
    • In the background - make the call to the new functionality and start collecting metrics. 
  4. If storage is involved - you can ignore it (if its dump) or do cleanup later (in case if there is some logic involved). Dump storage is preferred!


This practice allows you to collect some key metrics around correctness, robustness and performance of the new functionality even before the code is in use for real. Since the call to the new functionality happens in the background, it does not impact the original workflow! How awesome is that! You can ship the code faster to the production and get meaningful insights without breaking any of your clients! 

Feature Toggles
Feature toggles is a practice of enabling a new functionality for selected group of consumers at a time. This technique allows you to do selective rollout and also restricts the blast radius when things do go wrong with the newly deployed code. You can imagine dark-launch as a feature also and selectively enable it for set of clients. Toggles are created and stored external to the application and development team has full control over them. Feature toggles and branch-by-abstraction (which is well documented here by Martin Fowler) go hand in hand. In simple terms, all it is saying that before invoking a new piece of functionality, first check if the user has toggle-on for it. If yes - then call the new path else continue to call the old path. 

I work in Stock-market domain where one mistake could potentially cost our clients millions of dollars and I have used these techniques for all sort of work - from small rewrites/features to complex tasks of migrating storage from SQL-server to mysql. The results have been amazing and at the end, we have delivered a robust, stable and performant product offering. 




Popular posts from this blog

Break functional and orchestration responsibilities for better testability

Image
Microservice architecture paradigm fits well within the Unix’s first principle/philosophy: “Make one program do one thing well”.  We love creating small services which has identity of its own (within a bounded context). These services have their own CI/CD pipelines and provides loose coupling b/w components which significantly improves our ability to deliver code to production. In reality, a workflow can span multiple services. Its very common to find scenarios like below: In the above example, a workflow spans 3 services (Service-1, 2 and 3). Service1 receives the original request. In order to process this request, it GET(s) some data from its dependent services(d1/d2/d3). It further runs some business logic and produces an intermediate response #A. It then makes a state mutable call (POST/PUT/DELETE/PATCH) to Service-B using #A as input and waits for its response. Service-B follows the same paradigm and the workflow ends when all services returns.  The scenario above
Read more

Prefer Consumer over Integration Tests

Image
Recently I attended an RCA meeting for an incident that happened in prod few weeks back. The issue was somewhat of a 2nd degree failure. To put it in simple terms, lets say a workflow composed of two services - A and B. When user performed an action from the UI, it made a call to Service A which then internally called service B with some parameters based on user preferences. In this case, due to some unusual combination of preferences, Service-A ended making call to Service-B with parameters that Service-B didn’t fully understood. Solution - Write an integration test to cover this scenario. Reasoning - Since this involved 3 different components (User Preferences, Service A and Service B), it should be covered by an integration test. More Reasoning: This is sort of the usual way of thinking about the integration tests - meaning whenever there are multiple services involved, we immediately think about writing an integration test. But is this the right approach specially in microserv
Read more

Microservices and Tech Stack - Lessons Learned

Image
In this blog, I want to talk about how our technology stack has evolved over time while transitioning from Enterprise to adopting Microservice architecture as a main stream development practice and share some of the lessons learned. Tech stack for us has evolved from being a Monolithic Enterprise to Cloud to now Microservices. During Enterprise days, it looked something liked this: At the app layer, we had a desktop client written in Windows form and WPF. The app would talk to the Service Layer which was SOA architecture written completely in C#. It was the only language allowed for us to use. These were monolithic and stateful services communicating to each other over WCF. We used SQL server as backend storage. All this get deployed on premise meaning our clients buy our software and host it on their own hardware. I work for Finance industry (Stock Market to be precise) and they were sort of late adopter of public cloud. They absolutely hated the idea of putting their data on p
Read more