Why microservices fail?

Adoption of microservices is not easy and I have had my own share of painful experiences during the initial phase of the transition. In this blog, I will go over some of factors that can be critical to determine the outcome of such transition.

Lack of Engineering Culture - What’s an Engineering Culture anyway? Its a culture that practices lean-principles and further enables osmosis style learning for everyone. There is recognition for great work and people are willing to learn from each other. Its a culture where everyone is empowered and feels responsible to participate/evolve/improve all aspects of the SDLC irrespective of their primary role. You can have specialized teams for the different parts but everyones feedback is welcomed. Processes are not followed like a religion but evolved throughout the lifecycle.

Adoption of microservices is a steep learning curve for everyone. If an organization doesn’t have the right Engineering mindset, avoid doing microservices. You have even bigger battles to fight first.
Patterns to look for:
  • Not enough buy-in from other specialized teams or takes significant effort to get them involved. 
  • Hands off model followed b/w PM->Dev->QA->Release 
  • Process are followed like religion with minimum to no input from other teams. 
Microservice is just another architecture - The mindset that microservices is just another architecture that only concerns the Dev team is completely wrong. It has a widespread impact on the entire SDLC lifecycle and all the different teams involved. It will impact how you design, build, test, deploy and even market your product. For a successful adoption, It requires significant organizational team restructure in order to bring different specialized team closer and embrace for change.
Patterns to look for
  • Process teams over product teams. 
  • Lack of decision making power to make structural changes within the organization. 

Lack of Expertise - Few months back I was working on a home project of converting my basement’s carpeted floor into a hardwood floor. It was my first attempt and I made some horrible mistakes. I was sharing my experience with a colleague and he said something very interesting - “If you doing a home project for the first time - the first attempt should be at your enemy’s house because you will make lots of mistake while learning the tricks. Then do it on your friend’s house - reason being you may have learned the lessons but not an expert yet. The third attempt should be on your own house and at this time you will be a pro”. This is very true for microservices also. Its an acquired skill and I am sure teams will make lots of mistake when doing it for the first time. Don’t try to solve your hardest problem in the first attempt - go from easy areas and further move on as you gain more confidence. The lack of expertise could also be at the domain level meaning if you are exploring a new product for the market fit, don’t start with microservices. To get a hang of the domain and understand boundaries with bounded context will take time.
Patterns to look for:
  • Lack of domain expertise (Explorers versus Settlers). 
  • Level of expertise in building microservices 
    • General rule of thumb - If the team has crossed the mark of 5 services in their portfolio, then only they know something about microservices. 
Developer and Testing Experience - I have seen lots of examples in terms of blog post and videos, where companies tried to adopt microservices and have had some good initial success but later abandon it in the middle. The main complains so far has been around Developer Experience and lack of good isolated Component Tests. If you are doing microservice and require a developer to deploy the entire cloud on his local machine before he can be productive, you are doing it wrong! If you cannot test your service in isolation, you will never be able to gain enough confidence and things will fall when all come together. Invest into Developer experience early in the process. Create services that has well defined set of responsibilities and can be tested in isolation.
Patterns to look for:
  • Timespan b/w getting latest code to point where you have a working system (Ideally should be couple of mins.) 
  • Team’s confidence level in pushing code to prod intraday. 
  • Team’s confidence level in automated testing in place. 
DevOps at the core of the team work ethics- DevOps can be sliced and diced in different ways but at the core, it requires developer to be ready to support their code in production anytime of the day. Gone are the days when developer were only responsible for writing the code. Today, we need to think more like service providers and be ready to support both internal and external consumers with the same priority. Teams also need to monitor their services real time and clearly identify the good/bad/normal operating scenarios. Or put it in simple terms “You build it, you run it” - Werner Vogels
  • Teams can monitor realtime system-health and performance 
  • Realtime alerts are setup for errors. 
  • Space for blame-free postmortems 

Other than cultural implications, there are more complex technical problems to deal with around distributed transactions and embracing eventual consistency. So why adopt microservices when its so hard? I think of microservices as level of maturity. In the past when industry moved from 2-tier (Client-DB) application to SOA, it probably went through the same sort of arguments. In my mind, its the next level of maturity in building modern software. If adopted successfully - you are well on your way to transform the industry at an accelerated pace!

Popular posts from this blog

Break functional and orchestration responsibilities for better testability

Image
Microservice architecture paradigm fits well within the Unix’s first principle/philosophy: “Make one program do one thing well”.  We love creating small services which has identity of its own (within a bounded context). These services have their own CI/CD pipelines and provides loose coupling b/w components which significantly improves our ability to deliver code to production. In reality, a workflow can span multiple services. Its very common to find scenarios like below: In the above example, a workflow spans 3 services (Service-1, 2 and 3). Service1 receives the original request. In order to process this request, it GET(s) some data from its dependent services(d1/d2/d3). It further runs some business logic and produces an intermediate response #A. It then makes a state mutable call (POST/PUT/DELETE/PATCH) to Service-B using #A as input and waits for its response. Service-B follows the same paradigm and the workflow ends when all services returns.  The scenario above
Read more

Prefer Consumer over Integration Tests

Image
Recently I attended an RCA meeting for an incident that happened in prod few weeks back. The issue was somewhat of a 2nd degree failure. To put it in simple terms, lets say a workflow composed of two services - A and B. When user performed an action from the UI, it made a call to Service A which then internally called service B with some parameters based on user preferences. In this case, due to some unusual combination of preferences, Service-A ended making call to Service-B with parameters that Service-B didn’t fully understood. Solution - Write an integration test to cover this scenario. Reasoning - Since this involved 3 different components (User Preferences, Service A and Service B), it should be covered by an integration test. More Reasoning: This is sort of the usual way of thinking about the integration tests - meaning whenever there are multiple services involved, we immediately think about writing an integration test. But is this the right approach specially in microserv
Read more

Microservices and Tech Stack - Lessons Learned

Image
In this blog, I want to talk about how our technology stack has evolved over time while transitioning from Enterprise to adopting Microservice architecture as a main stream development practice and share some of the lessons learned. Tech stack for us has evolved from being a Monolithic Enterprise to Cloud to now Microservices. During Enterprise days, it looked something liked this: At the app layer, we had a desktop client written in Windows form and WPF. The app would talk to the Service Layer which was SOA architecture written completely in C#. It was the only language allowed for us to use. These were monolithic and stateful services communicating to each other over WCF. We used SQL server as backend storage. All this get deployed on premise meaning our clients buy our software and host it on their own hardware. I work for Finance industry (Stock Market to be precise) and they were sort of late adopter of public cloud. They absolutely hated the idea of putting their data on p
Read more