Instrumentation, Observability and Monitoring (IOM)

Terminology


Observability: Observability is the property of a system to answer either trivial or complex questions about itself. How easily you can find answers to your questions is a measure of how good the system’s observability is.


Monitoring: observing the quality of system performance over a time duration


Instrumentation: refers to metrics exposed by the internals of the system using code (a type of white-box monitoring)


Why IOM? 



  1. Analyzing long-term trends like

    1. User growth over time

    2. User time in the system

    3. System Performance over time

  2. Comparing over time or experiment groups

    1. How much faster are my queries after new version of a library

    2. Cache hit/miss ratio after adding more nodes

    3. Is my site slower than it was last week?

  3. Alerting - Something is broken, and somebody needs to fix it right now! 

  4. Building dashboards - Answer basic questions about your service - Latency, Traffic & Errors.

  5. Conducting ad hoc retrospective analysis - Our latency just shot up; what else happened around the same time?

How to observe?

The three pillars: 

  • Logs - A text that describes an event that happened at a certain time.

  • Metrics - Events with a name and a metric value, plus likely a low cardinality set of dimensions aggregated and stored for low cost, fast retrieval.

  • Tracing - A series of events with a parent/child relationship. Generally this tells the story of an entire user interaction and displayed in a Gantt-chart like view. This is usually achieved by adding a trace-id


What to Observe?





  1. Infrastructure Observability- Example include

    1. Lay of the land e.g (Hosts/regions, For serverless - memory/CPU/invocation)

    2. Message Bus stats (events published/captured, performance, limit, size..)

    3. Networking stats

    4. DB stats

    5. Cache stats

  2. Service health and performance: Examples include

    1. Service Uptime/health 

    2. P90/95 API performance, invocation count

    3. P90/95 performance for capturing events by event_type, 

    4. Captured Events count by event_type

    5. Performance breakdown by major blocks in the workflow (e.g time spend for DB operations, dependency api calls, actual business logic)

    6. P90/95 for DB Queries

    7. Error trends

    8. Input/Output to/from the service

  3. Product level observability.

    1. 80/20 rule for a product - Most used APIs/Most used features

    2. Tracing workflow through out the system

  4. Customer Experience: Examples include

    1. Customer UI experience - Time spent on page, clicks by user by workflow

    2. Workflow/API performance from the end user perspective.

    3. UI Errors

  5. Business level observability: Examples include

    1. User growth over time

    2. User time (session duration) in the system over time

    3. New Feature adoption rate

Best Practices

  • Focus on what and why? The "what’s broken" indicates the symptom; the "why" indicates a (possibly intermediate) cause. E.g a spike in 500 error codes is a symptom but possibly caused by a db server refusing connections.

  • Don't be afraid to write extra code in your service to detect and expose possible causes.

  • Always think about the user impact: “How will these metrics show me the user impact?”

  • Make it useful, then actionable: When a monitor triggers an alarm, it should first and foremost be “useful”. Secondly, it should be “actionable”. There should be something you can do to resolve the alarm and also be a set of steps (post-resolution) to prevent that alarm from triggering again.

  • If the alarm isn’t actionable, then it just becomes noise.

  • Having consistent log format makes it easier to parse and aggregate information across multiple services.

Popular posts from this blog

Break functional and orchestration responsibilities for better testability

Image
Microservice architecture paradigm fits well within the Unix’s first principle/philosophy: “Make one program do one thing well”.  We love creating small services which has identity of its own (within a bounded context). These services have their own CI/CD pipelines and provides loose coupling b/w components which significantly improves our ability to deliver code to production. In reality, a workflow can span multiple services. Its very common to find scenarios like below: In the above example, a workflow spans 3 services (Service-1, 2 and 3). Service1 receives the original request. In order to process this request, it GET(s) some data from its dependent services(d1/d2/d3). It further runs some business logic and produces an intermediate response #A. It then makes a state mutable call (POST/PUT/DELETE/PATCH) to Service-B using #A as input and waits for its response. Service-B follows the same paradigm and the workflow ends when all services returns.  The scenario above
Read more

An Effective TTL-Caching Pattern

Image
Microservice architecture comes with a high cost of service dependencies which impacts the overall performance of the system. In a typical setup, when a request hits the service, it first needs to grab data from its dependent apis and after which it will perform more operations and return the final result back to the caller. Dependent apis may further call other apis or talk to a DB in order the return the requested data and hence creating a complex graph of dependencies.  Caching is an effective way to improve the performance of such system. In many cases, you might be able to cache the response of the dependent apis locally and avoid making those extra calls specially if the data is somewhat static in nature. TTL is a great feature which allows you to cache a data for certain duration after which it expires. The pattern looks something like this: First check if the data is cached. If not, get the data from the original source and cache it for future calls. Recently, I worke
Read more

Microservices and Tech Stack - Lessons Learned

Image
In this blog, I want to talk about how our technology stack has evolved over time while transitioning from Enterprise to adopting Microservice architecture as a main stream development practice and share some of the lessons learned. Tech stack for us has evolved from being a Monolithic Enterprise to Cloud to now Microservices. During Enterprise days, it looked something liked this: At the app layer, we had a desktop client written in Windows form and WPF. The app would talk to the Service Layer which was SOA architecture written completely in C#. It was the only language allowed for us to use. These were monolithic and stateful services communicating to each other over WCF. We used SQL server as backend storage. All this get deployed on premise meaning our clients buy our software and host it on their own hardware. I work for Finance industry (Stock Market to be precise) and they were sort of late adopter of public cloud. They absolutely hated the idea of putting their data on p
Read more