Instrumentation, Observability and Monitoring (IOM)

Terminology


Observability: Observability is the property of a system to answer either trivial or complex questions about itself. How easily you can find answers to your questions is a measure of how good the system’s observability is.


Monitoring: observing the quality of system performance over a time duration


Instrumentation: refers to metrics exposed by the internals of the system using code (a type of white-box monitoring)


Why IOM? 



  1. Analyzing long-term trends like

    1. User growth over time

    2. User time in the system

    3. System Performance over time

  2. Comparing over time or experiment groups

    1. How much faster are my queries after new version of a library

    2. Cache hit/miss ratio after adding more nodes

    3. Is my site slower than it was last week?

  3. Alerting - Something is broken, and somebody needs to fix it right now! 

  4. Building dashboards - Answer basic questions about your service - Latency, Traffic & Errors.

  5. Conducting ad hoc retrospective analysis - Our latency just shot up; what else happened around the same time?

How to observe?

The three pillars: 

  • Logs - A text that describes an event that happened at a certain time.

  • Metrics - Events with a name and a metric value, plus likely a low cardinality set of dimensions aggregated and stored for low cost, fast retrieval.

  • Tracing - A series of events with a parent/child relationship. Generally this tells the story of an entire user interaction and displayed in a Gantt-chart like view. This is usually achieved by adding a trace-id


What to Observe?





  1. Infrastructure Observability- Example include

    1. Lay of the land e.g (Hosts/regions, For serverless - memory/CPU/invocation)

    2. Message Bus stats (events published/captured, performance, limit, size..)

    3. Networking stats

    4. DB stats

    5. Cache stats

  2. Service health and performance: Examples include

    1. Service Uptime/health 

    2. P90/95 API performance, invocation count

    3. P90/95 performance for capturing events by event_type, 

    4. Captured Events count by event_type

    5. Performance breakdown by major blocks in the workflow (e.g time spend for DB operations, dependency api calls, actual business logic)

    6. P90/95 for DB Queries

    7. Error trends

    8. Input/Output to/from the service

  3. Product level observability.

    1. 80/20 rule for a product - Most used APIs/Most used features

    2. Tracing workflow through out the system

  4. Customer Experience: Examples include

    1. Customer UI experience - Time spent on page, clicks by user by workflow

    2. Workflow/API performance from the end user perspective.

    3. UI Errors

  5. Business level observability: Examples include

    1. User growth over time

    2. User time (session duration) in the system over time

    3. New Feature adoption rate

Best Practices

  • Focus on what and why? The "what’s broken" indicates the symptom; the "why" indicates a (possibly intermediate) cause. E.g a spike in 500 error codes is a symptom but possibly caused by a db server refusing connections.

  • Don't be afraid to write extra code in your service to detect and expose possible causes.

  • Always think about the user impact: “How will these metrics show me the user impact?”

  • Make it useful, then actionable: When a monitor triggers an alarm, it should first and foremost be “useful”. Secondly, it should be “actionable”. There should be something you can do to resolve the alarm and also be a set of steps (post-resolution) to prevent that alarm from triggering again.

  • If the alarm isn’t actionable, then it just becomes noise.

  • Having consistent log format makes it easier to parse and aggregate information across multiple services.

Popular posts from this blog

Break functional and orchestration responsibilities for better testability

Image
Microservice architecture paradigm fits well within the Unix’s first principle/philosophy: “Make one program do one thing well”.  We love creating small services which has identity of its own (within a bounded context). These services have their own CI/CD pipelines and provides loose coupling b/w components which significantly improves our ability to deliver code to production. In reality, a workflow can span multiple services. Its very common to find scenarios like below: In the above example, a workflow spans 3 services (Service-1, 2 and 3). Service1 receives the original request. In order to process this request, it GET(s) some data from its dependent services(d1/d2/d3). It further runs some business logic and produces an intermediate response #A. It then makes a state mutable call (POST/PUT/DELETE/PATCH) to Service-B using #A as input and waits for its response. Service-B follows the same paradigm and the workflow ends when all services returns.  The scena...
Read more

Microservices and Tech Stack - Lessons Learned

Image
In this blog, I want to talk about how our technology stack has evolved over time while transitioning from Enterprise to adopting Microservice architecture as a main stream development practice and share some of the lessons learned. Tech stack for us has evolved from being a Monolithic Enterprise to Cloud to now Microservices. During Enterprise days, it looked something liked this: At the app layer, we had a desktop client written in Windows form and WPF. The app would talk to the Service Layer which was SOA architecture written completely in C#. It was the only language allowed for us to use. These were monolithic and stateful services communicating to each other over WCF. We used SQL server as backend storage. All this get deployed on premise meaning our clients buy our software and host it on their own hardware. I work for Finance industry (Stock Market to be precise) and they were sort of late adopter of public cloud. They absolutely hated the idea of putting their data on p...
Read more

Prefer Consumer over Integration Tests

Image
Recently I attended an RCA meeting for an incident that happened in prod few weeks back. The issue was somewhat of a 2nd degree failure. To put it in simple terms, lets say a workflow composed of two services - A and B. When user performed an action from the UI, it made a call to Service A which then internally called service B with some parameters based on user preferences. In this case, due to some unusual combination of preferences, Service-A ended making call to Service-B with parameters that Service-B didn’t fully understood. Solution - Write an integration test to cover this scenario. Reasoning - Since this involved 3 different components (User Preferences, Service A and Service B), it should be covered by an integration test. More Reasoning: This is sort of the usual way of thinking about the integration tests - meaning whenever there are multiple services involved, we immediately think about writing an integration test. But is this the right approach specially in microserv...
Read more