Five Things Developers Can Use To Go On-call (and Not Hate Their Life)

July 10, 2019 · 5 minute read


Why Are Developers Going On-Call These Days?

As service reliability and uptime become increasingly more important for organizations building applications, developers are now being exposed to new responsibilities and challenges.

Tiered ticketing processes, Network Operations Centers (NOC), along with traditional change and incident management technicques can’t provide the level of response necessary to support rapidly deployed applications requiring high availability and maximum uptime.

Urgency of downtime in the modern era requires a different approach to building and maintaining digital systems and services. Engineers must know about and respond to problems quickly.

Rapid response and remediation of service disruptions means those with intimate knowledge of the codebase as well as the build and release process are best suited as first responders - the the software engineers.

Today, developers with little experience outside of software delivery lifecycle are now being asked to go “on-call” and assume the First Responder role when problems inevitably arise in production systems.

Here are five things all software engineers should know about to accept the new challenges of building and maintaining highly available systems.

For more of my thoughts on the role of “On-call” including incident resonse and retrospective analysis, check out this talk from Microsoft Ignite | The Tour

1 - Application Insights

The worst part about being alerted when you are on-call is not knowing where to investigate or what data is even available to you.

Out of the box, Azure Monitor provides several great products to help build a more obvservable system and make it easier to know about and diagnose problems in our applications running in the cloud.

Instrumented with very few lines of code, Application Insights, instantly begins scraping tons of great data that can be queried, analyzed, and shared with other engineers.

Even if on-call isn’t a responsibility for you as a developer, understanding and leveraging Application Insights will give you a big advantage when it comes to building high performing applications on Azure.

2 - Azure Monitor For Containers

Containers are all the rage these days. But with so much abstraction it’s hard to know what’s actually going on with your applications inside of them. And when you start building large scale applications with components that dynamically come and go, the challenge is even harder.

Azure Monitor For Containers is the best way to gain visibility on exactly what’s going on within these abstracted pieces of modern applications.

3 - Log Analytics with KUSTO

Collecting data is one thing but how do you know where to look and how to ask what you want to know?

Log Analytics (a feature of Azure Monitor) stores a ton of helpful data regarding both the internals of your application as well as the underlying infrastructure.

Capturing data such as requests, exceptions, traces, and usage in Log Analytics, this data is stored in and made available by writing a simple query.

Log Analytics uses a version of the Data Explorer query language to retrieve and analyze log data in a variety of ways.

The basic structure of a query is a source table followed by a series of operators separated by a pipe character | .

You can chain together multiple operators to refine the data and perform advanced functions.

You use the query language KUSTO Query Language (KQL) to access this data.

For example, suppose you wanted to find the total number of successful HTTP requests in the last hour, along with a count of how many were successful and how many failed.

KUSTO Query Language (KQL)

| where timestamp > ago(1h)
| summarize succeed = count (success==true), failed = count (success == false), total = count() by bin(timestamp, 1h)

4 - Logic Apps & Teams

Azure Logic Apps is a service that helps you automate and orchestrate tasks and workflows by integrating apps, data, systems, and services across all kinds of places internally and externally to your company.

Using Logic Apps, new channels can be created automatically as problems arise. This allows for a clean workspace to respond to and collaborate over the issues at hand.

5 - Visual Studio Code Extensions

Visual Studio Code is now the most popular text/code editor according to “The State of JS”.

One reason, is because there are so many great extensions to make the lives of developers easier.

Here are a couple of extensions anyone developing in Azure should add to their VS-Code toolkit.

Azure Application Insights Visual Studio Code Extension

The Application Insights extension for Visual Studio Code brings information from your production services right into your code editor to help you find and fix issues faster.

Enable Application Insights telemetry collection in your Express app with a simple command in Visual Studio Code.

Azure Log Analytics (Kusto) Syntax Highlighting

Adds highlighting support for Azure Log Analytics (Kusto) (.csl and .kusto). This syntax is based on TextmateBundleInstaller - Kusto syntax.

Knowing about these five things should help upskill your development abilities and move you closer to building highly available applications without hating your life while “on-call”.