Five Tips and Tricks for Application Monitoring with Application Insights

I haven’t written much about Application Insights, even though I’ve been using it intensively for about five-six years now. In this post, I will share with you five tips and features from Application Insights that you can use to enhance your monitoring and logging experience.

If you don’t know what Application Insights is you can learn more about it from Microsoft Learn

Tweak Log Analytics Settings

When Application Insights was launched it stored the data inside the Application Insights service, but in recent years Microsoft has moved all the logs and metrics into Log Analytics Workspace. This is basically a database for logs. Previously we could set data sampling, data retention and daily caps directly at Application Insights settings, but now we need to do it in the Log Analytics Workspace settings. You can find the linked Log Analytics Workspace by going into “Usage and estimated costs” page at Application Insights page. There is link on top of the page to the linked service, but it won’t take you into correct page at Azure Portal. Copy the name of the workspace and search the resource by using Azure Portal search bar.

Log Analytics Workspace has same setting page called “Usage and estimated costs”, that can be used to set the data retention and daily cap. Notice that sampling cannot be set here. You need to set the data sampling at applicationinsights.config file, but I recommend to keep it at 100% level. If you have it lower, it can lead into strange results because some of the actual log data might be sampled (not send into Application Insights). So tip number one is to check your data retention, cap and sampling settings.

More information about workspace migration can be found from this url: https://learn.microsoft.com/en-us/azure/azure-monitor/app/convert-classic-resource

Filtering Data

If you are using Entity Framework there is a change that your application is sending lot of dependency data into Application Insights. You can check the data dispersion from Log Analytics Workspace (at Azure Portal). If you have lot of AppDependencies (like I had for one EF app), you might want to filter out some of it.

To filter out AI data, we need to write simple C# class that skips SQL dependencies. We need to create new class called NoSQLDependencies and implement ITelemetryProcessor interface. In this sample I will filter out all the SQL dependencies, but you can add more intelligence if you want filter only failed dependencies etc. This filter will affect how app map is shown in Application Insights (because there is no dependency into database), but I don’t care about that feature anyway. I know, that my app is using database.

    public class NoSQLDependencies : ITelemetryProcessor
    {
        private ITelemetryProcessor Next { get; set; }

        public NoSQLDependencies(ITelemetryProcessor next)
        {
            Next = next;
        }

        public void Process(ITelemetry item)
        {
            if (IsSQLDependency(item))
            {
                // Skip SQL Dependencies
                return;
            }

            Next.Process(item);
        }

        private bool IsSQLDependency(ITelemetry item)
        {
            var dependency = item as DependencyTelemetry;
            if (dependency?.Type == "SQL")
            {
                return true;
            }
            return false;
        }
    }

Then we need to add new telemetry processor into applicationinsights.config file. Add line inside TelemetryProcessors tag. Common is name of the namespace that holds the NoSQLDependencies class. You will have different namespace depending on your project name.

<Add Type="Common.ApplicationInsights.NoSQLDependencies, Common" />

Now when we fire the app, it won’t send those SQL dependencies anymore into Application Insights. You can set breakpoint into Process method and verify that it will fire when SQL dependency occurs. Tip number two is to filter out unnecessary data.

Alerts

Application Insights has very powerful alert system built-in. You can create custom alerts based on logs and metrics. If you set the check interval into max value, they will cost you like nothing (well 0,1$/month). My personal favorite is amount of exception, that is triggered when unusual amount of exception has occurred during interval. Usually you will always have at least some exception level logging happening inside your application, even though that they are not “real” issues. Or issues that you should immediately react on. To give us a notification if something is really wrong, we can set an alert rule that sends notification if we are having unusual amount of exceptions. For example if I’m normally having like 1-5 exceptions per hour (which is a lot I know), but anything above that is abnormal. I can create alert rule that is triggered when Exception signal is triggered more than 5 times within one hour.

This is great way to do alerts for systems that might have some exceptions occurring in normal usage. For example if you are using external service, which might have some regular downtimes, that causes exceptions.

Tip number three is to use powerful built-in alerting system.

Availability Checks

Availability check is relatively new feature in Application Insights. It has actually been there for few years, but I think it is still something that people often don’t know about. With availability checks we can verify that our web app is responding with valid response and if it doesn’t, trigger an alarm. Test is easy to setup by using Availability > Create Standard test feature. Test can be run from different locations, so you are not depended on specific Azure location (East US, West Europe etc.). There is also option to check for SSL certificate validity and receive an alert if it is going to expire.

Content match feature allows to inspect received result more carefully and make sure that web app is not just returning error page. Many organizations has other tools to do this same functionality, but I think they are redundant. AI can do all the same tricks and you can view the application logs at the same place.

Tip number four is to use availability checks directly from Azure. You don’t need to use third party services like SolarWinds anymore.

Error or Exception

Application Insights has categories for different kind of events. Log writes are called traces and if application captures exception, then they are categorized as exceptions. Normally when developers need to handle exceptions they add try-catch block with logging in the catch

try
{
 // Do something that might produce null reference exception
}
catch (NullReferenceException nullException)
{
    _logger.LogError("Failed to do something", nullException);
}

Application Insights creates exception from this logging as we would expect it to do. However if we don’t add the exception into LogError, then Application Insights creates only trace event from it. Now if we apply alert rules only for exceptions, this can lead into situations where we might get a lot of error traces, but alert does not trigger. There is no easy way to change this default behavior, so my suggestion here is to build KQL query that gets both exceptions and traces with loglevel of 3 or higher. This way we can pick both log errors with and without exceptions.

Tip number five is: Remember to handle trace errors.

Automatically create work items from Application Insights

If you want to create Azure DevOps work items automatically from Application Insights exceptions and error traces, please check out my old post about automating error reporting with Logic Apps.

Automatically create work items from Application Insights exceptions