Turn Application Insights Exceptions into Pull Requests: End-to-End Automation with Kilo Code

I’ve been diving deep into Kilo Code Cloud Agents lately, and I have to say that I absolutely love it. In a world where we’re moving towards cloud agents and no longer running code locally, this approach feels like a glimpse into the future. Today, I want to show you a fully automated solution that takes exceptions from Application Insights and automatically fixes it with a pull request. It’s a fully hands-off workflow that turns errors into code fixes.

Architecture

Our solution has four pieces:

1. Application that is connected into Application Insights
2. Application Insights instance in Azure
3. Kilo Code account with GitHub integration enabled
4. GitHub repository

Our application (for example, a .NET Web API app) is configured to send error messages to Application Insights. Application Insights has a custom rule, which triggers when a new exception is received and posts it to the Kilo Code webhook. Kilo Code launches an agent that reads the error message (+stacktrace etc.) and checks if this issue has already been reported. If this is a new issue the agent will create a new GitHub issue, fix the error and submit a Pull Request against our main branch.

Setting Up Kilo Code

At first we need to set up Kilo Code. Kilo Code started as a fork of Roo Code and Cursor, but since it has grown into an own platform with Cloud Agents, integrations and many other features. I like it because it is not provided by any AI model company (so it does not prefer its own models), it is VS Code extension is open source and it has really good integration against GitHub. For this example you can use some other system as well as replacement if you don’t use Kilo Code as long it has a support for webhooks.

After you have logged into Kilo Code dashboard you need to add GitHub integration to your account. This allows Kilo Code to interact with your GitHub repositories. You can allow all repos or just select the ones that you want to.

Next we need to configure a webhook that triggers when it receives a POST call. Fill all the mandatory fields and for model you can choose whatever you like. Of course the Claude models are really good, but Kilo Code supports many others. You also need to define an environment. If you don’t need anything specific for the environment, just create one without any settings.

For the Prompt Template I used this kind of prompt. It is not perfect, but it is a good start

This is application insight alert report. It contains details about exception that has occured. Create pull request that fixes the issue. Read first value from json array of property alertContext.context.condition.allOf and read value from event array that has name of problemId. Use the problemId as issue topic.
Put customDimensions as issue description. Then fix the issue and create pull request against main branch.

First check if issue about this topic has already been created and don’t duplicate issues. Check for open issues only.

{{bodyJson}}

After creating the webhook we can see an URL that we can put to Application Insights.

Setting Up Application Insights

First I have to say that I had lots of problems triggering the Application Insights event. You can use webhook.site to verify that Application Insights alert rule is really working, before pointing it against Kilo Code. This can save you some time if you have problems with triggering like I did.

OK, for Application Insights we are going to use Custom Alert Rule. If you don’t have existing Application Insights resource, you can create new one and then click Alerts > Create > Alert rule to create the new rule.

In alert rule page select custom log search as a signal. We need to use this, because we are interested from the content of the error messages, not just amount of errors.

Select Single event as Query type and In custom query add following KQL statement:

exceptions
| where timestamp >= ago(5m)
| project
TimeGenerated = timestamp,
problemId,
type,
outerMessage,
customDimensions,
details

This statement will pick all the errors during last 5 minutes and post them to webhook. If you are receiving LOTS of errors, you might want to fine tune this to be more precise and avoid flooding exceptions to the webhook. If you are receiving errors rarely (as you should!) this should work just fine.

For Actions you need to create new Action Group and configure it to use webhook URL, which we saw at Kilo Code. You can skip the Notifications part and just configure the Actions.

Testing the Solution

Now we should have everything set. Let’s test out the integration by causing an exception in our app. To demonstrate this behavior I created a simple bug into my order processing method. In this bug I always set the product to null, which causes ProductNotFoundException. This is clearly a bug as it breaks the whole ProcessOrderAsync method.

public Task ProcessOrderAsync(ProcessOrderRequest request)
{
    // Start tracking the operation
    using var operation = _telemetryClient.StartOperation<RequestTelemetry>("ProcessOrder");
    
    try
    {
        var product = _products.FirstOrDefault(p => p.Id == request.ProductId);
        
        // BUG: Force null reference exception
        product = null;
        
        if (product == null)
        {
            throw new ProductNotFoundException(request.ProductId);
        }

        // Process the order
        product.StockQuantity -= request.Quantity;

        // Track successful order
        _telemetryClient.TrackEvent("OrderProcessed", new Dictionary<string, string>
        {
            ["ProductId"] = request.ProductId.ToString(),
            ["Quantity"] = request.Quantity.ToString(),
            ["RemainingStock"] = product.StockQuantity.ToString()
        });

        operation.Telemetry.Success = true;
        _logger.LogInformation("Order processed successfully for product {ProductId}", request.ProductId);

        return Task.CompletedTask;
    }
    catch (Exception ex)
    {
        operation.Telemetry.Success = false;
        _telemetryClient.TrackException(ex);
        throw;
    }
}

After running the code we can verify from Application Insights alert page that it really detects the problem. Kilo Code has View Captured Request page (if you edit the webhook) which shows all the requests that came into webhook.

From session listing we can open the session run and view more details. Image below is from my first run and we can see that Agent has created a Pull Request in my repo.

Does it Really Work?

Yes it does. This is the Pull Request that the agent created. I also reproduced the same error again and verified that the Agent didn’t create PR from the same issue again.

Final Thoughts

It took me few hours to setup this whole thing. The only “code” that I had to write was that KQL query, which I created with help of Chat GPT. Over half of that few hours that I spent on this was used on configuring Application Insights and wondering why it does not send messages into webhook (I actually don’t know the reason it finally just started working…). Setting up these agentic workflows can be super easy with right tools.

The Cloud Agents are here. You don’t want to run these agentic workflows only on your machine. This bug fixing agent will work even though my personal computer is not powered on. It will fix issues and suggest pull requests when I am off the work. That is the true power of agentic workflows. It doesn’t restrict to hours when you are online.

Of course this solution is not perfect. I would maybe add database to keep track of handled exceptions and put Azure Function between Kilo Code and Application Insights. In Azure Function we could parse the error messages into more compact format, maybe add some extra context for Kilo Code Agent and log the agent usage.

One “guideline” or “thought” that I heard during this week was, that you need to stop coding your application. You need to start building agents which will do the coding for you.