Access Real-Time Data from LLM using Semantic Kernel

LLMs have been around for years now, and we have been using them for data that doesn’t change frequently. We don’t ask them for current data, like today’s weather or the balance in our bank account. Instead, we ask evergreen questions, such as who the members of the band ABBA are or whether we can eat ice cream in space (the answer is yes!).

However, since GPT-4, we have gained the ability to run our own functions, which are invoked by the large language model—and this is just great! In June 2023, GPT-4 introduced a feature called ‘function calling,’ which allows us to define functions that the LLM can call if it needs data that our functions can provide. It can even pass parameters into these functions!

Function calling in GPT models, like the one introduced in GPT-4 in June 2023, is implemented through a process that allows the model to recognize when to invoke specific functions during a conversation. For example:

User Input: “What’s the weather in New York?”
Model Output: The model identifies the need to call a weather API and generates a function call, getWeather(location="New York").
Function Execution: The app runs the getWeather function, retrieves the weather data, and passes it back.
Final Output: The model then responds with, “The weather in New York is currently 75°F with clear skies.”

The other models (than just GPT) supports this same kind of architecture. For example Anthropic’s Claude has this ability and so does the LLaMA from Meta.

Semantic Kernel

Using the function calling feature can be a bit rough to implement without any help, but luckily the Microsoft Semantic Kernel has support for this. Semantic Kernel abstracts away the need of defining functions for models. You just create normal C# function, decorate it with few attributes and link it into Semantic Kernel. That’s it.

Sample Code

In this sample code, I’m using the excellent train API available at rata.digitraffic.fi. Rata.digitraffic.fi has an API that returns arriving and departing trains from a given station. For example, I can query the trains from Helsinki Central Station by calling the URL https://rata.digitraffic.fi/api/v1/live-trains?station=HKI. Now, we can attach this API feature to our GPT-4o model and see how it can make calls when we ask about departing trains.

This sample is done with a .NET 8.0 Console App project.

First, we need to define some base classes: Invoke the live-trains API with a tool like Postman or Bruno and copy the response (you don’t need to worry about headers or authentication).

Then create a new class and use the brilliant Paste JSON As Classes feature from Visual Studio. You can do this if you have any JSON data in your clipboard. This feature will create a set of JSON classes, which describes the given JSON data. You can remove the RootObject class as we don’t need it in this sample (the API returns an array of trains).

Semantic Kernel

After creating the train classes, we can wire some Semantic Kernel code into our Program.cs file. In this sample I’m using a Semantic Kernel plugin feature and adding a plugin called Trains into the SK pipeline. I will also use ChatCompletionService to test the code.

Notice, that you have to set the ToolCallBehavior into AutoInvokeKernelFunctions to support the function calling. Default of this is null, which means that all tool calling is disabled => No Bueno for us.

var builder = Kernel.CreateBuilder();
builder.AddOpenAIChatCompletion("gpt-4o-mini", openAIApiKey, serviceId: "openai"); // I have variable called openAIApiKey defined above this....
// Add Train plugin
builder.Plugins.AddFromType<TrainPlugin>("Trains");

var kernel = builder.Build();

var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

// Create a history store the conversation
var history = new ChatHistory();
history.AddUserMessage("What trains are leaving from station HKI?");

// Enable planning
OpenAIPromptExecutionSettings openAIPromptExecutionSettings = new()
{
    ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
};

// Get the response from the AI
var result = await chatCompletionService.GetChatMessageContentAsync(
   history,
   executionSettings: openAIPromptExecutionSettings,
   kernel: kernel);

// Print the results
Console.WriteLine("Response to what trains are leaving from station HKI? > " + result);

// Add the message from the agent to the chat history
history.AddAssistantMessage(result.Items.First().ToString() ?? "");

Plugin and Function implementation

Finally we need to implement the TrainPlugin class, which contains the beef in this case. Create new class called TrainPlugin with following implementation:

 internal class TrainPlugin
 {
     private readonly RestClient _client;

     public TrainPlugin()
     {
         _client = new RestClient();
     }

     [KernelFunction("get_today_trains")]
     [Description("Get trains that are leaving or arriving today at the given station")]
     [return: Description("An array of departuring and arriving trains")]
     public async Task<List<Train>> GetTrainsAsync([Description("Station name")] string stationName)
     {
         // Make request to the Digitraffic API and get trains in json 
         var request = new RestRequest($"https://rata.digitraffic.fi/api/v1/live-trains?station={stationName}", Method.Get);
         request.AddHeader("Accept", "application/json");

         var response = await _client.ExecuteAsync(request);

         if (!response.IsSuccessStatusCode)
         {
             throw new Exception($"Failed to get trains from the Digitraffic API. Status code: {response.StatusCode}");
         }

         var content = response.Content ?? string.Empty;
         var trains = JsonConvert.DeserializeObject<List<Train>>(content);

         return trains ?? [];
     }
 }

Ok lets break this code into pieces. First things first. I am using two libraries here: RestSharp (which is great client library for JSON API’s) and NewtonSoft.JSON. I could do the serialization with System.Text.JSON library, but this is just old habit that I have.

GetTrainAsync is the function, that will be fed to our LLM when we run this app. Function has KernelFunction attribute, which defines what functions are included as KernelFunctions. The Description of course describes the situation where this function can be called and return description does the same thing for return value. I have also one parameter here called Station name, which is extracted from the user input.

The function implementation details are quite dull here. I’m just invoking API call with RestClient and returning the result. One little neat trick is, that in C# 12 you can use the [] -syntax to return empty collections like lists (not just arrays).

Result

And here is what the final result looks like:

You can attach debugger into GetTrainsAsync method so verify, that it is really called. You can also change the station into something like PSL to verify, that it returns data for specified station.

Summary

Function Calling feature enables developers to attach real time data into LLM solutions. You can implement database queries, API calls or simple mathematical operations to improve the model accuracy. I think this is very useful and maybe bit underrated feature in large language models like GPT-4o.