How to Securely Handle Secrets with Azure Key Vault in Microsoft Fabric Notebooks

Proper secret management is one of the security fundamentals in software development. You should never hardcode credentials, secrets or tokens into your Spark Notebooks, because it creates a security vulnerability and adds extra complexity. Hardcoding secrets into Notebooks means that if you need to update any of the secrets you have to go through all the notebooks.

Azure Key Vault is a centralized secret management system, which helps you to store secrets and credentials. By using centralized system, you can easily change the value of those secrets without need of going through all the Notebooks.

Now that we have established the ground of why we would should Key Vault for secret management, let’s see how we can implement it in Fabric

Create Azure Key Vault

First we of course need an Azure Key Vault. You can create one from Azure Portal or use the AZ CLI. Just remember to enable the RBAC authorization, as we are going to use Managed Identity to give the permissions for Fabric.

az keyvault create --location westus2 --name MyKeyVault --resource-group MyResourceGroup --enable-rbac-authorization

If you are using existing Key Vault, you can verify that RBAC is enabled by opening the Access Policies page in Azure Portal. It should state that Access policies are not available as access control should happen in IAM page.

We want to use RBAC as Fabric is using managed identity to access Key Vault

Create Workspace Identity

Updated 13.11.2024: Fabric does not yet support Workspace identities with notebookutils, so the Notebook will use the Notebook owner permissions to access the KeyVault. You can skip the Workspace identity step currently, but in the future I think it will be supported, so you can do it already.

Now that we have Azure Key Vault up and running, we can create managed identity for our Fabric Workspace. You can do that by opening the Workspace settings page for the Workspace, that you have your notebooks in it (or do it for all the Workspaces that are going to need to use Key Vault).

Navigate into Workspace identity tab and click + Workspace identity. This will greate new App Registration into Azure AD, which we can use to manage the access control in Azure.

Note: You cannot create workspace identity for My Workspace!

You can find more about restrictions and details of Workspace identity from Microsoft documentation.

Create new Workspace identity to access the Azure resources safely

Grant Permissions to Azure Key Vault

Now that we have our identity, we can grant permission to access the Azure Key Vault at Azure Portal (For the future). Copy the identity ID from workspace identity page and navigate into Azure Key Vault access control page at Azure. Add new Role assignment and use Key Vault Secrets User role (you can use other roles if you want to, but this gives access to Key Vault secret values). At members tab click “Select members” and paste the ID into search field. You should be able to find the workspace with ID or by using the workspace name in search field. If you cannot find the identity, wait for few minutes and try again as Azure AD might have some delays in user syncing.

Use Key Vault from PySpark Notebook

Now that we have our Key Vault ready, Workspace identity ready and access granted, we can start using the Key Vault from our notebooks. Microsoft has kindly provided us very easy way to access the key vault with notebookutils library. Just call the credentials.getSecret method with key vault address (which you can find from Azure Portal for example) and secret name. Currently the notebookutil will use the Notebook owner credentials to access the KeyVault, so you need to have at least Key Vault Secrets Users role for the owner.

secret = notebookutils.credentials.getSecret('https://<name>.vault.azure.net','secretname')

Note that you cannot print out the secret values, because they are stored safely into Fabric during runtime, but you can use them directly without any extra hazzle. For example you can call REST API, which has Basic auth enabled by just passing the secrets directly into auth.

user = notebookutils.credentials.getSecret('https://<name>;.vault.azure.net/','username')
pwd = notebookutils.credentials.getSecret('https://<name>.vault.azure.net/','password')

taskreq = requests.get("https://someapi.com/api/GetUser", auth=(user, pwd))

Summary

Using Azure Key Vault to store your secrets safely and use them in Microsoft Fabric is a very easy way to improve your code security and make it easier to control usage of those secrets. You can change the values of secrets by just updating them in Key Vault. No need to worry about updating multiple notebooks anymore.

By using Microsoft Fabric Workspace identity we can hopefully in the future control the access to Key Vault. Notebookutils library has built-in methods to read secrets from Key Vault without need of third party libraries or storing any secrets inside notebooks.