In Azure it is good practice (well I would say mandatory practice) to restrict database and Key Vault access by setting network restrictions for the Azure resource. Typically enterprises has their own virtual networks set and database/Key Vault access is restricted to those networks only. I recommend to disable the “Allow Azure services and resources to access this server” for SQL databases as it allows all Azure users to try to access your database. By disabling this feature we also disallow the Microsoft Fabric to access that Azure SQL database, which can be a problem.
Managed Private Endpoints
In February 2024 Microsoft announced public preview of Managed Private Endpoints feature. This feature allows Spark notebooks (yes, notebooks only) to access Azure resources by using the Azure Private Endpoint feature. This means that Fabric notebooks can access Azure SQL databases and Azure Key Vaults, which are restricted to virtual network access only.
Managed Private Endpoints support many data sources, including Azure Storage, Azure SQL Database, Azure Synapse Analytics, Azure Cosmos DB, Application gateway, Azure Key Vault, and more. Endpoints are supported for Fabric trial capacity and all Fabric F SKU capacities.
At least one private endpoint is required per resource type.
Endpoints are created under workspace. One workspace can have multiple endpoints for different Azure resources. All new endpoints needs to be approved from Azure Portal. Basically that means that after you have created the Managed private endpoint from Fabric you need to approve it at Networking > Private endpoint connections tab at Azure Portal.
When you are creating the endpoint, you can add custom approval message which is visible in Azure Portal. Endpoint is created with automatically generated name, that contains part of the name you gave it in Fabric. You cannot access the endpoint settings and I think that is the “managed” part in the endpoint name.
The create dialog for setting up the endpoint itself is simple, but there is one bit more complex field and that is “Resource identifier”. Easiest way to get value for this is to navigate into Azure Portal and copy Resource ID from Properties page. The value is usually in suitable format and the dialog accepts it wihtout need of manually editing the value. Resource Identifier also specifies the Target sub-resource type. It usually takes few minutes to fully setup the endpoint connection after it is approved.
Downside of Private Endpoints
As stated in this web page, you will lose the pre-warmed clusters when this feature is enabled. This can lead into longer spark session start-up times, but that is understandable as pre-warmed are hosted in shared network and you cannot use those with endpoints enabled.
Once the managed virtual network has been provisioned, the starter pools (default Compute option) for Spark are disabled, as these are pre-warmed clusters hosted in a shared virtual network. Spark jobs will run on custom pools created on-demand at the time of job submission within the dedicated managed virtual network of the workspace which could take up around 3 to 5 minutes to start sessions in Spark.
Cost of Private Endpoints
This thing is bit confusing. In general the Private Endpoint would cost something like 0,1€/h, so monthly fee is around 7€/month. However I haven’t seen any costs related to these Fabric managed private endpoints. Azure Portal cost analysis does not show any details about them. I don’t know if they are free to use, or if they are charged only when data is transferred. The traffic through managed endpoints is outbound (you access resources outside of Fabric). As I wasn’t able to find any cost details about this service, I will treat it as that it would be a normal Private Endpoint.
Conclusion
Managed Private Endpoints are simple and easy way to access network restricted Azure resources from Microsoft Fabric. Sadly they are currently supported only at Fabric Notebook resources, but I think this will improve over time. Pricing is still a baffle as I wasn’t able to find any costs from Azure Portal cost analysis page related to these endpoints. On a downside this feature disables pre-warmed spark pools, so it can increase the initial run time of notebook scripts.