All communication on Azure is over the public Internet by default. This leaves the infrastructure vulnerable to attacks as there is a much bigger attack window. Hence, your network in Azure needs to be secured by denying access over the public Internet and locking it down in a Virtual Network. Azure’s virtual network makes communication secure.
Virtual Network additionally allows:
- Communication with on-premises resources
- Filtering of Network traffic
All resources in a VNet can communicate outbound to the internet, by default by assigning a public IP address. To secure communication, the access needs to be locked by extending private address space using Service Endpoints. Service Endpoints ensure security by tying down communication only to that particular VNet.
When resources are in a VNet, they can be accessed in the following ways:
- Private Service Endpoint
- VNet Peering – You can connect virtual networks, enabling resources in either virtual network to communicate with each other. A low-latency, a high-bandwidth connection is established between resources in different virtual networks.
Private Service Endpoint
Private Service Endpoint provides secure and direct connectivity to Azure services over an optimized route over the backbone network in Azure . In simple terms, it establishes a Private link between 2 resources in a VNET. A Private Service Endpoint is deployed to a subnet within a VNet. It has several benefits:
- Optimal Routing – Traffic never goes to the public internet and stays on the Azure backbone.
- Less Organisational Overhead – Provides secure and direct connectivity to Azure services over an optimized route over the Azure backbone network.
- Protection Against Data Leakage – Endpoint only exposes an instance of a PaaS resource. This ensures that connection is only to a specific resource instead of the entire service.
Establishing Communication between Storage Account and Databricks
- Databricks needs 2 dedicated subnets:
- Public – allows communication with the Azure Databricks control pane
- Private – allows only intra-cluster communication.
- Create Private Endpoint on Storage Account Blob Service.
- Add the endpoint to Private DNS.
- Deny Public Access of Storage Account
- Now the data-lake is accessible from that endpoint. To verify, perform the Unix command of nslookup on the storage account from Databricks. It should resolve to its Private IP address.
- Observation – VNet properly registers to Databricks only during resource creation
Establishing Secure Communication between Azure Data Factory and SQL Database
- Create Private Endpoints of both ADF and SQL Server
- Create a Virtual Network-enabled Integration Runtime for SQL
- ADF requests a managed Private Endpoint from SQL Server that needs to be approved from the Server
- Reflects in 1-2 mins in ADF
In short, the following steps can be taken to secure your network in Azure and the communication between a pair of resources in a VNet via a Private Link:
- Create VNet and Subnets, Private DNS
- Add Resources to VNet
- Establish Private Endpoint connections between resources
- Configure Private DNS With Connection
- Disable Public Access
- Sharepoint Integration: How to Share and Ingest Data Automatically into a Data Platform - May 27, 2022
- How to Secure your Network in Azure - November 4, 2021