Written in collaboration with @ShabazShaik and @gusmodena.
Introduction:
Azure Firewall is a cloud-native and intelligent network firewall security that provides the best of breed threat protection for your cloud workloads running in Azure. It’s a fully stateful firewall with built-in high availability and unrestricted cloud scalability. With any firewall solution, it is imperative to log the traffic that is being processed by the firewall for auditing and troubleshooting purposes. To achieve this goal, you have Diagnostic Logs that you can configure on the firewall which helps you with monitoring of all the traffic and metrics on the firewall.
Today we see many IT Teams concerned about having better visibility of all the traffic passing through their firewalls and identifying vulnerabilities to act quickly.
In this blog we will discuss in detail about the new Structured Firewall Logs feature which helps you with enhanced logging capabilities.
Structured Firewall Logs:
We are excited to share that on June 2022 Microsoft announced the preview of the Structured Firewall Logs for Azure Firewall, allowing customers to choose using Resource Specific Tables instead of existing AzureDiagnostic table.
The following diagnostic log categories are currently available in Azure Firewall:
These log categories use Azure diagnostics mode in which all data from any diagnostic setting will be collected in the AzureDiagnostics table. Whereas with this new feature of Structured Firewall logs, you’ll be able to choose to use Resource Specific tables instead of existing AzureDiagnostics table. Structured Firewall logs is required for Policy Analytics. This new method helps you with better log querying and is recommended since it’s:
In Resource specific mode, individual tables in the selected workspace are created for each category selected in the diagnostic setting. This new mode helps you with better log querying capabilities along with reduction in ingestion latencies and query times.
The following are the new Resource Specific Tables that are now available in the Diagnostic Setting. Each link will take you to the public docs, giving details of the schema of each table.
Configuring Structured Firewall Logs:
By default, the new resource specific tables are disabled. You need to run the following Azure PowerShell commands to enable Azure Firewall Structured Logs:
Connect-AzAccount
Select-AzSubscription -Subscription "subscription_id or subscription_name"
Register-AzProviderFeature -FeatureName "AFWEnableStructuredLogs" -ProviderNamespace "Microsoft.Network"
Register-AzResourceProvider -ProviderNamespace "Microsoft.Network"
To check the status of the AzResourceProvider registration, you can run the PowerShell command:
Get-AzProviderFeature -FeatureName "AFWEnableStructuredLogs" -ProviderNamespace "Microsoft.Network"
Note: It can take several minutes for this to take effect. Once the feature is completely registered, perform an update on your Azure Firewall for the change to take effect.
As you can see, these commands are run at the subscription level and will be applicable to all the firewalls deployed under the Subscription. Once you have enabled this feature on the Subscription, you need to go to the diagnostic setting of the respective firewall and choose the Destination Table as ‘Resource Specific’ and select all the required categories as shown below.
Now, you can query the logs in a more granular and efficient way using the resource specific tables. You can clearly see the difference between the logs with Azure Diagnostics Table and Resource Specific Tables in the images below.
Note: To disable this feature, you can run the following command:
Unregister-AzProviderFeature -FeatureName "AFWEnableStructuredLogs" -ProviderNamespace "Microsoft.Network"
Logs with AzureDiagnostics Table
When querying the AzureDiagnostic you need to specify what category you want to look for, as ‘AzureFirewallNetworkRuleLog’. Then all the important information as source/destination IPs and ports, protocol, rule name and action are located at the column ‘msg_s’.
So, if you want to split the data in different columns, you will have to parse it by using a more complex KQL query as below:
Parsing the data may take longer depending on the amount of data available.
Logs with Resource Specific Table
To query the new resource specific tables, you don’t need to parse any data as each one of the tables have its own schema and the data is structured following the same.
So, if you want to query the network logs, the only thing you need is to know the columns and what data you are looking for. For example, if you want to look for all the deny logs on the port TCP 389 you can use the following query:
AZFWNetworkRule
| where Action contains "Deny"
| where DestinationPort == "389"
| where Protocol contains "TCP"
You can also use similar query to look for logs within AZFWApplicationRule table. Below you will find a query to looking for all the logs where the action is Allow. So, that way you will get to know all the FQDNs accessed in a period of time.
AZFWApplicationRule
| where Action contains "Allow"
Or if you are looking for IDPS alerts, and you want to know what the source/destination IPs and what categories and severity are, you can easily query the table AZFWIdpsSignature.
AZFWIdpsSignature
| take 100
If you are not sure how to write queries using the Resource Specific tables, you can check the built-in queries available in the Azure Portal, under Logs > Queries.
Conclusion:
With the new Structured Firewall logs you will have more control over the logs available for Azure Firewall, by enabling only what you really need. The new schemas will help you running queries faster with no parsing required and you can also use RBAC to restrict access based on your security requirements.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.