# What's actually crossing your ExpressRoute? Traffic Collector, with IaC

## Introduction

ExpressRoute traffic visibility is a gap most Azure observability doesn't fill - As a platform engineer i'll be sharing my findings and observations setting up ExpressRoute Traffic Collector using infrastructure as code with the AzApi provider. Hopefully by the end of this, you will be familiar with what it does, if it makes sense to set it up in your platform and see a practical IaC example

## What challenges does it solve?

Traffic Collector makes sense to evaluate when your operational question is specifically about the hybrid boundary.

1.  *What traffic is actually crossing my ExpressRoute?*
    
2.  *Who are the top talkers?*
    
3.  *Which source/destination pairs are consuming the circuit?*
    

It solves the visibility gap between on-premises and Azure. Traffic Collector observes *sampled flows* at the Microsoft Enterprise Edge (MSEE) with a rate of 1:4096 and aggregates flows - so it is not packet capture and shouldn't be treated as a complete record of every connection. It handles the observation independently of how traffic is routed inside Azure. For comparison, in my previous writing I covered VNet flow logs - those primarily solve the challenge of *What communicates inside our Azure VNets*.

I would highly recommend having a look at [Jose Morenos article on ExpressRoute traffic visibility](https://blog.cloudtrooper.net/2024/06/25/expressroute-traffic-visibility-flow-logs-or-traffic-collector/) to gain a deeper understanding about the differences between VNet Flow Logs and ExpressRoute Traffic Collector.

## Overview of Traffic Collector

The Collector sits at the MSEE routers and provides flow-level visibility for traffic traversing an ExpressRoute circuit. It samples flow metadata such as source and destination, ports, protocol and traffic volumes for traffic crossing the ExpressRoute private *or* Microsoft peering. The collected records are aggregated, and can be exported to Log Analytics, Storage, or Event Hubs.

## Infrastructure as code setup - AzApi

The setup has three parts to ensure the creation of the Traffic Collector instance, collector policy and a diagnostic setting to send the logs to the desired destination - in my example, Log Analytics Workspace.

```hcl
// Creating the ExpressRoute Traffic Collector resource on the circuit
resource "azapi_resource" "er_traffic_collector" {
  type      = "Microsoft.NetworkFunction/azureTrafficCollectors@2022-11-01"
  name      = var.er_traffic_collector_name
  parent_id = var.parent_id
  location  = var.location
  tags      = var.tags
}

// Creating the collector policy for the ExpressRoute Traffic Collector and defining its ingestion and emission policies
resource "azapi_resource" "er_traffic_collector_policy" {
  type      = "Microsoft.NetworkFunction/azureTrafficCollectors/collectorPolicies@2022-11-01"
  name      = var.er_traffic_collector_policy_name
  parent_id = azapi_resource.er_traffic_collector.id
  location  = var.location

  depends_on = [azapi_resource.er_traffic_collector]

  body = {
    properties = {
      ingestionPolicy = {
        ingestionType = "IPFIX"
        ingestionSources = [
          {
            resourceId = var.er_circuit_id
            sourceType = "Resource"
          }
        ]
      }
      emissionPolicies = [
        {
          emissionType = "IPFIX"
          emissionDestinations = [
            {
              destinationType = "AzureMonitor"
            }
          ]
        }
      ]
    }
  }
}


// Sending the IPFIX data to Log Analytics Workspace
resource "azapi_resource" "er_traffic_collector_to_law" {
  type      = "Microsoft.Insights/diagnosticSettings@2021-05-01-preview"
  name      = "er-traffic-collector-to-law"
  parent_id = azapi_resource.er_traffic_collector.id

  depends_on = [azapi_resource.er_traffic_collector_policy]

  body = {
    properties = {
      workspaceId = var.log_analytics_workspace_id
      logs = [
        {
          category = "ExpressRouteCircuitIpfix"
          enabled  = true
        }
      ]
    }
  }
}
```

A few things worth noticing:

*   `depends_on` is needed to ensure the policy is created after the Traffic Collector instance.
    
*   Traffic Collector — `parent_id` = var.parent\_id (the RG ID for the ExpressRoute Circuit)
    
*   \`\`destinationType\` in the collector policy accepts AzureMonitor. If you need Event Hubs or Storage, that's handled through a separate diagnostic setting rather than through this body.
    
*   Verify that you aren't already sending the log category with diagnostic settings, If you are, the diagnostic settings code block isn't needed
    

# Gotchas

*   Traffic Collector is currently supported for ExpressRoute Circuits with 1 gbps or greater - with private of Microsoft peering. No SKU restrictions.
    
*   The ExpressRoute circuit, the Traffic Collector, and the Log Analytics workspace must all sit in the same geo-political region.
    
*   The Traffic Collector resource itself is accessed from the ExpressRoute circuits blade. Diagnostic settings created by IaC land on the circuit.
    
*   Data starts flowing into Log Analytics within minutes, though how quickly depends on traffic volume — the 1:4096 sampling means low-volume periods may take longer to surface first records.
    

This is part 2 on Azure network observability. That closes the planned series — more network observability posts will come as I run into things worth writing about.

## Referances

## References

*   [ExpressRoute traffic visibility: Flow Logs or Traffic Collector? — Jose Moreno (Cloudtrooper)](https://blog.cloudtrooper.net/2024/06/25/expressroute-traffic-visibility-flow-logs-or-traffic-collector/)
    
*   [Microsoft.NetworkFunction/azureTrafficCollectors — Terraform reference](https://learn.microsoft.com/en-us/azure/templates/microsoft.networkfunction/2022-05-01/azuretrafficcollectors?pivots=deployment-language-terraform)
    
*   [azapi provider — NetworkFunction 2022-11-01 type definitions (GitHub)](https://github.com/Azure/terraform-provider-azapi/blob/0cbdf9e405df995ecc2da8d4d1e739c13333c186/internal/azure/generated/networkfunction/microsoft.networkfunction/2022-11-01/types.json#L602)
    
*   [Microsoft Learn Configure Traffic Collector for ExpressRoute](https://learn.microsoft.com/en-us/azure/expressroute/how-to-configure-traffic-collector)