Azure Functions: ServiceBusTrigger using Managed Identity doesn't wake up when running on consumption plan

Setup:

- .NET 6.0
- In-process/isolated v4 function apps
- Consumption plan
- Managed Identity

Background

Previously, I wrote a blog post "Azure Function secretless access to Azure Service Bus using Managed Identity", which describes how to use ServiceBusTriggers in an Azure function by using Managed Identity instead of a connection string containing secrets.

But something was not right

Life was good, until I discovered that some messages were not picked up. When running on a consumption plan, the functions will go idle if there are no activity in a given timeframe. But, new messages in the queue would not reactivate the function.

After going idle, the function would not pick up new messages, regardless of using queues or topics.

The only way I could get the Azure function to wake up was to navigate to the function app in the portal, which is not a feasible solution.

Investigating

As a seasoned developer, I googled the problem, until stumbling upon this GitHub issue. (There are a lot of similar issues, but this one is both consumption plan and for running isolated functions).

By reading this issue I learned about the scale controller, which is the mechanism responsible for reactivating functions after going idle.

Scale controller logging can be enabled by adding the following app setting for the function:

SCALE_CONTROLLER_LOGGING_ENABLED=AppInsights:Verbose

Then, the following query can be used in Application Insights to get the scale controller logs:

traces
| where customDimensions.Category == "ScaleControllerLogs"

When enabling scale controller logging, I got the following log entries:
[ManagedIdentity] SystemAssigned Managed Identity not found, but the chosen connection method requires one.

I was using User-assigned Managed Identity, not System-assigned, and the scale controller did not understand this.
To enable User-assigned Managed Identity for the scale controller I needed two additions to the app settings along with the fullyQualifiedNamespace:

Now, the scale controller logged:
[ManagedIdentity] Created NamespaceManager with ManagedIdentity
[ManagedIdentity] Created QueueClient with ManagedIdentity

This seems legit, but the problem still existed. The messages received when the function was idle would not be processed.

The missing part

The user boylec suggested adding the role Azure Service Bus Data Owner to the Managed Identity. The hypothesis was that the scale controller runs under the Managed Identity and needs visibility into the queue length to determine if there are any messages present. This requires the Data Owner role. NB! Roles might take some time to propagate.

This seemingly solved my problem. Messages started getting processed even if the function had gone idle. I had to add the Data Owner role to the Service Bus namespace, not the queue.

But, the cold start could take a while, and by a while I mean everything from 30 seconds to to 5 minutes. I think that's an issue with the framework. The GitHub site for Azure Functions .NET Worker states one single known problem:

Optimizations are not all in place in the consumption plan and you may experience longer cold starts.

Hopefully this will be solved in .NET7, and also Microsoft should make their documentation clearer. The Service Bus extensions docs does not mention scale controller or required roles at all.

TDLR

To enable Managed Identity to work for an Azure Function ServiceBusTrigger on a consumption plan:

• Add the following roles to the Managed Identity:
  Azure Service Bus Data Owner for the Service Bus namespace.
  Azure Service Bus Data Receiver for the queue/topic to read from.
  
• Add the following connection string to use Managed Identity without secrets:
  ServiceBusConnection__fullyQualifiedNamespace (ServiceBus namespace)
  
• If using User-assigned Managed Identity, add the following settings:
  ServiceBusConnection__credential ("managedIdentity")
  ServiceBusConnection__clientId (ClientId for the User-assigned Managed Identity)