Use managed identity to store data in storage account queue from API Management

In a previous post, I described how to create a SharedKey which could be used to drop data directly into a storage account queue from API management. Although that trick works fine, you still needed to have the storage account access key. Of course, this one is stored in a keyvault and referenced via a named value, but still something to maintain and could possible leak (it is visible in the trace though).

A better alternative is to use a managed identity, which works pretty easy. So let’s adjust our sample to use this instead.

Assuming you have an API Management instead rolled out with a storage account, you will need to add the below policy:

<policies>
    <inbound>
        <base />

        <set-variable name="APIVersion" value="2017-11-09" />

        <set-header name="x-ms-version" exists-action="override">
            <value>@( context.Variables.GetValueOrDefault<string>("APIVersion") )</value>
        </set-header>

        <authentication-managed-identity resource="https://storage.azure.com/" />

        <set-header name="content-type" exists-action="override">
            <value>application/xml</value>
        </set-header>
        
        <set-backend-service base-url="https://.queue.core.windows.net/" />
        <set-body>@{ 
            JObject inBody = context.Request.Body.As<JObject>(); 

            string base64Body = Convert.ToBase64String(Encoding.UTF8.GetBytes(inBody.ToString()));
            string content = "{ \"transaction\": \""+ base64Body + "\", \"version\": \"" + context.Api.Version + "\", \"subscription\": \"" + context.Subscription.Id +"\"}";
            string contentBase64 = Convert.ToBase64String(Encoding.UTF8.GetBytes(content));
            return  "<QueueMessage><MessageText>" + contentBase64 + "</MessageText></QueueMessage>"; 

          }</set-body>

        <rewrite-uri template="/messages" copy-unmatched-params="true" />

        <!--  Don't expose APIM subscription key to the backend. -->
        <set-header name="ocp-apim-subscription-key" exists-action="delete" />

        <!--  Don't expose api-version query string to the backend. -->
        <set-query-parameter name="api-version" exists-action="delete" />
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
        <choose>
        <when condition="@(context.Response.StatusCode == 201)">
        <return-response>
            <set-status code="202" reason="Transaction queued for processing" />
            <set-header name="Content-Type" exists-action="override">
              <value>application/text</value>
            </set-header>
            <set-body template="none">@{
              return context.Response.Body.As<XDocument>()
              .Element("QueueMessagesList").Element("QueueMessage").Element("MessageId")    
              .Value.ToString();
                }
            </set-body>       
            </return-response>
        </when>
        </choose>
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

As you can see, I removed the authentication header generation code in favor of this line:

<authentication-managed-identity resource="https://storage.azure.com/" />

The policy will make sure we get a token so API Management can access resources (in this case, storage). Which storage does not matter; the IAM of Azure will handle if the identity of API Management can indeed access the storage account. I do need to specify the API version of the call we make to the backend for it to work. If you omit this, it falls back to the earliest version, and this fails. You also need this to do the trick in the outbound section. Here I retrieve the message-id generated by the storage queue and return it to the caller.

Make also sure that you base64 encode your message before it enters the queue. I even double encode it as I envelop the contents with some other properties.

The API Management instance can only access the storage account when it has permission to do so. So you will need to provide it access, with for example, the below Bicep. The apiManagementPrincipalId is the guid you can find on the Identity page of API Management. The storageAccountName is the name of the storage account.

// Assign Storage Queue Data Owner for the API management identity
module storageQueueContributorApiManagementRoleAssignment './modules/storage-account-role-assignment.bicep' = {
  name: 'storageQueueContributorApiManagementRoleAssignment'
  scope: rg
  params: {
    principalId: apiManagementPrincipalId
    storageAccountName: storageAccountName
    roleDefinitionId: '/subscriptions/${subscription().subscriptionId}/providers/Microsoft.Authorization/roleDefinitions/974c5e8b-45b9-4653-ba55-5f855dd0fb88'
  }
  dependsOn:[    
    storageaccount
  ]
}

Where the storage-account-role-assignment.bicep module contains the actual logic to link the given role to the resource:

param principalId string
param roleDefinitionId string
param storageAccountName string

resource resource 'Microsoft.Storage/storageAccounts@2019-04-01' existing = {
  name: storageAccountName
}

resource roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid(resource.id, principalId, roleDefinitionId)
  scope: resource
  properties: {
    roleDefinitionId: roleDefinitionId
    principalId: principalId    
  }
}

Be aware that this gives the API Management instance permission to perform the given operations on the storage account. If you have other APIs running on the same API Management instance, then they can do the same, so you will need to trust these endpoints to behave correctly.