After upgrading an Azure Function, the Event Hub trigger stopped working

August 4, 2022    Azure

I upgraded an Azure Function from .net 3.1, v3 to .net 6.0 v4 and my Event Hub trigger stopped working

I often say, “Let’s get that updated, it won’t be too hard”. However, there is usually something that pops up to make it more difficult. The difficulty leads many developers and teams to put off upgrading until something is forced (security issues) and then it takes days or even weeks to complete the task.

I’ve learned the less often a program is updated or longer you wait to update the more problems you run into. The more often you do the updates, then less problems you’ll have (your batch size of changes is smaller, there’ll be less to change. “if it hurts, do that more often” ~ DevOps). Here’s a great story from DOES 2021 .

I’ve been through many .Net version updates, Nuget Package updates, npm package updates (Angular and others).

Upgrading our Azure Function from .Net 3.1 to .Net 6.0 and all the Nuget packages lead to a few bumps in the road and an opportunity to learn.

Here’s what happened, how I fixed it and a plan for upgrading production

Azure Function Version

After deploying the upgraded code from .Net 3.1 to .Net 6.0 to our dev version, the function couldn’t start. I needed to change the version of the function from 3 to 4.

I first saw the exception Could not load file or assembly ‘Microsoft.Extensions.Configuration.Abstractions, Version=5.0.0.0 in Azure Functions. StackOverflow pointed me to setting the version <AzureFunctionsVersion>v4</AzureFunctionsVersion>. After deploying that I still had problems. I did not see version 4 in the Portal UI, so I used the Azure CLI to set the version az functionapp config set --net-framework-version v6.0 -g -n funcap-MyCompany-prod --slot stage

portal ui does not show version 4

I had to go to config > Runtime settings and select 4 as the runtime after running the cli command (I’m not sure if that will move with a slot deploy).

More about Azure Function Versions which included a guide to upgrade and a pre-upgrade validator.

Running locally

When I hit it with the tester I was getting an exception about EntityPath (Microsoft.Azure.WebJobs.Host: Error indexing method ‘MyFunction’. Microsoft.Azure.WebJobs.Host: Multiple properties named ‘ContentType’ found in type ‘EventData’.)

Solution: upgrade you cli tools: npm install -g [email protected] --unsafe-perm

from https://github.com/Azure/azure-sdk-for-net/issues/28850 which lead to https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local .

Event Hub Trigger stopped working

We had an Azure Event Hub Trigger for this function, after getting the function to start, I never got requests.

public Task Run([EventHubTrigger("local", Connection = "MyConn")] EventData @event, ILogger logger)

I eventually found that there was an error at runtime about the connection string for the EventHub (“The connection string could not be parsed;"). I realized that the Nuget Package went from version 4.1.1 to 5.1.1 and EntityPath can’t be a part of the connection string. I removed that then things were running.

I still didn’t get requests. I embarked on asking questions and learning and having someone find a log that it was getting put on the Event Hub. I really wish it was easier to capture messages off of Event Hub and see the contents of that message, that would have helped.

Solution: Since the EntityPath was removed and I had “local” as the Event Hub name hard-coded, but it should have been “job” no triggers were happening. I changed to a configurable Event Hub name

I changed the Event Hub Name to come from the settings (local.settings.json for local, Portal UI settings for Azure). Now local and each slot can point to the desired Event Hub “bucket”.

public Task Run([EventHubTrigger("%EventHubName%", Connection = "MyConn")] EventData @event, ILogger logger)

Live Metrics stopped working

Upgraded <PackageReference Include="Microsoft.ApplicationInsights" Version="2.12.2" /> to version 2.21.0

Get Debug output locally

I didn’t really solve this. I’m able to stream the output with Log Stream , but Live Metrics still doesn’t work. Maybe this answer would help, but I didn’t think packages were needed. Maybe my App Insights needs to be reconfigured?

This is nice query for App Insights logs to see that requests are still coming in.

requests
| summarize totalCount=sum(itemCount) by bin(timestamp, 15m)
| render timechart

Deployment With Slots to reduce downtime when we release to production

Now that I knew the upgrade would require some downtime, I investigated deployment slots.

If 2 slots are pointing at the same Event Hub Name and running at the same, you’ll be double processing. So I Leveraged slot sticky app settings to point one slot at a different Event Hub Name for testing (see above for making that possible).

We do several steps for a function slot deploy and swap.

  1. Deploy the built code artifact to a storage blob
  2. Deploy that code to the Azure Function stage slot
  3. Swap the slots. Automatically swap for dev, Manual action swap for production.
  4. I chose to stop the slot after the swap

This is a Gitlab pipeline yml example where each job is ran in a Docker container.


# Templates
.Azure-Func-Release:
  image: ourdocker.reposiitory/ci/azure-docker
  stage: release
  script:
    - BASENAME="$(basename $PROJECT_FILE .csproj)"
    - ARTIFACT="$BASENAME-$CI_COMMIT_SHORT_SHA.zip"
    - cd "$(dirname -- $PROJECT_FILE)/output"
    - zip --recurse-paths "$ARTIFACT" *
    - az login --service-principal --username $azure_service_principal --password ${HOME}/.azure/gitlab-sp.pem --tenant $azure_tenant
    - az storage blob upload --account-name mySAAccount --container-name build-artifacts --file "$ARTIFACT" --name "$BASENAME/$ARTIFACT"
  artifacts:
    paths:
      - ./$(dirname -- $PROJECT_FILE)/output/*.zip

.Azure-Func-Deploy:
  image: ourdocker.reposiitory/ci/azure-docker
  stage: deploy
  script:
    - BASENAME="$(basename $PROJECT_FILE .csproj)"
    - ARTIFACT="$BASENAME-$CI_COMMIT_SHORT_SHA.zip"
    - OUTPUT="$(dirname -- "$PROJECT_FILE")/output"
    - az login --service-principal --username $azure_service_principal --password ${HOME}/.azure/gitlab-sp.pem --tenant $azure_tenant
    - az functionapp deployment source config-zip --resource-group="$FUNCTION_GROUP" --name="$FUNCTION_NAME" --slot="$FUNCTION_SLOT" --src="$OUTPUT/$ARTIFACT";

.Azure-Func-Deploy-Swap:
  image: ourdocker.reposiitory/ci/azure-docker
  stage: deploy
  script:
    - az login --service-principal --username $azure_service_principal --password ${HOME}/.azure/gitlab-sp.pem --tenant $azure_tenant
    - echo "Swapping for $FUNCTION_GROUP->$FUNCTION_NAME to slot production";
      az functionapp start --resource-group="$FUNCTION_GROUP" --name="$FUNCTION_NAME" --slot=stage;
      az functionapp deployment slot swap --resource-group "$FUNCTION_GROUP" --name "$FUNCTION_NAME" --slot stage --target-slot production --verbose --debug;
      az functionapp stop --resource-group="$FUNCTION_GROUP" --name="$FUNCTION_NAME" --slot=stage;

# Pipeline Jobs
MyFunction-build:
  extends: .Azure-Func-Build
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
  only:
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
      - tests/MyCompanyFunctions.MyFunction.Tests/**/*

MyFunction-release:
  extends: .Azure-Func-Release
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
  only:
    refs:
      - master
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
  dependencies:
      - MyFunction-build

MyFunction-deploy-ci:
  extends: .Azure-Func-Deploy
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
    FUNCTION_GROUP: rg-MyCompany
    FUNCTION_NAME: funcap-MyCompany-test
    FUNCTION_SLOT: stage
  only:
    refs:
      - master
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
  dependencies:
      - MyFunction-release

MyFunction-deploy-ci-swap:
  extends: .Azure-Func-Deploy-Swap
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
    FUNCTION_GROUP: rg-MyCompany
    FUNCTION_NAME: funcap-MyCompany-test
  only:
    refs:
      - master
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
  dependencies:
      - MyFunction-deploy-ci

MyFunction-deploy:
  extends: .Azure-Func-Deploy
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
    FUNCTION_GROUP: rg-MyCompany
    FUNCTION_NAME: funcap-MyCompany-prod
    FUNCTION_SLOT: stage
  only:
    refs:
      - master
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
  dependencies:
      - MyFunction-release

MyFunction-deploy-swap:
  extends: .Azure-Func-Deploy-Swap
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
    FUNCTION_GROUP: rg-MyCompany
    FUNCTION_NAME: funcap-MyCompany-prod
  only:
    refs:
      - master
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
  when: manual
  dependencies:
      - MyFunction-deploy

One time steps after upgrade release

  • I think I’ll still need to change from v3 to v4 after swapping the slots, but I’m hoping that the slot swap will handle that.

That’s all

After we finish testing, we’ll be ready for an almost 0 downtime deployment (this upgrade takes a few more steps) and 0 downtime deployment in the future with slots. As always, check your logs after swapping the slots to make sure your trigger is working after the update.

I hope this helps you reduce the time to upgrade your Function and you’ll upgrade more often and sooner.



Watch the Story for Good News

Please consider using Brave and adding me to your BAT payment ledger. Then you won't have to see ads!

Use Brave

Also check out my Resources Page for referrals that would help me.


Swan Bitcoin referral image
Use Swan Bitcoin to onramp with low fees and automatic daily cost averaging and get $10 in BTC when you sign up.