I upgraded an Azure Function from .net 3.1, v3 to .net 6.0 v4 and my Event Hub trigger stopped working

I often say, “Let’s get that updated, it won’t be too hard”. However, there is usually something that pops up to make it more difficult. The difficulty leads many developers and teams to put off upgrading until something is forced (security issues) and then it takes days or even weeks to complete the task.

I’ve learned the less often a program is updated or longer you wait to update the more problems you run into. The more often you do the updates, then less problems you’ll have (your batch size of changes is smaller, there’ll be less to change. “if it hurts, do that more often” ~ DevOps). Here’s a great story from DOES 2021.

I’ve been through many .Net version updates, Nuget Package updates, npm package updates (Angular and others).

Upgrading our Azure Function from .Net 3.1 to .Net 6.0 and all the Nuget packages lead to a few bumps in the road and an opportunity to learn.

Here’s what happened, how I fixed it and a plan for upgrading production

Azure Function Version

After deploying the upgraded code from .Net 3.1 to .Net 6.0 to our dev version, the function couldn’t start. I needed to change the version of the function from 3 to 4.

I first saw the exception Could not load file or assembly ‘Microsoft.Extensions.Configuration.Abstractions, Version=5.0.0.0 in Azure Functions. StackOverflow pointed me to setting the version <AzureFunctionsVersion>v4</AzureFunctionsVersion>. After deploying that I still had problems. I did not see version 4 in the Portal UI, so I used the Azure CLI to set the version az functionapp config set --net-framework-version v6.0 -g -n funcap-MyCompany-prod --slot stage

I had to go to config > Runtime settings and select 4 as the runtime after running the cli command (I’m not sure if that will move with a slot deploy).

More about Azure Function Versions which included a guide to upgrade and a pre-upgrade validator.

Running locally

When I hit it with the tester I was getting an exception about EntityPath (Microsoft.Azure.WebJobs.Host: Error indexing method ‘MyFunction’. Microsoft.Azure.WebJobs.Host: Multiple properties named ‘ContentType’ found in type ‘EventData’.)

Solution: upgrade you cli tools: npm install -g azure-functions-core-tools@4 --unsafe-perm

from https://github.com/Azure/azure-sdk-for-net/issues/28850 which lead to https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local.

Event Hub Trigger stopped working

We had an Azure Event Hub Trigger for this function, after getting the function to start, I never got requests.

public Task Run([EventHubTrigger("local", Connection = "MyConn")] EventData @event, ILogger logger)

I eventually found that there was an error at runtime about the connection string for the EventHub (“The connection string could not be parsed;”). I realized that the Nuget Package went from version 4.1.1 to 5.1.1 and EntityPath can’t be a part of the connection string. I removed that then things were running.

I still didn’t get requests. I embarked on asking questions and learning and having someone find a log that it was getting put on the Event Hub. I really wish it was easier to capture messages off of Event Hub and see the contents of that message, that would have helped.

Solution: Since the EntityPath was removed and I had “local” as the Event Hub name hard-coded, but it should have been “job” no triggers were happening. I changed to a configurable Event Hub name

I changed the Event Hub Name to come from the settings (local.settings.json for local, Portal UI settings for Azure). Now local and each slot can point to the desired Event Hub “bucket”.

public Task Run([EventHubTrigger("%EventHubName%", Connection = "MyConn")] EventData @event, ILogger logger)

Live Metrics stopped working

Upgraded <PackageReference Include="Microsoft.ApplicationInsights" Version="2.12.2" /> to version 2.21.0

Get Debug output locally

I didn’t really solve this. I’m able to stream the output with Log Stream, but Live Metrics still doesn’t work. Maybe this answer would help, but I didn’t think packages were needed. Maybe my App Insights needs to be reconfigured?

This is nice query for App Insights logs to see that requests are still coming in.

requests
| summarize totalCount=sum(itemCount) by bin(timestamp, 15m)
| render timechart

Deployment With Slots to reduce downtime when we release to production

Now that I knew the upgrade would require some downtime, I investigated deployment slots.

If 2 slots are pointing at the same Event Hub Name and running at the same, you’ll be double processing. So I Leveraged slot sticky app settings to point one slot at a different Event Hub Name for testing (see above for making that possible).

We do several steps for a function slot deploy and swap.

1. Deploy the built code artifact to a storage blob
2. Deploy that code to the Azure Function stage slot
3. Swap the slots. Automatically swap for dev, Manual action swap for production.
   1. I chose to stop the slot after the swap

This is a Gitlab pipeline yml example where each job is ran in a Docker container.

# Templates
.Azure-Func-Release:
  image: ourdocker.reposiitory/ci/azure-docker
  stage: release
  script:
    - BASENAME="$(basename $PROJECT_FILE .csproj)"
    - ARTIFACT="$BASENAME-$CI_COMMIT_SHORT_SHA.zip"
    - cd "$(dirname -- $PROJECT_FILE)/output"
    - zip --recurse-paths "$ARTIFACT" *
    - az login --service-principal --username $azure_service_principal --password ${HOME}/.azure/gitlab-sp.pem --tenant $azure_tenant
    - az storage blob upload --account-name mySAAccount --container-name build-artifacts --file "$ARTIFACT" --name "$BASENAME/$ARTIFACT"
  artifacts:
    paths:
      - ./$(dirname -- $PROJECT_FILE)/output/*.zip

.Azure-Func-Deploy:
  image: ourdocker.reposiitory/ci/azure-docker
  stage: deploy
  script:
    - BASENAME="$(basename $PROJECT_FILE .csproj)"
    - ARTIFACT="$BASENAME-$CI_COMMIT_SHORT_SHA.zip"
    - OUTPUT="$(dirname -- "$PROJECT_FILE")/output"
    - az login --service-principal --username $azure_service_principal --password ${HOME}/.azure/gitlab-sp.pem --tenant $azure_tenant
    - az functionapp deployment source config-zip --resource-group="$FUNCTION_GROUP" --name="$FUNCTION_NAME" --slot="$FUNCTION_SLOT" --src="$OUTPUT/$ARTIFACT";

.Azure-Func-Deploy-Swap:
  image: ourdocker.reposiitory/ci/azure-docker
  stage: deploy
  script:
    - az login --service-principal --username $azure_service_principal --password ${HOME}/.azure/gitlab-sp.pem --tenant $azure_tenant
    - echo "Swapping for $FUNCTION_GROUP->$FUNCTION_NAME to slot production";
      az functionapp start --resource-group="$FUNCTION_GROUP" --name="$FUNCTION_NAME" --slot=stage;
      az functionapp deployment slot swap --resource-group "$FUNCTION_GROUP" --name "$FUNCTION_NAME" --slot stage --target-slot production --verbose --debug;
      az functionapp stop --resource-group="$FUNCTION_GROUP" --name="$FUNCTION_NAME" --slot=stage;

# Pipeline Jobs
MyFunction-build:
  extends: .Azure-Func-Build
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
  only:
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
      - tests/MyCompanyFunctions.MyFunction.Tests/**/*

MyFunction-release:
  extends: .Azure-Func-Release
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
  only:
    refs:
      - master
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
  dependencies:
      - MyFunction-build

MyFunction-deploy-ci:
  extends: .Azure-Func-Deploy
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
    FUNCTION_GROUP: rg-MyCompany
    FUNCTION_NAME: funcap-MyCompany-test
    FUNCTION_SLOT: stage
  only:
    refs:
      - master
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
  dependencies:
      - MyFunction-release

MyFunction-deploy-ci-swap:
  extends: .Azure-Func-Deploy-Swap
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
    FUNCTION_GROUP: rg-MyCompany
    FUNCTION_NAME: funcap-MyCompany-test
  only:
    refs:
      - master
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
  dependencies:
      - MyFunction-deploy-ci

MyFunction-deploy:
  extends: .Azure-Func-Deploy
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
    FUNCTION_GROUP: rg-MyCompany
    FUNCTION_NAME: funcap-MyCompany-prod
    FUNCTION_SLOT: stage
  only:
    refs:
      - master
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
  dependencies:
      - MyFunction-release

MyFunction-deploy-swap:
  extends: .Azure-Func-Deploy-Swap
  variables: 
    PROJECT_FILE: src/MyCompanyFunctions.MyFunction/MyCompanyFunctions.MyFunction.csproj
    FUNCTION_GROUP: rg-MyCompany
    FUNCTION_NAME: funcap-MyCompany-prod
  only:
    refs:
      - master
    changes:
      - src/MyCompanyFunctions.MyFunction/**/*
  when: manual
  dependencies:
      - MyFunction-deploy

One time steps after upgrade release

• I think I’ll still need to change from v3 to v4 after swapping the slots, but I’m hoping that the slot swap will handle that.

That’s all

After we finish testing, we’ll be ready for an almost 0 downtime deployment (this upgrade takes a few more steps) and 0 downtime deployment in the future with slots. As always, check your logs after swapping the slots to make sure your trigger is working after the update.

I hope this helps you reduce the time to upgrade your Function and you’ll upgrade more often and sooner.

.Net 7 Update

In February, 2023 we moved it to .Net 7 and it stopped working. We figured out that a publish from Visual Studio worked, but our Gitlab pipeline failed with this error. c# - Azure Functions in .NET 7 (isolated) published to Azure, 0 functions loaded - Stack Overflow.

You also need to use ‘dotnet-isolated’ Guide for running C# Azure Functions in an isolated worker process | Microsoft Learn and set your function runtime to .Net 7.

Later we learned that our EventHub triggered Function App needed to be always on or it stopped processing messages off of our hub.

VS Publish Output

2>WorkerExtensions -> C:\Users\me\AppData\Local\Temp\vvdslmsy.4bi\publishout\
2>Publishing C:\git\functions\src\Function1\obj\Release\net7.0\win-x64\PubTmp\Function1 - 20230228162702499.zip to https://-test.scm.azurewebsites.net/api/zipdeploy...

Once we found Create a C# function from the command line - Azure Functions | Microsoft Learn, we tried the cli func azure functionapp publish funcap-test and it started running. It was fast and easy with one command! Make sure you have the updated versions mentioned in the post.

The yml looks like this (with Azure-Func-Deploy and Azure-Func-Release changed from above)

.Azure-Func-Release:
  image: yourImage
  -- needs to have az cli and func cli and .net 7
  stage: release
  script:
    - cd $BUILD_OUTPUT_PATH
    - az login --service-principal --username $azure_service_principal --password ${HOME}/.azure/gitlab-sp.pem --tenant $azure_tenant
        echo "Deploying to $FUNCTION_GROUP->$FUNCTION_NAME";
        func azure functionapp publish $FUNCTION_NAME --slot stage

in local.settings.json

  "Values": {
    "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
    ... your values
  }

I could not find a pre-built azure functions tool image, so I fought through it and built my own.

# Contains az cli, azure function tools cli and dotnet-sdk-6.0
# this is 2.4 GB so it could be optimized in the future
# Created for ICCForwarder after the Azure function was updated to .Net 7

# FROM ubuntu:22.04
# sdk is  Debian GNU/Linux 11 (bullseye), 11, bullseye
FROM mcr.microsoft.com/dotnet/sdk:7.0
RUN apt update
# RUN apt update && apt install -y dotnet6
RUN apt install -y zip unzip
# https://stackoverflow.com/questions/65520596/deploy-azure-function-with-an-alpine-unhandled-error-event

# Install az cli and func ap cli
# .Net 7+ Azure Functions require isolated and the seperate func cli tools: 
# see https://learn.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-csharp?tabs=azure-cli%2Cisolated-process#deploy-the-function-project-to-azure
RUN apt-get install -y ca-certificates curl apt-transport-https lsb-release gnupg
RUN curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | tee /etc/apt/trusted.gpg.d/microsoft.gpg
RUN AZ_REPO=$(lsb_release -cs) && \
    echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | tee /etc/apt/sources.list.d/azure-cli.list
RUN apt install -y azure-cli

# for bullseye
RUN echo "deb [arch=amd64] https://packages.microsoft.com/debian/$(lsb_release -rs | cut -d'.' -f 1)/prod $(lsb_release -cs) main" > /etc/apt/sources.list.d/dotnetdev.list
## for Ubuntu: echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-$AZ_REPO-prod $AZ_REPO main" | tee /etc/apt/sources.list.d/dotnetdev.list
RUN apt-get update && apt-get install -y azure-functions-core-tools-4
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1

Your slot needs to be running or you’ll get

Getting site publishing info...
Creating archive for current directory...
Uploading 14.88 MB []Upload completed successfully.
Deployment completed successfully.
Syncing triggers...
Syncing triggers...
Syncing triggers...
Syncing triggers...
Syncing triggers...
Syncing triggers...
Error calling sync triggers (BadRequest). Request ID = '9772eb9b-ce1a-46dd-9399-3eeffb084216'.

I’ve asked for help on Github.