.Net Parallel.ForEachAsync for Http calls

October 12, 2023    Development .Net

.Net Parallel.ForEachAsync for Http calls

I had to make multiple HTTP calls at the same time, but limit the how many are at the same time. It would have been better to change the call to batch it into one or less, but I didn’t have that option.

As normal, we had a discussion at Omnitech during lunch.

We found this SO question and answer .

“Is he saying ParallelForEachAsync is ok for IO bound? but not Parallel.ForEach?” ~ me

“I believe so yes. Ultimately, the Parallel class has been associated with CPU bound work until .NET 6 when the Parallel.ForEachAsync was introduced. I think it’s confusing because the Task Parallel Library stuff is typically the go to for IO and network bound requests. The gut reaction is that Parallel is for CPU-bound work, but I think this is the one exception….” ~ John

Scott Hanselman also has a blog post on it from 2021 .

Here’s some code

 var products = new List<Product>();
 var productsInScope = await _repository.GetAllProductsInScope<Vertex>(scopeId);

 // too many requests overwhelms the pool size, limit to 5 at a time
 await Parallel.ForEachAsync(
     new ParallelOptions { MaxDegreeOfParallelism = 5, CancellationToken = cancellationToken  },
     async (product, ct) =>
            // make your extra api calls to get nested data
         catch (Exception ex)
             _logger.LogError(ex, "product id: {productId}", product.Id);
 return products;

Watch the Story for Good News
I gladly accept BTC Lightning Network tips at [email protected]

Please consider using Brave and adding me to your BAT payment ledger. Then you won't have to see ads! (when I get to $100 in Google Ads for a payout, I pledge to turn off ads)

Use Brave

Also check out my Resources Page for referrals that would help me.

Swan logo
Use Swan Bitcoin to onramp with low fees and automatic daily cost averaging and get $10 in BTC when you sign up.