c# - .net Core Parallel.ForEach issues

Question

I've switched over to .net Core for some projects and am now having a problem with Parallel.ForEach. In the past I often had a List of id values which I then would use to make web requests in order to get the full data. It would look something like this:

Parallel.ForEach(myList, l =>
{
    // make web request using l.id 
    // process the data somehow
});

Well, in .net Core the web requests must all be tagged await which means the Parallel.ForEach action must be tagged with async. But, tagging a Parallel.ForEach action as async means we have a void async method which causes issues. In my particular case that means the response returns to the application before all of the web requests in the Parallel loop are completed which is both awkward and causes errors.

Question: What are the alternatives to using Parallel.ForEach here?

One possible solution I found was to wrap the Parallel loop inside of a Task and await the task:

await Task.Run(() => Parallel.ForEach(myList, l =>
{
    // stuff here
}));

(found here:Parallel.ForEach vs Task.Run and Task.WhenAll)

But, that isn't working for me. When I use that I still end up returning to the application before the loop is completed.

Another option:

var tasks = new List<Task>();
foreach (var l in myList)
{
    tasks.Add(Task.Run(async () =>
    {
         // stuff here
    }));
}
await Task.WhenAll(tasks);

This appears to work, but is that the only option? It seems that the new .net Core has rendered Parallel.ForEach virtually useless (at least when it comes to nested web calls).

Any assistance/advice is appreciated.

Answer 1

Why Parallel.ForEach is not good for this task is explained in comments: it's designed for CPU-bound (CPU-intensive) tasks. If you use it for IO-bound operations (like making web requests) - you will waste thread pool thread blocked while waiting for response, for nothing good. It's possible to use it still, but it's not best for this scenario.

What you need is to use asynchronous web request methods (like HttpWebRequest.GetResponseAsync), but here comes another problem - you don't want to execute all your web requests at once (like another answer suggests). There may be thousands urls (ids) in your list. So you can use thread synchronization constructs designed for that, for example Semaphore. Semaphore is like queue - it allows X threads to pass, and the rest should wait until one of busy threads will finish it's work (a bit simplified description). Here is an example:

static async Task ProcessUrls(string[] urls) {
    var tasks = new List<Task>();
    // semaphore, allow to run 10 tasks in parallel
    using (var semaphore = new SemaphoreSlim(10)) {
        foreach (var url in urls) {
            // await here until there is a room for this task
            await semaphore.WaitAsync();
            tasks.Add(MakeRequest(semaphore, url));
        }
        // await for the rest of tasks to complete
        await Task.WhenAll(tasks);
    }
}

private static async Task MakeRequest(SemaphoreSlim semaphore, string url) {
    try {
        var request = (HttpWebRequest) WebRequest.Create(url);

        using (var response = await request.GetResponseAsync().ConfigureAwait(false)) {
            // do something with response    
        }
    }
    catch (Exception ex) {
        // do something
    }
    finally {
        // don't forget to release
        semaphore.Release();
    }
}