Home / Forums / Streaming Large Datasets in Real-Time with IAsyncEnumerable<T> & yield return

Tutorial

Streaming Large Datasets in Real-Time with IAsyncEnumerable<T> & yield return

LinQ_Lover

C# Developer

MEMBER

Rep: 313

Join Date: Jul 2020

Posts: 8

Thanks: 27

2y ago · Oct 14, 2023 4:30 PM

Before C# 8, returning multiple items asynchronously required fetching the entire list into memory first (

CODE

Task<List<T>>

).

With
CODE
IAsyncEnumerable<T>
, data streams item-by-item as it arrives:

CSHARP

using System.Runtime.CompilerServices;

public async IAsyncEnumerable<SensorData> StreamSensorReadingsAsync([EnumeratorCancellation] CancellationToken ct = default)
{
    while (!ct.IsCancellationRequested)
    {
        SensorData data = await ReadHardwareSensorAsync(ct);
        yield return data; // Streams to consumer immediately!
        
        await Task.Delay(100, ct); // 10Hz tick
    }
}

// Consuming with await foreach
await foreach (SensorData reading in StreamSensorReadingsAsync(cancellationToken))
{
    Console.WriteLine($"Temperature: {reading.Temp}°C");
}

The

CODE

[EnumeratorCancellation]

attribute ensures the consumer's

CODE

WithCancellation(ct)

token propagates into the async generator method properly!

LinQ_Lover · C# Developer
Clean code, fluent LINQ expressions, and expressive C# patte...

The following users thanked LinQ_Lover for this post:

AsyncMaster

Concurrency Geek

MEMBER

Rep: 72

Join Date: Jul 2020

Posts: 13

Thanks: 83

2y ago · Oct 14, 2023 8:02 PM

CODE

await foreach

with

CODE

IAsyncEnumerable

is great for database cursors (like EF Core

CODE

AsAsyncEnumerable()

). It streams rows directly from the TDS socket without buffering 100,000 entities in RAM!

AsyncMaster · Concurrency Geek
Task Parallel Library (TPL), async/await internals, and lock...

DevDan

.NET Core & Cloud

MEMBER

Rep: 324

Join Date: Feb 2021

Posts: 16

Thanks: 65

2y ago · Oct 15, 2023 4:02 AM

Also check out

CODE

System.Linq.Async

NuGet package. It adds all the standard LINQ operators (

CODE

Where

CODE

Select

CODE

Take

) to

CODE

IAsyncEnumerable<T>

DevDan · .NET Core & Cloud
Writing clean C# code and microservices since .NET Core 2.1....

Destination Subforum

Thread Prefix

Thread Title

Post Content (BBCode Enabled)

File Attachment

UnreliableCode Community

Streaming Large Datasets in Real-Time with IAsyncEnumerable<T> & yield return

Create Forum Thread

Add Reputation