I have a chain of TPL Dataflow blocks and would like to observe progress somewhere inside the system.
I am aware that I could just jam a TransformBlock
into the mesh where I want to observe, get it to post to a progress updater of some variety and then return the message unchanged to the next block. I don't love this solution as the block would be purely there for its side-effect and I would also have to change the block linking logic wherever I want to observe.
So I wondered if I could use ISourceBlock<T>.AsObservable
to observe the passing of messages within the mesh without altering it and without consuming the messages. This seems both a purer and more practical solution, if it worked.
From my (limited) understanding of Rx that means that I need the observable to be hot rather than cold, so that my progress
updater sees the message but doesn't consume it. And .Publish().RefCount()
seems to be the way to make an observable hot. However, it simply does not work as intended - instead either block2
or progress
receives and consumes each message.
// Set up mesh
var block1 = new TransformBlock<int, int>(i => i + 20, new ExecutionDataflowBlockOptions() { BoundedCapacity = 1 });
var block2 = new ActionBlock<int>(i => Debug.Print("block2:" + i.ToString()), new ExecutionDataflowBlockOptions() { BoundedCapacity = 1 });
var obs = block1.AsObservable().Publish().RefCount(); // Declare this here just in case it makes a difference to do it before the LinkTo call.
var l1 = block1.LinkTo(block2, new DataflowLinkOptions() { PropagateCompletion = true});
// Progress
obs.ForEachAsync(i => Debug.Print("progress:" + i.ToString()));
// Start
var vals = Enumerable.Range(1, 5);
foreach (var v in vals)
{
block1.Post(v);
}
block1.Complete();
Result is non-deterministic but I get something mixed like this:
block2:21
progress:22
progress:24
block2:23
progress:25
So, am I doing something wrong, or is this impossible due to the way the way TPL Dataflow AsObservable
is implemented?
I realise I could also replace the LinkTo
between block1
and block2
with an Observable/Observer pair and that might work, but LinkTo
with downstream BoundedCapacity = 1
is the whole reason I'm using TPL Dataflow in the first place.
edit:
A few clarifications:
- I did intend to set
BoundedCapacity=1
in block2. While it's unnecessary in this trivial example, the downstream-constrained case is where I find TPL Dataflow really useful.
To clarify the solution I rejected in my second paragraph, it would be to add the following block linked in between block1 and block2:
var progressBlock = new TransformBlock<int, int>( i => {SomeUpdateProgressMethod(i); return i;});
I would also like to maintain back-pressure so that if a further-upstream block was distributing work to block1
and also other equivalent workers, it wouldn't send work to block1
if that chain was already busy.
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…