Error handling with node.js streams

Question

What's the correct way to handle errors with streams? I already know there's an 'error' event you can listen on, but I want to know some more details about arbitrarily complicated situations.

For starters, what do you do when you want to do a simple pipe chain:

input.pipe(transformA).pipe(transformB).pipe(transformC)...

And how do you properly create one of those transforms so that errors are handled correctly?

More related questions:

• when an error happens, what happens to the 'end' event? Does it never get fired? Does it sometimes get fired? Does it depend on the transform/stream? What are the standards here?
• are there any mechanisms for propogating errors through the pipes?
• do domains solve this problem effectively? Examples would be nice.
• do errors that come out of 'error' events have stack traces? Sometimes? Never? is there a way to get one from them?

Answer 1

transform

Transform streams are both readable and writeable, and thus are really good 'middle' streams. For this reason, they are sometimes referred to as through streams. They are similar to a duplex stream in this way, except they provide a nice interface to manipulate the data rather than just sending it through. The purpose of a transform stream is to manipulate the data as it is piped through the stream. You may want to do some async calls, for example, or derive a couple of fields, remap some things, etc.

---

---

For how to create a transform stream see here and here. All you have to do is :

1. include the stream module
2. instantiate ( or inherit from) the Transform class
3. implement a _transform method which takes a (chunk, encoding, callback).

The chunk is your data. Most of the time you won't need to worry about encoding if you are working in objectMode = true. The callback is called when you are done processing the chunk. This chunk is then pushed on to the next stream.

If you want a nice helper module that will enable you to do through stream really really easily, I suggest through2.

For error handling, keep reading.

pipe

In a pipe chain, handling errors is indeed non-trivial. According to this thread .pipe() is not built to forward errors. So something like ...

var a = createStream();
a.pipe(b).pipe(c).on('error', function(e){handleError(e)});

... would only listen for errors on the stream c. If an error event was emitted on a, that would not be passed down and, in fact, would throw. To do this correctly:

var a = createStream();
a.on('error', function(e){handleError(e)})
.pipe(b)
.on('error', function(e){handleError(e)})
.pipe(c)
.on('error', function(e){handleError(e)});

Now, though the second way is more verbose, you can at least keep the context of where your errors happen. This is usually a good thing.

One library I find helpful though if you have a case where you only want to capture the errors at the destination and you don't care so much about where it happened is event-stream.

end

When an error event is fired, the end event will not be fired (explicitly). The emitting of an error event will end the stream.

domains

In my experience, domains work really well most of the time. If you have an unhandled error event (i.e. emitting an error on a stream without a listener), the server can crash. Now, as the above article points out, you can wrap the stream in a domain which should properly catch all errors.

var d = domain.create();
 d.on('error', handleAllErrors);
 d.run(function() {
     fs.createReadStream(tarball)
       .pipe(gzip.Gunzip())
       .pipe(tar.Extract({ path: targetPath }))
       .on('close', cb);
 });

• the above code sample is from this post

The beauty of domains is that they will preserve the stack traces. Though event-stream does a good job of this as well.

For further reading, check out the stream-handbook. Pretty in depth, but super useful and gives some great links to lots of helpful modules.