python - Handle generator exceptions in its consumer

Question

This is a follow-up to Handle an exception thrown in a generator and discusses a more general problem.

I have a function that reads data in different formats. All formats are line- or record-oriented and for each format there's a dedicated parsing function, implemented as a generator. So the main reading function gets an input and a generator, which reads its respective format from the input and delivers records back to the main function:

def read(stream, parsefunc):
    for record in parsefunc(stream):
        do_stuff(record)

where parsefunc is something like:

def parsefunc(stream):
    while not eof(stream):
        rec = read_record(stream)
        do some stuff
        yield rec

The problem I'm facing is that while parsefunc can throw an exception (e.g. when reading from a stream), it has no idea how to handle it. The function responsible for handling exceptions is the main read function. Note that exceptions occur on a per-record basis, so even if one record fails, the generator should continue its work and yield records back until the whole stream is exhausted.

In the previous question I tried to put next(parsefunc) in a try block, but as turned out, this is not going to work. So I have to add try-except to the parsefunc itself and then somehow deliver exceptions to the consumer:

def parsefunc(stream):
    while not eof(stream):
        try:
            rec = read_record()
            yield rec
        except Exception as e:
            ?????

I'm rather reluctant to do this because

• it makes no sense to use try in a function that isn't intended to handle any exceptions
• it's unclear to me how to pass exceptions to the consuming function
• there going to be many formats and many parsefunc's, I don't want to clutter them with too much helper code.

Has anyone suggestions for a better architecture?

A note for googlers: in addition to the top answer, pay attention to senderle's and Jon's posts - very smart and insightful stuff.

Answer 1

You can return a tuple of record and exception in the parsefunc and let the consumer function decide what to do with the exception:

import random

def get_record(line):
  num = random.randint(0, 3)
  if num == 3:
    raise Exception("3 means danger")
  return line


def parsefunc(stream):
  for line in stream:
    try:
      rec = get_record(line)
    except Exception as e:
      yield (None, e)
    else:
      yield (rec, None)

if __name__ == '__main__':
  with open('temp.txt') as f:
    for rec, e in parsefunc(f):
      if e:
        print "Got an exception %s" % e
      else:
        print "Got a record %s" % rec