Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
514 views
in Technique[技术] by (71.8m points)

python - pyPdf ignores newlines in PDF file

I'm trying to extract each page of a PDF as a string:

import pyPdf

pages = []
pdf = pyPdf.PdfFileReader(file('g-reg-101.pdf', 'rb'))
for i in range(0, pdf.getNumPages()):
    this_page = pdf.getPage(i).extractText() + "
"
    this_page = " ".join(this_page.replace(u"xa0", " ").strip().split())
    pages.append(this_page.encode("ascii", "xmlcharrefreplace"))
for page in pages:
    print '*' * 80
    print page

But this script ignore newline characters, leaving me with messy strings like information concerning an individual which, because of name, identifyingnumber, mark or description (i.e, this should read identifying number, not identifyingumber).

Here's an example of the type of PDF I'm trying to parse.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

I don't know much about PDF encoding, but I think you can solve your particular problem by modifying pdf.py. In the PageObject.extractText method, you see what's going on:

def extractText(self):
    [...]
    for operands,operator in content.operations:
        if operator == "Tj":
            _text = operands[0]
            if isinstance(_text, TextStringObject):
                text += _text
        elif operator == "T*":
            text += "
"
        elif operator == "'":
            text += "
"
            _text = operands[0]
            if isinstance(_text, TextStringObject):
                text += operands[0]
        elif operator == '"':
            _text = operands[2]
            if isinstance(_text, TextStringObject):
                text += "
"
                text += _text
        elif operator == "TJ":
            for i in operands[0]:
                if isinstance(i, TextStringObject):
                    text += i

If the operator is Tj or TJ (it's Tj in your example PDF) then the text is simply appended and no newline is added. Now you wouldn't necessarily want to add a newline, at least if I'm reading the PDF reference right: Tj/TJ are simply the single and multiple show-string operators, and the existence of a separator of some kind isn't mandatory.

Anyway, if you modify this code to be something like

def extractText(self, Tj_sep="", TJ_sep=""):

[...]

        if operator == "Tj":
            _text = operands[0]
            if isinstance(_text, TextStringObject):
                text += Tj_sep
                text += _text

[...]

        elif operator == "TJ":
            for i in operands[0]:
                if isinstance(i, TextStringObject):
                    text += TJ_sep
                    text += i

then the default behaviour should be the same:

In [1]: pdf.getPage(1).extractText()[1120:1250]
Out[1]: u'ing an individual which, because of name, identifyingnumber, mark or description can be readily associated with a particular indiv'

but you can change it when you want to:

In [2]: pdf.getPage(1).extractText(Tj_sep=" ")[1120:1250]
Out[2]: u'ta" means any information concerning an individual which, because of name, identifying number, mark or description can be readily '

or

In [3]: pdf.getPage(1).extractText(Tj_sep="
")[1120:1250]
Out[3]: u'ta" means any information concerning an individual which, because of name, identifying
number, mark or description can be readily '

Alternatively, you could simply add the separators yourself by modifying the operands themselves in-place, but that could break something else (methods like get_original_bytes make me nervous).

Finally, you don't have to edit pdf.py itself if you don't want to: you could simply pull out this method into a function.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...