You can also do this:
def spans(txt):
tokens=nltk.word_tokenize(txt)
offset = 0
for token in tokens:
offset = txt.find(token, offset)
yield token, offset, offset+len(token)
offset += len(token)
s = "And now for something completely different and."
for token in spans(s):
print token
assert token[0]==s[token[1]:token[2]]
And get:
('And', 0, 3)
('now', 4, 7)
('for', 8, 11)
('something', 12, 21)
('completely', 22, 32)
('different', 33, 42)
('.', 42, 43)
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…