Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
672 views
in Technique[技术] by (71.8m points)

python - Split unicode string into 300 byte chunks without destroying characters

I want to split u"an arbitrary unicode string" into chunks of say 300 bytes without destroying any characters. The strings will be written to a socket that expects utf8 using unicode_string.encode("utf8"). I don't want to destroy any characters. How would I do this?

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

UTF-8 is designed for this.

def split_utf8(s, n):
    """Split UTF-8 s into chunks of maximum length n."""
    while len(s) > n:
        k = n
        while (ord(s[k]) & 0xc0) == 0x80:
            k -= 1
        yield s[:k]
        s = s[k:]
    yield s

Not tested. But you find a place to split, then backtrack until you reach the beginning of a character.

However, if a user might ever want to see an individual chunk, you may want to split on grapheme cluster boundaries instead. This is significantly more complicated, but not intractable. For example, in "é", you might not want to split apart the "e" and the "′". Or you might not care, as long as they get stuck together again in the end.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...