Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
236 views
in Technique[技术] by (71.8m points)

python - process socket data that ends with a line break

What is the best approach to process a socket connection where I need var data to end with a line break ? I'm using the code below but sometimes the tcp packets get chunked and it takes a long time to match data.endswith(" "). I've also tried other approaches, like saving the last line if it doesn't end with and append it to dataon the next loop. but this also doesn't work because multiple packets get chunked and the 1st and 2nd part don't match. I've no control over the other end, it basically sends multiple lines that end in .

Any suggestion will be welcome, as I don't have much knowledge on socket connections.

def receive_bar_updates(s):
    global all_bars
    data = ''
    buffer_size = 4096
    while True:
        data += s.recv(buffer_size)
        if not data.endswith("
"):
            continue
        lines = data.split("
")
        lines = filter(None, lines)
        for line in lines:
            if line.startswith("BH") or line.startswith("BC"):
                symbol = str(line.split(",")[1])
                all_bars[symbol].append(line)
                y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
                y.start()
        data = ""

Example of "normal" data:

line1

line2

line3

Example of chunked data:

line1

line2

lin
See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

If you have a raw input that you want to process as line, the io module is your friend because it will do the low level assembling of packets in lines.

You could use:

class SocketIO(io.RawIOBase):
    def __init__(self, sock):
        self.sock = sock
    def read(self, sz=-1):
        if (sz == -1): sz=0x7FFFFFFF
        return self.sock.recv(sz)
    def seekable(self):
        return False

It is more robust than endswith(' ') because if one packet contains an embedded newline ('ab cd'), the io module will correctly process it. Your code could become:

def receive_bar_updates(s):
    global all_bars
    data = ''
    buffer_size = 4096
    fd = SocketIO(s)  # fd can be used as an input file object

    for line in fd:
        if should_be_rejected_by_filter(line): continue # do not know what filter does...
        if line.startswith("BH") or line.startswith("BC"):
            symbol = str(line.split(",")[1])
            all_bars[symbol].append(line)
            y = Thread(target=proccess_bars, kwargs={'symbol': symbol})
            y.start()

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...