networking - different tcp packets captured on sender and receiver

Question

I am using tcpdump/wireshark to capture tcp packets while tcp client sending data to tcp server. The client simply sends 4096 bytes to server in one "send()" call. And I get different tcp packets on two sides, two packets on the sender side seem to be "compacted" on the receiver side, this conflicts with how i understand the tcp protocol and I stuck on this issue for a few days and really need some help.

Please notice the packet length in following packets:

client (sender) sends 2 packets 0Xbcac (4) and 0xbcae (5), sends 2896 + 1200 = 4096 bytes in all.

(0xbcac) 4  14:31:33.838305 192.168.91.194  192.168.91.193  TCP 2962    59750 > 9877 [ACK] Seq=1 Ack=1 Win=14720 **Len=2896** TSval=260728 TSecr=3464603    0
(0xbcae) 5  14:31:33.838427 192.168.91.194  192.168.91.193  TCP 1266    59750 > 9877 [PSH, ACK] Seq=2897 Ack=1 Win=14720 **Len=1200** TSval=260728 TSecr=3464603    0

However on the server (receiver) side, only one packet is presented, with ip.id=0xbcac and length = 4096 (receiver.packet.0xbcac = sender.packet.0xbcac + 0xbcae):

(0xbcac) 4  14:31:33.286296 192.168.91.194  192.168.91.193  TCP 4162    59750 > 9877 [PSH, ACK] Seq=1 Ack=1 Win=14720 **Len=4096** TSval=260728 TSecr=3464603   0

I'm aware that tcp is a stream protocol and data sent can be divided into packets according to MSS (or MTU), but i guess the division happens before packets are sent to NIC, thus before captured. I'm also aware that the PSH flag in packet 0xbcae lead to writing data from buffer to NIC, but that cannot explain the "compacted" packet. Also I tried in client to send 999999 bytes in one "send" call and the data are divided into small packets and sent, but still mismatch the packets captured on server side. At last I disable tcp nagle, get the same result, and ruled out that reason.

So my question is the mismatching i encountered normal? If it is, what caused this? If not, i'm using ubuntu 12.04 and ubuntu 13.10 in LAN, and what is the possible reason to this "compacted" packet?

Thanks in advance for any help!

Answer 1

two packets on the sender side seem to be "compacted" on the receiver side

It looks like a case of generic receive offload or large receive offload. Long story short, the receiving network card does some smart stuff and coalesces segments before they hit the kernel, which improves performance.

To check if this is the case you can try to disable it using:

$ ethtool -K eth0 gro off
$ ethtool -K eth0 lro off

Something complementary happens on the sending side: tcp segmentation offload or generic segmentation offload.

After disabling these don't forget to reenable them: they seriously improve performance.