HP MSR Routers - TCP Connectivity Through the GRE Over IPSec Tunnel Gets Dropped or Slow

Issue

A GRE over IPSec tunnel is established between two MSR routers and the user traffic between two sites is carried by the tunnel. During testing it is found only ping can work properly. When try to sync email with exchange server, browse Internet and RDP to remote desktop through the tunnel, the connectivity is very slow and even gets dropped.

The GRE over IPSec tunnel add quite a lot overhead to the original data packet. If the original TCP packet from the LAN is 1400-byte large, then both the GRE and IPSec add another 76 bytes (when running as transport mode) and 96 bytes (when running as tunnel mode). This makes the packet size out from the router WAN port to 1476 bytes or 1496 bytes, which almost reaches the 1500-byte TCP limit. It is very much possible that such a large packet gets dropped between two routers.

Solution

The easy way is to reduce the TCP packet before reaching the MSR LAN port: A command tcp mss applied to the LAN ports can suggest the client reduce the TCP packet sent. To reduce the TCP packet from client to 1200-byte large, then use the following command:

# interface e0/0  

    tcp mss 1200

This command changes the mss to 1200-byte in the TCP sync + ack packet out from the LAN port (sent to the client PC) so that when the PC client received the TCP sync + ack packet, it understands the other side (TCP peer) has set the TCP packet to 1200-byte maximum. As a result it will reduce its maximum TCP packet to 1200 as well. So once this command is applied to the LAN ports of both sides of MSR, both original TCP packet sent by client will be reduced to 1200. Therefore even with the GRE and IPSec overhead, the packet out of the WAN port will not be too large to get dropped.