The setup

A linux server connected to a VPN, using iptables to forward external ports (TCP) to a service behing the VPN. The servers behind the VPN do not answer ICMP requests, they are out of control from our setup.

When launching a script to test the connection to a service behind a VPN:

• It works locally on the server connected to the VPN
• It works remotely via a SSH tunnel to the server connected to the VPN
• It does not work when connecting to the server externally via iptables

Finding the root cause

We were a bit puzzled as to why we could connect to the remote service via a SSH tunnel, but not when using iptables. After some investigation with tcpdump, we received weird messages like IP truncated-ip in the tcpdump outputs of the computer running the test script.
This message appears when the “size of the IP packet (for example, IP header + TCP header + TCP payload) is shorter than the IP packet’s total length indicator” (see here).

It turns out that the virtual interface (tun0) of the VPN has an MTU of 1300, while it was 1500 on our computer running the script. This explained why packets were truncated but not why our computer did not receive any notification. Testing lowering the MTU on the computer running the script makes the connection to the service behind the VPN work.

After some googling, I found this article which explains that Path MTU Discovery (PMTUD) uses ICMP packets to notify hosts if a packet size exceeds the expected size.

Fixing the issue

Our iptables setup only did forward TCP packets, and not related ICMP packets. We then added the following to our iptables setup:

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

However, after retrying the script to connect to the remote service, it still did not work. The servers behind the VPN do not answer to ICMP requests, so our computer cannot be notified of packet too big errors.
Our other option was to force the Maximum Segment Size (MSS) of TCP packets on our iptables to a value lower than the virtual interface MTU + VPN encryptions.
At the end, we used the following rule to fix the problem, not the best, but it does the trick:

iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o tun0 -j TCPMSS --set-mss 1200

We did not use the --clamp-mss-to-pmtu because the VPN encryption needs to be taken into account in the payload size. While 1200 may not be the correct value, it works.