2014年1月12日日曜日

How to drop some tcp packets in a http response to force retransmission with iptables

I heard that a problem that one of our web server sometime has timeouts. We get timeouts more often when I access from EC2 instance like 3 out of 10 tries. I checked the result of tcpdump on the web server side and the client side and it seemed that retransmitted packet cannot reach to the client.

In order to check if it is EC2 specific problem or not, I decided to try to drop one of packet in response from server and if retransmission works.

This is how to drop one of packet.

iptables -N LOGDROP
iptables -A LOGDROP -j LOG
iptables -A LOGDROP -j DROP
iptables -A INPUT --source ${webserver_ip} -p tcp --sport 80 -m state --state ESTABLISHED -m limit --limit 1/s --limit-burst ${n-1} -j ACCEPT
iptables -A INPUT --source ${webserver_ip} -p tcp --sport 80 -m state --state ESTABLISHED -m limit --limit 1/s --limit-burst ${j} -j LOGDROP
iptables -A INPUT --source ${webserver_ip} -p tcp --sport 80 -j ACCEPT

it will accept before $nth packet and after that $j packets will be logged and dropped, all packets after that will be accepted again.
This rules should be applied on the client side. if it is executed on web server, it causes huge problems, so do not do that.

With this rule, I have checked the problem that is related to retransmit may happen from other environment.
I'm still investigating the exact reason of the problem.

Normally, even if you apply this filter, you won't have any problem because you will get retransmitted packet.
It should be quite rare problem, though, If you have any problem with retransmission, you can try to use this for investigation.

0 件のコメント: