• Welcome to Hurricane Electric's IPv6 Tunnel Broker Forums.

IPv6 tunnel keeps timing out while trying to connect to IRC

Started by st0ner, July 25, 2013, 07:07:28 AM

Previous topic - Next topic

st0ner

Hello All,

I hope everyone is enjoying this lovely hot summer. We just got a level 3 heat wave warning for the next 2-3 days here, so HOORAY!

I've been running my IPv6 tunnel for few years now and even managed to move it from one place to another couple of times with no issues.

I noticed something yesterday though, whenever I try to connect to an IRC server using my IPv6 tunnel, it times out before connecting.

I have 2 IRC bots that are running on EFNet to protect the channel and they are working fine.

The problem with IRC (when connecting to EFNet/Freenode) using mIRC or irssi, it times out after a while, here is a telnet of whats happening:

@earth:~$ telnet -6 efnet.xs4all.nl 6667
Trying 2001:888:0:2::2...
Connected to efnet.xs4all.nl.
Escape character is '^]'.
NOTICE AUTH :*** Processing connection to efnet.xs4all.nl
NOTICE AUTH :*** Looking up your hostname...
NOTICE AUTH :*** Checking Ident
NOTICE AUTH :*** Couldn't look up your hostname
NOTICE AUTH :*** Got Ident response
ERROR :Closing Link: 127.0.0.1 (Connection timed out)
Connection closed by foreign host.


Any help or suggestions are appreciated.

Thanks in advance.

kasperd

The server is waiting for you to send a command. Since it never receives any commands, it times out and close the connection. Have you tried with a real IRC client?

I see the same output. If I then send valid NICK and USER commands, I get more output.

st0ner

Quote from: kasperd on July 25, 2013, 10:16:15 AM
The server is waiting for you to send a command. Since it never receives any commands, it times out and close the connection. Have you tried with a real IRC client?

I see the same output. If I then send valid NICK and USER commands, I get more output.

Thanks for the reply.

I tried with both, mIRC and irssi, both with the same result.

What real IRC client do you suggest?

I only pasted the telnet log to show how it is when I'm connecting from an IRC client as it is the exact same output and times out in the same spot.

kasperd

Quote from: st0ner on July 25, 2013, 02:38:36 PMWhat real IRC client do you suggest?
I use irssi. That works fine for me through an HE tunnel.

Quote from: st0ner on July 25, 2013, 02:38:36 PMI only pasted the telnet log to show how it is when I'm connecting from an IRC client as it is the exact same output and times out in the same spot.
That sounds unlikely since when I run irssi, it sends commands to the server immediately on connecting without waiting for any messages from the server. So, why would irssi send nothing to the server, when you run it?

You should get a packet capture of the connection for us to look at, then we might be able to spot the problem. If you use tcpdump, a command like this might work:tcpdump -pni eth0 'proto 41 || icmp' -s0 -Uw dumpfile.pcap


kasperd

In that trace I see 10 different machines on your LAN each doing IRC, and I didn't notice problems with any of them.

st0ner

I honestly dont know how to read the dump file in a 'readable' way.

But i can provide some details maybe that can get you more information.

Ubuntu server that i have the HE tunnel running from is 192.168.1.200.

I'm testing with trying to connect to with efnet.xs4all.nl using the ipv6 2001:470:26:71::19.

FYI, i removed the file from the link above, let me know if you need it again.

kasperd

Quote from: st0ner on July 26, 2013, 04:47:37 AMI honestly dont know how to read the dump file in a 'readable' way.
Wireshark is the best tool I have seen for this.

Quote from: st0ner on July 26, 2013, 04:47:37 AMBut i can provide some details maybe that can get you more information.

Ubuntu server that i have the HE tunnel running from is 192.168.1.200.
That much I could tell from looking at the pcap file.

Quote from: st0ner on July 26, 2013, 04:47:37 AMI'm testing with trying to connect to with efnet.xs4all.nl using the ipv6 2001:470:26:71::19.
That is not one of the 10 IRC clients communicating while you had tcpdump running.

Maybe you didn't actually try while tcpdump was running. Could you try the steps in the following order.

  • Start the same tcpdump command as before.
  • Start the IRC client.
  • Wait until the connection has been hanging for a few seconds.
  • Stop tcpdump
  • Post the new pcap file.

You can attach files to postings on the HE forum, you don't need to use a third party site for it.

st0ner

Thanks for the Wireshark tip.

I did it he way you suggested, here is what i thought might be interesting from the dump file:

14   3.596526   2001:470:26:71::4   2a01:270:0:666f::1   IRC   119   Request (NICK)
15   3.665249   2a01:270:0:666f::1   2001:470:26:71::4   IRC   169   Response (NOTICE)
20   3.665393   2001:470:26:71::4   2a01:270:0:666f::1   IRC   150   Request (USER)
21   3.733970   2a01:270:0:666f::1   2001:470:26:71::4   IRC   222   Response (NOTICE) (NOTICE)
32   3.841195   2a01:270:0:666f::1   2001:470:26:71::4   IRC   160   Response (NOTICE)
34   3.910217   2a01:270:0:666f::1   2001:470:26:71::4   IRC   246   [TCP Previous segment not captured] Response (identify) (NOTICE)

I uploaded the file to this post just in case i missed something.

Thanks again for the help.

st0ner

I forgot to mention that i've tested with Freenode this time (irc.freenode.net [2a01:270:0:666f::1]) instead of the efnet server so its easier to find from the rest of the packets.

Tested irssi with IPv6 2001:470:26:71::4

kasperd

Quote from: st0ner on July 26, 2013, 04:33:03 PMI uploaded the file to this post just in case i missed something.
You clearly missed something important. As long as you don't know what to look for, you really can't do much better than just provide the entire dump.

I'll explain what relevant information I found in this dump. First of all packet number 11 is your SYN packet to the server. In this packet the MSS option is interesting. It tells the server that it is not allowed to send more than 1420 bytes to the client in each packet. From this value we expect to subtract 12 bytes for a timestamp option, thus the server can send 1408 bytes of IRC payload in each TCP packet.

Now we look at packet 32, which is a packet from the server to the client. Based on the packet contents Wireshark will compute the expected sequence number of the next packet. If Wireshark is configured to display relative sequence numbers, the value shows up as 234.

Packet number 33 is an acknowledgement of packet 32, which in itself is not very interesting.

Then packet number 34 is the next packet from the server. The sequence number of this packet does not match the next sequence number, which Wireshark computed. Rather it shows 5866. This means some packets were missing.

Packet number 35 is another acknowledgement of packet 32. This is sent to hint the server about the gap in the sequence. The server is most likely retransmitting the lost packets, but they keep getting lost.

If we subtract the two sequence numbers, we can figure out how much data is missing. 5866-234 = 5632, so 5632 bytes are missing. It happens to be the case that 5632 = 4*1408. So the missing data corresponds to exactly four packets of maximum size.

So what happens is that the server needed to send a payload, which was too large for one packet, so it was split into four packets of maximum size and one smaller packet with the rest of the data. However the maximum size, which could be handled by both ends of the connection could not be handled by all the intermediate routers.

Some intermediate router could not handle this large packet and was required to send an error message back to the server indicating how large packets it could handle. This error message never reached the server. This is a typical PMTU discovery problem. Pinpointing the exact location of the problem and getting it fixed may not be possible for you, so you may need to look for a workaround. The most reliable workaround is MSS clamping. That means you let any one of the intermediate routers reduce the value of the MSS option in any TCP SYN packets, which has a larger value than what you expect to work.

Since you are running the tunnel on an Ubuntu machine, you should be able to use ip6tables to do that. Here is an example of what the rule could look like (loosely based on an example from the documentation):ip6tables -t mangle -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220I think this rule would solve the problem for you, but I am not 100% sure, so let us know how it turns out.

st0ner

heya kasperd,

i loved your reply, it was very informative and it had few things that i didnt know about.

i tried to add the ip6tables rule and unfortunately things havent changed and still facing the same issue  ???

Heres the results of ip6tables -L if it helps

sudo ip6tables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all      anywhere             anywhere             MAC 00:0C:29:48:D8:8F

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination


kasperd

Quote from: st0ner on July 28, 2013, 06:38:34 AMHeres the results of ip6tables -L if it helps
Not much use. The output from that command leaves out too many details. Try posting the output of ip6tables-save instead. By default ip6tables-save writes all rules in full to stdout.

st0ner

Here is the results for ip6tables-save:

@earth:~$ sudo ip6tables-save
[sudo] password for nizar:
# Generated by ip6tables-save v1.4.12 on Sun Jul 28 19:02:32 2013
*mangle
:PREROUTING ACCEPT [252675:22261628]
:INPUT ACCEPT [252675:22261628]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [258405:22585290]
:POSTROUTING ACCEPT [258405:22585290]
-A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220
-A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220
COMMIT
# Completed on Sun Jul 28 19:02:32 2013
# Generated by ip6tables-save v1.4.12 on Sun Jul 28 19:02:32 2013
*filter
:INPUT ACCEPT [85421302:7503885582]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [89259258:7766407466]
-A INPUT -m mac --mac-source 00:0C:29:48:D8:8F -j DROP
COMMIT
# Completed on Sun Jul 28 19:02:32 2013

kasperd

Quote from: st0ner on July 28, 2013, 10:05:04 AMHere is the results for ip6tables-save
You got two copies of the rule, but that shouldn't cause any problems. So now we need to figure out why it is still not working. I can come up with a list of possible explanations:

  • You are testing IRC connectivity from the Ubuntu machine, but the rule only applies to packets routed through that machine. As a first test you could try to insert the rule into the four other chains as well. If that helps, we can narrow it down afterwards.
  • It may be there is another problem as well, so maybe the rule did fix the first issue and now we need to find out what the second issue is.
  • It could be ip6tables has been buggy on some kernel versions.
So my first two suggestions is, see if adding such a rule to the other chains help. And if not, then try to produce another packet dump to see if TCPMSS did in fact reduce the value in the MSS option.