Hurricane Electric's IPv6 Tunnel Broker Forums

General IPv6 Topics => IPv6 Basics & Questions & General Chatter => Topic started by: st0ner on July 25, 2013, 07:07:28 AM

Title: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: st0ner on July 25, 2013, 07:07:28 AM
Hello All,

I hope everyone is enjoying this lovely hot summer. We just got a level 3 heat wave warning for the next 2-3 days here, so HOORAY!

I've been running my IPv6 tunnel for few years now and even managed to move it from one place to another couple of times with no issues.

I noticed something yesterday though, whenever I try to connect to an IRC server using my IPv6 tunnel, it times out before connecting.

I have 2 IRC bots that are running on EFNet to protect the channel and they are working fine.

The problem with IRC (when connecting to EFNet/Freenode) using mIRC or irssi, it times out after a while, here is a telnet of whats happening:

@earth:~$ telnet -6 efnet.xs4all.nl 6667
Trying 2001:888:0:2::2...
Connected to efnet.xs4all.nl.
Escape character is '^]'.
NOTICE AUTH :*** Processing connection to efnet.xs4all.nl
NOTICE AUTH :*** Looking up your hostname...
NOTICE AUTH :*** Checking Ident
NOTICE AUTH :*** Couldn't look up your hostname
NOTICE AUTH :*** Got Ident response
ERROR :Closing Link: 127.0.0.1 (Connection timed out)
Connection closed by foreign host.


Any help or suggestions are appreciated.

Thanks in advance.
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: kasperd on July 25, 2013, 10:16:15 AM
The server is waiting for you to send a command. Since it never receives any commands, it times out and close the connection. Have you tried with a real IRC client?

I see the same output. If I then send valid NICK and USER commands, I get more output.
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: st0ner on July 25, 2013, 02:38:36 PM
Quote from: kasperd on July 25, 2013, 10:16:15 AM
The server is waiting for you to send a command. Since it never receives any commands, it times out and close the connection. Have you tried with a real IRC client?

I see the same output. If I then send valid NICK and USER commands, I get more output.

Thanks for the reply.

I tried with both, mIRC and irssi, both with the same result.

What real IRC client do you suggest?

I only pasted the telnet log to show how it is when I'm connecting from an IRC client as it is the exact same output and times out in the same spot.
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: kasperd on July 25, 2013, 03:56:16 PM
Quote from: st0ner on July 25, 2013, 02:38:36 PMWhat real IRC client do you suggest?
I use irssi. That works fine for me through an HE tunnel.

Quote from: st0ner on July 25, 2013, 02:38:36 PMI only pasted the telnet log to show how it is when I'm connecting from an IRC client as it is the exact same output and times out in the same spot.
That sounds unlikely since when I run irssi, it sends commands to the server immediately on connecting without waiting for any messages from the server. So, why would irssi send nothing to the server, when you run it?

You should get a packet capture of the connection for us to look at, then we might be able to spot the problem. If you use tcpdump, a command like this might work:tcpdump -pni eth0 'proto 41 || icmp' -s0 -Uw dumpfile.pcap
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: st0ner on July 26, 2013, 02:41:02 AM

Hello,

Here is the the file you requested:

https://www.dropbox.com/s/wy3xghekd3hdvgg/dumpfile.pcap

Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: kasperd on July 26, 2013, 04:21:27 AM
In that trace I see 10 different machines on your LAN each doing IRC, and I didn't notice problems with any of them.
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: st0ner on July 26, 2013, 04:47:37 AM
I honestly dont know how to read the dump file in a 'readable' way.

But i can provide some details maybe that can get you more information.

Ubuntu server that i have the HE tunnel running from is 192.168.1.200.

I'm testing with trying to connect to with efnet.xs4all.nl using the ipv6 2001:470:26:71::19.

FYI, i removed the file from the link above, let me know if you need it again.
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: kasperd on July 26, 2013, 10:30:19 AM
Quote from: st0ner on July 26, 2013, 04:47:37 AMI honestly dont know how to read the dump file in a 'readable' way.
Wireshark is the best tool I have seen for this.

Quote from: st0ner on July 26, 2013, 04:47:37 AMBut i can provide some details maybe that can get you more information.

Ubuntu server that i have the HE tunnel running from is 192.168.1.200.
That much I could tell from looking at the pcap file.

Quote from: st0ner on July 26, 2013, 04:47:37 AMI'm testing with trying to connect to with efnet.xs4all.nl using the ipv6 2001:470:26:71::19.
That is not one of the 10 IRC clients communicating while you had tcpdump running.

Maybe you didn't actually try while tcpdump was running. Could you try the steps in the following order.
You can attach files to postings on the HE forum, you don't need to use a third party site for it.
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: st0ner on July 26, 2013, 04:33:03 PM
Thanks for the Wireshark tip.

I did it he way you suggested, here is what i thought might be interesting from the dump file:

14   3.596526   2001:470:26:71::4   2a01:270:0:666f::1   IRC   119   Request (NICK)
15   3.665249   2a01:270:0:666f::1   2001:470:26:71::4   IRC   169   Response (NOTICE)
20   3.665393   2001:470:26:71::4   2a01:270:0:666f::1   IRC   150   Request (USER)
21   3.733970   2a01:270:0:666f::1   2001:470:26:71::4   IRC   222   Response (NOTICE) (NOTICE)
32   3.841195   2a01:270:0:666f::1   2001:470:26:71::4   IRC   160   Response (NOTICE)
34   3.910217   2a01:270:0:666f::1   2001:470:26:71::4   IRC   246   [TCP Previous segment not captured] Response (identify) (NOTICE)

I uploaded the file to this post just in case i missed something.

Thanks again for the help.
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: st0ner on July 26, 2013, 04:34:50 PM
I forgot to mention that i've tested with Freenode this time (irc.freenode.net [2a01:270:0:666f::1]) instead of the efnet server so its easier to find from the rest of the packets.

Tested irssi with IPv6 2001:470:26:71::4
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: kasperd on July 27, 2013, 03:44:23 AM
Quote from: st0ner on July 26, 2013, 04:33:03 PMI uploaded the file to this post just in case i missed something.
You clearly missed something important. As long as you don't know what to look for, you really can't do much better than just provide the entire dump.

I'll explain what relevant information I found in this dump. First of all packet number 11 is your SYN packet to the server. In this packet the MSS option is interesting. It tells the server that it is not allowed to send more than 1420 bytes to the client in each packet. From this value we expect to subtract 12 bytes for a timestamp option, thus the server can send 1408 bytes of IRC payload in each TCP packet.

Now we look at packet 32, which is a packet from the server to the client. Based on the packet contents Wireshark will compute the expected sequence number of the next packet. If Wireshark is configured to display relative sequence numbers, the value shows up as 234.

Packet number 33 is an acknowledgement of packet 32, which in itself is not very interesting.

Then packet number 34 is the next packet from the server. The sequence number of this packet does not match the next sequence number, which Wireshark computed. Rather it shows 5866. This means some packets were missing.

Packet number 35 is another acknowledgement of packet 32. This is sent to hint the server about the gap in the sequence. The server is most likely retransmitting the lost packets, but they keep getting lost.

If we subtract the two sequence numbers, we can figure out how much data is missing. 5866-234 = 5632, so 5632 bytes are missing. It happens to be the case that 5632 = 4*1408. So the missing data corresponds to exactly four packets of maximum size.

So what happens is that the server needed to send a payload, which was too large for one packet, so it was split into four packets of maximum size and one smaller packet with the rest of the data. However the maximum size, which could be handled by both ends of the connection could not be handled by all the intermediate routers.

Some intermediate router could not handle this large packet and was required to send an error message back to the server indicating how large packets it could handle. This error message never reached the server. This is a typical PMTU discovery problem. Pinpointing the exact location of the problem and getting it fixed may not be possible for you, so you may need to look for a workaround. The most reliable workaround is MSS clamping. That means you let any one of the intermediate routers reduce the value of the MSS option in any TCP SYN packets, which has a larger value than what you expect to work.

Since you are running the tunnel on an Ubuntu machine, you should be able to use ip6tables to do that. Here is an example of what the rule could look like (loosely based on an example from the documentation):ip6tables -t mangle -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220I think this rule would solve the problem for you, but I am not 100% sure, so let us know how it turns out.
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: st0ner on July 28, 2013, 06:38:34 AM
heya kasperd,

i loved your reply, it was very informative and it had few things that i didnt know about.

i tried to add the ip6tables rule and unfortunately things havent changed and still facing the same issue  ???

Heres the results of ip6tables -L if it helps

sudo ip6tables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
DROP       all      anywhere             anywhere             MAC 00:0C:29:48:D8:8F

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: kasperd on July 28, 2013, 07:34:53 AM
Quote from: st0ner on July 28, 2013, 06:38:34 AMHeres the results of ip6tables -L if it helps
Not much use. The output from that command leaves out too many details. Try posting the output of ip6tables-save instead. By default ip6tables-save writes all rules in full to stdout.
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: st0ner on July 28, 2013, 10:05:04 AM
Here is the results for ip6tables-save:

@earth:~$ sudo ip6tables-save
[sudo] password for nizar:
# Generated by ip6tables-save v1.4.12 on Sun Jul 28 19:02:32 2013
*mangle
:PREROUTING ACCEPT [252675:22261628]
:INPUT ACCEPT [252675:22261628]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [258405:22585290]
:POSTROUTING ACCEPT [258405:22585290]
-A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220
-A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1220
COMMIT
# Completed on Sun Jul 28 19:02:32 2013
# Generated by ip6tables-save v1.4.12 on Sun Jul 28 19:02:32 2013
*filter
:INPUT ACCEPT [85421302:7503885582]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [89259258:7766407466]
-A INPUT -m mac --mac-source 00:0C:29:48:D8:8F -j DROP
COMMIT
# Completed on Sun Jul 28 19:02:32 2013
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: kasperd on July 28, 2013, 10:23:59 AM
Quote from: st0ner on July 28, 2013, 10:05:04 AMHere is the results for ip6tables-save
You got two copies of the rule, but that shouldn't cause any problems. So now we need to figure out why it is still not working. I can come up with a list of possible explanations:So my first two suggestions is, see if adding such a rule to the other chains help. And if not, then try to produce another packet dump to see if TCPMSS did in fact reduce the value in the MSS option.
Title: Re: IPv6 tunnel keeps timing out while trying to connect to IRC
Post by: st0ner on July 31, 2013, 03:15:44 AM
Heya kasperd.

Just wanted to update the post. I found out other issues on my home network that i need to sort out first before i continue troubleshooting the issue im having (which still exists).

Lack of time is not helping.

Will update this post again once i get things sorted out. I hope you'll be around then to continue this.