TCP Reset packets being injected in downloads?

[woei]

I just sent the email below to HE. I'm curious if someone is seeing something alike? Could you perhaps test downloads from:

- ipv6.download1.thinkbroadband.com
- snapshots.pfsense.org
- mirrors.dotsrc.org

And see if you have the same behavior?

Thanks,

Dennis

=================================================

Dear HE employee,

I've been a happy tunnelbroker user for a couple of years now. Lately i'm seeing more and more problems with downloads. After some researching i found this was to do with IPv6 downloads. Obviously more and more sites are IPv6 enabled and I believe that must be the reason why the issue is increasing.

I have made some packet captures and I notice after a few MB's of downloading, it appears the server starts sending tcp reset packets which kills the download. The weird thing is though, that the server is also still sending data. I see this behavior from different IPv6 (if not all) sites. I have tested on windows 7 and linux debian.

This leads me to believe that there might be tcp reset packets injected, perhaps even to prevent a free service as tunnelbroker, to overload HE's network with IPv6 downloads. This is all fine with me... being a free service, but I need to have a clear understanding to whether this is indeed happening so i can take this into account, or that there is perhaps something else going on...

Any ideas would be very welcome!

Please find a few examples of packet captures at: http://www.ipaddr.nl/HE-tcp-reset.pcap.zip

Kind regards,

Dennis Hagens

[broquea]

I've never seen something in the broker configured to "inject RST". If you get RST from the server, you are getting RST from the server. I looked at your pfsense PCAP, and the only RSTs are at the end and sourced from the webserver you connected to. Everything else prior to the end of your capture are: TCP DUP ACK, TCP WINDOW FULL, TCP FAST RETRANSMISSION. Not a single RST prior to the end of that PCAP.

Looking at your dotsrc.org PCAP, I don't see any mid-transaction RSTs. I only see RST at the end of that PCAP, as sourced from the webserver.
The same can be said of the thinkbroadband PCAP.

At least when I left I hadn't written anything to inject RST packets, and I very much doubt that changed, let alone has happened. That would be a serious degredation of service and definitely NOT what HE.NET is trying to do with any of their paid or unpaid services.

[woei]

Hi Alex,

Thanks for your response. I would feel pretty much the same about it, and i might have set things up a bit provocative to get some attention ;-).
If i would really have suspected this, i would not have requested a quote for 10GE port with 3Gbit commit IP transit from HE last week, if i would not trust them at all ;-)

The very strange thing is though, that i see it with all downloads on IPv6 i have captured, and the downloads all end after a couple of MB's while they are supposed to be hundreds of MB's... So it just leaves me puzzled.... It feels the downloads are simply "killed" somehow and i had to think back of the Comcast debacle with peer to peer traffic a couple of years back.

[kcochran]

Yeah, we're not injecting RSTs, unless the broker servers have become sentient.  If so, we're all in trouble.

[woei]

Yeah, we're not injecting RSTs, unless the broker servers have become sentient.  If so, we're all in trouble.

That's great :) now for suggestions anyone? Cause i'm getting lost here... why are all my IPv6 downloads getting messed up, cross OS and hardware...
Is my Juniper SRX210 HE messing with me? Junos bugs on IPv6 over IPv4 tunneling have been numerous... Are all the servers i download from sending me rst packets? I find that highly doubt able...

Can anyone suggest some test sites for me to download from which they reference as just fine to download 500+ MB files from over v6?

[kasperd]

This leads me to believe that there might be tcp reset packets injected

If TCP reset packets are being injected, the most likely place for that to happen is in a firewall right in front of the server itself. And to even think that TCP reset packets are being injected, I'd have to see packet traces from both ends of the connection.

It feels the downloads are simply "killed" somehow and i had to think back of the Comcast debacle with peer to peer traffic a couple of years back.

It is technically possible for your ISP to do that with your tunnelled traffic. There are no integrity checks on a plain 6in4 tunnel. You can use IPsec to get integrity checks, but I don't know of any 6in4 provider with IPsec support. IPsec between your client and the webserver is a possibility, if the webserver supports it. Alternatively you can look into other tunnel protocols than 6in4. AYIYA would be an option, but I only know of one provider supporting it.

Can anyone suggest some test sites for me to download from which they reference as just fine to download 500+ MB files from over v6?

The best idea I could come up with for your was to download the latest ISO of your favourite Linux distribution from a nearby mirror.

I just went and downloaded http://ubuntu.mirrors.proxad.net/quantal/ubuntu-12.10-beta2-desktop-i386.iso through an HE tunnel. No problems whatsoever. 787787776 bytes downloaded without interruptions.

[kasperd]

I just sent the email below to HE. I'm curious if someone is seeing something alike? Could you perhaps test downloads from:

- ipv6.download1.thinkbroadband.com
- snapshots.pfsense.org
- mirrors.dotsrc.org

And see if you have the same behavior?

I tested a download of http://snapshots.pfsense.org/FreeBSD_RELENG_8_3/i386/pfSense_HEAD/livecd_installer/pfSense-memstick-2.1-BETA0-i386-20121005-1942.img.gz through an HE tunnel. I only got about 1MB/s, but all 87874004 bytes were downloaded without interruptions.

[woei]

Thanks for your suggestion kasperd. I've tried that mirror and the same thing happens... Definitely something else going on than the servers hosting the file being the issue.

Capture: http://www.ipaddr.nl/HE-tcp-reset-ubuntu.mirrors.proxad.net.pcap.zip < windows 7 machine download in chrome

I have also made a capture on a debian machine in my internal network with wget. Different hardware.
Wget shows the connection beig reset all the time:

Code Select Expand

dennish@source:~$ wget http://ubuntu.mirrors.proxad.net/quantal/ubuntu-12.10-beta2-desktop-i386.iso
--2012-10-07 11:25:38--  http://ubuntu.mirrors.proxad.net/quantal/ubuntu-12.10-beta2-desktop-i386.iso
Resolving ubuntu.mirrors.proxad.net... 2a01:e0c:1:1598::1, 212.27.60.27
Connecting to ubuntu.mirrors.proxad.net|2a01:e0c:1:1598::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 787787776 (751M) [application/octet-stream]
Saving to: “ubuntu-12.10-beta2-desktop-i386.iso.1”

1% [>                                                                                          ] 10,265,937   446K/s   in 19s    

2012-10-07 11:25:57 (531 KB/s) - Read error at byte 10265937/787787776 (Connection reset by peer). Retrying.

--2012-10-07 11:25:58--  (try: 2)  http://ubuntu.mirrors.proxad.net/quantal/ubuntu-12.10-beta2-desktop-i386.iso
Connecting to ubuntu.mirrors.proxad.net|2a01:e0c:1:1598::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 787787776 (751M) [application/octet-stream]
Saving to: “ubuntu-12.10-beta2-desktop-i386.iso.1”

1% [>                                                                                          ] 13,568,016   493K/s   in 19s    

2012-10-07 11:26:17 (700 KB/s) - Read error at byte 13568016/787787776 (Connection reset by peer). Retrying.

--2012-10-07 11:26:19--  (try: 3)  http://ubuntu.mirrors.proxad.net/quantal/ubuntu-12.10-beta2-desktop-i386.iso
Connecting to ubuntu.mirrors.proxad.net|2a01:e0c:1:1598::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 787787776 (751M) [application/octet-stream]
Saving to: “ubuntu-12.10-beta2-desktop-i386.iso.1”

0% [                                                                                           ] 2,950,931    646K/s  eta 20m 34s


Also the capture for this download: http://www.ipaddr.nl/HE-tcp-reset-ubuntu.mirrors.proxad.net-debian.pcap.zip

The last occurrence of reset packets is off course me aborting the download...

Would someone be prepared to set up a download for me, to be able to capture on both sides? You can also email me root at ipaddr dot nl to set a time :-) I would very much appreciate this!

[kasperd]

Would someone be prepared to set up a download for me, to be able to capture on both sides?

I can setup a tunnel server using RFC 4193 address space and a webserver behind it. But I might not have time to set it up today. Meanwhile you can try using the reflector, which I set up a while ago. It is called reflector.easyv6.net. Any TCP or UDP packet send to that host will simply be echoed back to you by swapping source and destination IP and portnumber. So just start your own webserver on the same host as your HTTP client.

[woei]

Found the leak....

Is my Juniper SRX210 HE messing with me? Junos bugs on IPv6 over IPv4 tunneling have been numerous...

It appears to be so... When i was troubleshooting why the inbound connection via the reflection server (cool thing for testing btw, thanks for that) wasn't working, i noticed the initial syn packet never hit the "slow path" (session setup)... Or to be more precise, the first inbound packet of a session via the 6in4 tunnel seems to miss the "slow path" all together... For inbound connections it means it misses the first syn and all else fails... no half open state is achieved.

For outbound connections this causes the behavior that a half open session is created and stays open for 20 seconds, waiting for the syn-ack.. which gets missed... During those 20 seconds though i can download a couple of MB's since the end to end (between client and server) session did come to be established. After the 20 seconds (default tcp-initial-timeout) the firewall assumed the session was not established and flushed the thing by sending rst's to the initiator and cleaning out the session table.

After increasing the tcp initial timeout (set security flow tcp-session tcp-initial-timeout 150) i noticed that the session cut-off happened after 150 seconds indeed, in stead of 20 seconds. By disabling syn checking in tunneled traffic (set security flow tcp-session no-syn-check-in-tunnel), i can successfully download large files over IPv6. This poses great security risks however...

Seems to me to be a bug. Currently running 12.1r1.9 will try to test on 12.1R3.5 later today.

Thanks for your efforts and i hope i have not hit anyone against the head by setting this thread up a bit provocative ;-)!

Dennis

[woei]

Still broken in Junos 12.1r3.5 :-(

[kasperd]

By disabling syn checking in tunneled traffic (set security flow tcp-session no-syn-check-in-tunnel), i can successfully download large files over IPv6. This poses great security risks however...

Should be possible to disable the stateful inspection and instead apply a stateless filter to just SYN packets.

[woei]

By disabling syn checking in tunneled traffic (set security flow tcp-session no-syn-check-in-tunnel), i can successfully download large files over IPv6. This poses great security risks however...

Should be possible to disable the stateful inspection and instead apply a stateless filter to just SYN packets.

Thanks for your suggestion, but it don't believe thats going to help me... The SYN already does reach my PC and the SYN-ACK the server on the internet. It's just the firewall that doesn't take notice of it and because of that the state is not correct in the session table.

[kasperd]

It's just the firewall that doesn't take notice of it and because of that the state is not correct in the session table.

That problem won't happen, if it is configured like I suggested. If the firewall is stateless, it cannot mess up its state, because there is no state to mess up.