can't ping hosts behind openbsd - blocked ICMPv6?

[arader]

Hi all,

I'm having a frustrating issue right now where I'm unable to ping any of hosts from the external internet. I've set up my pf.conf file (below) to allow ICMP and ICMPv6 traffic on all interfaces, but I'm still not getting anything. The really frustrating part is that if I allow ALL traffic on my tunnel interface, I'm suddenly able to ping my hosts just fine! Obviously I don't want to allow all traffic in from the tunnel, but I can't figure out how to tell openbsd to stop blocking ICMP! Any help?

A host of mine that I've been pinging is: 2001:470:1f05:3dd::12. With the below pf.conf used as is, I'm unable to ping it at all. If I add pass in on $tun_if everything starts working, but that obviously nullifies the point of a firewall. Additionally, I took the below pf.conf, added a "allow port 22" to the above host, and telnetted to reflector.easyv6.net, and it successfully connected back to my machine.

thoughts?

Code Select Expand

# cat /etc/pf.conf
ext_if="sis0"
int_if="sis1"
tun_if="gif0"

set block-policy drop
set loginterface egress
set skip on lo

match out on egress inet from ! (egress:network) to any nat-to (egress:0)

block in log

pass out quick

antispoof quick for { lo $int_if }

pass in proto icmp
pass in proto ipv6-icmp

pass in on $int_if


[passport123]

I don't allow that type of traffic on my home network, so I've not used the following.   But from memory, the syntax seems to be closer to:

  pass in inet6 proto icmp6 from any to any icmp6-type echoreq  keep state


The "keep state" should provide a path for the outbound echoreply.  And there's probably some redundant keywords in that rule.   :)

[kasperd]

The "keep state" should provide a path for the outbound echoreply.

It should be possible to configure the firewall to permit the outgoing packet even without the state. That would allow reducing the amount of state it needs to keep.

[passport123]

Good point.

Yes, it is possible to avoid the keep state by writing another rule for the outbound direction.  My firewall is not that busy, so the size of the state table is not a concern for me at this point.

[kasperd]

Memory usage is not the only problem you get by tracking state. State means restarting the firewall is going to cause disruption. If your firewall is stateless it can be restarted without any impact apart from a few lost packets while it is restarting. This is no problem for something as short lived as an echo request. But it would be a problem for anything based on TCP. Tracking state also makes it harder to have redundant firewalls. In general one should avoid state if it isn't strictly necessary.

[arader]

I don't allow that type of traffic on my home network, so I've not used the following.   But from memory, the syntax seems to be closer to:

  pass in inet6 proto icmp6 from any to any icmp6-type echoreq  keep state


The "keep state" should provide a path for the outbound echoreply.  And there's probably some redundant keywords in that rule.   :)

I tried this but it doesn't appear to make any difference. After reading through PF's docs, I think this is the same exact thing as pass in proto icmp6 that I already have:

• inet6: icmp6 would only appear over inet6, so it's not really necessary
• from any to any: it appears this is the default
• keep state: according to http://openbsd.org/faq/pf/filter.html#stateopts keep state is implicit for all filter rules

Some more info: I updated my pf.conf back to only having pass in log proto icmp6 (note the added log) and went to work today. Before heading home I decided to try and ping my machine from there, and it worked a few times, then immediately stopped. I tried a few "ping test" websites to see if it was maybe a corporate firewall issue or something, but they all had the same failure. When I came home I checked the logs, and sure enough:

Code Select Expand

Apr 10 16:06:44.044062 rule 14/(match) pass in on gif0: 2001:4898:d8:f08d:7919:d268:208:4f2 > 2001:470:1f04:3dd::2: icmp6: echo request
Apr 10 16:06:50.508284 rule 14/(match) pass in on gif0: 2001:4898:d8:f08d:7919:d268:208:4f2 > 2001:470:1f05:3dd::1: icmp6: echo request
Apr 10 16:06:54.843874 rule 14/(match) pass in on gif0: 2001:4898:d8:f08d:7919:d268:208:4f2 > 2001:470:1f05:3dd:12::1: icmp6: echo request
Apr 10 16:08:15.070451 rule 1/(match) block in on sis0: 2001:4898:d8:f08d:7919:d268:208:4f2 > 2001:470:1f05:3dd:12::1: [|tcp] (encap)
Apr 10 16:08:18.080226 rule 1/(match) block in on sis0: 2001:4898:d8:f08d:7919:d268:208:4f2 > 2001:470:1f05:3dd:12::1: [|tcp] (encap)
Apr 10 16:08:22.237299 rule 1/(match) block in on sis0: 2001:4898:d8:f08d:7919:d268:208:4f2 > 2001:470:1f05:3dd:12::1: [|icmp6] (encap)
Apr 10 16:08:24.079758 rule 1/(match) block in on sis0: 2001:4898:d8:f08d:7919:d268:208:4f2 > 2001:470:1f05:3dd:12::1: [|tcp] (encap)
Apr 10 16:08:33.365432 rule 1/(match) block in on sis0: 2001:4898:d8:f08d:7919:d268:208:4f2 > 2001:470:1f04:3dd::2: [|icmp6] (encap)
Apr 10 16:08:38.193635 rule 1/(match) block in on sis0: 2001:4898:d8:f08d:7919:d268:208:4f2 > 2001:470:1f04:3dd::2: [|icmp6] (encap)
Apr 10 16:08:52.509733 rule 1/(match) block in on sis0: 2001:4898:d8:f08d:7919:d268:208:4f2 > 2001:470:1f04:3dd::2: [|icmp6] (encap)
Apr 10 16:09:28.696189 rule 1/(match) block in on sis0: 2001:4d48:1337:5afe::2 > 2001:470:1f05:3dd::1: [|icmp6] (encap)
Apr 10 16:09:29.696704 rule 1/(match) block in on sis0: 2001:4d48:1337:5afe::2 > 2001:470:1f05:3dd::1: [|icmp6] (encap)
Apr 10 16:09:30.696862 rule 1/(match) block in on sis0: 2001:4d48:1337:5afe::2 > 2001:470:1f05:3dd::1: [|icmp6] (encap)
Apr 10 16:09:31.696800 rule 1/(match) block in on sis0: 2001:4d48:1337:5afe::2 > 2001:470:1f05:3dd::1: [|icmp6] (encap)
Apr 10 16:09:32.696815 rule 1/(match) block in on sis0: 2001:4d48:1337:5afe::2 > 2001:470:1f05:3dd::1: [|icmp6] (encap)
Apr 10 16:09:33.696591 rule 1/(match) block in on sis0: 2001:4d48:1337:5afe::2 > 2001:470:1f05:3dd::1: [|icmp6] (encap)
Apr 10 16:09:34.695612 rule 1/(match) block in on sis0: 2001:4d48:1337:5afe::2 > 2001:470:1f05:3dd::1: [|icmp6] (encap)
Apr 10 16:09:35.696978 rule 1/(match) block in on sis0: 2001:4d48:1337:5afe::2 > 2001:470:1f05:3dd::1: [|icmp6] (encap)
Apr 10 16:09:36.696659 rule 1/(match) block in on sis0: 2001:4d48:1337:5afe::2 > 2001:470:1f05:3dd::1: [|icmp6] (encap)
Apr 10 16:09:37.696775 rule 1/(match) block in on sis0: 2001:4d48:1337:5afe::2 > 2001:470:1f05:3dd::1: [|icmp6] (encap)
Apr 10 16:11:03.484070 rule 1/(match) block in on sis0: 2a02:348:82:cb69::5 > 2001:470:1f05:3dd:12::1: [|icmp6] (encap)
Apr 10 16:11:04.493082 rule 1/(match) block in on sis0: 2a02:348:82:cb69::5 > 2001:470:1f05:3dd:12::1: [|icmp6] (encap)
Apr 10 16:11:05.493106 rule 1/(match) block in on sis0: 2a02:348:82:cb69::5 > 2001:470:1f05:3dd:12::1: [|icmp6] (encap)
Apr 10 16:11:06.493027 rule 1/(match) block in on sis0: 2a02:348:82:cb69::5 > 2001:470:1f05:3dd:12::1: [|icmp6] (encap)


You can see in the above:

• 3 passed packets: First I pinged my tunnel's client ipv6 address, then my router, then my server. All were "passed" thanks to the icmp6 rule I have
• A ton of blocked ping requests on my router's external interface

Using the above log, it looks like PF is blocking the ICMP packets while they're still encapsulated (before they even get to gif0). My next guess is going to try allowing all "proto encap" traffic to see if that works, but does anyone have any other thoughts? Why would the first 3 seemingly work fine, but all the others fail?

[kasperd]

icmp6 would only appear over inet6, so it's not really necessary

It is possible to send ICMP over IPv6 or ICMPv6 over IPv4 to your firewall. If that happens you really want the firewall to block it and not send any ICMP or ICMPv6 error back to the sender.

Packets where IP and ICMP versions don't match are not standards compliant. So dropping them and logging that fact is a sensible treatment.

Using the above log, it looks like PF is blocking the ICMP packets while they're still encapsulated (before they even get to gif0). My next guess is going to try allowing all "proto encap" traffic to see if that works, but does anyone have any other thoughts?

I don't know the specifics of that firewall implementation. But the most obvious implementation would have such a packet go through the firewall rules multiple times.

First pass will treat the tunnelled packet as an incoming IPv4 packet. At this stage you'd expect the rules to only look at the IPv4 header. The contents of the IPv6 header would not be inspected unless you enable some degree of deep-packet-inspection in the firewall. If the packet is permitted at the first pass it proceeds to the next pass.

The second pass will treat the packet as an incoming IPv6 packet, which is likely a packet intended to be forwarded through the firewall. At this stage the IPv4 header may no longer be available to the filtering rules, but an implementation could keep the encapsulation headers available at this stage, just like it could make MAC addresses available to IP filtering rules.

A packet going the other direction would first be treated as a forwarded IPv6 packet, and next as an outgoing IPv4 packet. If you were to forward between two tunnels, one packet could go through filtering rules three times. First as an incoming IPv4 packet, then as a forwarded IPv6 packet, finally as an outgoing IPv4 packet.

A firewall might like to allow for rules that correlate IPv4 and IPv6 addresses to allow rejecting packets with unacceptable combinations of addresses. This could be done with DPI on the IPv4 packet or by making IPv4 headers available to the IPv6 filtering, or both.

Why would the first 3 seemingly work fine, but all the others fail?

Sounds like rate limiting. Can you figure out the numbering of the rules? I don't see how rule numbers 1 and 14 match up with your config.

[arader]

icmp6 would only appear over inet6, so it's not really necessary

It is possible to send ICMP over IPv6 or ICMPv6 over IPv4 to your firewall. If that happens you really want the firewall to block it and not send any ICMP or ICMPv6 error back to the sender.

Packets where IP and ICMP versions don't match are not standards compliant. So dropping them and logging that fact is a sensible treatment.

Good point, I'll update my rules to only allow what makes sense.

First pass will treat the tunnelled packet as an incoming IPv4 packet. At this stage you'd expect the rules to only look at the IPv4 header. The contents of the IPv6 header would not be inspected unless you enable some degree of deep-packet-inspection in the firewall. If the packet is permitted at the first pass it proceeds to the next pass.

The second pass will treat the packet as an incoming IPv6 packet, which is likely a packet intended to be forwarded through the firewall. At this stage the IPv4 header may no longer be available to the filtering rules, but an implementation could keep the encapsulation headers available at this stage, just like it could make MAC addresses available to IP filtering rules.

A packet going the other direction would first be treated as a forwarded IPv6 packet, and next as an outgoing IPv4 packet. If you were to forward between two tunnels, one packet could go through filtering rules three times. First as an incoming IPv4 packet, then as a forwarded IPv6 packet, finally as an outgoing IPv4 packet.

Yep, after some experimentation this looks like what is happening. My firewall rules were blocking the IPv4 traffic so the IPv6 traffic never had a chance to be "allowed". The odd part still though is that why would my ICMP6 packets be allowed sometimes, but not others? My only guess here is that since I'm doing NAT on IPv4, some previous packet from the network was causing some router state to allow these packets. Thoughts?

Why would the first 3 seemingly work fine, but all the others fail?

Sounds like rate limiting. Can you figure out the numbering of the rules? I don't see how rule numbers 1 and 14 match up with your config.

Rule #1 is the block in log rule, which is my "default deny" rule. Rule #14 is the pass in proto ipv6-icmp. The reason they don't match up is the "antispoof" entry expands to multiple rules, throwing off the count. As above, I'm still confused as to why I would sometimes see successful pings before I set up a rule to allow the tunneled IPv6 packets in via IPv4?

So now I've added a rule that explicitly allows all traffic from the "tunnel server ipv4 address" from the tunnel config page. This appears to give me much greater odds of successfully pinging and connecting to my machines at home, but it's not 100%. Even worse than the ping issue though is that my ssh sessions routinely timeout...

[kasperd]

My only guess here is that since I'm doing NAT on IPv4, some previous packet from the network was causing some router state to allow these packets.

That sort of situation certainly happens sometimes. I am periodically using my laptop in places where I have to rely on such NAT behaviour to get an IPv6 tunnel out. For that explanation to match up with your log, the timeout would have to be less than 81 seconds. That's a quite small timeout for a NAT.

I have a script that can try to estimate the NAT timeout by pinging from my end. Do you want me to give it a shot at one of your IP addresses?

So now I've added a rule that explicitly allows all traffic from the "tunnel server ipv4 address" from the tunnel config page.

Is there really any security to be gained from restricting the source IP on that rule?

[arader]

My only guess here is that since I'm doing NAT on IPv4, some previous packet from the network was causing some router state to allow these packets.

That sort of situation certainly happens sometimes. I am periodically using my laptop in places where I have to rely on such NAT behaviour to get an IPv6 tunnel out. For that explanation to match up with your log, the timeout would have to be less than 81 seconds. That's a quite small timeout for a NAT.

I have a script that can try to estimate the NAT timeout by pinging from my end. Do you want me to give it a shot at one of your IP addresses?

I'd definitely be curious to see what the results are, yeah. Can you just try pinging 2001:470:1f05:3dd::1?

So now I've added a rule that explicitly allows all traffic from the "tunnel server ipv4 address" from the tunnel config page.

Is there really any security to be gained from restricting the source IP on that rule?

You're absolutely right, it isn't really secure at all. I tried limiting it only to traffic that was using the "encap" protocol (Protocol-41) but I still had issues pinging/connecting. What is the best solution then? how are people here usually accessing their network over the IPv6 tunnel? Or is this a pretty rare setup?

[kasperd]

I'd definitely be curious to see what the results are, yeah. Can you just try pinging 2001:470:1f05:3dd::1?

The script works by pinging once per second until it gets the first reply, then it performs exponential backoff until it no longer gets a reply. The idea is that the script will be pinging once per second until a packet from the inside opens a connection through the NAT. And at that point the script will notice and start backing off until there is a time out on the connection, at which point the script will no longer get replies.

I started it some minutes ago, and it got a reply on the very first ping it send to that IP. At this time it is waiting three minutes between two successive pings, and it still hasn't seen any loss of connectivity. I'll leave it running.

I tried limiting it only to traffic that was using the "encap" protocol (Protocol-41) but I still had issues pinging/connecting. What is the best solution then? how are people here usually accessing their network over the IPv6 tunnel? Or is this a pretty rare setup?

Accessing your own network from other locations is one of the cases, where you can benefit a lot from IPv6. So I don't think that is rare.

On my network I try to not filter protocol 41 at all. I permit incoming protocol 41 regardless of what source addresses it uses. I can filter the IPv6 packets once they have been decapsulated.

My reasoning behind that is that filtering which protocol 41 packets are permitted really only serves as a guard against spoofing of source IPv6 address, and a very ineffective guard at that, since there are so many other ways to spoof the source address at either the IPv4 or IPv6 layer. The best protection against source address spoofing is for higher layer protocols to actually verify that the party you are communicating with is able to receive packets you send to their IP address.

Additionally avoiding filtering of protocol 41 packets means it is easier to route packets on a more direct path. When I am somewhere else with my laptop, my laptop and my router at home will autodetect each other's IPv4 address to tunnel packets directly between them. That part I do however think is rare. I guess most users in that situation would have let the traffic between their laptop and their own home take the route through the tunnel server.

[kasperd]

I started it some minutes ago, and it got a reply on the very first ping it send to that IP. At this time it is waiting three minutes between two successive pings, and it still hasn't seen any loss of connectivity.

The script still didn't see any loss of connectivity to 2001:470:1f05:3dd::1.

[arader]

Thanks for your help (once again!) kasperd! Your mention of not filtering protocol-41 got me thinking. Why wasn't my firewall letting packets in after I added the "allow encapsulated traffic" rule? Well, it turns out the "encap" protocol is actually a protocol, but it's not protocol-41!

My confusion came from the fact that all of my pf logs show the tunneled traffic with they keyword 'encap' - I foolishly though that that was the correct protocol name for protocol-41. After reading your post I dug into /etc/protocols and found that 'encap' is actually protocol-98, the name for protocol-41 is... "ipv6", which is a bit confusing.

So my final solution: Added pass in inet proto ipv6 to pf.conf. This allows all protocol-41 traffic to pass through my external interface, which then comes in to my firewall a second time as ipv6 traffic via the tunnel interface gif0. I'm now able to ping my various IPv6 hosts without issue, and SSH connections work just fine.

thanks!

[plugwash]

My reasoning behind that is that filtering which protocol 41 packets are permitted really only serves as a guard against spoofing of source IPv6 address, and a very ineffective guard at that, since there are so many other ways to spoof the source address at either the IPv4 or IPv6 layer. The best protection against source address spoofing is for higher layer protocols to actually verify that the party you are communicating with is able to receive packets you send to their IP address.

IMO just because we can't totally prevent packets with spoofed source addresses from arriving (source address filtering in the core of the internet is impractical and unfortunately some ISPs fail to implement source filtering at their end user boundaries) doesn't mean we should introduce new ways to inject them. Sending packets with spoofed source addresses to certain services can be used to singificant amplify DDoS attacks.

[kasperd]

IMO just because we can't totally prevent packets with spoofed source addresses from arriving (source address filtering in the core of the internet is impractical and unfortunately some ISPs fail to implement source filtering at their end user boundaries) doesn't mean we should introduce new ways to inject them.

If you can be 100% sure your filter is not going to drop legitimate packets, I would agree, it should be applied. But if there is a risk of filtering legitimate packets, is it really worth applying such a filter, given how easy it would be to bypass it?

Sending packets with spoofed source addresses to certain services can be used to singificant amplify DDoS attacks.

DNS is the only widely deployed service, I know of with this property. Sadly EDNS + DNSSEC makes the problem worse rather than better. A minor protocol change could solve that problem. It may not be easy to get such a fix deployed, but I still think it is more feasible to get a fix deployed in DNS than it is to get rid of all the means by which source IP can be spoofed. Given that DNSSEC is not widely deployed yet, trying to get it fixed may be worthwhile.