- Welcome to Hurricane Electric's IPv6 Tunnel Broker Forums.
News:
Welcome to Hurricane Electric's Tunnelbroker.net forums!
Recent posts
#71
Questions & Answers / Re: Google forcing ReCAPTCHA o...
Last post by Jenick - June 23, 2023, 10:55:18 AMQuote from: cshilton on June 23, 2023, 07:12:32 AMQuote from: Jenick on June 22, 2023, 09:21:55 PMQuote from: cshilton on June 22, 2023, 09:08:56 AM...snip...
I'd bet dollars to donuts that some a**hat has found a way to abuse the tunnels again and is causing problems for google and this is just a response. This was the case with Netflix and it's also the problem with Wikipedia. As such, I think that we are going to be dealing with problems like this until IPv6 is much more relevant. Right now, kick/banning an entire ISP is an easy way to deal with the abuse if you look at it from the application provider's side.
I'm on pfSense Plus 23.05 latest release with the latest patch set. There's an option in pfBlockerNG under DNSBL Python mode to disable AAAA lookups against a list right there in the UI. I just spotted it, enabled it, and added www.google.com and google.com to the list. Got google access back now, not that we use it much anyhow but we do sometimes go there when duckduckgo isn't giving enough relevant results. Hope this helps some pfSense users. I'm not sure if the pfSense CE 2.6.0 version has this option since that version used to require the pfBlockerNG-devel branch package to get the extra features. It's easy enough to upgrade to 23.05 Plus from CE 2.6.0 anyhow for personal use. Even with a work-around place, HE.net has to continue the pushback on Google to STOP doing this crap!
OpenBSD is distantly related to, and more primitive than pfSense and opnSense. It's like the OS from those products without the web based UI. Your solutions is much simpler than mine, would you mind posting it to the general solutions thread - https://forums.he.net/index.php?topic=4263.0 so that other people only have to look in one place to find it?
I'll do that but have to test a few more computers and devices with the change in place. It seems to work only if you implement the chromium/firefox tweaks to disable the browser's built in DNS client DOH/DOT. Our iPhones/iPads are still getting the block error even with the pfSense pfBlockerNG changes in place because they're likely resolving via their own built-in DNS client implementations. This is so frustrating!
Yeah, I recall attempting OpenBSD about 15 years ago, had some luck getting it up and running but ended up going with pfSense when it first came out and just stuck with that. I'm mainly a Linux/Windows/VMware/Cisco guy. I can manage my way around the FreeBSD cli side of pfSense but only go there as needed. So, more frustration, I just rebooted my iPhone, then hit google.com no issues inside the duckduckgo browser app. As soon as I tried hitting it in the Safari browser I got kicked again. So iOS is caching the IPv6 from the Safari attempt and then I get kicked from duckduckgo app on the next attempt. This is ridiculous! Here we are downgrading security, kludging work-arounds, etc. all to visit a search engine that's become way too involved in our private lives. I'll still post the work-arounds I put in place likely over the weekend.
#72
Questions & Answers / Re: Unable to use Google Searc...
Last post by snarked - June 23, 2023, 09:48:29 AMGoogle doesn't play well with others.
As noted, Google sees a lot of activity from HE tunnels; so much that it thinks there are web spiders from HE (and there may be). When it thinks there's a spider (or other automated program due to "high volume"), it throws the captcha at it. Continued high volume then yields the 403 response. Google doesn't care whether this blocks similar IP addresses or not. This is not a 2023 problem. It's happened before.
Other google issues they won't change (examples):
1) Googlebot will make requests with bad HTTP header data, including invalid mailbox syntax in the optional "From:" header and for nonexistent Google-site-verification keys. The latter is OK, intentionally seeking a 404 response. The "From:" header problem started in 2018. I have even complained to their representatives at CES in person in 2019, yet they won't fix it.
2) Gmail sometimes generates invalid "Received:" email headers for messages claiming SMTP compliance by omitting the required "from" clause. This is usually for messages originating via Gmail.
Deviations from Internet standards is bad, but Google thinks it's large enough that it may do what it wants, and that's the real problem. HE may not be a large-enough player to be able to fight back.
As noted, Google sees a lot of activity from HE tunnels; so much that it thinks there are web spiders from HE (and there may be). When it thinks there's a spider (or other automated program due to "high volume"), it throws the captcha at it. Continued high volume then yields the 403 response. Google doesn't care whether this blocks similar IP addresses or not. This is not a 2023 problem. It's happened before.
Other google issues they won't change (examples):
1) Googlebot will make requests with bad HTTP header data, including invalid mailbox syntax in the optional "From:" header and for nonexistent Google-site-verification keys. The latter is OK, intentionally seeking a 404 response. The "From:" header problem started in 2018. I have even complained to their representatives at CES in person in 2019, yet they won't fix it.
2) Gmail sometimes generates invalid "Received:" email headers for messages claiming SMTP compliance by omitting the required "from" clause. This is usually for messages originating via Gmail.
Deviations from Internet standards is bad, but Google thinks it's large enough that it may do what it wants, and that's the real problem. HE may not be a large-enough player to be able to fight back.
#73
Questions & Answers / Re: Google forcing ReCAPTCHA o...
Last post by cshilton - June 23, 2023, 07:12:32 AMQuote from: Jenick on June 22, 2023, 09:21:55 PMQuote from: cshilton on June 22, 2023, 09:08:56 AM...snip...
I'd bet dollars to donuts that some a**hat has found a way to abuse the tunnels again and is causing problems for google and this is just a response. This was the case with Netflix and it's also the problem with Wikipedia. As such, I think that we are going to be dealing with problems like this until IPv6 is much more relevant. Right now, kick/banning an entire ISP is an easy way to deal with the abuse if you look at it from the application provider's side.
I'm on pfSense Plus 23.05 latest release with the latest patch set. There's an option in pfBlockerNG under DNSBL Python mode to disable AAAA lookups against a list right there in the UI. I just spotted it, enabled it, and added www.google.com and google.com to the list. Got google access back now, not that we use it much anyhow but we do sometimes go there when duckduckgo isn't giving enough relevant results. Hope this helps some pfSense users. I'm not sure if the pfSense CE 2.6.0 version has this option since that version used to require the pfBlockerNG-devel branch package to get the extra features. It's easy enough to upgrade to 23.05 Plus from CE 2.6.0 anyhow for personal use. Even with a work-around place, HE.net has to continue the pushback on Google to STOP doing this crap!
OpenBSD is distantly related to, and more primitive than pfSense and opnSense. It's like the OS from those products without the web based UI. Your solutions is much simpler than mine, would you mind posting it to the general solutions thread - https://forums.he.net/index.php?topic=4263.0 so that other people only have to look in one place to find it?
#74
Questions & Answers / Re: Google forcing ReCAPTCHA o...
Last post by Jenick - June 22, 2023, 09:21:55 PMQuote from: cshilton on June 22, 2023, 09:08:56 AMI'm not sure what everyone is using for a firewall. I'm using OpenBSD. For the purposes of this discussion it might as well be a variant of pfSense / opnSense.. When I had to deal with the netflix problem, I went with defense-in-depth and stopped quad-A searches for netflix's domains. As a second layer, I also added all of their IP space relative to me to a table. I block new connections at my edge to addresses in the table with a TCP RST / UDP port unreachable. The pros and cons are: pro - this plays well with happy eyeballs; the target never sees traffic from HE's ASN coming from your tunnelbroker space; cons - the addresses that you are blocking are dynamic so you need to have a mechanism to keep them up to date; you also may be blocking content temporarily that you don't need to block. I added manually added www.google.com to this space when this first started happening since I hoped that the problem would get resolved. Apparently, it hasn't so I'll have to automate this process.
I'd bet dollars to donuts that some a**hat has found a way to abuse the tunnels again and is causing problems for google and this is just a response. This was the case with Netflix and it's also the problem with Wikipedia. As such, I think that we are going to be dealing with problems like this until IPv6 is much more relevant. Right now, kick/banning an entire ISP is an easy way to deal with the abuse if you look at it from the application provider's side.
I'm on pfSense Plus 23.05 latest release with the latest patch set. There's an option in pfBlockerNG under DNSBL Python mode to disable AAAA lookups against a list right there in the UI. I just spotted it, enabled it, and added www.google.com and google.com to the list. Got google access back now, not that we use it much anyhow but we do sometimes go there when duckduckgo isn't giving enough relevant results. Hope this helps some pfSense users. I'm not sure if the pfSense CE 2.6.0 version has this option since that version used to require the pfBlockerNG-devel branch package to get the extra features. It's easy enough to upgrade to 23.05 Plus from CE 2.6.0 anyhow for personal use. Even with a work-around place, HE.net has to continue the pushback on Google to STOP doing this crap!
#75
Questions & Answers / General solutions to the curre...
Last post by cshilton - June 22, 2023, 01:59:20 PMI solved this with straight up boring packet filtering. My firewall is pf under OpenBSD so my solution should work under any firewall built against a recent pf with possibly minor syntax changes. The theory holds for any firewall that allows you match IP addresses against the values representing hosts and networks stored in a table. The pf firewall works top down so your block rules are usually at the top of your config.
The goal of the rule at the bottom is to prevent your clients from ever using IPv6 to reach hosts or networks hashed into the table no_ipv6_for_us. Those packets should be dropped with a TCP RESET. This should cause Happy Eyeballs at your client to conclude that whatever site you are addressing is currently IPv4 only. There are lots of ways to solve this problem. My network is somewhat large and I don't feel like running around to each client turning off IPv6 or changing the protocol preference from IPv6 to IPv4. For me, this seemed to be the best way to solve this problem.
The shell script above uses dig to find out what the current IP address of www.google.com where you are in the world. I'm assuming that Google is using different IP addresses in different places for the service. They may be doing tricks with routing to get you to closest server as a function of your physical location. I assumed that /48 is a good guess for the network size based on what I've seen in my DevOps career.
I still consider this to be problem that's going to get solved unlike Netflix and Wikipedia. I'm guessing that there's a bad actor in the wild and that she's figured out a way to influence Google's search results by flooding Google with a bunch of queries from different IP addresses. The bad actor has figured out that he can grab a new /64 or /48 somehow, possibly using the tunnelbroker api, and then spam Google from the resulting IP space. As with Netflix, HE will figure out a way to stop the abuse.
-- Chris
Code Select
## General firewall policy
block drop log all
...
pass on $v6_ext_if proto icmp6
...
table <no_ipv6_for_us> persist
...
block return in log quick on $int_if inet6 proto {tcp,udp} from ($int_if:network) to <no_ipv6_for_us>
The goal of the rule at the bottom is to prevent your clients from ever using IPv6 to reach hosts or networks hashed into the table no_ipv6_for_us. Those packets should be dropped with a TCP RESET. This should cause Happy Eyeballs at your client to conclude that whatever site you are addressing is currently IPv4 only. There are lots of ways to solve this problem. My network is somewhat large and I don't feel like running around to each client turning off IPv6 or changing the protocol preference from IPv6 to IPv4. For me, this seemed to be the best way to solve this problem.
Code Select
$ dig +short AAAA www.google.com
2607:f8b0:4006:dead::beef
$ sudo pfctl -t no_ipv6_for_us -T add 2607:f8b0:4006::/48
1/1 addresses added.
$ sudo pfctl -k 2607:f8b0:4006::/48
killed 0 states from 1 sources and 0 destinations
The shell script above uses dig to find out what the current IP address of www.google.com where you are in the world. I'm assuming that Google is using different IP addresses in different places for the service. They may be doing tricks with routing to get you to closest server as a function of your physical location. I assumed that /48 is a good guess for the network size based on what I've seen in my DevOps career.
I still consider this to be problem that's going to get solved unlike Netflix and Wikipedia. I'm guessing that there's a bad actor in the wild and that she's figured out a way to influence Google's search results by flooding Google with a bunch of queries from different IP addresses. The bad actor has figured out that he can grab a new /64 or /48 somehow, possibly using the tunnelbroker api, and then spam Google from the resulting IP space. As with Netflix, HE will figure out a way to stop the abuse.
-- Chris
#76
Questions & Answers / Re: abuse warning but tunnel w...
Last post by aboaboit - June 22, 2023, 10:35:59 AMWell, actually, other than this warning I have no issues at all.
Even the warning isn't actually causing problems, at least not as far as I can tell!
Even the warning isn't actually causing problems, at least not as far as I can tell!
#77
Questions & Answers / Re: Google forcing ReCAPTCHA o...
Last post by doktornotor - June 22, 2023, 10:28:44 AMQuote from: cshilton on June 22, 2023, 09:08:56 AMRight now, kick/banning an entire ISP is an easy way to deal with the abuse if you look at it from the application provider's side.
Right now, Google has got pretty much a monopoly in the search engines market. Similar behavior is completely unacceptable. Apparently, they are asking for more trouble in the EU.
#78
Questions & Answers / Re: Unable to use Google Searc...
Last post by cshilton - June 22, 2023, 09:24:04 AMThis has been an ongoing problem for a few weeks. At the start, www.google.com forced you through a CAPTCHA to use the site. Now they are 403 / blocking things. It's a good bet that there's some a**hat on the internet who is using HE.net tunnelbroker IP space to abuse www.google.com in some way. Once you pass the tests, you get a tunnel and you can get a new tunnel with new IP space through the api. Someone could run many kinds of attack through that and they could change their IP address nearly at will through the API while doing it.
I started a thread on this three or four weeks ago: "Google forcing ReCAPTCHA..." This issue responds well to the solutions for the Netflix problem, basically change your edge router so you don't go to www.google.com over your HE.net IPv6 tunnel. I'm not sure what you are using for a router but your solution probably lies there.
I started a thread on this three or four weeks ago: "Google forcing ReCAPTCHA..." This issue responds well to the solutions for the Netflix problem, basically change your edge router so you don't go to www.google.com over your HE.net IPv6 tunnel. I'm not sure what you are using for a router but your solution probably lies there.
#79
Questions & Answers / Re: Google forcing ReCAPTCHA o...
Last post by cshilton - June 22, 2023, 09:08:56 AMI'm not sure what everyone is using for a firewall. I'm using OpenBSD. For the purposes of this discussion it might as well be a variant of pfSense / opnSense.. When I had to deal with the netflix problem, I went with defense-in-depth and stopped quad-A searches for netflix's domains. As a second layer, I also added all of their IP space relative to me to a table. I block new connections at my edge to addresses in the table with a TCP RST / UDP port unreachable. The pros and cons are: pro - this plays well with happy eyeballs; the target never sees traffic from HE's ASN coming from your tunnelbroker space; cons - the addresses that you are blocking are dynamic so you need to have a mechanism to keep them up to date; you also may be blocking content temporarily that you don't need to block. I added manually added www.google.com to this space when this first started happening since I hoped that the problem would get resolved. Apparently, it hasn't so I'll have to automate this process.
I'd bet dollars to donuts that some a**hat has found a way to abuse the tunnels again and is causing problems for google and this is just a response. This was the case with Netflix and it's also the problem with Wikipedia. As such, I think that we are going to be dealing with problems like this until IPv6 is much more relevant. Right now, kick/banning an entire ISP is an easy way to deal with the abuse if you look at it from the application provider's side.
I'd bet dollars to donuts that some a**hat has found a way to abuse the tunnels again and is causing problems for google and this is just a response. This was the case with Netflix and it's also the problem with Wikipedia. As such, I think that we are going to be dealing with problems like this until IPv6 is much more relevant. Right now, kick/banning an entire ISP is an easy way to deal with the abuse if you look at it from the application provider's side.
#80
Questions & Answers / Unable to use Google Search
Last post by nelgin - June 22, 2023, 08:20:06 AMI've had to switch my systems to prefer ipv4 because I'm getting an error accessing Google with ipv6. On my cellphone I get
403. That's an error.
Your client does not have permissions to get URL /search?..... from this server. That's all we know.
HE need to sort this out in a hurry since it seems there's no way to prefer ipv4 over ipv6 on Android (tho I'm willing to be educated).
403. That's an error.
Your client does not have permissions to get URL /search?..... from this server. That's all we know.
HE need to sort this out in a hurry since it seems there's no way to prefer ipv4 over ipv6 on Android (tho I'm willing to be educated).