Hurricane Electric's IPv6 Tunnel Broker Forums

General IPv6 Topics => IPv6 on Linux & BSD & Mac => Topic started by: bottswana on January 31, 2011, 12:54:11 PM

Title: Apache & IPv6
Post by: bottswana on January 31, 2011, 12:54:11 PM
Hey guys, having a bit of a nightmare here
The server Ip seems to be completely accessable. I can ping it from both sides of the tunnel, however, apache dosent like it.
THere are no errors when it starts up, and it works fine over IPv4, however, when you try it on IPv6, the page never loads. It establishes a connection, and then just hangs on "waiting" for an indefinate period of time.
Apache's error logs report nothing, and the access log reports a code 200 for every connection (All OK)
Can't understand why it never loads anything?
DNS works fine, the Quad-A records are resolving as normal, so it cant be anything wrong with the IPv6 configuration?
Any Apache geniuses know whats going on here?
Title: Re: Apache & IPv6
Post by: cholzhauer on January 31, 2011, 12:55:16 PM
Need more information...what's your domain?

Is Apache actually listening on IPv6?  what's "netstat -a" show?
Title: Re: Apache & IPv6
Post by: bottswana on January 31, 2011, 01:38:45 PM
Yes, apache is binded and is listening on the port.
Shutting down the apache service stops the "waiting for" issue and it fails on connecting, which is logical.

v4: 81.138.181.108
v6: [2001:470:95bb::2]

Netstat output:
Code: [Select]
root@marge:~# netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 *:29000                 *:*                     LISTEN
tcp        0      0 *:mysql                 *:*                     LISTEN
tcp        0      0 *:submission            *:*                     LISTEN
tcp        0      0 *:imap2                 *:*                     LISTEN
tcp        0      0 host81-138-181-1:webmin *:*                     LISTEN
tcp        0      0 host81-138-181-108.:www *:*                     LISTEN
tcp        0      0 192.168.10.2:domain     *:*                     LISTEN
tcp        0      0 host81-138-181-1:domain *:*                     LISTEN
tcp        0      0 localhost:domain        *:*                     LISTEN
tcp        0      0 *:ssh                   *:*                     LISTEN
tcp        0      0 *:19000                 *:*                     LISTEN
tcp        0      0 localhost:11000         *:*                     LISTEN
tcp        0      0 *:smtp                  *:*                     LISTEN
tcp        0      0 localhost:953           *:*                     LISTEN
tcp        0      0 host81-138-181-10:https *:*                     LISTEN
tcp        0      0 host81-138-181-10:20000 *:*                     LISTEN
tcp        0      0 *:imaps                 *:*                     LISTEN
tcp        0      1 host81-138-181-10:51104 clamav.swishmail.co:www SYN_SENT
tcp        0     52 host81-138-181-108.:ssh cpc1-lanc6-2-0-cu:49506 ESTABLISHED
tcp6       0      0 marge.bottswanam:webmin [::]:*                  LISTEN
tcp6       0      0 marge.bottswanamedi:www [::]:*                  LISTEN
tcp6       0      0 [::]:ftp                [::]:*                  LISTEN
tcp6       0      0 [::]:domain             [::]:*                  LISTEN
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN
tcp6       0      0 localhost:953           [::]:*                  LISTEN
tcp6       0      0 marge.bottswaname:https [::]:*                  LISTEN
tcp6       0      0 marge.bottswaname:20000 [::]:*                  LISTEN
udp        0      0 *:27500                 *:*
udp        0      0 192.168.10.2:domain     *:*
udp        0      0 host81-138-181-1:domain *:*
udp        0      0 localhost:domain        *:*
udp6       0      0 [::]:domain             [::]:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     4841     private/bsmtp
unix  2      [ ACC ]     STREAM     LISTENING     4845     private/scalemail-backend
unix  2      [ ACC ]     STREAM     LISTENING     4849     private/mailman
unix  2      [ ACC ]     STREAM     LISTENING     4749     private/tlsmgr

netstat -tulpn | grep :80
Code: [Select]
root@marge:~# netstat -tulpn | grep :80
tcp        0      0 81.138.181.108:80       0.0.0.0:*               LISTEN      1316/apache2
tcp6       0      0 2001:470:95bb::2:80     :::*                    LISTEN      1316/apache2
Title: Re: Apache & IPv6
Post by: cconn on January 31, 2011, 01:57:55 PM
ip6tables is blocking your HTTP port?
Title: Re: Apache & IPv6
Post by: bottswana on January 31, 2011, 02:00:17 PM
If it was a firewall issue, then surely it would not connect in the first place?
Plus an IPv6 port scan shows port 80 as open.
Title: Re: Apache & IPv6
Post by: jrocha on January 31, 2011, 04:01:34 PM
What does your virtualhost config look like? What does your ip6tables config look like? What happens when you telnet to port 80 using the ipv6 address?
Title: Re: Apache & IPv6
Post by: bottswana on February 01, 2011, 02:29:17 PM
Telnetting in results in a black screen in which you can do nothing (at least, I don't know what to do with it).
It didnt display any errors though.

ip6tables: (This is probably where the problem is. I hate iptables!)
Code: [Select]

# Generated by ip6tables-save v1.4.4 on Tue Feb  1 22:57:59 2011
*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [27:2807]
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 19000:19010 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 29000 -j ACCEPT
-A INPUT -p tcp -m tcp -m multiport --dports 20,21 -j ACCEPT
-A INPUT -p tcp -m tcp -m multiport --dports 10000,20000 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 5000:5100 -j ACCEPT
-A INPUT -p tcp -m tcp -m multiport --dports 993,25,3306 -j ACCEPT
-A INPUT -s ::1/128 -j ACCEPT
-A INPUT -d ::1/128 -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Tue Feb  1 22:57:59 2011
# Generated by ip6tables-save v1.4.4 on Tue Feb  1 22:57:59 2011
*mangle
:PREROUTING ACCEPT [19:2299]
:INPUT ACCEPT [19:2299]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [27:2807]
:POSTROUTING ACCEPT [27:2807]
COMMIT
# Completed on Tue Feb  1 22:57:59 2011


Virtual Servers:
Code: [Select]

NameVirtualHost 81.138.181.108:80
NameVirtualHost [2001:470:95bb::2]:80

<VirtualHost [2001:470:95bb::2]:80>
ServerAdmin webmaster@localhost

DocumentRoot /var/www
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>

ErrorLog /var/log/apache2/error.log
LogLevel warn
CustomLog /var/log/apache2/access.log combined

</VirtualHost>

<VirtualHost 81.138.181.108:80>
ServerAdmin webmaster@localhost

DocumentRoot /var/www
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>

ErrorLog /var/log/apache2/error.log
LogLevel warn
CustomLog /var/log/apache2/access.log combined

</VirtualHost>

Edit: Added new Ip6tables config to reflect icmp > icmpv6
Title: Re: Apache & IPv6
Post by: broquea on February 01, 2011, 02:33:04 PM
Can't seem to ping6 or reach that host, not even from the tunnel-server itself:

Code: [Select]
~$ telnet 2001:470:95bb::2 80
Trying 2001:470:95bb::2...
telnet: Unable to connect to remote host: No route to host

--- 2001:470:95bb::2 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3017ms

--- 2001:470:1f08:13b3::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms

I've confirmed the static route is in place on our side.
Title: Re: Apache & IPv6
Post by: bottswana on February 01, 2011, 02:37:50 PM
Very, very strange. Just used an online IPv6 ping utility and it reports the same thing. Can't be my router because I can ping other hosts in my /48 prefix, and I can ping it from within the prefix.
I will check my routing

Edit: No, that was me blocking all ICMP requests with Ip6tables. Fixed now :/

Ping6 to google from troublesome server:
Code: [Select]
root@marge:~# ping6 2a00:1450:8006::93
PING 2a00:1450:8006::93(2a00:1450:8006::93) 56 data bytes
64 bytes from 2a00:1450:8006::93: icmp_seq=1 ttl=56 time=36.4 ms
64 bytes from 2a00:1450:8006::93: icmp_seq=2 ttl=56 time=23.1 ms
64 bytes from 2a00:1450:8006::93: icmp_seq=3 ttl=56 time=24.5 ms
64 bytes from 2a00:1450:8006::93: icmp_seq=4 ttl=56 time=25.1 ms
^C
--- 2a00:1450:8006::93 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 23.141/27.294/36.421/5.317 ms

So, after spending a while tweaking ip6tables, apache is still displaying a blank white page. Running out of ideas here :(
Title: Re: Apache & IPv6
Post by: schvin on February 01, 2011, 06:31:47 PM
do your apache access logs show the v6 clients? just curious if by any chance something else is intercepting the traffic.

i can open a socket to it, but it doesn't appear to respond to http requests, just closes the questions. if it shows up in the apache logs then at least it is something within apache (though your apache config looks sane to me).

alternately you could open a v6 connection from somewhere else (i.e. printf "GET / HTTP/1.0\r\nHost: bottswanamedia.info\r\n\r\n" | telnet 2001:470:95bb::2 80), or, even just the telnet portion and let it hang, will give you some more time, and see if you see a netstat connection from the client.
Title: Re: Apache & IPv6
Post by: bottswana on February 02, 2011, 09:50:06 AM
So, I switched logs mode from errors to 'debug' and loaded up the site again. This is the only line that is output:

Code: [Select]
Wed Feb 02 17:43:42 2011] [debug] mod_deflate.c(615): [client 2001:470:95bb::4] Zlib: Compressed 177 to 128 : URL /index.html

access.log outputs:
Code: [Select]
2001:470:95bb::4 - - [02/Feb/2011:17:43:42 +0000] "GET / HTTP/1.1" 200 485 "-" (Loads of browser info)

Still not getting any joy on page loads though.

using netstat when trying to connect does show a connection:

Code: [Select]
tcp6       0    485 2001:470:95bb::2:80     2001:470:95bb::4:1028   ESTABLISHED 14042/apache2
Title: Re: Apache & IPv6
Post by: schvin on February 02, 2011, 01:06:33 PM
Ah, the gzipping definitely makes it hard to debug w/ telnet. Can you try disabling the mod_deflate as a test. Any other modules of relevance that it might be hanging on (ssl, etc.)?
Title: Re: Apache & IPv6
Post by: bottswana on February 02, 2011, 02:51:16 PM
I disabled that deflate module. Theres quite a few modules loaded as its set  up as a shared hosting environment with Virtualmin.

Heres the module output from virtualmin:
Code: [Select]
actions
alias
auth_basic
auth_digest
authn_file
authz_default
authz_groupfile
authz_host
authz_user
autoindex
cgi
dav
dav_fs
dav_svn
dir
env
fcgid
mime
negotiation
php5
proxy
proxy_balancer
proxy_connect
proxy_html
proxy_http
reqtimeout
rewrite
ruby
setenvif
ssl
status
suexec
Title: Re: Apache & IPv6
Post by: schvin on February 02, 2011, 03:12:52 PM
got it. connection refused currently, but guessing you are working on the system :)

it's not expecting a proxy backend or anything for that vhost, right?
Title: Re: Apache & IPv6
Post by: bottswana on February 02, 2011, 03:17:20 PM
I tried disabling every module I could but still got the same result.
That included disabling all the proxy modules.

(Servers back as it was now)
Title: Re: Apache & IPv6
Post by: schvin on February 02, 2011, 03:24:05 PM
Really weird. So ICMP6 both ways are fine. Just to check, can you get ssh with 6 to work on the server, and can you go to an http site with 6? Just  to ensure that you have tcp connectivity both ways working OK, and it really is an Apache issue?

Just as a test, I tried sshing to the same IP and it hangs as well, which implies that maybe it is something upstream, like packet size getting hung up. Can you sniff the connection upstream and see if maybe some icmp6 responses aren't making it all the way downstream, like packet-too-large, or something like that?

Another thought is that perhaps apache & sshd are trying to query DNS for my inbound ip and failing badly, but that should give up eventually and let it through...
Title: Re: Apache & IPv6
Post by: bottswana on February 02, 2011, 04:40:04 PM
Heres a thought. It seems it isnt apache

This is my web control panel (Not run by apache)
It does exactly the same thing. Chance of that happening is quite slim as it is a completely different web serving software.

http://[2001:470:95bb::2]:19000

How bizarre.

It seems ICMP traffic is getting through, but not web traffic, for example, one of the update servers ubuntu hits has IPv6.
Updates from that server fail.
Yet a WinServer2003 machine on the same netowkr (With the same routing) accesses all IPv6 netowrking fine
There is something odd going on here...
Title: Re: Apache & IPv6
Post by: schvin on February 02, 2011, 08:01:00 PM
i'd try sniffing it upstream to see if any icmp6 messages getting back to the box are not getting through...
Title: Re: Apache & IPv6
Post by: bottswana on February 03, 2011, 05:45:19 AM
Any recommendations on how to do that via ssh? :)
Title: Re: Apache & IPv6
Post by: schvin on February 03, 2011, 09:35:48 AM
sure, just tcpdump. you can write it to a file and open that file somewhere else with wireshark, if you prefer the visual approach.

to monitor ipv6 traffic on interface eth0:

tcpdump -l -n -i eth0 -s 8192 ip6

to write it to a file rather than displaying it on screen, add "-w filename" before the "ip6" portion. the "ip6" portion is just a match, you can do lots of stuff with the matching to cut up the traffic as you see fit, if you like.
Title: Re: Apache & IPv6
Post by: bombcar on February 17, 2011, 06:20:51 PM
Shut down apache. Run netcat on port 80 (google it if needed).

See if you can connect.

If you can, it's something in Apache that's making it mad. Virtualmin by default doesn't listen on IPv6; so you may have to poke it a bit.