Hey guys, having a bit of a nightmare here
The server Ip seems to be completely accessable. I can ping it from both sides of the tunnel, however, apache dosent like it.
THere are no errors when it starts up, and it works fine over IPv4, however, when you try it on IPv6, the page never loads. It establishes a connection, and then just hangs on "waiting" for an indefinate period of time.
Apache's error logs report nothing, and the access log reports a code 200 for every connection (All OK)
Can't understand why it never loads anything?
DNS works fine, the Quad-A records are resolving as normal, so it cant be anything wrong with the IPv6 configuration?
Any Apache geniuses know whats going on here?
Need more information...what's your domain?
Is Apache actually listening on IPv6? what's "netstat -a" show?
Yes, apache is binded and is listening on the port.
Shutting down the apache service stops the "waiting for" issue and it fails on connecting, which is logical.
v4: 81.138.181.108
v6: [2001:470:95bb::2]
Netstat output:
root@marge:~# netstat -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:29000 *:* LISTEN
tcp 0 0 *:mysql *:* LISTEN
tcp 0 0 *:submission *:* LISTEN
tcp 0 0 *:imap2 *:* LISTEN
tcp 0 0 host81-138-181-1:webmin *:* LISTEN
tcp 0 0 host81-138-181-108.:www *:* LISTEN
tcp 0 0 192.168.10.2:domain *:* LISTEN
tcp 0 0 host81-138-181-1:domain *:* LISTEN
tcp 0 0 localhost:domain *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
tcp 0 0 *:19000 *:* LISTEN
tcp 0 0 localhost:11000 *:* LISTEN
tcp 0 0 *:smtp *:* LISTEN
tcp 0 0 localhost:953 *:* LISTEN
tcp 0 0 host81-138-181-10:https *:* LISTEN
tcp 0 0 host81-138-181-10:20000 *:* LISTEN
tcp 0 0 *:imaps *:* LISTEN
tcp 0 1 host81-138-181-10:51104 clamav.swishmail.co:www SYN_SENT
tcp 0 52 host81-138-181-108.:ssh cpc1-lanc6-2-0-cu:49506 ESTABLISHED
tcp6 0 0 marge.bottswanam:webmin [::]:* LISTEN
tcp6 0 0 marge.bottswanamedi:www [::]:* LISTEN
tcp6 0 0 [::]:ftp [::]:* LISTEN
tcp6 0 0 [::]:domain [::]:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN
tcp6 0 0 localhost:953 [::]:* LISTEN
tcp6 0 0 marge.bottswaname:https [::]:* LISTEN
tcp6 0 0 marge.bottswaname:20000 [::]:* LISTEN
udp 0 0 *:27500 *:*
udp 0 0 192.168.10.2:domain *:*
udp 0 0 host81-138-181-1:domain *:*
udp 0 0 localhost:domain *:*
udp6 0 0 [::]:domain [::]:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 4841 private/bsmtp
unix 2 [ ACC ] STREAM LISTENING 4845 private/scalemail-backend
unix 2 [ ACC ] STREAM LISTENING 4849 private/mailman
unix 2 [ ACC ] STREAM LISTENING 4749 private/tlsmgr
netstat -tulpn | grep :80
root@marge:~# netstat -tulpn | grep :80
tcp 0 0 81.138.181.108:80 0.0.0.0:* LISTEN 1316/apache2
tcp6 0 0 2001:470:95bb::2:80 :::* LISTEN 1316/apache2
ip6tables is blocking your HTTP port?
If it was a firewall issue, then surely it would not connect in the first place?
Plus an IPv6 port scan shows port 80 as open.
What does your virtualhost config look like? What does your ip6tables config look like? What happens when you telnet to port 80 using the ipv6 address?
Telnetting in results in a black screen in which you can do nothing (at least, I don't know what to do with it).
It didnt display any errors though.
ip6tables: (This is probably where the problem is. I hate iptables!)
# Generated by ip6tables-save v1.4.4 on Tue Feb 1 22:57:59 2011
*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [27:2807]
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 19000:19010 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 29000 -j ACCEPT
-A INPUT -p tcp -m tcp -m multiport --dports 20,21 -j ACCEPT
-A INPUT -p tcp -m tcp -m multiport --dports 10000,20000 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 5000:5100 -j ACCEPT
-A INPUT -p tcp -m tcp -m multiport --dports 993,25,3306 -j ACCEPT
-A INPUT -s ::1/128 -j ACCEPT
-A INPUT -d ::1/128 -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Tue Feb 1 22:57:59 2011
# Generated by ip6tables-save v1.4.4 on Tue Feb 1 22:57:59 2011
*mangle
:PREROUTING ACCEPT [19:2299]
:INPUT ACCEPT [19:2299]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [27:2807]
:POSTROUTING ACCEPT [27:2807]
COMMIT
# Completed on Tue Feb 1 22:57:59 2011
Virtual Servers:
NameVirtualHost 81.138.181.108:80
NameVirtualHost [2001:470:95bb::2]:80
<VirtualHost [2001:470:95bb::2]:80>
ServerAdmin webmaster@localhost
DocumentRoot /var/www
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>
ErrorLog /var/log/apache2/error.log
LogLevel warn
CustomLog /var/log/apache2/access.log combined
</VirtualHost>
<VirtualHost 81.138.181.108:80>
ServerAdmin webmaster@localhost
DocumentRoot /var/www
<Directory />
Options FollowSymLinks
AllowOverride None
</Directory>
ErrorLog /var/log/apache2/error.log
LogLevel warn
CustomLog /var/log/apache2/access.log combined
</VirtualHost>
Edit: Added new Ip6tables config to reflect icmp > icmpv6
Can't seem to ping6 or reach that host, not even from the tunnel-server itself:
~$ telnet 2001:470:95bb::2 80
Trying 2001:470:95bb::2...
telnet: Unable to connect to remote host: No route to host
--- 2001:470:95bb::2 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3017ms
--- 2001:470:1f08:13b3::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
I've confirmed the static route is in place on our side.
Very, very strange. Just used an online IPv6 ping utility and it reports the same thing. Can't be my router because I can ping other hosts in my /48 prefix, and I can ping it from within the prefix.
I will check my routing
Edit: No, that was me blocking all ICMP requests with Ip6tables. Fixed now :/
Ping6 to google from troublesome server:
root@marge:~# ping6 2a00:1450:8006::93
PING 2a00:1450:8006::93(2a00:1450:8006::93) 56 data bytes
64 bytes from 2a00:1450:8006::93: icmp_seq=1 ttl=56 time=36.4 ms
64 bytes from 2a00:1450:8006::93: icmp_seq=2 ttl=56 time=23.1 ms
64 bytes from 2a00:1450:8006::93: icmp_seq=3 ttl=56 time=24.5 ms
64 bytes from 2a00:1450:8006::93: icmp_seq=4 ttl=56 time=25.1 ms
^C
--- 2a00:1450:8006::93 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 23.141/27.294/36.421/5.317 ms
So, after spending a while tweaking ip6tables, apache is still displaying a blank white page. Running out of ideas here :(
do your apache access logs show the v6 clients? just curious if by any chance something else is intercepting the traffic.
i can open a socket to it, but it doesn't appear to respond to http requests, just closes the questions. if it shows up in the apache logs then at least it is something within apache (though your apache config looks sane to me).
alternately you could open a v6 connection from somewhere else (i.e. printf "GET / HTTP/1.0\r\nHost: bottswanamedia.info\r\n\r\n" | telnet 2001:470:95bb::2 80), or, even just the telnet portion and let it hang, will give you some more time, and see if you see a netstat connection from the client.
So, I switched logs mode from errors to 'debug' and loaded up the site again. This is the only line that is output:
Wed Feb 02 17:43:42 2011] [debug] mod_deflate.c(615): [client 2001:470:95bb::4] Zlib: Compressed 177 to 128 : URL /index.html
access.log outputs:
2001:470:95bb::4 - - [02/Feb/2011:17:43:42 +0000] "GET / HTTP/1.1" 200 485 "-" (Loads of browser info)
Still not getting any joy on page loads though.
using netstat when trying to connect does show a connection:
tcp6 0 485 2001:470:95bb::2:80 2001:470:95bb::4:1028 ESTABLISHED 14042/apache2
Ah, the gzipping definitely makes it hard to debug w/ telnet. Can you try disabling the mod_deflate as a test. Any other modules of relevance that it might be hanging on (ssl, etc.)?
I disabled that deflate module. Theres quite a few modules loaded as its set up as a shared hosting environment with Virtualmin.
Heres the module output from virtualmin:
actions
alias
auth_basic
auth_digest
authn_file
authz_default
authz_groupfile
authz_host
authz_user
autoindex
cgi
dav
dav_fs
dav_svn
dir
env
fcgid
mime
negotiation
php5
proxy
proxy_balancer
proxy_connect
proxy_html
proxy_http
reqtimeout
rewrite
ruby
setenvif
ssl
status
suexec
got it. connection refused currently, but guessing you are working on the system :)
it's not expecting a proxy backend or anything for that vhost, right?
I tried disabling every module I could but still got the same result.
That included disabling all the proxy modules.
(Servers back as it was now)
Really weird. So ICMP6 both ways are fine. Just to check, can you get ssh with 6 to work on the server, and can you go to an http site with 6? Just to ensure that you have tcp connectivity both ways working OK, and it really is an Apache issue?
Just as a test, I tried sshing to the same IP and it hangs as well, which implies that maybe it is something upstream, like packet size getting hung up. Can you sniff the connection upstream and see if maybe some icmp6 responses aren't making it all the way downstream, like packet-too-large, or something like that?
Another thought is that perhaps apache & sshd are trying to query DNS for my inbound ip and failing badly, but that should give up eventually and let it through...
Heres a thought. It seems it isnt apache
This is my web control panel (Not run by apache)
It does exactly the same thing. Chance of that happening is quite slim as it is a completely different web serving software.
http://[2001:470:95bb::2]:19000
How bizarre.
It seems ICMP traffic is getting through, but not web traffic, for example, one of the update servers ubuntu hits has IPv6.
Updates from that server fail.
Yet a WinServer2003 machine on the same netowkr (With the same routing) accesses all IPv6 netowrking fine
There is something odd going on here...
i'd try sniffing it upstream to see if any icmp6 messages getting back to the box are not getting through...
Any recommendations on how to do that via ssh? :)
sure, just tcpdump. you can write it to a file and open that file somewhere else with wireshark, if you prefer the visual approach.
to monitor ipv6 traffic on interface eth0:
tcpdump -l -n -i eth0 -s 8192 ip6
to write it to a file rather than displaying it on screen, add "-w filename" before the "ip6" portion. the "ip6" portion is just a match, you can do lots of stuff with the matching to cut up the traffic as you see fit, if you like.
Shut down apache. Run netcat on port 80 (google it if needed).
See if you can connect.
If you can, it's something in Apache that's making it mad. Virtualmin by default doesn't listen on IPv6; so you may have to poke it a bit.