Hello
Here is the the script https://github.com/tactmaster/HECT
It is still a bit rough round the edges, as it was just put together in small little bits.
So consider this an "alpha" release, and let me know bugs etc and I will tidy it up.
Thanks
Ed
Ok running on ubuntu 11.04 get this error
PHP Fatal error: Call to undefined function curl_init() in /home/johnpoz/cert/he.php on line 44
I have curl installed
curl -V
curl 7.21.3 (i686-pc-linux-gnu) libcurl/7.21.3 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
Quote from: johnpoz on May 23, 2011, 09:16:50 AM
Ok running on ubuntu 11.04 get this error
PHP Fatal error: Call to undefined function curl_init() in /home/johnpoz/cert/he.php on line 44
I have curl installed
But do you have the CURL php extension enabled?
It does not matter whether you have CURL binaries, you need libcurl (http://www.php.net/manual/en/curl.requirements.php).
Check the output of
<? phpinfo() ?> (script) or
php -i (command line) to see if CURL is enabled.
no I didn't quick apt-get install php5-curl fixed it right up!
Well just ran it and worked great!
Will play around with it in the next few days before I set it up for cron, but worked great getting me my 5 points for today.
edit: BTW any reason to change the timezone setting from GB, which is clearly not my timezone..
Quoteedit: BTW any reason to change the timezone setting from GB, which is clearly not my timezone..
For some reason the new version of PHP I am running on my Gentoo system, give me warning that I should set timezone because for some reason you can't rely on a straight date().
The time is outputted, when in debug mode at the start and the end of the tests. It is also outputted in resultspass.csv. Which give a nice log of success and failure of the tests.
GB is mine ;)
To other people that use Gentoo you may need to recompile php with a USE of curl. I had too.
Let me know how the setting up cron goes.
well ran the script again today and worked great, 5 points. But not sure how to script this with cron. There is a 24 clock on submission of tests, so if schedule to ran at same time everyday is going to have issues.
need to cron for say 24 hours plus 5 minutes, not sure how to do that if even possible?
Well mine got a bit out of sync. So I set it run every few hours. I am not sure weather HE would consider that abusive or not.
In README there is the cron setting for every 24 hours
The code itself checks to see test has already run. So I won't try submit the a test that has already run the last 24 hours, but it does check the HE website.
As the RSS feed reader is slow and inefficient. It will take very slightly longer every time it get new IPs to use. It could be enough to push it a bit 24 in processing time. So try it every 24 hours and see if it work. Check in the resultspass.csv to see.
Also HE maybe nice and have set there timer to be slighly less that 24 so you can submit new test at the same time each day.
Let me know how it does. If it doesn't work. I can always add an increasing sleep function
Maybe I need to add in a sleep function, to the code.
Ed
maybe you could have it look in the resultspass.csv for the last timestamp and then sleep say 30 seconds past that.
I will run it via cron tmrw, and but wont know how its working or not for a few days of it running on cron.
Ok setup cron to run at 1355 everyday..
Ran great! 5 points for today.. Now will just keep an eye on it next couple of days to see how it works with their 24 hour clock, etc.
Loving it sofar!!! Once it has run say for 3 or 4 days getting 5 points a day I say its golden!
Thanks again for posting your script!
well I can tell you that the cron did not work at 1355 today.. I have not had chance to get in and look at log to verify it kicked off and what the script logged, etc. But it did not work - if I had to guess, I would say because of the 24 hour thing. But I will verify that cron kicked off and post up the log from the script when I get a chance.
OK. Well let me know when you have checked the log. Just incase there is something odd, command not running etc, as there was no Path set up when running cron. So all the command has to be absolutely referenced. This was on the slug, running busybox so it may not be set up ideally.
Has it worked at all from cron? If not the scheduled run tomorrow should clear that issue up.
If it is the case it needs a sleep.
I will add on, probably add one to check if the test has been performed and then run a binary back off algorithm. The it has been performed. I will record that sleep length and add that to the next days. That way it should be a nice balance between abuse and minimising repeated test delay.
I am off on holiday tomorrow soon. So may not get a change to get started on it for another week or so.
Thanks
Ed
Well it clearly ran on schedule -- but points did not go up.
Here is from the he log for today when it kicked off
Start:Thu, 26 May 11 19:55:02 +0100
RSS Check adding:ipv6.xo.com 2610:18:8102:102:2::1 aXB2Ni54by5jb20= MjYxMDoxODo4MTAyOjEwMjoyOjox
adding:cnet6.ccnu.edu.cn 2001:da8:3001:3::19 Y25ldDYuY2NudS5lZHUuY24= MjAwMTpkYTg6MzAwMTozOjoxOQ==
adding:lawrenceabel.name 2607:f740:0:3f::f08 bGF3cmVuY2VhYmVsLm5hbWU= MjYwNzpmNzQwOjA6M2Y6OmYwOA==
adding:live.sdu6.edu.cn 2001:da8:7000:15:202:194:60:60 bGl2ZS5zZHU2LmVkdS5jbg== MjAwMTpkYTg6NzAwMDoxNToyMDI6MTk0OjYwOjYw
adding:sdcert.sdu6.edu.cn 2001:da8:7000:15:202:194:15:21 c2RjZXJ0LnNkdTYuZWR1LmNu MjAwMTpkYTg6NzAwMDoxNToyMDI6MTk0OjE1OjIx
adding:chelloo.com 2a02:d28:1:1001:230:48ff:fe90:faa Y2hlbGxvby5jb20= MmEwMjpkMjg6MToxMDAxOjIzMDo0OGZmOmZlOTA6ZmFh
adding:aqfw.ncu6.edu.cn 2001:250:6c00:4::3:60 YXFmdy5uY3U2LmVkdS5jbg== MjAwMToyNTA6NmMwMDo0OjozOjYw
adding:ultradns.com 2610:a0:16:4::4 dWx0cmFkbnMuY29t MjYxMDphMDoxNjo0Ojo0
adding:ncsu.edu 2001:470:e2ba::10 bmNzdS5lZHU= MjAwMTo0NzA6ZTJiYTo6MTA=
adding:chameleon.eu 2a01:348:64::1:2 Y2hhbWVsZW9uLmV1 MmEwMTozNDg6NjQ6OjE6Mg==
adding:blox.eu 2001:470:d75b:1::2 YmxveC5ldQ== MjAwMTo0NzA6ZDc1YjoxOjoy
adding:tdc.dk 2a02:108:301:1::6 dGRjLmRr MmEwMjoxMDg6MzAxOjE6OjY=
adding:tdc.se 2a02:108:301:1::9 dGRjLnNl MmEwMjoxMDg6MzAxOjE6Ojk=
adding:unc.edu 2610:28:3090:3002::11 dW5jLmVkdQ== MjYxMDoyODozMDkwOjMwMDI6OjEx
adding:ontsi.red.es 2001:67c:21cc:2000::64:70 b250c2kucmVkLmVz MjAwMTo2N2M6MjFjYzoyMDAwOjo2NDo3MA==
adding:red.es 2001:67c:21cc:2000::64:50 cmVkLmVz MjAwMTo2N2M6MjFjYzoyMDAwOjo2NDo1MA==
adding:cccnt.ca 2607:f0b0:0:202:209:87:231:40 Y2NjbnQuY2E= MjYwNzpmMGIwOjA6MjAyOjIwOTo4NzoyMzE6NDA=
adding:orion.on.ca 2607:fd78:304:2::11 b3Jpb24ub24uY2E= MjYwNzpmZDc4OjMwNDoyOjoxMQ==
Count:38
Number:2
IP:2001:250:6803:3::34 Host:fzu6.edu.cn
Whois
Done
Ping
Done
dig PTR
Done
dig AAAA
Done
Traceroute
Done
Done:Thu, 26 May 11 19:55:17 +0100
I reschedule the cron for 1710, should kick off in like 1 minute.
Yup that went off just fine, here is the log of that.
Start:Thu, 26 May 11 23:10:01 +0100
RSS Check adding:wlan.net.id 2001:470:35:245::2 d2xhbi5uZXQuaWQ= MjAwMTo0NzA6MzU6MjQ1Ojoy
adding:ateliersnowflake.com 2001:41d0:1:1b00:213:186:33:19 YXRlbGllcnNub3dmbGFrZS5jb20= MjAwMTo0MWQwOjE6MWIwMDoyMTM6MTg2OjMzOjE5
Count:40
Number:2
IP:2001:250:6803:3::34 Host:fzu6.edu.cn
Whois
Not Done
Performing again : 0
Performing Test..
Fail - whois same netblock
Number:2
Next number:3 Written bytes:3
Number:3
Performing again : 1
Performing Test..
Pass
Ping
Not Done
Performing again : 0
Performing Test..
Pass
dig PTR
Not Done
Performing again : 0
Performing Test..
Fail - error in submission
Number:3
Next number:4 Written bytes:4
Number:4
Performing again : 1
Performing Test..
Pass
dig AAAA
Not Done
Performing again : 0
Performing Test..
Pass
Traceroute
Not Done
Performing again : 0
Performing Test..
Pass
Number:4
Next number:5 Written bytes:5
Done:Thu, 26 May 11 23:10:43 +0100
Need some kind of sleeper to wait from last time it ran or something. Or some method to change the cron to kick off say 5 minutes later than the day before.
edit: since I will be most likely busy tmrw, I changed the cron to be 1715 tmrw - bet it works just fine. but how do you automate cron to kick off 5 minutes later every day??
I know that you can set CRON with intervals like this: */5 in the minutes column means every five minutes.
Could you put */25 in the hours column (i.e. every 25 hours)?
Edit: well, I put it in my Ubuntu 10.04 VM and I will see if it works.
hmmm should be able to do 1445 in minutes then which would be 24 hours 5 minutes.
edit: well did not get a chance to test this yet was out all day yesterday and the script ran -- but did not get the tracroute point.. Here is the log. Now out of sync for points which kind of sucks. But will just wait til tmrw morning to catch up all five again.
here is the log from the traceroute failure. did this like 49 times.
But also in the middle see a failure to sixy.ch
Performing again : 46
Performing Test..
Fail - unknown reason
Can't find any IPv6 hops or this is an invalid IPv6 traceroute!<br />Number:15
Next number:16 Written bytes:16
Number:16
Performing again : 47
Performing Test..
Fail - unknown reason
Can't find any IPv6 hops or this is an invalid IPv6 traceroute!<br />Number:16
Next number:17 Written bytes:17
Number:17
Performing again : 48
Performing Test..
Fail - unknown reason
Can't find any IPv6 hops or this is an invalid IPv6 traceroute!<br />Number:17
Next number:18 Written bytes:18
Number:18
Performing again : 49
Performing Test..
Fail - unknown reason
Can't find any IPv6 hops or this is an invalid IPv6 traceroute!<br />Number:18
Next number:19 Written bytes:19
Number:19
Done:Sat, 28 May 11 02:10:17 +0100
Next number:40 Written bytes:40
Number:40
Ran Out
RSS Check PHP Warning: fsockopen(): unable to connect to sixy.ch:80 (Connection timed out) in /home/johnpoz/cert/getips.php on line 111
Feed download failed
bool(false)
Count:40
Performing again : 31
Performing Test..
Fail - unknown reason
Can't find any IPv6 hops or this is an invalid IPv6 traceroute!<br />Number:0
Next number:1 Written bytes:1
Number:1
Maybe the ipv6 connection went down? This morning I could not talk to the lan IPv6 address of the pfsense box, pfsense could talk to ipv6 addresses on the net, etc. but clients could not talk to my pfsense box on ipv6. I just took the op to update to latest release and upon reboot everything working again. So that might of been the reason, I can not be sure. Since I was not around to check when the script kicked off.
Im going to schedule the cron to be 24 hours from now so that all 5 will be open again.
edit: Ok the cron ran fine this morning 5 points. I have now scheduled it for */1445 which should run every 24 and 5 minutes so should not have to worry about the 24 hour clock.
Quote from: yakatz on May 26, 2011, 03:17:43 PM
I know that you can set CRON with intervals like this: */5 in the minutes column means every five minutes.
Could you put */25 in the hours column (i.e. every 25 hours)?
Edit: well, I put it in my Ubuntu 10.04 VM and I will see if it works.
On my Ubuntu 10.04 VM, it runs every 24 hours, not 25 hours.
I will try 1445 minutes next.
to johnpoz
does sound like you had a ipv6 failure.
I have noticed that sixy.ch is slow over ipv4, and often timesout. (You can change the timeout somewhere in the code. I think it is set to 30 seconds) but yeah if you ipv6 is not working better check that :-P
You can see what was submitted to HE by looking at tracertraw.txt (or named something simliar) also check fail.html. If you have look at that it may explain your errors.
"Fail - unknown error" is just when the error isn't one I haven't come across before. But then it extracts the reason from the returned page on HE and display it. Hence the "Can't find any IPv6 hops or this is an invalid IPv6 traceroute!" I will add it to the list of know errors...
Have fun
Ed
well that */1445 is not working its running every hour on the hour with that.
Which is kind of hammering he.net if you ask me -- I would be kind of pissed if everyone was doing that. So turning it off.
So still need a way to run the script 24 hours and few minutes later every day to get around the 24 hour clock on he.net they really should make it like a 23:55 clock.
But got my 5 points today from the script running.
Quote from: johnpoz on May 30, 2011, 08:11:19 AM
well that */1445 is not working its running every hour on the hour with that.
Mine too. I looked at having the script use a scheduled job (AT) instead of CRON.
(I moved this to a new post so that the email notifications will activate.)
#!/bin/bash
php he.php
echo "$0" | at tomorrow + 5minutes
I named this he.sh and put it in the same directory as everything else.
chmod +x he.shExecute it once to set it up/
I also added it to my user crontab so it will run if the server is restarted:
Quote from: crontab -e# m h dom mon dow user command
@reboot cd /home/yehuda/he; ./he.sh
yeah that looks like a great idea, will give it a go tmrw after my current 24 hour clock has expired.
I would like to report that this script along with my AT job addition has been working well for over a week.
I did run in to a minor issue with my jobs not being run, but that was a Ubuntu bug (https://bugs.launchpad.net/ubuntu/+source/pam/+bug/790538)
Security fix:
diff --git a/getips.php b/getips.php
index 5256ca0..74d1b0a 100644
--- a/getips.php
+++ b/getips.php
@@ -22,7 +22,7 @@ if ($xml && !empty($xml))
foreach($xml->entry as $entry) {
$title = base64_encode($entry->title);
//getting ip from dig
- $arip = preg_split("/((?<!\\\|\r)\n)|((?<!\\\)\r\n)/",trim(shell_exec("dig $entry->title AAAA +short")));
+ $arip = preg_split("/((?<!\\\|\r)\n)|((?<!\\\)\r\n)/",trim(shell_exec("dig ".escapeshellcmd($entry->title)." AAAA +short")));
//may have multiply ips so adds them a different entries.
foreach($arip as $ipraw)
{
That is because I would hate if an RSS feeded commands to my shell.
Quote from: juhoostman on June 19, 2011, 05:09:41 AM
Security fix:
- $arip = preg_split("/((?<!\\\|\r)\n)|((?<!\\\)\r\n)/",trim(shell_exec("dig $entry->title AAAA +short")));
+ $arip = preg_split("/((?<!\\\|\r)\n)|((?<!\\\)\r\n)/",trim(shell_exec("dig ".escapeshellcmd($entry->title)." AAAA +short")));
That is because I would hate if an RSS feeded commands to my shell.
I put in a pull request for this change to edmundwatson on GitHub.
https://github.com/tactmaster/HECT/pull/1
Sorry for not poking at this for a while, Tallinn is a very nice city I would recommend it for a short trip :-P
Cool. Thanks very much for this. That should now be in the code.
Also line 23
$title = base64_encode($entry->title)
has the same problem as that is used later when the real test are used. That has been fixed also.
I have tided up the code a bit.
The settings have been moved in to a setting.json file.
Which gets created on first run.
This means that in future update you can just update from the git repo and you won't have to reset you settings.
(It also means I don't accidently commit my username and password ;))
The at solution looks good.
I am writing another php script at the moment which needs to be daemonised. So I may add this to code in the near future.
Just wanted to poke my head in and say "Thank You" for making this script!
;D
Quote from: edmundwatson on June 20, 2011, 08:03:38 AM
Also line 23
$title = base64_encode($entry->title)
has the same problem as that is used later when the real test are used. That has been fixed also.
Was that not taken care of in
getAddress:
$address = array('host' => escapeshellcmd(base64_decode($ipdata[0])), 'ip' => escapeshellcmd(base64_decode($ipdata[1])));
It seems that the hostname would be escaped twice, not that it matters for any valid input.
As for me, I would escape the strings just before they are used to form a command line.
Yeap fair enough! I forgot about that.
i only get errors, why?
root@orev [~/HE]# php he.php
Current Dir: HECT/
Start:Thu, 23 Jun 11 09:50:15 -0600
RSS Check adding:pokeritieto.fi 2a01:4f8:110:3162:0:a:bad:beaf cG9rZXJpdGlldG8uZ
adding:iki.fi 2a00:1c30:1c1::51 aWtpLmZp MmEwMDoxYzMwOjFjMTo6NTE=
Count:54
IP:2001:8c0:3202:1::3 Host:ventelo.no
Whois
Not Done
Performing again : 0
Performing Test..
whois 2001:8c0:3202:1::3
Fail - invalid submission
IP:2001:5c0:1000:b::57e9 Host:galactic.no
Performing again : 1
Performing Test..
whois 2001:5c0:1000:b::57e9
Fail - invalid submission
IP:2a01:4f8:110:3162:0:a:bad:beaf Host:pokeritieto.fi
Performing again : 2
Performing Test..
whois 2a01:4f8:110:3162:0:a:bad:beaf
Fail - invalid submission
IP:2a00:1c30:1c1::51 Host:iki.fi
Performing again : 3
Performing Test..
whois 2a00:1c30:1c1::51
Fail - invalid submission
Ran Out
RSS Check Count:54
IP:2001:638:804:a082::1:225 Host:wwwiuk.informatik.uni-rostock.de
Performing again : 4
Performing Test..
whois 2001:638:804:a082::1:225
In ~/HE/HECT you should have a file called whoisraw.txt
Can you let me know what is in that file. That should display your error.
Thanks.
Ed