Script for Certification tests

[edmundwatson]

Hello

Here is the the script https://github.com/tactmaster/HECT
It is still a bit rough round the edges, as it was just put together in small little bits.
So consider this an "alpha" release, and let me know bugs etc and I will tidy it up.

Thanks
Ed

[johnpoz]

Ok running on ubuntu 11.04 get this error

PHP Fatal error:  Call to undefined function curl_init() in /home/johnpoz/cert/he.php on line 44

I have curl installed

curl -V
curl 7.21.3 (i686-pc-linux-gnu) libcurl/7.21.3 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

[yakatz]

Ok running on ubuntu 11.04 get this error
PHP Fatal error:  Call to undefined function curl_init() in /home/johnpoz/cert/he.php on line 44
I have curl installed

But do you have the CURL php extension enabled?
It does not matter whether you have CURL binaries, you need libcurl.
Check the output of

Code Select Expand

<? phpinfo() ?> (script) or

Code Select Expand

php -i (command line) to see if CURL is enabled.

[johnpoz]

no I didn't quick apt-get install php5-curl fixed it right up!

Well just ran it and worked great!

Will play around with it in the next few days before I set it up for cron, but worked great getting me my 5 points for today.

edit: BTW any reason to change the timezone setting from GB, which is clearly not my timezone..

[edmundwatson]

edit: BTW any reason to change the timezone setting from GB, which is clearly not my timezone..

For some reason the new version of PHP I am running on my Gentoo system, give me warning that I should set timezone because for some reason you can't rely on a straight date(). 

The time is outputted, when in debug mode at the start and the end of the tests.  It is also outputted in resultspass.csv. Which give a nice log of success and failure of the tests.

GB is mine  ;)

To other people that use Gentoo you may need to recompile php with a USE of curl. I had too.

Let me know how the setting up cron goes.

[johnpoz]

well ran the script again today and worked great, 5 points.  But not sure how to script this with cron.  There is a 24 clock on submission of tests, so if schedule to ran at same time everyday is going to have issues.

need to cron for say 24 hours plus 5 minutes, not sure how to do that if even possible?

[edmundwatson]

Well mine got a bit out of sync. So I set it run every few hours. I am not sure weather HE would consider that abusive or not.

In README there is the cron setting for every 24 hours

The code itself checks to see test has already run. So I won't try submit the a test that has already run the last 24 hours, but it does check the HE website.

As the RSS feed reader is slow and inefficient. It will take very slightly longer every time it get new IPs to use. It could be enough to push it a bit 24 in processing time. So try it every 24 hours and see if it work. Check in the resultspass.csv to see.

Also HE maybe nice and have set there timer to be slighly less that 24 so you can submit new test at the same time each day.

Let me know how it does. If it doesn't work. I can always add an increasing sleep function

Maybe I need to add in a sleep function, to the code.

Ed

[johnpoz]

maybe you could have it look in the resultspass.csv for the last timestamp and then sleep say 30 seconds past that.

I will run it via cron tmrw, and but wont know how its working or not for a few days of it running on cron.

[johnpoz]

Ok setup cron to run at 1355 everyday..

Ran great!  5 points for today..  Now will just keep an eye on it next couple of days to see how it works with their 24 hour clock, etc.

Loving it sofar!!!  Once it has run say for 3 or 4 days getting 5 points a day I say its golden!

Thanks again for posting your script!

[johnpoz]

well I can tell you that the cron did not work at 1355 today.. I have not had chance to get in and look at log to verify it kicked off and what the script logged, etc.  But it did not work - if I had to guess, I would say because of the 24 hour thing.  But I will verify that cron kicked off and post up the log from the script when I get a chance.

[edmundwatson]

OK. Well let me know when you have checked the log. Just incase there is something odd, command not running etc, as there was no Path set up when running cron. So all the command has to be absolutely referenced. This was on the slug, running busybox so it may not be set up ideally.

Has it worked at all from cron? If not the scheduled run tomorrow should clear that issue up.

If it is the case it needs a sleep.

I will add on, probably add one to check if the test has been performed and then run a binary back off algorithm. The it has been performed. I will record that sleep length and add that to the next days. That way it should be a nice balance between abuse and minimising repeated test delay.

I am off on holiday tomorrow soon. So may not get a change to get started on it for another week or so.

Thanks

Ed

[johnpoz]

Well it clearly ran on schedule -- but points did not go up.

Here is from the he log for today when it kicked off

Start:Thu, 26 May 11 19:55:02 +0100
RSS Check adding:ipv6.xo.com 2610:18:8102:102:2::1 aXB2Ni54by5jb20=  MjYxMDoxODo4MTAyOjEwMjoyOjox
adding:cnet6.ccnu.edu.cn 2001:da8:3001:3::19 Y25ldDYuY2NudS5lZHUuY24=  MjAwMTpkYTg6MzAwMTozOjoxOQ==
adding:lawrenceabel.name 2607:f740:0:3f::f08 bGF3cmVuY2VhYmVsLm5hbWU=  MjYwNzpmNzQwOjA6M2Y6OmYwOA==
adding:live.sdu6.edu.cn 2001:da8:7000:15:202:194:60:60 bGl2ZS5zZHU2LmVkdS5jbg==  MjAwMTpkYTg6NzAwMDoxNToyMDI6MTk0OjYwOjYw
adding:sdcert.sdu6.edu.cn 2001:da8:7000:15:202:194:15:21 c2RjZXJ0LnNkdTYuZWR1LmNu  MjAwMTpkYTg6NzAwMDoxNToyMDI6MTk0OjE1OjIx
adding:chelloo.com 2a02:d28:1:1001:230:48ff:fe90:faa Y2hlbGxvby5jb20=  MmEwMjpkMjg6MToxMDAxOjIzMDo0OGZmOmZlOTA6ZmFh
adding:aqfw.ncu6.edu.cn 2001:250:6c00:4::3:60 YXFmdy5uY3U2LmVkdS5jbg==  MjAwMToyNTA6NmMwMDo0OjozOjYw
adding:ultradns.com 2610:a0:16:4::4 dWx0cmFkbnMuY29t  MjYxMDphMDoxNjo0Ojo0
adding:ncsu.edu 2001:470:e2ba::10 bmNzdS5lZHU=  MjAwMTo0NzA6ZTJiYTo6MTA=
adding:chameleon.eu 2a01:348:64::1:2 Y2hhbWVsZW9uLmV1  MmEwMTozNDg6NjQ6OjE6Mg==
adding:blox.eu 2001:470:d75b:1::2 YmxveC5ldQ==  MjAwMTo0NzA6ZDc1YjoxOjoy
adding:tdc.dk 2a02:108:301:1::6 dGRjLmRr  MmEwMjoxMDg6MzAxOjE6OjY=
adding:tdc.se 2a02:108:301:1::9 dGRjLnNl  MmEwMjoxMDg6MzAxOjE6Ojk=
adding:unc.edu 2610:28:3090:3002::11 dW5jLmVkdQ==  MjYxMDoyODozMDkwOjMwMDI6OjEx
adding:ontsi.red.es 2001:67c:21cc:2000::64:70 b250c2kucmVkLmVz  MjAwMTo2N2M6MjFjYzoyMDAwOjo2NDo3MA==
adding:red.es 2001:67c:21cc:2000::64:50 cmVkLmVz  MjAwMTo2N2M6MjFjYzoyMDAwOjo2NDo1MA==
adding:cccnt.ca 2607:f0b0:0:202:209:87:231:40 Y2NjbnQuY2E=  MjYwNzpmMGIwOjA6MjAyOjIwOTo4NzoyMzE6NDA=
adding:orion.on.ca 2607:fd78:304:2::11 b3Jpb24ub24uY2E=  MjYwNzpmZDc4OjMwNDoyOjoxMQ==
Count:38
Number:2
IP:2001:250:6803:3::34 Host:fzu6.edu.cn
Whois
Done
Ping
Done
dig PTR
Done
dig AAAA
Done
Traceroute
Done
Done:Thu, 26 May 11 19:55:17 +0100

I reschedule the cron for 1710, should kick off in like 1 minute.

Yup that went off just fine, here is the log of that.

Start:Thu, 26 May 11 23:10:01 +0100
RSS Check adding:wlan.net.id 2001:470:35:245::2 d2xhbi5uZXQuaWQ=  MjAwMTo0NzA6MzU6MjQ1Ojoy
adding:ateliersnowflake.com 2001:41d0:1:1b00:213:186:33:19 YXRlbGllcnNub3dmbGFrZS5jb20=  MjAwMTo0MWQwOjE6MWIwMDoyMTM6MTg2OjMzOjE5
Count:40
Number:2
IP:2001:250:6803:3::34 Host:fzu6.edu.cn
Whois
Not Done
Performing again : 0
Performing Test..
Fail - whois same netblock
Number:2
Next number:3 Written bytes:3
Number:3
Performing again : 1
Performing Test..
Pass
Ping
Not Done
Performing again : 0
Performing Test..
Pass
dig PTR
Not Done
Performing again : 0
Performing Test..
Fail - error in submission
Number:3
Next number:4 Written bytes:4
Number:4
Performing again : 1
Performing Test..
Pass
dig AAAA
Not Done
Performing again : 0
Performing Test..
Pass
Traceroute
Not Done
Performing again : 0
Performing Test..
Pass
Number:4
Next number:5 Written bytes:5
Done:Thu, 26 May 11 23:10:43 +0100

Need some kind of sleeper to wait from last time it ran or something.  Or some method to change the cron to kick off say 5 minutes later than the day before.

edit: since I will be most likely busy tmrw, I changed the cron to be 1715 tmrw - bet it works just fine.  but how do you automate cron to kick off 5 minutes later every day??

[yakatz]

I know that you can set CRON with intervals like this: */5 in the minutes column means every five minutes.
Could you put */25 in the hours column (i.e. every 25 hours)?

Edit: well, I put it in my Ubuntu 10.04 VM and I will see if it works.

[johnpoz]

hmmm should be able to do 1445 in minutes then which would be 24 hours 5 minutes.

edit: well did not get a chance to test this yet was out all day yesterday and the script ran -- but did not get the tracroute point..  Here is the log.  Now out of sync for points which kind of sucks.  But will just wait til tmrw morning to catch up all five again.

here is the log from the traceroute failure.  did this like 49 times.

But also in the middle see a failure to sixy.ch

Performing again : 46
Performing Test..
Fail - unknown reason
Can't find any IPv6 hops or this is an invalid IPv6 traceroute!<br />Number:15
Next number:16 Written bytes:16
Number:16
Performing again : 47
Performing Test..
Fail - unknown reason
Can't find any IPv6 hops or this is an invalid IPv6 traceroute!<br />Number:16
Next number:17 Written bytes:17
Number:17
Performing again : 48
Performing Test..
Fail - unknown reason
Can't find any IPv6 hops or this is an invalid IPv6 traceroute!<br />Number:17
Next number:18 Written bytes:18
Number:18
Performing again : 49
Performing Test..
Fail - unknown reason
Can't find any IPv6 hops or this is an invalid IPv6 traceroute!<br />Number:18
Next number:19 Written bytes:19
Number:19
Done:Sat, 28 May 11 02:10:17 +0100


Next number:40 Written bytes:40
Number:40
Ran Out
RSS Check PHP Warning:  fsockopen(): unable to connect to sixy.ch:80 (Connection timed out) in /home/johnpoz/cert/getips.php on line 111
Feed download failed
bool(false)
Count:40
Performing again : 31
Performing Test..
Fail - unknown reason
Can't find any IPv6 hops or this is an invalid IPv6 traceroute!<br />Number:0
Next number:1 Written bytes:1
Number:1

Maybe the ipv6 connection went down?  This morning I could not talk to the lan IPv6 address of the pfsense box, pfsense could talk to ipv6 addresses on the net, etc. but clients could not talk to my pfsense box on ipv6.  I just took the op to update to latest release and upon reboot everything working again.  So that might of been the reason, I can not be sure. Since I was not around to check when the script kicked off.

Im going to schedule the cron to be 24 hours from now so that all 5 will be open again.

edit: Ok the cron ran fine this morning 5 points.  I have now scheduled it for */1445 which should run every 24 and 5 minutes so should not have to worry about the 24 hour clock.

[yakatz]

I know that you can set CRON with intervals like this: */5 in the minutes column means every five minutes.
Could you put */25 in the hours column (i.e. every 25 hours)?

Edit: well, I put it in my Ubuntu 10.04 VM and I will see if it works.

On my Ubuntu 10.04 VM, it runs every 24 hours, not 25 hours.
I will try 1445 minutes next.