DNS ACME challenge. (Let's encrypt validation)

[finalbeta]

Hello,

I've had a look (used) at the let's encrypt project. it allows everyone to obtain (free) certificates for their website (and other services).
To retrieve a certificate, they require you to validate that you actually control the service/domain.
Two methods exist that allow this validation.

1) Place a challenge accessible on your web site. Port 80 or 433, so the let's encrypt servers can validate that you control the server the certificate points to.
2) Place a challenge inside a TXT record. This has the added advantage that validation can happen for services other then webservers running on port 80/443. (I'm thinking of VPN, alternative port webservers, media servers etc etc).
Validity of let's encrypt certificates is 90 days. Thus renewing of certificates can happen +- every 60 days. Automation is a must.

I would like to use this functionality (DNS validation) for my HE hosted domain. (I believe this question will become more and more frequent)
To make a long story short, can you please extend the dynamic DNS functionality to TXT records? This will allow me to script an update of a TXT record so validation can happen.

(Some more info https://letsencrypt.github.io/acme-spec/#rfc.section.7.4 )
PS: Thank you for providing me with great dynamic DNS for years!

[HQuest]

While at it, static CAA records can be another alternative to dynamic TXT records.

example.org. CAA 1 issue "letsencrypt.org" <-- req'd
example.org. CAA 1 iodef "mailto:caa@example.org" <-- currently optional/not yet supported by LE

[dig1234]

+1 this would be great. I really want to be able to use LE certs with HE.net dynamic dns. It's already supported by many other DNS providers but no one that rocks the house like HE!

[seeed]

So first of all, i am well aware that this topic is quite old.
However, the issue still persists and HE does not provide an API to update TXT records dynamically.

Therefore i took the time to create a rudimentary  bash script which basically logs into the Website,
parses the actual HTML code (very ugly) and finally updates the desired DNS record.

Use at your own risk, improvements are welcome.
Note, that the _acme-challenge.$host TXT record has to exist beforehand!

Code Select Expand

#!/bin/bash

#
# $1 is supposed to be the hostname
# $2 is supposed to be the acme-response, edited in the TXT record
#
# Note: The _acme-challenge.$host TXT record has to exist beforehand!

HENET='https://dns.he.net/'
HENET_USERNAME=''
HENET_PASS='''

#
# Get initial cookie from HE.net
#
cookie=$(mktemp)
wget --save-cookies $cookie --keep-session-cookies -q $HENET

#
# Log in using your username and password
# store the resulting page
#
result=$(mktemp)
wget --load-cookies $cookie  --post-data "email=$HENET_USERNAME&pass=$HENET_PASS" -q -O $result $HENET

#
# Every zone has its own key we need to parse, e.g.
# 'alt="delete"  onclick="delete_dom(this);" name="example.org" value="123456789"'
# save in the format 'example.org;123456789'
#
domains=$(mktemp)
grep 'alt="delete"  onclick="delete_dom(this);" name="' $result | sed -e 's/.*alt="delete"  onclick="delete_dom(this);" name="//g' -e 's/" value="/;/g' -e 's/".*//g' > $domains

# Find host in domainlist and return key
domain_key=$(awk -v host=$1 -F\; '
host ~ $1 {r=$2}
END {print r}' $domains)

#
# Every dns entry has its own key we need to parse from the actual domain page
# 'onclick="event.cancelBubble=true;deleteRecord('1103350666','_acme-challenge.www.kleinet.at','TXT')" '
#
wget --load-cookies $cookie -q -O $result "$HENET?hosted_dns_zoneid=$domain_key&menu=edit_zone&hosted_dns_editzone"
host_key=$(grep "_acme-challenge.$1','TXT')" $result | sed -e 's/.*onclick="event.cancelBubble=true;deleteRecord(.//g' -e "s/','_acme-challenge.$1','TXT').*//g")

#
# Perform the actual 'edit'
#
wget --load-cookies $cookie --post-data "menu=edit_zone&Type=txt&hosted_dns_zoneid=$domain_key&hosted_dns_recordid=$host_key&hosted_dns_editzone=1&Name=_acme-challenge.$1&Content=%22$2%22&TTL=600&hosted_dns_editrecord=Update" -q -O $result $HENET

#
# On success, the Website returns the following String, if you explicitly want to return 0 or 1
# 'Successfully updated record'
#

rm $cookie $result $domains


Honestly, i'm posting this to push HE to implement the actual API ;)

[passport123]

...Therefore i took the time to create a rudimentary  bash script which basically logs into the Website,
parses the actual HTML code (very ugly) and finally updates the desired DNS record.
...

When I've had to do things like this, I've used QA automation scripts.  There was a free (i.e., no-charge) QA automation tool available a few years ago when I last needed to do this.  I don't remember its name, but google should be helpful...

[spil]

I've been playing with this over the weekend using Python. Both to get dns-01 validation working with HE.net and also to learn a bit about Python.

Parsing the HTML is indeed arduous as noted by seeed. ;D

This isn't fully functional yet, but I have been able to get the list of domains, delete and add records...

Code Select Expand

#!/usr/local/bin/python2.7
from requests import Request, Session
from lxml import html, etree

# All classes will work with the session
sess = Session()

class HEsession(object):

    def __init__(self, sess, user_id, user_pw):
        self.user_id = user_id
        self.user_pw = user_pw
        # Authenticate to HE.net DNS service
        post_data = { 'email' : self.user_id,
                      'pass'  : self.user_pw,
                      'submit': 'Login!' }
        resp = sess.post('https://dns.he.net/',
                         data = post_data)
        # Store landing page for later reference
        self.landing = html.fromstring(resp.content)
        # Check for login error
        selector = "//div[@class='caption']/../div[@id='dns_err']/text()"
        nodes = self.landing.xpath(selector)
        if nodes == 'Incorrect':
            print "Login incorrect"
            quit()
        # Get list of domains + id's (not reverse domains)
        selector = "//img[@alt='delete']" \
                        "[not(substring(@name, string-length(@name) - 4) = '.arpa')]" \
                   "/@*[name()='value' or name()='name']"
        nodes = self.landing.xpath(selector)
        # Transform to a dictionary of domain: id
        self.domains = dict(zip(nodes[0::2], nodes[1::2]))

    def getDomains(self):
        return self.domains
       
    def getZoneId(self,domain):
        return self.domains[domain]
   
    def getRR(self,domain, type):
        if self.domains is None:
            self.login()

    def delACMERR(self,domain):
        post_data = { 'hosted_dns_zoneid':   zoneid,
                      'menu':                "edit_zone",
                      'hosted_dns_editzone': ""}
        resp = self.sess.post('https://dns.he.net/',
                              data = post_data)
        tree = html.fromstring(resp.content)
        selector = "//tr[td='_acme-challenge." \
                   + domain \
                   + "']/td/img[@data='TXT']/../../td[@class='dns_delete']/@onclick"
        nodes = tree.xpath(selector)

class ResourceRecord(object):
        def __init__(self, id, name, type, data)
            self.id   = id
            self.name = name
            self.type = type
            self.data = data

class HEdomain(object):
    Domain = etree.Element('Domain')

    def __init__(self, sess, HEnet, domain):
        # Get the ZoneID
        self.zoneid = HEnet.getZoneId(domain)
        post_data = { 'hosted_dns_zoneid':   self.zoneid,
                      'menu':                "edit_zone",
                      'hosted_dns_editzone': ""}
        resp = sess.post('https://dns.he.net/',
                         data = post_data )
        tree = html.fromstring(resp.content)
        # Extract Id, name, type and data
        selector =  "//tr[@class='dns_tr']/td[2]/text()" \
                    "|//tr[@class='dns_tr']/td[3]/text()" \
                    "|//tr[@class='dns_tr']/td[4]/img/@data" \
                    "|//tr[@class='dns_tr']/td[7]/@data"
        nodes = tree.xpath(selector)
        # Transform into a simple XML structure
        Ids   = nodes[0::4]
        names = nodes[1::4]
        types = nodes[2::4]
        datas = nodes[3::4]
        for i in range(len(Ids)):
            ResR = etree.Subelement(self.Domain, 'Record', id=Ids[i])
            Name = etree.SubElement(RR, 'Name')
            Name.text = names[i]
            Type = etree.SubElement(RR, 'Type')
            Type.text = types[i]
            Data = etree.Subelement(RR, 'Data')
            Data.text = datas[i]
       
    def listRRs():
        for i in range(len(self.Ids)):
            print "Id: %s, Name: %s, Type: %s, Data %s" % \
                (self.Ids[i], self.names[i], self.types[i], self.datas[i])

    def getRecord(id)
        selector = "//Record[Type='TXT'][Name='_acme.brnrd.eu']"
       
    def delRecord(recordid)
        post_data = { 'hosted_dns_zoneid':     self.zoneid,
                      'hosted_dns_recordid':   recordid,
                      'menu':                  "edit_zone",
                      'hosted_dns_delconfirm': "delete",
                      'hosted_dns_delrecord':  "1",
                      'hosted_dns_editzone':   "1"}

    def addRecord()
        post_data = {
            'menu': "edit_zone"
            'Type': "TXT"
            'hosted_dns_zoneid':501311
            'hosted_dns_editzone':1
            'Priority':
            'Name': "_acme-challenge"
            'Content': "testing123"
            'TTL': "900"
            'hosted_dns_editrecord': "Submit" }


[garyfowler]

Added an additional DNS api called dns_manual..
I just modified the dns_myapi.sh

The allows the following command to work effectively.

acme.sh --signcsr --csr /somedir/someweb.csr --dns dns_manual

The result is that the FQDM you need to modify and the associated key string are output for you to manually key into your DNS interface.
The script pauses for you press ENTER.. and the acme.sh waits an additional 120 seconds for DNS records to sync etc.


----

Code Select Expand

########  Public functions #####################

#Usage: dns_myapi_add   _acme-challenge.www.domain.com   "XKrxpRBosdIKFzxW_CT3KLZNf6q0HG9i01zxXp5CPBs"
dns_manual_add() {
  fulldomain=$1
  txtvalue=$2
  _info "Using myapi"
  _debug fulldomain "$fulldomain"
  _debug txtvalue "$txtvalue"
  _err "Not implemented!"

  echo
  echo "Create TXT Record ${fulldomain}  with value  ${txtvalue}"
  echo
  echo
  echo "Press ENTER when done."
  read somevariable
  return 0
}

#Usage: fulldomain txtvalue
#Remove the txt record after validation.
dns_manual_rm() {
  fulldomain=$1
  txtvalue=$2
  _info "Using myapi"
  _debug fulldomain "$fulldomain"
  _debug txtvalue "$txtvalue"
  echo
  echo "Remove TXT Record ${fulldomain}  with value  ${txtvalue}"
  echo
  echo
  echo "Press ENTER when done."
  read somevariable
}

####################  Private functions below ##################################


[adamel]

I created a simple script that can create/update/delete records in HE DNS, which I'm now using with https://github.com/lukas2511/dehydrated

In the deploy_challenge hook I have:

Code Select Expand

hednsupdate _acme-challenge.${DOMAIN}. TXT "${TOKEN_VALUE}" 300

And in clean_challenge:

Code Select Expand

hednsupdate _acme-challenge.${DOMAIN}. DELETE

Code Select Expand

#!/bin/sh

url=https://dns.he.net/

# These should be set in .hedns-credentials
login=
password=
zoneid=
# Optionally override these.
ttl=300

if [ -e ${HOME}/.hedns-credentials ]; then
    . ${HOME}/.hedns-credentials
fi

function usage()
{
    prog=$(basename $0)
    cat <<EOF
Create/update:
$prog NAME TYPE CONTENT [TTL]
Delete:
$prog NAME "DELETE"
EOF
}

record=$1
type=$2
content=$3
usettl=${4-${ttl}}

if [ $# -lt 2 ] || [ $# -lt 3 -a "${type}" != DELETE ]; then
    usage
    exit 1
fi

cookies=${HOME}/.hedns-cookies.txt
touch ${cookies}
chmod 0600 ${cookies}

function get_id()
{
    curl -sS --cookie ${cookies} "${url}?hosted_dns_zoneid=${zoneid}&menu=edit_zone&hosted_dns_editzone" | grep -B5 ">${record}" | sed -nre 's/.*dns_tr.* id="([^"]*)".*/\1/p' | head -n 1
}

# Get any initial cookies.
curl -sS --cookie-jar ${cookies} ${url} -o /dev/null
# Login.
curl -sS --cookie ${cookies} --cookie-jar ${cookies} --data "email=${login}&pass=${password}&submit=Login%21" ${url} -o /dev/null

# Check if we should update or create new.
recordid=$(get_id)
if [ "${type}" == DELETE ]; then
    if [ ! "${recordid}" ]; then
echo "No such record"
exit 1
    fi
    curl -sS --cookie ${cookies} --data "menu=edit_zone&hosted_dns_zoneid=${zoneid}&hosted_dns_recordid=${recordid}&hosted_dns_editzone=1&hosted_dns_delrecord=1" "${url}index.cgi" | grep "^${record}"
    [ $? -ne 0 ]
    exit
fi
if [ "${recordid}" ]; then
    op=Update
else
    op=Submit
fi
curl -sS --cookie ${cookies} --data "account=&menu=edit_zone&Type=${type}&hosted_dns_zoneid=${zoneid}&hosted_dns_recordid=${recordid}&hosted_dns_editzone=1&Priority=&Name=${record}&Content=${content}&TTL=${usettl}&hosted_dns_editrecord=${op}" "${url}?hosted_dns_zoneid=${zoneid}&menu=edit_zone&hosted_dns_editzone" | grep "^${record}"


[matthiaspfaller]

Hi,

thank's for your script. I modified it a little to make it possible to specify more than one zoneid. You can put something like this into your credentials file:

Code Select Expand


zoneids[my.domain.]=123456
zoneids[sub.my.domain.]=123457

Please note that the trailing "." is necessary. When you just specify

Code Select Expand


zoneid=123456

everything should work as before. The downside is that it's bash specific now.

Code Select Expand


#!/usr/local/bin/bash

url=https://dns.he.net/

# These should be set in .hedns-credentials
login=
password=
zoneid=
# Optionally override these.
ttl=300

declare -A zoneids
if [ -e /usr/local/etc/dehydrated/hedns-credentials ]; then
    . /usr/local/etc/dehydrated/hedns-credentials
fi

function usage()
{
    prog=$(basename $0)
    cat <<EOF
Create/update:
$prog NAME TYPE CONTENT [TTL]
Delete:
$prog NAME "DELETE"
EOF
}

function hasdot()
{
    case $1 in
        *.*) return 0 ;;
        *)   return 1 ;;
    esac
}

record=$1
type=$2
content=$3
usettl=${4-${ttl}}

if [ $# -lt 2 ] || [ $# -lt 3 -a "${type}" != DELETE ]; then
    usage
    exit 1
fi

cookies=/usr/local/etc/dehydrated/hedns-cookies.txt
touch ${cookies}
chmod 0600 ${cookies}

i=$record
while hasdot $i; do
    if [ "${zoneids[$i]}" != "" ]; then
        zoneid=${zoneids[$i]}
        break
    fi
    i=${i#*.}
done

function get_id()
{
    curl -sS --cookie ${cookies} "${url}?hosted_dns_zoneid=${zoneid}&menu=edit_zone&hosted_dns_editzone" | grep -B5 ">${record}" | sed -nre 's/.*dns_tr.* id="([^"]*)".*/\1/p' | head -n 1
}

# Get any initial cookies.
curl -sS --cookie-jar ${cookies} ${url} -o /dev/null
# Login.
curl -sS --cookie ${cookies} --cookie-jar ${cookies} --data "email=${login}&pass=${password}&submit=Login%21" ${url} -o /dev/null

# Check if we should update or create new.
recordid=$(get_id)
if [ "${type}" == DELETE ]; then
    if [ ! "${recordid}" ]; then
        echo "No such record"
        exit 1
    fi
    curl -sS --cookie ${cookies} --data "menu=edit_zone&hosted_dns_zoneid=${zoneid}&hosted_dns_recordid=${recordid}&hosted_dns_editzone=1&hosted_dns_delrecord=1" "${url}index.cgi" | grep "^${record}"
    [ $? -ne 0 ]
    exit
fi
if [ "${recordid}" ]; then
    op=Update
else
    op=Submit
fi
curl -sS --cookie ${cookies} --data "account=&menu=edit_zone&Type=${type}&hosted_dns_zoneid=${zoneid}&hosted_dns_recordid=${recordid}&hosted_dns_editzone=1&Priority=&Name=${record}&Content=${content}&TTL=${usettl}&hosted_dns_editrecord=${op}" "${url}?hosted_dns_zoneid=${zoneid}&menu=edit_zone&hosted_dns_editzone" | grep "^${record}"


[angel333]

Here's my quick and dirty take on this - I needed to renew all my domains so I wrote a hook script certbot. Note that it takes a session ID instead of login details. You can obtain a session id from your browser (look for a cookie named CGISESSID). On the other hand, you don't have to fiddle with zone IDs, the script will figure them out.

Here's how you renew all domains:

Code Select Expand


HE_SESSID=<session_id> certbot renew --preferred-challenges dns --manual-auth-hook /path/to/the/certbot-he-hook.sh  --manual-public-ip-logging-ok


Validating a new domain works too:

Code Select Expand


HE_SESSID=<session_id> certbot certonly --manual -d <requested.domain.com> -m your@email.com --preferred-challenges dns --manual-auth-hook /path/to/certbot-he-hook.sh  --manual-public-ip-logging-ok


Here's the script:

Code Select Expand


#!/bin/bash

TLD=$(echo $CERTBOT_DOMAIN | grep -Eo '[a-z0-9]+$')
SLD=$(echo $CERTBOT_DOMAIN | grep -Eo '[a-z0-9]+\.[a-z0-9]+$' | grep -Eo '^[a-z0-9]+')
HE_ZONEID=$(curl --stderr - --cookie CGISESSID=$HE_SESSID https://dns.he.net/index.cgi \
  | grep -Eo "delete_dom.*name=\"$SLD\.$TLD\" value=\"[0-9]+" | grep -Eo "[0-9]+$")

curl --stderr - -o /dev/null --cookie CGISESSID=$HE_SESSID https://dns.he.net/index.cgi \
  -d "account=&menu=edit_zone&Type=TXT&hosted_dns_zoneid=$HE_ZONEID&hosted_dns_recordid=&hosted_dns_editzone=1&Priority=&Name=_acme-challenge.$CERTBOT_DOMAIN&Content=$CERTBOT_VALIDATION&TTL=300&hosted_dns_editrecord=Submit"


It's also on gist: https://gist.github.com/angel333/1aae17b1ea53a9cd538966979781d8aa

Certbot also supports a cleanup script (so you're not left with "_acme-challenge" records) but I haven't written one yet.

[angel333]

Here's a somewhat better version of my certbot hook: https://github.com/angel333/certbot-he-hook

It now also supports passing login credentials (instead of session id) so you can do something like this to renew all domains:

Code Select Expand


HE_USER=<username> HE_PASS=<password> certbot renew \
  --preferred-challenges dns \
  --manual-auth-hook /path/to/certbot-he-hook.sh  \
  --manual-public-ip-logging-ok


[EddyBeaupre]

acme.sh support dns.he.net out of the box... Basically all you have to do is:

First install acme.sh

wget -O -  https://get.acme.sh | sh

Next, you need to provide your credential (acme.sh will save them automatically to ~/.acme.sh/account.conf). This is only needed for the first run:

export HE_Username="yourusername"
export HE_Password="password"

or set them directly into ~/.acme.sh/account.conf by adding theses lines:

HE_Username='yourusername'
HE_Password='password'

Then you can issue your certificate:

acme.sh --issue --dns dns_he -d example.com -d www.example.com

The HE_Username and HE_Password settings will be saved in ~/.acme.sh/account.conf and will be reused when needed. That may be a security concern...

[divad27182]

wget -O -  https://get.acme.sh | sh

Of course, if you have ANY security concerns, you will not do this at all!

[PJSalt]

Any progress on getting a *real* API from HE.net itself?

Or at least something that let's us more easily add (and remove!) TXT records via a HTTPS call instead of these (nice and clever, by the way) hacks by parsing the HTML code, which are prone to fail when they make a breaking change.

They already have something in place for dynamic DNS updates via curl, so in theory they could go and take that and adapt it for doing TXT stuff.

There is something on the homepage under "Upcoming Features!" named "Expanding our DDNS service to support TXT records" but that seems to be related to DDNS (I assume that abbreviation stands for dynamic DNS), and thus not "normal" hostnames. Unless it is just poorly worded by the person that typed out that line of text. It is confusing to me. I don't use DDNS.

[notzac]

There is something on the homepage under "Upcoming Features!" named "Expanding our DDNS service to support TXT records" but that seems to be related to DDNS (I assume that abbreviation stands for dynamic DNS), and thus not "normal" hostnames. Unless it is just poorly worded by the person that typed out that line of text. It is confusing to me. I don't use DDNS.

The way I read the note about expanding the DDNS service to cover TXT records is that it will allow exactly what is being described in this thread - a method of using an API to update a TXT record.

This would be especially helpful for me, as all these awesome scripts don't work with 2FA.

:)