Manage FreeIPA certificates with ACME on vSphere (or not)

 | 4 May 2023 16:28

Some days ‘in the office’ are great, others … not so much. But we learn from both, right?!

While writing this article, I ended up ditching the original plan and gave up altogether on running an ACME client on vCenter. Instead, I’m going to use an IPA client to request the certificate and upload it through the REST API, just like I do for FreeIPA certificates for Palo Alto firewalls and Panorama. But, you’ll have to wait for the next article to see how that works out.

My attempts (original ramblings)

This article gave me the inspiration to use ACME to automate FreeIPA certificate renewal for vCenter (vSphere Client UI).

However, I ran into some issues. First, with acme.sh as I couldn’t work out how to get it to update the DNS record on FreeIPA. Let me explain the requirement for dns-01 when using an ACME client on vCenter.

ACME certificate verification requires a challenge-response to authorise the certificate request, else anyone could request a certificate for any domain they like completely defeating the point of SSL as a mechanism of verifying source authenticity. FreeIPA ACME supports two mechanisms for this challenge-response: http-01 and dns-01. http-01 is based on the client hosting a temporary key on port 80 on the domain the certificate is being requested for, thus the ACME client does not need to communicate with the IPA server nor does the client need to be IPA enrolled. dns-01 on the other hand requires the client to control a TXT record in the domain’s DNS zone. If IPA is the nameserver for this zone, then the client will need an IPA account and privileges to manage the DNS entries. The default means of authenticating is Kerberos.

If not acme.sh, then what?

Well, acme.sh isn’t the only ACME client out there and over the last couple of years, I’ve actually shifted some deployments of acme.sh to dehydrated, which supports hooks for ACME servers. Enter ipa-dns-hook by Jimmy Hedman, which uses the Python library HTTPKerberosAuth to authenticate against IPA using stored credentials. I’m not a fan of storing credentials in a text file, but IPA does have decent controls for delegating privileges, which should limit the impact if the account was ever compromised.

What about IPA principals for certificates?

It’s important to realise that traditional certificate requests (IPA web-UI, ipa-getcert etc.) require an IPA principal for the FQDN, ACME does not. Note that the validity lengths are different as well: ‘Traditional’ IPA host and service certificates are valid for 2 years while ACME certificates are valid for 90 days.

How can an ACME client create/change DNS records on a FreeIPA server?

Fraser Tweedale gives this example using a custom hook for Certbot, which uses the ipapython Python library to edit DNS records with Kerberos authentication. You can download his certbot-dns-ipa.py GitHub Gist here. The code is small, this is it in its entirety:

#!/usr/bin/python3
import os
from dns import resolver
from ipalib import api 
from ipapython import dnsutil

certbot_domain = os.environ['CERTBOT_DOMAIN']
certbot_validation = os.environ['CERTBOT_VALIDATION']
if 'CERTBOT_AUTH_OUTPUT' in os.environ:
    command = 'dnsrecord_del'
else:
    command = 'dnsrecord_add'

validation_domain = f'_acme-challenge.{certbot_domain}'
fqdn = dnsutil.DNSName(validation_domain).make_absolute()
zone = dnsutil.DNSName(resolver.zone_for_name(fqdn))
name = fqdn.relativize(zone)

api.bootstrap(context='cli')
api.finalize()
api.Backend.rpcclient.connect()

api.Command[command](zone, name, txtrecord=[certbot_validation], dnsttl=60)

And Jimmy Hedman kindly provides this hook for dehydrated, which uses json calls to IPA using either a userid and password or Kerberos. The relevant section:

def _call_freeipa(json_operation):
    headers = {'content-type': 'application/json',
               'referer': 'https://%s/ipa' % IPA_SERVER}
    if IPA_USER:
        # Login and keep a cookie
        login_result = requests.post("https://%s/ipa/session/login_password" % IPA_SERVER,
                                     data="user=%s&password=%s" % (IPA_USER, IPA_PASSWORD),
                                     headers={'content-Type':'application/x-www-form-urlencoded',
                                              'referer': 'https://%s/ipa' % IPA_SERVER},
                                     verify='/etc/ipa/ca.crt')

        # No auth
        auth = None
        # Use cookies
        cookies=login_result.cookies
    else:
        # Use kerberos authentication
        auth = HTTPKerberosAuth(mutual_authentication=REQUIRED,
                                sanitize_mutual_error_response=False)
        # No cookies
        cookies = None

    result = requests.post("https://%s/ipa/session/json" % IPA_SERVER,
                           data=json_operation,
                           headers=headers,
                           auth=auth,
                           cookies=cookies,
                           verify='/etc/ipa/ca.crt')

    retval = result.json()

While these two examples are by no means exhaustive, they do shed light on what’s possible using different approaches.

The (not) solution?

While dehydrated seemed to be an obvious candidate. I’ve used it elsewhere, it runs in plain bash, doesn’t require much and is extendible with hooks. But this is where the dream ended for me, because though dehydrated will run on vCenter, the hook won’t due to requiring the requests-kerberos Python library: It won’t build on my vCenter 7.0.

So while ipa-dns-hook allows for the use of dehydrated on systems that do not have native Kerberos ability by using a userid and password. I’m not prepared to hack vCenter to the point where things might break or official support could end up being affected.

Install dehydrated on vCenter 7

If you want to try it yourself then this is how to get dehydrated and the hook onto vCenter and install the requirements for ipa-dns-hook

WARNING – try this at your own peril!

# Assumed login as root
yum install python3-pip
mkdir -p ~/dehydrated/hooks/ipa-dns && cd ~/dehydrated
wget https://github.com/dehydrated-io/dehydrated/raw/master/dehydrated
chmod +x dehydrated
wget -P hooks/ipa-dns https://github.com/HeMan/ipa-dns-hook/raw/master/{ipa-dns-hook.py,requirements.txt}
pip install -r hooks/ipa-dns/requirements.txt

The problem

requests-kerberos wants to install or update wheels and then things break catastrophically. It didn’t break vCenter for me, but we don’t end up with requests-kerberos either.

root@vcenter [ ~/dehydrated ]# pip3 install -r hooks/ipa-dns/requirements.txt
Requirement already satisfied: requests>=2.21.0 in /usr/lib/python3.7/site-packages (from -r hooks/ipa-dns/requirements.txt (line 1)) (2.24.0)
Collecting requests-kerberos>=0.12.0
  Downloading requests_kerberos-0.14.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: urllib3>=1.25.2 in /usr/lib/python3.7/site-packages (from -r hooks/ipa-dns/requirements.txt (line 3)) (1.25.11)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (2.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (2018.8.24)
Collecting pyspnego[kerberos]
  Downloading pyspnego-0.9.0-py3-none-any.whl (132 kB)
     |????????????????????????????????| 132 kB 10.6 MB/s
Requirement already satisfied: cryptography>=1.3 in /usr/lib/python3.7/site-packages (from requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (2.8)
Requirement already satisfied: six>=1.4.1 in /usr/lib/python3.7/site-packages (from cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (1.12.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/lib/python3.7/site-packages (from cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (1.11.5)
Requirement already satisfied: pycparser in /usr/lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (2.18)
Collecting krb5>=0.3.0
  Downloading krb5-0.5.0.tar.gz (220 kB)
     |????????????????????????????????| 220 kB 14.0 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting gssapi>=1.6.0
  Downloading gssapi-1.8.2.tar.gz (94 kB)
     |????????????????????????????????| 94 kB 3.9 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting decorator
  Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)
Building wheels for collected packages: gssapi, krb5
  Building wheel for gssapi (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /bin/python3 /usr/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmptj6zqrpb
       cwd: /tmp/pip-install-rgl5qe_t/gssapi_84b8dab848e24790aee0f6754af32bc2
  Complete output (58 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-cpython-37
  creating build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/sec_contexts.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/names.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/mechs.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/exceptions.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/creds.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/_win_config.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/_utils.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi
  creating build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/named_tuples.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  creating build/lib.linux-x86_64-cpython-37/gssapi/raw/_enum_extensions
  copying gssapi/raw/_enum_extensions/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw/_enum_extensions
  creating build/lib.linux-x86_64-cpython-37/gssapi/tests
  copying gssapi/tests/test_raw.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests
  copying gssapi/tests/test_high_level.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests
  copying gssapi/tests/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests
  copying gssapi/py.typed -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/raw/types.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/sec_contexts.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/oids.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/names.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/misc.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/message.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/mech_krb5.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_set_cred_opt.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_s4u.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc6680_comp_oid.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc6680.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc5801.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc5588.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc5587.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc4178.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_password_add.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_password.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_krb5.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_iov_mic.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_ggf.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_dce_aead.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_dce.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_cred_store.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_cred_imp_exp.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/exceptions.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/creds.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/chan_bindings.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  running build_ext
  building 'gssapi.raw.misc' extension
  creating build/temp.linux-x86_64-cpython-37
  creating build/temp.linux-x86_64-cpython-37/gssapi
  creating build/temp.linux-x86_64-cpython-37/gssapi/raw
  x86_64-unknown-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -O2 -g -fPIC -Igssapi/raw -I./gssapi/raw -I/usr/include/python3.7m -c gssapi/raw/misc.c -o build/temp.linux-x86_64-cpython-37/gssapi/raw/misc.o
  error: command 'x86_64-unknown-linux-gnu-gcc' failed: No such file or directory: 'x86_64-unknown-linux-gnu-gcc'
  ----------------------------------------
  ERROR: Failed building wheel for gssapi
  Building wheel for krb5 (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /bin/python3 /usr/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp63v6q_gf
       cwd: /tmp/pip-install-rgl5qe_t/krb5_1498451d36dc4f77a08707fabdd34711
  Complete output (76 lines):
  Using krb5-config at 'krb5-config'
  Using /usr/lib/libkrb5.so as Kerberos module for platform checks
  Compiling src/krb5/_ccache.pyx
  Compiling src/krb5/_ccache_mit.pyx
  Compiling src/krb5/_ccache_match.pyx
  Compiling src/krb5/_ccache_support_switch.pyx
  Compiling src/krb5/_cccol.pyx
  Compiling src/krb5/_context.pyx
  Compiling src/krb5/_context_mit.pyx
  Compiling src/krb5/_creds.pyx
  Compiling src/krb5/_creds_opt.pyx
  Skipping src/krb5/_creds_opt_heimdal.pyx as it is not supported by the selected Kerberos implementation.
  Compiling src/krb5/_creds_opt_mit.pyx
  Compiling src/krb5/_creds_opt_set_in_ccache.pyx
  Compiling src/krb5/_creds_opt_set_pac_request.pyx
  Compiling src/krb5/_exceptions.pyx
  Compiling src/krb5/_keyblock.pyx
  Compiling src/krb5/_kt.pyx
  Compiling src/krb5/_kt_mit.pyx
  Skipping src/krb5/_kt_heimdal.pyx as it is not supported by the selected Kerberos implementation.
  Compiling src/krb5/_kt_have_content.pyx
  Compiling src/krb5/_principal.pyx
  Skipping src/krb5/_principal_heimdal.pyx as it is not supported by the selected Kerberos implementation.
  Compiling src/krb5/_string.pyx
  Compiling src/krb5/_string_mit.pyx
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-cpython-37
  creating build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/__init__.py -> build/lib.linux-x86_64-cpython-37/krb5
  running egg_info
  writing src/krb5.egg-info/PKG-INFO
  writing dependency_links to src/krb5.egg-info/dependency_links.txt
  writing top-level names to src/krb5.egg-info/top_level.txt
  reading manifest file 'src/krb5.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files found matching '.coverage'
  warning: no previously-included files found matching '.gitignore'
  warning: no previously-included files found matching '.pre-commit-config.yaml'
  warning: no previously-included files matching '*.pyc' found under directory 'tests'
  adding license file 'LICENSE'
  writing manifest file 'src/krb5.egg-info/SOURCES.txt'
  copying src/krb5/_ccache.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_ccache_match.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_ccache_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_ccache_support_switch.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_cccol.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_context.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_context_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds_opt.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds_opt_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds_opt_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds_opt_set_in_ccache.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds_opt_set_pac_request.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_exceptions.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_keyblock.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_kt.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_kt_have_content.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_kt_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_kt_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_principal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_principal_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_string.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_string_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/py.typed -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/python_krb5.h -> build/lib.linux-x86_64-cpython-37/krb5
  running build_ext
  building 'krb5._ccache' extension
  creating build/temp.linux-x86_64-cpython-37
  creating build/temp.linux-x86_64-cpython-37/src
  creating build/temp.linux-x86_64-cpython-37/src/krb5
  x86_64-unknown-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -O2 -g -fPIC -Isrc/krb5 -I/usr/include/python3.7m -c src/krb5/_ccache.c -o build/temp.linux-x86_64-cpython-37/src/krb5/_ccache.o
  error: command 'x86_64-unknown-linux-gnu-gcc' failed: No such file or directory: 'x86_64-unknown-linux-gnu-gcc'
  ----------------------------------------
  ERROR: Failed building wheel for krb5
Failed to build gssapi krb5
ERROR: Could not build wheels for gssapi, krb5 which use PEP 517 and cannot be installed directly
root@vcenter [ ~/dehydrated ]#

The solution?

Use the REST API to install certificates. I figure that an additional benefit is that upgrades of vCenter won’t break or remove the ACME client. A downside is that the key will be stored on the IPA client machine.

No Responses to “Manage FreeIPA certificates with ACME on vSphere (or not)”

Care to comment?