Manage FreeIPA certificates with ACME on vSphere (or not)
Djerk | 4 May 2023 16:28Some days ‘in the office’ are great, others … not so much. But we learn from both, right?!
While writing this article, I ended up ditching the original plan and gave up altogether on running an ACME client on vCenter. Instead, I’m going to use an IPA client to request the certificate and upload it through the REST API, just like I do for FreeIPA certificates for Palo Alto firewalls and Panorama. But, you’ll have to wait for the next article to see how that works out.
My attempts (original ramblings)
This article gave me the inspiration to use ACME to automate FreeIPA certificate renewal for vCenter (vSphere Client UI).
However, I ran into some issues. First, with acme.sh as I couldn’t work out how to get it to update the DNS record on FreeIPA. Let me explain the requirement for dns-01 when using an ACME client on vCenter.
ACME certificate verification requires a challenge-response to authorise the certificate request, else anyone could request a certificate for any domain they like completely defeating the point of SSL as a mechanism of verifying source authenticity. FreeIPA ACME supports two mechanisms for this challenge-response: http-01 and dns-01. http-01 is based on the client hosting a temporary key on port 80 on the domain the certificate is being requested for, thus the ACME client does not need to communicate with the IPA server nor does the client need to be IPA enrolled. dns-01 on the other hand requires the client to control a TXT record in the domain’s DNS zone. If IPA is the nameserver for this zone, then the client will need an IPA account and privileges to manage the DNS entries. The default means of authenticating is Kerberos.
If not acme.sh, then what?
Well, acme.sh isn’t the only ACME client out there and over the last couple of years, I’ve actually shifted some deployments of acme.sh to dehydrated, which supports hooks for ACME servers. Enter ipa-dns-hook by Jimmy Hedman, which uses the Python library HTTPKerberosAuth to authenticate against IPA using stored credentials. I’m not a fan of storing credentials in a text file, but IPA does have decent controls for delegating privileges, which should limit the impact if the account was ever compromised.
What about IPA principals for certificates?
It’s important to realise that traditional certificate requests (IPA web-UI, ipa-getcert etc.) require an IPA principal for the FQDN, ACME does not. Note that the validity lengths are different as well: ‘Traditional’ IPA host and service certificates are valid for 2 years while ACME certificates are valid for 90 days.
How can an ACME client create/change DNS records on a FreeIPA server?
Fraser Tweedale gives this example using a custom hook for Certbot, which uses the ipapython Python library to edit DNS records with Kerberos authentication. You can download his certbot-dns-ipa.py GitHub Gist here. The code is small, this is it in its entirety:
#!/usr/bin/python3
import os
from dns import resolver
from ipalib import api
from ipapython import dnsutil
certbot_domain = os.environ['CERTBOT_DOMAIN']
certbot_validation = os.environ['CERTBOT_VALIDATION']
if 'CERTBOT_AUTH_OUTPUT' in os.environ:
command = 'dnsrecord_del'
else:
command = 'dnsrecord_add'
validation_domain = f'_acme-challenge.{certbot_domain}'
fqdn = dnsutil.DNSName(validation_domain).make_absolute()
zone = dnsutil.DNSName(resolver.zone_for_name(fqdn))
name = fqdn.relativize(zone)
api.bootstrap(context='cli')
api.finalize()
api.Backend.rpcclient.connect()
api.Command[command](zone, name, txtrecord=[certbot_validation], dnsttl=60)
And Jimmy Hedman kindly provides this hook for dehydrated, which uses json calls to IPA using either a userid and password or Kerberos. The relevant section:
def _call_freeipa(json_operation):
headers = {'content-type': 'application/json',
'referer': 'https://%s/ipa' % IPA_SERVER}
if IPA_USER:
# Login and keep a cookie
login_result = requests.post("https://%s/ipa/session/login_password" % IPA_SERVER,
data="user=%s&password=%s" % (IPA_USER, IPA_PASSWORD),
headers={'content-Type':'application/x-www-form-urlencoded',
'referer': 'https://%s/ipa' % IPA_SERVER},
verify='/etc/ipa/ca.crt')
# No auth
auth = None
# Use cookies
cookies=login_result.cookies
else:
# Use kerberos authentication
auth = HTTPKerberosAuth(mutual_authentication=REQUIRED,
sanitize_mutual_error_response=False)
# No cookies
cookies = None
result = requests.post("https://%s/ipa/session/json" % IPA_SERVER,
data=json_operation,
headers=headers,
auth=auth,
cookies=cookies,
verify='/etc/ipa/ca.crt')
retval = result.json()
While these two examples are by no means exhaustive, they do shed light on what’s possible using different approaches.
The (not) solution?
While dehydrated seemed to be an obvious candidate. I’ve used it elsewhere, it runs in plain bash, doesn’t require much and is extendible with hooks. But this is where the dream ended for me, because though dehydrated will run on vCenter, the hook won’t due to requiring the requests-kerberos Python library: It won’t build on my vCenter 7.0.
So while ipa-dns-hook allows for the use of dehydrated on systems that do not have native Kerberos ability by using a userid and password. I’m not prepared to hack vCenter to the point where things might break or official support could end up being affected.
Install dehydrated on vCenter 7
If you want to try it yourself then this is how to get dehydrated and the hook onto vCenter and install the requirements for ipa-dns-hook
WARNING – try this at your own peril!
# Assumed login as root
yum install python3-pip
mkdir -p ~/dehydrated/hooks/ipa-dns && cd ~/dehydrated
wget https://github.com/dehydrated-io/dehydrated/raw/master/dehydrated
chmod +x dehydrated
wget -P hooks/ipa-dns https://github.com/HeMan/ipa-dns-hook/raw/master/{ipa-dns-hook.py,requirements.txt}
pip install -r hooks/ipa-dns/requirements.txt
The problem
requests-kerberos wants to install or update wheels and then things break catastrophically. It didn’t break vCenter for me, but we don’t end up with requests-kerberos either.
root@vcenter [ ~/dehydrated ]# pip3 install -r hooks/ipa-dns/requirements.txt
Requirement already satisfied: requests>=2.21.0 in /usr/lib/python3.7/site-packages (from -r hooks/ipa-dns/requirements.txt (line 1)) (2.24.0)
Collecting requests-kerberos>=0.12.0
Downloading requests_kerberos-0.14.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: urllib3>=1.25.2 in /usr/lib/python3.7/site-packages (from -r hooks/ipa-dns/requirements.txt (line 3)) (1.25.11)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (2.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (2018.8.24)
Collecting pyspnego[kerberos]
Downloading pyspnego-0.9.0-py3-none-any.whl (132 kB)
|????????????????????????????????| 132 kB 10.6 MB/s
Requirement already satisfied: cryptography>=1.3 in /usr/lib/python3.7/site-packages (from requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (2.8)
Requirement already satisfied: six>=1.4.1 in /usr/lib/python3.7/site-packages (from cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (1.12.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/lib/python3.7/site-packages (from cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (1.11.5)
Requirement already satisfied: pycparser in /usr/lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (2.18)
Collecting krb5>=0.3.0
Downloading krb5-0.5.0.tar.gz (220 kB)
|????????????????????????????????| 220 kB 14.0 MB/s
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing wheel metadata ... done
Collecting gssapi>=1.6.0
Downloading gssapi-1.8.2.tar.gz (94 kB)
|????????????????????????????????| 94 kB 3.9 MB/s
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing wheel metadata ... done
Collecting decorator
Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)
Building wheels for collected packages: gssapi, krb5
Building wheel for gssapi (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /bin/python3 /usr/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmptj6zqrpb
cwd: /tmp/pip-install-rgl5qe_t/gssapi_84b8dab848e24790aee0f6754af32bc2
Complete output (58 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-37
creating build/lib.linux-x86_64-cpython-37/gssapi
copying gssapi/sec_contexts.py -> build/lib.linux-x86_64-cpython-37/gssapi
copying gssapi/names.py -> build/lib.linux-x86_64-cpython-37/gssapi
copying gssapi/mechs.py -> build/lib.linux-x86_64-cpython-37/gssapi
copying gssapi/exceptions.py -> build/lib.linux-x86_64-cpython-37/gssapi
copying gssapi/creds.py -> build/lib.linux-x86_64-cpython-37/gssapi
copying gssapi/_win_config.py -> build/lib.linux-x86_64-cpython-37/gssapi
copying gssapi/_utils.py -> build/lib.linux-x86_64-cpython-37/gssapi
copying gssapi/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi
creating build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/named_tuples.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw
creating build/lib.linux-x86_64-cpython-37/gssapi/raw/_enum_extensions
copying gssapi/raw/_enum_extensions/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw/_enum_extensions
creating build/lib.linux-x86_64-cpython-37/gssapi/tests
copying gssapi/tests/test_raw.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests
copying gssapi/tests/test_high_level.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests
copying gssapi/tests/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests
copying gssapi/py.typed -> build/lib.linux-x86_64-cpython-37/gssapi
copying gssapi/raw/types.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/sec_contexts.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/oids.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/names.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/misc.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/message.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/mech_krb5.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_set_cred_opt.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_s4u.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_rfc6680_comp_oid.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_rfc6680.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_rfc5801.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_rfc5588.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_rfc5587.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_rfc4178.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_password_add.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_password.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_krb5.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_iov_mic.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_ggf.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_dce_aead.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_dce.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_cred_store.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/ext_cred_imp_exp.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/exceptions.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/creds.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
copying gssapi/raw/chan_bindings.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
running build_ext
building 'gssapi.raw.misc' extension
creating build/temp.linux-x86_64-cpython-37
creating build/temp.linux-x86_64-cpython-37/gssapi
creating build/temp.linux-x86_64-cpython-37/gssapi/raw
x86_64-unknown-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -O2 -g -fPIC -Igssapi/raw -I./gssapi/raw -I/usr/include/python3.7m -c gssapi/raw/misc.c -o build/temp.linux-x86_64-cpython-37/gssapi/raw/misc.o
error: command 'x86_64-unknown-linux-gnu-gcc' failed: No such file or directory: 'x86_64-unknown-linux-gnu-gcc'
----------------------------------------
ERROR: Failed building wheel for gssapi
Building wheel for krb5 (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /bin/python3 /usr/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp63v6q_gf
cwd: /tmp/pip-install-rgl5qe_t/krb5_1498451d36dc4f77a08707fabdd34711
Complete output (76 lines):
Using krb5-config at 'krb5-config'
Using /usr/lib/libkrb5.so as Kerberos module for platform checks
Compiling src/krb5/_ccache.pyx
Compiling src/krb5/_ccache_mit.pyx
Compiling src/krb5/_ccache_match.pyx
Compiling src/krb5/_ccache_support_switch.pyx
Compiling src/krb5/_cccol.pyx
Compiling src/krb5/_context.pyx
Compiling src/krb5/_context_mit.pyx
Compiling src/krb5/_creds.pyx
Compiling src/krb5/_creds_opt.pyx
Skipping src/krb5/_creds_opt_heimdal.pyx as it is not supported by the selected Kerberos implementation.
Compiling src/krb5/_creds_opt_mit.pyx
Compiling src/krb5/_creds_opt_set_in_ccache.pyx
Compiling src/krb5/_creds_opt_set_pac_request.pyx
Compiling src/krb5/_exceptions.pyx
Compiling src/krb5/_keyblock.pyx
Compiling src/krb5/_kt.pyx
Compiling src/krb5/_kt_mit.pyx
Skipping src/krb5/_kt_heimdal.pyx as it is not supported by the selected Kerberos implementation.
Compiling src/krb5/_kt_have_content.pyx
Compiling src/krb5/_principal.pyx
Skipping src/krb5/_principal_heimdal.pyx as it is not supported by the selected Kerberos implementation.
Compiling src/krb5/_string.pyx
Compiling src/krb5/_string_mit.pyx
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-37
creating build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/__init__.py -> build/lib.linux-x86_64-cpython-37/krb5
running egg_info
writing src/krb5.egg-info/PKG-INFO
writing dependency_links to src/krb5.egg-info/dependency_links.txt
writing top-level names to src/krb5.egg-info/top_level.txt
reading manifest file 'src/krb5.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching '.coverage'
warning: no previously-included files found matching '.gitignore'
warning: no previously-included files found matching '.pre-commit-config.yaml'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
adding license file 'LICENSE'
writing manifest file 'src/krb5.egg-info/SOURCES.txt'
copying src/krb5/_ccache.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_ccache_match.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_ccache_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_ccache_support_switch.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_cccol.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_context.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_context_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_creds.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_creds_opt.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_creds_opt_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_creds_opt_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_creds_opt_set_in_ccache.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_creds_opt_set_pac_request.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_exceptions.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_keyblock.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_kt.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_kt_have_content.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_kt_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_kt_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_principal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_principal_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_string.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/_string_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/py.typed -> build/lib.linux-x86_64-cpython-37/krb5
copying src/krb5/python_krb5.h -> build/lib.linux-x86_64-cpython-37/krb5
running build_ext
building 'krb5._ccache' extension
creating build/temp.linux-x86_64-cpython-37
creating build/temp.linux-x86_64-cpython-37/src
creating build/temp.linux-x86_64-cpython-37/src/krb5
x86_64-unknown-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -O2 -g -fPIC -Isrc/krb5 -I/usr/include/python3.7m -c src/krb5/_ccache.c -o build/temp.linux-x86_64-cpython-37/src/krb5/_ccache.o
error: command 'x86_64-unknown-linux-gnu-gcc' failed: No such file or directory: 'x86_64-unknown-linux-gnu-gcc'
----------------------------------------
ERROR: Failed building wheel for krb5
Failed to build gssapi krb5
ERROR: Could not build wheels for gssapi, krb5 which use PEP 517 and cannot be installed directly
root@vcenter [ ~/dehydrated ]#
The solution?
Use the REST API to install certificates. I figure that an additional benefit is that upgrades of vCenter won’t break or remove the ACME client. A downside is that the key will be stored on the IPA client machine.
No Responses to “Manage FreeIPA certificates with ACME on vSphere (or not)”
Care to comment?