Manage FreeIPA certificates with ACME on vSphere (or not)
Djerk | 4 May 2023 16:28Some days ‘in the office’ are great, others … not so much. But we learn from both, right?!
While writing this article, I ended up ditching the original plan and gave up altogether on running an ACME client on vCenter. Instead, I’m going to use an IPA client to request the certificate and upload it through the REST API, just like I do for FreeIPA certificates for Palo Alto firewalls and Panorama. But, you’ll have to wait for the next article to see how that works out.
My attempts (original ramblings)
This article gave me the inspiration to use ACME to automate FreeIPA certificate renewal for vCenter (vSphere Client UI).
However, I ran into some issues. First, with acme.sh as I couldn’t work out how to get it to update the DNS record on FreeIPA. Let me explain the requirement for dns-01 when using an ACME client on vCenter.
ACME certificate verification requires a challenge-response to authorise the certificate request, else anyone could request a certificate for any domain they like completely defeating the point of SSL as a mechanism of verifying source authenticity. FreeIPA ACME supports two mechanisms for this challenge-response: http-01 and dns-01. http-01 is based on the client hosting a temporary key on port 80 on the domain the certificate is being requested for, thus the ACME client does not need to communicate with the IPA server nor does the client need to be IPA enrolled. dns-01 on the other hand requires the client to control a TXT record in the domain’s DNS zone. If IPA is the nameserver for this zone, then the client will need an IPA account and privileges to manage the DNS entries. The default means of authenticating is Kerberos.
If not acme.sh, then what?
Well, acme.sh isn’t the only ACME client out there and over the last couple of years, I’ve actually shifted some deployments of acme.sh to dehydrated, which supports hooks for ACME servers. Enter ipa-dns-hook by Jimmy Hedman, which uses the Python library HTTPKerberosAuth to authenticate against IPA using stored credentials. I’m not a fan of storing credentials in a text file, but IPA does have decent controls for delegating privileges, which should limit the impact if the account was ever compromised.
What about IPA principals for certificates?
It’s important to realise that traditional certificate requests (IPA web-UI, ipa-getcert etc.) require an IPA principal for the FQDN, ACME does not. Note that the validity lengths are different as well: ‘Traditional’ IPA host and service certificates are valid for 2 years while ACME certificates are valid for 90 days.
How can an ACME client create/change DNS records on a FreeIPA server?
Fraser Tweedale gives this example using a custom hook for Certbot, which uses the ipapython Python library to edit DNS records with Kerberos authentication. You can download his certbot-dns-ipa.py GitHub Gist here. The code is small, this is it in its entirety:
#!/usr/bin/python3 import os from dns import resolver from ipalib import api from ipapython import dnsutil certbot_domain = os.environ['CERTBOT_DOMAIN'] certbot_validation = os.environ['CERTBOT_VALIDATION'] if 'CERTBOT_AUTH_OUTPUT' in os.environ: command = 'dnsrecord_del' else: command = 'dnsrecord_add' validation_domain = f'_acme-challenge.{certbot_domain}' fqdn = dnsutil.DNSName(validation_domain).make_absolute() zone = dnsutil.DNSName(resolver.zone_for_name(fqdn)) name = fqdn.relativize(zone) api.bootstrap(context='cli') api.finalize() api.Backend.rpcclient.connect() api.Command[command](zone, name, txtrecord=[certbot_validation], dnsttl=60)
And Jimmy Hedman kindly provides this hook for dehydrated, which uses json calls to IPA using either a userid and password or Kerberos. The relevant section:
def _call_freeipa(json_operation): headers = {'content-type': 'application/json', 'referer': 'https://%s/ipa' % IPA_SERVER} if IPA_USER: # Login and keep a cookie login_result = requests.post("https://%s/ipa/session/login_password" % IPA_SERVER, data="user=%s&password=%s" % (IPA_USER, IPA_PASSWORD), headers={'content-Type':'application/x-www-form-urlencoded', 'referer': 'https://%s/ipa' % IPA_SERVER}, verify='/etc/ipa/ca.crt') # No auth auth = None # Use cookies cookies=login_result.cookies else: # Use kerberos authentication auth = HTTPKerberosAuth(mutual_authentication=REQUIRED, sanitize_mutual_error_response=False) # No cookies cookies = None result = requests.post("https://%s/ipa/session/json" % IPA_SERVER, data=json_operation, headers=headers, auth=auth, cookies=cookies, verify='/etc/ipa/ca.crt') retval = result.json()
While these two examples are by no means exhaustive, they do shed light on what’s possible using different approaches.
The (not) solution?
While dehydrated seemed to be an obvious candidate. I’ve used it elsewhere, it runs in plain bash, doesn’t require much and is extendible with hooks. But this is where the dream ended for me, because though dehydrated will run on vCenter, the hook won’t due to requiring the requests-kerberos Python library: It won’t build on my vCenter 7.0.
So while ipa-dns-hook allows for the use of dehydrated on systems that do not have native Kerberos ability by using a userid and password. I’m not prepared to hack vCenter to the point where things might break or official support could end up being affected.
Install dehydrated on vCenter 7
If you want to try it yourself then this is how to get dehydrated and the hook onto vCenter and install the requirements for ipa-dns-hook
WARNING – try this at your own peril!
# Assumed login as root yum install python3-pip mkdir -p ~/dehydrated/hooks/ipa-dns && cd ~/dehydrated wget https://github.com/dehydrated-io/dehydrated/raw/master/dehydrated chmod +x dehydrated wget -P hooks/ipa-dns https://github.com/HeMan/ipa-dns-hook/raw/master/{ipa-dns-hook.py,requirements.txt} pip install -r hooks/ipa-dns/requirements.txt
The problem
requests-kerberos wants to install or update wheels and then things break catastrophically. It didn’t break vCenter for me, but we don’t end up with requests-kerberos either.
root@vcenter [ ~/dehydrated ]# pip3 install -r hooks/ipa-dns/requirements.txt Requirement already satisfied: requests>=2.21.0 in /usr/lib/python3.7/site-packages (from -r hooks/ipa-dns/requirements.txt (line 1)) (2.24.0) Collecting requests-kerberos>=0.12.0 Downloading requests_kerberos-0.14.0-py2.py3-none-any.whl (11 kB) Requirement already satisfied: urllib3>=1.25.2 in /usr/lib/python3.7/site-packages (from -r hooks/ipa-dns/requirements.txt (line 3)) (1.25.11) Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (3.0.4) Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (2.7) Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (2018.8.24) Collecting pyspnego[kerberos] Downloading pyspnego-0.9.0-py3-none-any.whl (132 kB) |????????????????????????????????| 132 kB 10.6 MB/s Requirement already satisfied: cryptography>=1.3 in /usr/lib/python3.7/site-packages (from requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (2.8) Requirement already satisfied: six>=1.4.1 in /usr/lib/python3.7/site-packages (from cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (1.12.0) Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/lib/python3.7/site-packages (from cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (1.11.5) Requirement already satisfied: pycparser in /usr/lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (2.18) Collecting krb5>=0.3.0 Downloading krb5-0.5.0.tar.gz (220 kB) |????????????????????????????????| 220 kB 14.0 MB/s Installing build dependencies ... done Getting requirements to build wheel ... done Installing backend dependencies ... done Preparing wheel metadata ... done Collecting gssapi>=1.6.0 Downloading gssapi-1.8.2.tar.gz (94 kB) |????????????????????????????????| 94 kB 3.9 MB/s Installing build dependencies ... done Getting requirements to build wheel ... done Installing backend dependencies ... done Preparing wheel metadata ... done Collecting decorator Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB) Building wheels for collected packages: gssapi, krb5 Building wheel for gssapi (PEP 517) ... error ERROR: Command errored out with exit status 1: command: /bin/python3 /usr/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmptj6zqrpb cwd: /tmp/pip-install-rgl5qe_t/gssapi_84b8dab848e24790aee0f6754af32bc2 Complete output (58 lines): running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-cpython-37 creating build/lib.linux-x86_64-cpython-37/gssapi copying gssapi/sec_contexts.py -> build/lib.linux-x86_64-cpython-37/gssapi copying gssapi/names.py -> build/lib.linux-x86_64-cpython-37/gssapi copying gssapi/mechs.py -> build/lib.linux-x86_64-cpython-37/gssapi copying gssapi/exceptions.py -> build/lib.linux-x86_64-cpython-37/gssapi copying gssapi/creds.py -> build/lib.linux-x86_64-cpython-37/gssapi copying gssapi/_win_config.py -> build/lib.linux-x86_64-cpython-37/gssapi copying gssapi/_utils.py -> build/lib.linux-x86_64-cpython-37/gssapi copying gssapi/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi creating build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/named_tuples.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw creating build/lib.linux-x86_64-cpython-37/gssapi/raw/_enum_extensions copying gssapi/raw/_enum_extensions/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw/_enum_extensions creating build/lib.linux-x86_64-cpython-37/gssapi/tests copying gssapi/tests/test_raw.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests copying gssapi/tests/test_high_level.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests copying gssapi/tests/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests copying gssapi/py.typed -> build/lib.linux-x86_64-cpython-37/gssapi copying gssapi/raw/types.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/sec_contexts.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/oids.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/names.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/misc.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/message.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/mech_krb5.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_set_cred_opt.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_s4u.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_rfc6680_comp_oid.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_rfc6680.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_rfc5801.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_rfc5588.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_rfc5587.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_rfc4178.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_password_add.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_password.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_krb5.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_iov_mic.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_ggf.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_dce_aead.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_dce.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_cred_store.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/ext_cred_imp_exp.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/exceptions.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/creds.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw copying gssapi/raw/chan_bindings.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw running build_ext building 'gssapi.raw.misc' extension creating build/temp.linux-x86_64-cpython-37 creating build/temp.linux-x86_64-cpython-37/gssapi creating build/temp.linux-x86_64-cpython-37/gssapi/raw x86_64-unknown-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -O2 -g -fPIC -Igssapi/raw -I./gssapi/raw -I/usr/include/python3.7m -c gssapi/raw/misc.c -o build/temp.linux-x86_64-cpython-37/gssapi/raw/misc.o error: command 'x86_64-unknown-linux-gnu-gcc' failed: No such file or directory: 'x86_64-unknown-linux-gnu-gcc' ---------------------------------------- ERROR: Failed building wheel for gssapi Building wheel for krb5 (PEP 517) ... error ERROR: Command errored out with exit status 1: command: /bin/python3 /usr/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp63v6q_gf cwd: /tmp/pip-install-rgl5qe_t/krb5_1498451d36dc4f77a08707fabdd34711 Complete output (76 lines): Using krb5-config at 'krb5-config' Using /usr/lib/libkrb5.so as Kerberos module for platform checks Compiling src/krb5/_ccache.pyx Compiling src/krb5/_ccache_mit.pyx Compiling src/krb5/_ccache_match.pyx Compiling src/krb5/_ccache_support_switch.pyx Compiling src/krb5/_cccol.pyx Compiling src/krb5/_context.pyx Compiling src/krb5/_context_mit.pyx Compiling src/krb5/_creds.pyx Compiling src/krb5/_creds_opt.pyx Skipping src/krb5/_creds_opt_heimdal.pyx as it is not supported by the selected Kerberos implementation. Compiling src/krb5/_creds_opt_mit.pyx Compiling src/krb5/_creds_opt_set_in_ccache.pyx Compiling src/krb5/_creds_opt_set_pac_request.pyx Compiling src/krb5/_exceptions.pyx Compiling src/krb5/_keyblock.pyx Compiling src/krb5/_kt.pyx Compiling src/krb5/_kt_mit.pyx Skipping src/krb5/_kt_heimdal.pyx as it is not supported by the selected Kerberos implementation. Compiling src/krb5/_kt_have_content.pyx Compiling src/krb5/_principal.pyx Skipping src/krb5/_principal_heimdal.pyx as it is not supported by the selected Kerberos implementation. Compiling src/krb5/_string.pyx Compiling src/krb5/_string_mit.pyx running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-cpython-37 creating build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/__init__.py -> build/lib.linux-x86_64-cpython-37/krb5 running egg_info writing src/krb5.egg-info/PKG-INFO writing dependency_links to src/krb5.egg-info/dependency_links.txt writing top-level names to src/krb5.egg-info/top_level.txt reading manifest file 'src/krb5.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no previously-included files found matching '.coverage' warning: no previously-included files found matching '.gitignore' warning: no previously-included files found matching '.pre-commit-config.yaml' warning: no previously-included files matching '*.pyc' found under directory 'tests' adding license file 'LICENSE' writing manifest file 'src/krb5.egg-info/SOURCES.txt' copying src/krb5/_ccache.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_ccache_match.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_ccache_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_ccache_support_switch.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_cccol.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_context.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_context_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_creds.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_creds_opt.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_creds_opt_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_creds_opt_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_creds_opt_set_in_ccache.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_creds_opt_set_pac_request.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_exceptions.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_keyblock.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_kt.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_kt_have_content.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_kt_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_kt_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_principal.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_principal_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_string.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/_string_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/py.typed -> build/lib.linux-x86_64-cpython-37/krb5 copying src/krb5/python_krb5.h -> build/lib.linux-x86_64-cpython-37/krb5 running build_ext building 'krb5._ccache' extension creating build/temp.linux-x86_64-cpython-37 creating build/temp.linux-x86_64-cpython-37/src creating build/temp.linux-x86_64-cpython-37/src/krb5 x86_64-unknown-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -O2 -g -fPIC -Isrc/krb5 -I/usr/include/python3.7m -c src/krb5/_ccache.c -o build/temp.linux-x86_64-cpython-37/src/krb5/_ccache.o error: command 'x86_64-unknown-linux-gnu-gcc' failed: No such file or directory: 'x86_64-unknown-linux-gnu-gcc' ---------------------------------------- ERROR: Failed building wheel for krb5 Failed to build gssapi krb5 ERROR: Could not build wheels for gssapi, krb5 which use PEP 517 and cannot be installed directly root@vcenter [ ~/dehydrated ]#
The solution?
Use the REST API to install certificates. I figure that an additional benefit is that upgrades of vCenter won’t break or remove the ACME client. A downside is that the key will be stored on the IPA client machine.
Tags: FreeIPA,Security,SSL,VMware,vSphere
Categories: Security, Systems, Work
No Comments »
Recent Comments