Archive for the 'Systems' category

Manage FreeIPA certificates with ACME on vSphere (or not)

 | 4 May 2023 16:28

Some days ‘in the office’ are great, others … not so much. But we learn from both, right?!

While writing this article, I ended up ditching the original plan and gave up altogether on running an ACME client on vCenter. Instead, I’m going to use an IPA client to request the certificate and upload it through the REST API, just like I do for FreeIPA certificates for Palo Alto firewalls and Panorama. But, you’ll have to wait for the next article to see how that works out.

My attempts (original ramblings)

This article gave me the inspiration to use ACME to automate FreeIPA certificate renewal for vCenter (vSphere Client UI).

However, I ran into some issues. First, with acme.sh as I couldn’t work out how to get it to update the DNS record on FreeIPA. Let me explain the requirement for dns-01 when using an ACME client on vCenter.

ACME certificate verification requires a challenge-response to authorise the certificate request, else anyone could request a certificate for any domain they like completely defeating the point of SSL as a mechanism of verifying source authenticity. FreeIPA ACME supports two mechanisms for this challenge-response: http-01 and dns-01. http-01 is based on the client hosting a temporary key on port 80 on the domain the certificate is being requested for, thus the ACME client does not need to communicate with the IPA server nor does the client need to be IPA enrolled. dns-01 on the other hand requires the client to control a TXT record in the domain’s DNS zone. If IPA is the nameserver for this zone, then the client will need an IPA account and privileges to manage the DNS entries. The default means of authenticating is Kerberos.

If not acme.sh, then what?

Well, acme.sh isn’t the only ACME client out there and over the last couple of years, I’ve actually shifted some deployments of acme.sh to dehydrated, which supports hooks for ACME servers. Enter ipa-dns-hook by Jimmy Hedman, which uses the Python library HTTPKerberosAuth to authenticate against IPA using stored credentials. I’m not a fan of storing credentials in a text file, but IPA does have decent controls for delegating privileges, which should limit the impact if the account was ever compromised.

What about IPA principals for certificates?

It’s important to realise that traditional certificate requests (IPA web-UI, ipa-getcert etc.) require an IPA principal for the FQDN, ACME does not. Note that the validity lengths are different as well: ‘Traditional’ IPA host and service certificates are valid for 2 years while ACME certificates are valid for 90 days.

How can an ACME client create/change DNS records on a FreeIPA server?

Fraser Tweedale gives this example using a custom hook for Certbot, which uses the ipapython Python library to edit DNS records with Kerberos authentication. You can download his certbot-dns-ipa.py GitHub Gist here. The code is small, this is it in its entirety:

#!/usr/bin/python3
import os
from dns import resolver
from ipalib import api 
from ipapython import dnsutil

certbot_domain = os.environ['CERTBOT_DOMAIN']
certbot_validation = os.environ['CERTBOT_VALIDATION']
if 'CERTBOT_AUTH_OUTPUT' in os.environ:
    command = 'dnsrecord_del'
else:
    command = 'dnsrecord_add'

validation_domain = f'_acme-challenge.{certbot_domain}'
fqdn = dnsutil.DNSName(validation_domain).make_absolute()
zone = dnsutil.DNSName(resolver.zone_for_name(fqdn))
name = fqdn.relativize(zone)

api.bootstrap(context='cli')
api.finalize()
api.Backend.rpcclient.connect()

api.Command[command](zone, name, txtrecord=[certbot_validation], dnsttl=60)

And Jimmy Hedman kindly provides this hook for dehydrated, which uses json calls to IPA using either a userid and password or Kerberos. The relevant section:

def _call_freeipa(json_operation):
    headers = {'content-type': 'application/json',
               'referer': 'https://%s/ipa' % IPA_SERVER}
    if IPA_USER:
        # Login and keep a cookie
        login_result = requests.post("https://%s/ipa/session/login_password" % IPA_SERVER,
                                     data="user=%s&password=%s" % (IPA_USER, IPA_PASSWORD),
                                     headers={'content-Type':'application/x-www-form-urlencoded',
                                              'referer': 'https://%s/ipa' % IPA_SERVER},
                                     verify='/etc/ipa/ca.crt')

        # No auth
        auth = None
        # Use cookies
        cookies=login_result.cookies
    else:
        # Use kerberos authentication
        auth = HTTPKerberosAuth(mutual_authentication=REQUIRED,
                                sanitize_mutual_error_response=False)
        # No cookies
        cookies = None

    result = requests.post("https://%s/ipa/session/json" % IPA_SERVER,
                           data=json_operation,
                           headers=headers,
                           auth=auth,
                           cookies=cookies,
                           verify='/etc/ipa/ca.crt')

    retval = result.json()

While these two examples are by no means exhaustive, they do shed light on what’s possible using different approaches.

The (not) solution?

While dehydrated seemed to be an obvious candidate. I’ve used it elsewhere, it runs in plain bash, doesn’t require much and is extendible with hooks. But this is where the dream ended for me, because though dehydrated will run on vCenter, the hook won’t due to requiring the requests-kerberos Python library: It won’t build on my vCenter 7.0.

So while ipa-dns-hook allows for the use of dehydrated on systems that do not have native Kerberos ability by using a userid and password. I’m not prepared to hack vCenter to the point where things might break or official support could end up being affected.

Install dehydrated on vCenter 7

If you want to try it yourself then this is how to get dehydrated and the hook onto vCenter and install the requirements for ipa-dns-hook

WARNING – try this at your own peril!

# Assumed login as root
yum install python3-pip
mkdir -p ~/dehydrated/hooks/ipa-dns && cd ~/dehydrated
wget https://github.com/dehydrated-io/dehydrated/raw/master/dehydrated
chmod +x dehydrated
wget -P hooks/ipa-dns https://github.com/HeMan/ipa-dns-hook/raw/master/{ipa-dns-hook.py,requirements.txt}
pip install -r hooks/ipa-dns/requirements.txt

The problem

requests-kerberos wants to install or update wheels and then things break catastrophically. It didn’t break vCenter for me, but we don’t end up with requests-kerberos either.

root@vcenter [ ~/dehydrated ]# pip3 install -r hooks/ipa-dns/requirements.txt
Requirement already satisfied: requests>=2.21.0 in /usr/lib/python3.7/site-packages (from -r hooks/ipa-dns/requirements.txt (line 1)) (2.24.0)
Collecting requests-kerberos>=0.12.0
  Downloading requests_kerberos-0.14.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: urllib3>=1.25.2 in /usr/lib/python3.7/site-packages (from -r hooks/ipa-dns/requirements.txt (line 3)) (1.25.11)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (2.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3.7/site-packages (from requests>=2.21.0->-r hooks/ipa-dns/requirements.txt (line 1)) (2018.8.24)
Collecting pyspnego[kerberos]
  Downloading pyspnego-0.9.0-py3-none-any.whl (132 kB)
     |????????????????????????????????| 132 kB 10.6 MB/s
Requirement already satisfied: cryptography>=1.3 in /usr/lib/python3.7/site-packages (from requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (2.8)
Requirement already satisfied: six>=1.4.1 in /usr/lib/python3.7/site-packages (from cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (1.12.0)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/lib/python3.7/site-packages (from cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (1.11.5)
Requirement already satisfied: pycparser in /usr/lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=1.3->requests-kerberos>=0.12.0->-r hooks/ipa-dns/requirements.txt (line 2)) (2.18)
Collecting krb5>=0.3.0
  Downloading krb5-0.5.0.tar.gz (220 kB)
     |????????????????????????????????| 220 kB 14.0 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting gssapi>=1.6.0
  Downloading gssapi-1.8.2.tar.gz (94 kB)
     |????????????????????????????????| 94 kB 3.9 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting decorator
  Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)
Building wheels for collected packages: gssapi, krb5
  Building wheel for gssapi (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /bin/python3 /usr/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmptj6zqrpb
       cwd: /tmp/pip-install-rgl5qe_t/gssapi_84b8dab848e24790aee0f6754af32bc2
  Complete output (58 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-cpython-37
  creating build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/sec_contexts.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/names.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/mechs.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/exceptions.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/creds.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/_win_config.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/_utils.py -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi
  creating build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/named_tuples.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  creating build/lib.linux-x86_64-cpython-37/gssapi/raw/_enum_extensions
  copying gssapi/raw/_enum_extensions/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/raw/_enum_extensions
  creating build/lib.linux-x86_64-cpython-37/gssapi/tests
  copying gssapi/tests/test_raw.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests
  copying gssapi/tests/test_high_level.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests
  copying gssapi/tests/__init__.py -> build/lib.linux-x86_64-cpython-37/gssapi/tests
  copying gssapi/py.typed -> build/lib.linux-x86_64-cpython-37/gssapi
  copying gssapi/raw/types.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/sec_contexts.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/oids.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/names.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/misc.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/message.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/mech_krb5.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_set_cred_opt.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_s4u.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc6680_comp_oid.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc6680.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc5801.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc5588.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc5587.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_rfc4178.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_password_add.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_password.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_krb5.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_iov_mic.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_ggf.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_dce_aead.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_dce.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_cred_store.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/ext_cred_imp_exp.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/exceptions.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/creds.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  copying gssapi/raw/chan_bindings.pyi -> build/lib.linux-x86_64-cpython-37/gssapi/raw
  running build_ext
  building 'gssapi.raw.misc' extension
  creating build/temp.linux-x86_64-cpython-37
  creating build/temp.linux-x86_64-cpython-37/gssapi
  creating build/temp.linux-x86_64-cpython-37/gssapi/raw
  x86_64-unknown-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -O2 -g -fPIC -Igssapi/raw -I./gssapi/raw -I/usr/include/python3.7m -c gssapi/raw/misc.c -o build/temp.linux-x86_64-cpython-37/gssapi/raw/misc.o
  error: command 'x86_64-unknown-linux-gnu-gcc' failed: No such file or directory: 'x86_64-unknown-linux-gnu-gcc'
  ----------------------------------------
  ERROR: Failed building wheel for gssapi
  Building wheel for krb5 (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /bin/python3 /usr/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp63v6q_gf
       cwd: /tmp/pip-install-rgl5qe_t/krb5_1498451d36dc4f77a08707fabdd34711
  Complete output (76 lines):
  Using krb5-config at 'krb5-config'
  Using /usr/lib/libkrb5.so as Kerberos module for platform checks
  Compiling src/krb5/_ccache.pyx
  Compiling src/krb5/_ccache_mit.pyx
  Compiling src/krb5/_ccache_match.pyx
  Compiling src/krb5/_ccache_support_switch.pyx
  Compiling src/krb5/_cccol.pyx
  Compiling src/krb5/_context.pyx
  Compiling src/krb5/_context_mit.pyx
  Compiling src/krb5/_creds.pyx
  Compiling src/krb5/_creds_opt.pyx
  Skipping src/krb5/_creds_opt_heimdal.pyx as it is not supported by the selected Kerberos implementation.
  Compiling src/krb5/_creds_opt_mit.pyx
  Compiling src/krb5/_creds_opt_set_in_ccache.pyx
  Compiling src/krb5/_creds_opt_set_pac_request.pyx
  Compiling src/krb5/_exceptions.pyx
  Compiling src/krb5/_keyblock.pyx
  Compiling src/krb5/_kt.pyx
  Compiling src/krb5/_kt_mit.pyx
  Skipping src/krb5/_kt_heimdal.pyx as it is not supported by the selected Kerberos implementation.
  Compiling src/krb5/_kt_have_content.pyx
  Compiling src/krb5/_principal.pyx
  Skipping src/krb5/_principal_heimdal.pyx as it is not supported by the selected Kerberos implementation.
  Compiling src/krb5/_string.pyx
  Compiling src/krb5/_string_mit.pyx
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-cpython-37
  creating build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/__init__.py -> build/lib.linux-x86_64-cpython-37/krb5
  running egg_info
  writing src/krb5.egg-info/PKG-INFO
  writing dependency_links to src/krb5.egg-info/dependency_links.txt
  writing top-level names to src/krb5.egg-info/top_level.txt
  reading manifest file 'src/krb5.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files found matching '.coverage'
  warning: no previously-included files found matching '.gitignore'
  warning: no previously-included files found matching '.pre-commit-config.yaml'
  warning: no previously-included files matching '*.pyc' found under directory 'tests'
  adding license file 'LICENSE'
  writing manifest file 'src/krb5.egg-info/SOURCES.txt'
  copying src/krb5/_ccache.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_ccache_match.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_ccache_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_ccache_support_switch.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_cccol.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_context.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_context_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds_opt.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds_opt_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds_opt_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds_opt_set_in_ccache.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_creds_opt_set_pac_request.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_exceptions.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_keyblock.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_kt.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_kt_have_content.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_kt_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_kt_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_principal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_principal_heimdal.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_string.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/_string_mit.pyi -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/py.typed -> build/lib.linux-x86_64-cpython-37/krb5
  copying src/krb5/python_krb5.h -> build/lib.linux-x86_64-cpython-37/krb5
  running build_ext
  building 'krb5._ccache' extension
  creating build/temp.linux-x86_64-cpython-37
  creating build/temp.linux-x86_64-cpython-37/src
  creating build/temp.linux-x86_64-cpython-37/src/krb5
  x86_64-unknown-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -O2 -g -fPIC -Isrc/krb5 -I/usr/include/python3.7m -c src/krb5/_ccache.c -o build/temp.linux-x86_64-cpython-37/src/krb5/_ccache.o
  error: command 'x86_64-unknown-linux-gnu-gcc' failed: No such file or directory: 'x86_64-unknown-linux-gnu-gcc'
  ----------------------------------------
  ERROR: Failed building wheel for krb5
Failed to build gssapi krb5
ERROR: Could not build wheels for gssapi, krb5 which use PEP 517 and cannot be installed directly
root@vcenter [ ~/dehydrated ]#

The solution?

Use the REST API to install certificates. I figure that an additional benefit is that upgrades of vCenter won’t break or remove the ACME client. A downside is that the key will be stored on the IPA client machine.

Automating FreeIPA certificates on Palo Alto devices

 | 1 May 2023 10:12

pan_getcert is a script that uses ipa-getcert to request a certificate from FreeIPA CA and then uploads it via Palo Alto XML API to a Palo Alto firewall or Panorama, optionally updating one or two SSL/TLS Profiles with the new certificate and commits to activate the changes.

It’s hosted on GitHub: https://github.com/dmgeurts/getcert_paloalto


Introduction

Palo Alto SSL/TLS Profiles

Some uses of FreeIPA certificates on a Palo Alto firewall or Panorama:

  • Global Protect Gateway
  • Global Protect Portal
  • Management UI

Should one use an internal certificate for an external service?

There’s no need to get a publicly signed certificate as long as all Global Protect clients trust the FreeIPA (root) CA. A nice bonus is not having to permit inbound HTTP-01 traffic, which in Let’s Encrypt’s case is cloud-hosted (what else is hosted there?). Or exposing internal domains, see: Terence Eden’s Blog – Should you use Let’s Encrypt for internal hostnames?

FreeIPA CA

FreeIPA with Dogtag PKI supports certificate requests and renewals from Certmonger via ipa-getcert and since FreeIPA v4.9 also via ACME.

ACME vs Certmonger

Palo Alto firewalls nor Panorama natively support ACME, nor would I expect them to. For my lab environment, ipa-getcert is a natural choice as the server in use for certificate management is FreeIPA enrolled already, hence I have no need for anonymous ACME.

Prerequisites

pan_getcert uses ipa-getcert, the requirements are identical as far as FreeIPA is concerned:

  • An enrolled FreeIPA client with reachability to the Palo Alto firewall or Panorama.
    • Test with: nc -zv fw-mgmt.domain.local 443
    • pan-python installed
  • A manually added host (with the Service hostname).
    • The manual host must be ‘managed by’ the host on which pan_getcert will be executed.
  • A Service Principal for the service domain
    • The Service Principal must be ‘managed by’ the host on which pan_getcert will be executed.
  • An API key for the Palo Alto firewall or Panorama
    • Ideally, use a system account to tie the API key to, users tend to churn and break their API keys.
    • Store the API key in /etc/ipa/.panrc
  • IPA CA root certificate manually installed on the Palo Alto firewall or Panorama, as a trusted CA.

To install pan-python on Ubuntu 22.04:

sudo apt install python3-pip
sudo pip install pan-python

To generate the API key; first, create a user account for/on the Palo Alto and then run this from the Linux host:

panxapi.py -h PAN_MGMT_IP_OR_FQDN -l USERNAME:'PASSWORD' -k

Copy the key and paste it into /etc/ipa/.panrc as follows:

api_key=C2M1P2h1tDEz8zF3SwhF2dWC1gzzhnE1qU39EmHtGZM=

And secure the file:

sudo chmod 600 /etc/ipa/.panrc

To install getcert_paloalto:

wget https://github.com/dmgeurts/getcert_paloalto/edit/master/pan_{get,inst}cert
chmod +x pan_{get,inst}cert
sudo cp pan_{get,inst}cert /usr/local/bin/
rm pan_{get,inst}cert

Automating the two

Now that the scope is clear, it’s time to explain how getcert_paloalto automates certificate requests, deployment and renewals.

Certmonger supports has pre- and post-save commands, these can be used to run things like systemctl restart apache but also more complex commands like the name of a script with its various options. It’s this post-save option that automates the renewal process but also makes certificate deployment interesting as the same command is executed when the certificate is first created (the first save).

This is the reason there are two scripts for the solution. Also note that, as opposed to ACME, for example, no crontab entry is needed for the renewal of FreeIPA certificates. Cernmonger takes care of monitoring certificates and renewing them before they expire.

pan_getcert – Introduction

pan_getcert uses ipa-getcert (part of freeipa-client) to request a certificate from IPA CA and then sets the post-save command to pan_instcert with options based on the parameters parsed to pan_getcert. The nodes involved and the processes used are as follows:

Process overview: FreeIPA <-- Linux --> Palo Alto.

*) My OS of choice is Ubuntu hence the Ubuntu logo for the FreeIPA client machine.

pan_getcert – Options

The bare minimum options to parse are the certificate Subject (aka Common Name) [-c] and the Palo Alto device hostname.

Note: pan_getcert must be run as root, ideally using sudo. This enables admins to give certificate managers the delegated privilege to run pan_getcert as root rather than all the commands in it that require elevated privileges.

Optional arguments:

  • -n Certificate name in the Palo Alto configuration, if none is given the Certificate Subject will be used.
  • -Y Certificate name postfix of a four-digit year, prevents the existing certificate from being replaced. <certificate.name_2023>
  • -p Name of the ‘primary’ SSL/TLS Profile, will see the currently configured certificate replaced with the new certificate. if none is given, no SSL/TLS Profile will be updated.
  • -s Name of the ‘secondary’ SSL/TLS Profile, will see the currently configured certificate replaced with the new certificate. Requires [-p] to be set.

The secondary Profile option is useful in cases where the same certificate must be updated on two different SSL/TLS Profiles. It is not possible to request more than one certificate using pan_getcert from FreeIPA.

Usage: pan_getcert [-hv] -c CERT_CN [-n CERT_NAME] [-Y] [OPTIONS] FQDN
This script requests a certificate from FreeIPA using ipa-getcert and calls a partner
script to deploy the certificate to a Palo Alto firewall or Panorama.

    FQDN              Fully qualified name of the Palo Alto firewall or Panorama
                      interface. Must be reachable from this host on port TCP/443.
    -c CERT_CN        REQUIRED. Common Name (Subject) of the certificate (must be a
                      FQDN). Will also present in the certificate as a SAN.

OPTIONS:
    -n CERT_NAME      Name of the certificate in PanOS configuration. Defaults to the
                      certificate Common Name.
    -Y                Parsed to pan_instcert to append the current year '_YYYY' to
                      the certificate name.

    -p PROFILE_NAME   Apply the certificate to a (primary) SSL/TLS Service Profile.
    -s PROFILE_NAME   Apply the certificate to a (secondary) SSL/TLS Service Profile.

    -h                Display this help and exit.
    -v                Verbose mode.

pan_getcert – Actions

  1. Uses the privileges set in FreeIPA (managed by) to call ipa-getcert and request a certificate from FreeIPA.
  2. ipa-getcert will automatically renew a certificate when it’s due, as long as the FQDN DNS record resolves, and the host and Service Principal still exist in FreeIPA.
  3. Sets the post-save command to pan_instcert with the same parameters as issued to pan_getcert, for automated installation of renewed certificates.
    • Post-save will run on the first certificate save, using pan_instcert for certificate installation.

pan_instcert – Introduction

Uses panxapi.py from pan-python and can be used on its own. For example, if the certificate is created without pan_getcert. Or one might choose to use it as the post-save command of a certificate already monitored by Certmonger.

Note: pan_instcert must be run as root, ideally using sudo. This enables admins to give certificate managers the delegated privilege to run pan_instcert as root rather than all the commands in it that require elevated privileges.

pan_getcert – Options

pan_instcert options and arguments are deliberately identical to pan_getcert.

Usage: pan_instcert [-hv] -c CERT_CN [-n CERT_NAME] [OPTIONS] FQDN
This script uploads a certificate issued by ipa-getcert to a Palo Alto firewall
or Panorama and optionally adds it to up to two SSL/TLS Profiles.

    FQDN              Fully qualified name of the Palo Alto firewall or Panorama
                      interface. Must be reachable from this host on port TCP/443.
    -c CERT_CN        REQUIRED. Common Name (Subject) of the certificate, to find
                      the certificate and key files.

OPTIONS:
    -n CERT_NAME      Name of the certificate in PanOS configuration. Defaults to the
                      certificate Common Name.
    -Y                Append the current year '_YYYY' to the certificate name.

    -p PROFILE_NAME   Apply the certificate to a (primary) SSL/TLS Service Profile.
    -s PROFILE_NAME   Apply the certificate to a (secondary) SSL/TLS Service Profile.

    -h                Display this help and exit.
    -v                Verbose mode.

pan_instcert – Actions

  1. Randomly generates a certificate passphrase using “openssl rand”.
  2. Creates a temporary, password-protected PKCS12 cert file /tmp/getcert_pkcs12.pfx from the individual private and public keys issued by ipa-getcert.
  3. Uploads the temporary PKCS12 file to the firewall using the randomly-generated passphrase.
    • (Optionally) adds a year (in 4-digit notation) to the certificate name.
  4. Deletes the temporary PKCS12 certificate from the Linux host.
  5. (Optionally) applies the certificate to up to two SSL/TLS Profiles.
    • Single SSL/TLS Profile: For example for the Management UI SSL/TLS profile.
    • Two SSL/TLS Profiles: For example for GlobalProtect Portal and GlobalProtect Gateway SSL/TLS Profiles.
  6. Commits the candidate configuration (synchronously) and reports the commit result.
  7. Logs all output to `/var/log/pan_instcert.log`.

Command execution

The expected output when requesting a certificate with pan_getcert is:

$ sudo pan_getcert -v -c gp.domain.com -Y -p GP_PORTAL_PROFILE -s GP_EXT_GW_PROFILE fw01.domain.local
Certificate Common Name: gp.domain.com
  verbose=1
  CERT_CN: gp.domain.com
  CERT_NAME: gp.domain.com_2023
  PAN_FQDN: fw01.domain.local
Primary SSL/TLS Profile name: GP_PORTAL_PROFILE
Secondary SSL/TLS Profile name: GP_EXT_GW_PROFILE
New signing request "20230427151532" added.
Certificate requested for: gp.domain.com
  Certificate issue took 6 seconds, waiting for the post-save process to finish.
  Certificate install and commit by the post-save process on: fw01.domain.local took 84 seconds.
FINISHED: Check the Palo Alto firewall or Panorama to check the commit succeeded.

And pan_instcert will log to /var/log/pan_instcert.log.

[2023-04-27 15:15:33+00:00]: START of pan_instcert.
[2023-04-27 15:15:33+00:00]: Certificate Common Name: gp.domain.com
[2023-04-27 15:15:34+00:00]: XML API output for crt: <response status="success"><result>Successfully imported gp.domain.com_2023 into candidate configuration</result></response>
[2023-04-27 15:15:35+00:00]: XML API output for key: <response status="success"><result>Successfully imported gp.domain.com_2023 into candidate configuration</result></response>
[2023-04-27 15:15:35+00:00]: Finished uploading certificate: gp.domain.com_2023
[2023-04-27 15:15:37+00:00]: Starting commit, please be patient.
[2023-04-27 15:17:01+00:00]: commit: success: "Configuration committed successfully"
[2023-04-27 15:17:01+00:00]: The commit took 84 seconds to complete.
[2023-04-27 15:17:01+00:00]: END - Finished certificate installation to: fw01.domain.local

Both pan_getcert and pan_instcert will report back how long it took to do certain tasks:

  • pan_getcert
    • Time spent waiting for the certificate to be issued.
    • Time spent waiting for pan_instcert to complete.
  • pan_instcert
    • Time spent waiting for the commit to finish.

Logrotate for pan_instcert.log

This log file shouldn’t grow quickly unless there’s a problem or the number of monitored certificates grows very large. By default, IPA certificates have a two-year validity, thus monthly log lines will average at about 0.375 per certificate (9 lines / 24 months). Generally speaking, you should not expect this file to grow by more than a few lines a year.

The bigger risk is misconfiguration or changes of ipa-getcert and the Palo Alto API. And from experience, I can confirm that getcert is very persistent in retrying failed post-save commands. A few times while testing my code the log file grew to >20GB within minutes, containing only repeated pan_instcert usage instructions.

Count yourself warned! A suggestion for a logrotate.d file is:

# /etc/logrotate.d/pan_instcert
/var/log/pan_instcert.log { 
	missingok
	rotate 5
	yearly
	size 50M
	notifempty
	create
}

Verify Certificate Status and Post-save Command

Use ipa-getcert to check that the certificate ended up in status MONITORING and that the post-save command is set according to the parameters and values parsed to pan_getcert:

$ sudo ipa-getcert list
Number of certificates and requests being tracked: 2.
Request ID '20230427151532':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/etc/ssl/private/gp.domain.com.key'
        certificate: type=FILE,location='/etc/ssl/certs/gp.domain.com.crt'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.LOCAL
        subject: CN=gp.domain.com,O=IPA.LOCAL
        issued: 2023-04-27 16:15:33 BST
        expires: 2025-04-27 16:15:33 BST
        dns: gp.domain.com
        principal name: HTTP/gp.domain.com@MM.EU
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/local/bin/pan_instcert -c gp.domain.com -n gp.domain.com -Y -p GP_PORTAL_PROFILE -s GP_EXT_GW_PROFILE fw01.domain.local
        track: yes
        auto-renew: yes
Request ID '20230428001508':
        status: MONITORING
        stuck: no
        key pair storage: type=FILE,location='/etc/ssl/private/fw01.domain.local.key'
        certificate: type=FILE,location='/etc/ssl/certs/fw01.domain.local.crt'
        CA: IPA
        issuer: CN=Certificate Authority,O=IPA.LOCAL
        subject: CN=fw01.domain.local,O=IPA.LOCAL
        issued: 2023-04-27 16:15:33 BST
        expires: 2025-04-27 16:15:33 BST
        dns: fw01.domain.local
        principal name: HTTP/fw01.domain.local@IPA.LOCAL
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/local/bin/pan_instcert -c fw01.domain.local -n fw01.domain.local -Y -p MGMT_UI_PROFILE fw01.domain.local
        track: yes
        auto-renew: yes

Note the status, location of the saved files and the post-save command. Tracking and auto-renew are enabled by default by ipa-getcert.

pan_instcert will only log to stdout when executed directly. However, it will always log to /var/log/pan_instcert.log. When requesting certificates, it can be helpful to run a tail to see the post-save command logging in real-time. If the log file doesn’t yet exist the tail will fail.

sudo touch -a /var/log/pan_instcert.log
sudo tail -f /var/log/pan_instcert.log

Wrong SSL/TLS profile name

If the SSL/TLS Service Profile doesn’t exist it will be created, but the following error will be shown in /var/log/pan_instcert.log and the commit will fail:

commit: success: "Validation Error:
 ssl-tls-service-profile -> Test_profile  is missing 'protocol-settings'
 ssl-tls-service-profile is invalid"

Verify the API calls on the Palo Alto Firewall or Panorama

Check the following locations on the Palo Alto firewall for additional confirmation: Monitor >> Logs >> Configuration There should be 3-5 operations shown, depending on whether or not the SSL/TLS service profile(s) are being updated.

  1. A web upload to /config/shared/certificate.
  2. A web upload to /config/shared/certificate/entry[@name=’FQDN(_YYYY)’], under the FreeIPA root CA certificate.
  3. One or more web “set” commands to /config/shared/ssl-tls-service-profile/entry[@name=’YOUR_PROFILE(S)’]
  4. And a web “commit” operation.

To see all API actions, filter by the admin username used for API key: ( admin eq [api-admin] )

Under Device >> Certificate Management >> Certificates the new certificate should be shown with a valid status and under the manually imported IPA CA root certificate.

If any SSL/TLS Profiles were parsed, then under Device >> Certificate Management >> SSL/TLS Service Profile the respective profiles should show the new certificate has replaced the previous certificate.

Recent script changes

[2023-05-02] Looking into issuing a subordinate certificate from FreeIPA for a Palo Alto firewall, for user VPN certificates and ideally SSL interception. I found that I needed to specify a Certificate Profile in the ipa-getcert command, I thus went about adding option -T to pan_getcert.

Furthermore, I also added the following options:

  • -b Key bit length, 2048 is the default but it’s good to be able to request 3072 and 4096 length RSA certificates.`-b Key bit length, 2048 is the default but it’s good to be able to request 3072 and 4096 length RSA certificates.
  • -G Certificate type, currently only RSA is supported by FreeIPA. But it’s good to be ready for when EC and ECDSA certificates can be issued from FreeIPA. I don’t think this will be any time soon, but the code is there now.
  • -S Service type. The subordinate certificate isn’t for an HTTP service, in fact it’s best suited to being tied to a host rather than a service. So now there’s an option to specify a service, if omitted HTTP is assumed.