TLS certificate setup

Updated 6 Days AgoPublic

Actions

Pacemaker optionally supports encrypting communications with TLS certificates for two things:

1. Pacemaker Remote node connections
2. Remote CIB operations (CIB_user=... CIB_encrypted=... CIB_port=... cibadmin -Q)

This is especially important for remote CIB operations which have no encryption at all otherwise.

Testing this requires a fair bit of setup. We assume that users will use TLS certificates as follows:

• They will set up their own Certificate Authority (CA). In the real world, you pay a real CA a lot of money to sign your credentials. This is because web browsers have lists of CAs they trust, and so if you use one of those then your credentials will be trusted. Otherwise the user will get a scary error about self-signed certificates and things like that. However, we expect that people will keep their clusters on private networks and connect via our own command line tools, so there's no real need for a trusted, real CA.

• They will generate one set of credentials for each server. If they have ten Pacemaker Remote nodes (remember - remote nodes are the server, the cluster is the client) then each would have its own server cert and key.

• They will generate one set of credentials for each client. If they have ten admin machines that need to run cibadmin to do remote CIB operations, then each would have its own client cert and key. This makes it possible to revoke a single client's credentials if necessary without a lot of annoying work.

In addition, this document will assume there is not an intermediate CA involved. In the real world, you would create a root CA and only use that to sign credentials for intermediate CAs. The intermediate CA would then sign the client/server credentials. The reason for this is that if your CA gets compromised, everything it has ever signed is now compromised as well. The root CA credentials would be locked away and only taken out on the rare time when you need a new intermediate. Pacemaker should support this just fine as long as the root CA and intermediate CA credentials get bundled together. I'm not going to cover how to do that here.

If you look around, you'll find tons of explanations for how to set this stuff up. These instructions worked for me. There very well may be much simpler ways to do all of this.

To generate TLS credentials, you need to do all of the following...

Create your own CA

Set up the directory structure

$ mkdir -p ~/cert-stuff/ca/{certs,newcerts,private,csr}
$ mkdir -p ~/cert-stuff/{clients,servers}
$ cd ~/cert-stuff/ca
$ echo 01 > serial
$ echo 01 > crlnumber
$ touch index.txt

Create openssl.cnf

I'm not sure how much of this is necessary at all. I haven't experimented with simplification. At the least, check dir, countryName, stateOrProvinceName, localityName, and organizationName and set them to something that makes sense for you.

[ ca ]                                                   # The default CA section
default_ca = CA_default                                  # The default CA name

[ CA_default ]                                           # Default settings for the CA
dir               = /root/cert-stuff/ca                  # CA directory
certs             = $dir/certs                           # Certificates directory
crl_dir           = $dir/crl                             # CRL directory
new_certs_dir     = $dir/newcerts                        # New certificates directory
database          = $dir/index.txt                       # Certificate index file
serial            = $dir/serial                          # Serial number file
RANDFILE          = $dir/private/.rand                   # Random number file
private_key       = $dir/private/ca.key.pem              # Root CA private key
certificate       = $dir/certs/ca.cert.pem               # Root CA certificate
crl               = $dir/crl/ca.crl.pem                  # Root CA CRL
crlnumber         = $dir/crlnumber                       # Root CA CRL number
crl_extensions    = crl_ext                              # CRL extensions
default_crl_days  = 30                                   # Default CRL validity days
default_md        = sha256                               # Default message digest
preserve          = no                                   # Preserve existing extensions
email_in_dn       = no                                   # Exclude email from the DN
name_opt          = ca_default                           # Formatting options for names
cert_opt          = ca_default                           # Certificate output options
policy            = policy_strict                        # Certificate policy
unique_subject    = no                                   # Allow multiple certs with the same DN

[ policy_strict ]                                        # Policy for stricter validation
countryName             = match                          # Must match the issuer's country
stateOrProvinceName     = match                          # Must match the issuer's state
organizationName        = match                          # Must match the issuer's organization
organizationalUnitName  = optional                       # Organizational unit is optional
commonName              = optional                       # Must provide a common name
emailAddress            = optional                       # Email address is optional

[ req ]                                                  # Request settings
default_bits        = 2048                               # Default key size
distinguished_name  = req_distinguished_name             # Default DN template
string_mask         = utf8only                           # UTF-8 encoding
default_md          = sha256                             # Default message digest
prompt              = no                                 # Non-interactive mode

[ req_distinguished_name ]                               # Template for the DN in the CSR
countryName                     = US
stateOrProvinceName             = NH
localityName                    = Merrimack
organizationName                = Cluster

[ v3_ca ]                                           # Root CA certificate extensions
subjectKeyIdentifier = hash                         # Subject key identifier
authorityKeyIdentifier = keyid:always,issuer        # Authority key identifier
basicConstraints = critical, CA:true                # Basic constraints for a CA
keyUsage = critical, keyCertSign, cRLSign           # Key usage for a CA

[ crl_ext ]                                         # CRL extensions
authorityKeyIdentifier = keyid:always,issuer        # Authority key identifier

Generate the root CA certificate

Set days to something appropriate - this is how many days until the root CA expires. For testing, shorter is fine. For real world use, make it very long. When the root CA expires, everything it signed expires too. Years would be appropriate.

Also set subj appropriately too. I'm not actually sure if this needs to be set at all given that it's already in the config file.

$ openssl genrsa -out private/ca.key.pem 4096
$ openssl req -config openssl.cnf -key private/ca.key.pem -new -x509 -days 100 -sha256 -extensions v3_ca -out certs/ca.cert.pem -subj "/C=US/ST=NH/L=Merrimack/O=Cluster/CN=Root CA"

Installation

For real world use, keep the private ca.key.pem file private. Set ownership and permissions restrictively. Maybe store it on a USB key instead of permanently on the system. The ca.cert.pem file should be installed to every client and server system and readable by the cluster user (hacluster:haclient most likely). This file should also be referenced in /etc/sysconfig/pacemaker as PCMK_ca_file.

Generate server stuff

For each server (Pacemaker Remote node or hosts that you want to be able to access via cibadmin) do the following:

Generate private key and Certificate Signing Request (CSR)

Replace <servername> with the hostname (not FQDN) of the server. You can name it whatever you want, actually, but using the hostname makes keeping track of these things easier if there's a bunch of them.

$ openssl genrsa -out ../servers/<servename>r.key.pem 4096
$ openssl req -new -key ../servers/<servername>.key.pem -out ../servers/<servername>.csr

Answer any questions openssl asks. If it asks for a passphrase, don't provide one. This has not been tested yet and it's likely the cluster will hang asking for the passphrase when it tries to read the key.

Create server_cert_ext.cnf

Some things can't be set on the command line so they go in this file instead.

basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth

Create and sign the server certificate

Again, change <servername> to the hostname and set days to something that makes sense.

$ openssl x509 -req -in ../servers/<servername>.csr -CA certs/ca.cert.pem -CAkey private/ca.key.pem -out ../servers/<servername>.cert.pem -CAcreateserial -days 100 -sha256 -extfile server_cert_ext.cnf
$ openssl ca -extfile server_cert_ext.cnf -config openssl.cnf -days 100 -md sha256 -out ../servers/<servername>.cert.pem -infiles ../servers/<servername>.csr

Installation

Copy <servername>.cert.pem and <servername>.key.pem to the server. Both should be owned by the cluster user (hacluster:haclient most likely). The key should only be readable by the cluster user. I think it's okay if the cert is world-readable. These files should also be referenced in /etc/sysconfig/pacemaker as PCMK_cert_file and PCMK_key_file.

Generate client stuff

Everything is exactly the same on clients, except the paths will be different and there's a different config file. I'm not going to cover any of this in depth, just list the commands. Refer to the server section for details.

Generate private key and CSR

$ openssl genrsa -out ../clients/<clientname>.key.pem 4096
$ openssl req -new -key ../clients/<clientname>.key.pem -out ../clients/<clientname>.csr

Create client_cert_ext.cnf

basicConstraints = CA:FALSE
nsCertType = client, email
nsComment = "OpenSSL Generated Client Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, emailProtection

Create and sign the client certificate

$ openssl x509 -req -in ../clients/<clientname>.csr -CA certs/ca.cert.pem -CAkey private/ca.key.pem -out ../clients/<clientname>.cert.pem -CAcreateserial -days 100 -sha256 -extfile client_cert_ext.cnf
$ openssl ca -extfile client_cert_ext.cnf -config openssl.cnf -days 100 -md sha256 -out ../clients/<clientname>.cert.pem -infiles ../clients/<clientname>.csr

Installation

Installation of the client side is the same as the server side.

Having done all that, using it is a lot less complicated. Pacemaker Administration and Pacemaker Explained cover this in more depth, but the quick explanation is:

• For remote CIB operations, the admin machine is the client and the cluster machine is the server. Make sure the server has the server credentials and they are listed in /etc/sysconfig/pacemaker. Command line tools (like cibadmin, most commonly used for remote CIB operations) do not read the sysconfig file so all the credentials must be passed by the environment. Note that in addition to the TLS parameters, you must still give a username and password. See Pacemaker Administration for more details.

• For Pacemaker Remote nodes, the cluster machine running the remote resource is the client and the system running pacemaker_remoted is the server. Here, both sides are running a daemon so make sure the appropriate credentials are listed in /etc/sysconfig/pacemaker on both ends of the connection. There is no need to pass anything in the environment.