LetsEncrypt with Webroot

All righty then, time to get working with LetsEncrypt. Again.

So there are several different ways to have LetsEncrypt verify that the domain you wish to have a certificate for is actually yours.

I usually use the webroot version.

But in the past, I found it irksome that I had to let certbot spin up a http server to verify my domain.

What was easier was to have my application server have a part on the filesystem mapped to the Webpages¹.

Then certbot could simply use that webroot directory. Which is option 2.

So let's try that.

~]# certbot certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Plugins selected: Authenticator webroot, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Please enter in your domain name(s) (comma and/or space separated) (Enter 'c'
to cancel): www.mrbear.org
Cert is due for renewal, auto-renewing...
Renewing an existing certificate for www.mrbear.org
Performing the following challenges:
http-01 challenge for www.mrbear.org
Input the webroot for www.mrbear.org: (Enter 'c' to cancel): /home/jelastic/media
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/www.mrbear.org/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/www.mrbear.org/privkey.pem
Your certificate will expire on 2021-11-10. To obtain a new or
tweaked version of this certificate in the future, simply run
certbot again. To non-interactively renew *all* of your
certificates, run "certbot renew"
- If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le

It will put stuff in /home/jelastic/media/.well-known. Usually in the acme-challenge directory. It'll put some random file in there, for example 366_oWEFaizWeQqOts4KhuTBQcCsFH5dBgG7-jNX32k with a similar string inside, but usually a lot longer.

This file is accessed by the LetsEncrypt server, and if that works, the domain is yours and you get your certificate.

"certbot renew" is preferably easier, as it takes into account the previous configuration settings. Nice for cron jobs.

Also, I found this great script for importing the ssl certificates on the Forums of LetsEncrypt² for my Java Application Server!

My Payara MicroProfile server uses the following commandline to access the new certificates:

java -server -XX:+UnlockExperimentalVMOptions -javaagent:/java_agent/jelastic-gc-agent.jar=period=300,debug=0 -XX:+UseStringDeduplication -XX:+UseG1GC -Xmaxf0.3 \
-Xminf0.1 -Xmx2048M -Xms32M \
-Djavax.net.ssl.trustStore="/home/jelastic/cacerts.jks" \
-Djavax.net.ssl.keyStore="/home/jelastic/keystore.jks" \
-Djavax.net.ssl.trustStorePassword="changeit" \
-Djavax.net.ssl.keyStorePassword="changeit" \
-jar ./payara-micro-5.2021.1.jar --postbootcommandfile ./postboot --addlibs ../libs/mariadb-java-client-2.5.2.jar --port 8080 --sslport 8743 \
--deploy ../release/mrbear.war \
--deploy ../release/mrbearportal-2.0.9.war \

Please pay special attention to the four "javax.net.ssl.*" properties that take care of the SSL properties.

And that's it!

References

[1] MyBlog - Alternate docroots

http://randomthoughtsonjavaprogramming.blogspot.com/search/label/alternate%20docroot

[2] LetsEncrypt Community Forum - Importing LetsEncrypt into Java and Glassfish

https://community.letsencrypt.org/t/importing-letsencrypt-into-java-and-glassfish/9711/11

Java Keytool Essentials: Working with Java Keystores - Mitchell Anicas

https://www.digitalocean.com/community/tutorials/java-keytool-essentials-working-with-java-keystores

Automating Importing Let's Encrypt Certificates into Payara/Glassfish

Well, I already made a blogpost on letsencrypt certificates in payara¹.

In this blogpost I mentioned a python script that automatically renews certificates for payara². Unfortunately, it's not quite a perfect fit for me, as I do not run payara on port 80, which the script requires.

My setup has an nginx running on port 80, that basically forwards all requests to the https version of the website.

So I could simply turn off nginx, re-certify and turn nginx back on.

This simple bash script is as follows:

Running it will give you:

$ source renew_letsencrypt_certificates.sh
---- Starting.
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.mrbear.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.mrbear.org
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.mrbear.org/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/www.mrbear.org/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
---- Adding certificates...
/home/payara/payara5/glassfish/domains/production/config/keystore.jks
Private key with alias [s1as] added to keystore /home/payara/payara5/glassfish/domains/production/config/keystore.jks.
Command add-pkcs8 executed successfully.
---- Disabling http listener...
server.network-config.network-listeners.network-listener.http-listener-2.enabled=false
Command set executed successfully.
---- Enabling http listener...
server.network-config.network-listeners.network-listener.http-listener-2.enabled=true
Command set executed successfully.
---- Done.

And your certificate is renewed. I am most pleased that this is soo easy in Payara/Glassfish.

In order to properly automate this, the master password must be stored somehow, and it's possible to do this via the --passwordfile=passwordfile.txt commandline parameter.

I found a good blogpost on how to do this at .Lost in Coding³.

And also the blogpost on [4] to get me started on how to do this.

Verifying that the keystore contains the appropriate stuff, can be done as follows:

# keytool -list -keystore /home/payara/payara5/glassfish/domains/production/config/keystore.jks
Enter keystore password:
Keystore type: jks
Keystore provider: SUN

Your keystore contains 2 entries

glassfish-instance, Jan 22, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): WZ:B6:4N:8V:AT:YP:QC:9N:VT:HA:WI:NQ:B[:V8:0W:YT:B8:AW:YT:VQ
s1as, Sep 25, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): N8:5Y:AN:SV:N8:9W:4T:Y7:9W:V9:W4:VT:8Y:WV:4N:W4:VT:FB:8S:4E

To get more information, try:

keytool -list -v -keystore /home/payara/payara5/glassfish/domains/production/config/keystore.jks

Or for a specific alias:

keytool -list -v -keystore /home/payara/payara5/glassfish/domains/production/config/keystore.jks -alias s1as

References

[1] Enabling SSL in Payara with certificate from Let's Encrypt.

https://randomthoughtsonjavaprogramming.blogspot.com/2019/01/enabling-ssl-in-payara-with-certificate.html

[2] github - Let's encrypt script to retrieve and upload keys/certificates to Payara

https://gist.github.com/ratcashdev/1b09877d37e02ef5170bf9e60c377f34

[3] .Lost in Coding - Configure Payara Server passwords in Docker

https://ondro.inginea.eu/index.php/configure-passwords-in-payara-server-and-glassfish/

[4] Payara Blog - TLS certificates with Payara Server and Let's Encrypt

https://blog.payara.fish/configuring-ssl/tls-certificates-with-payara-server-and-lets-encrypt

Enabling SSL in Payara with certificate from Let's Encrypt.

In most cases the steps here are exactly the same as for the Glassfish application server¹.

So, it might be a bit superfluous, but in the interest of being complete.

Bear in mind that when you change the master password, this password is also used to re-encrypt all your keystores!

Letsencrypt in cacerts.jks

I'm still using the Keystore Explorer for this. When you open up the cacerts.jks file, you'll see Let's Encrypt is already in there.

It's called 'letsencryptisrgx1 [jdk]'.

Add private key to keystore.jks

1. "Open an existing keystore" in Keystore Explorer.
2. Choose keystore.jks. Put in your master password to access the keystore.
3. Deleted the old "s1as" Key Pair.
4. Import Key Pair
5. Select format PKCS#8
6. Encrypted private key checkbox should be unchecked
7. PKCS#8 Private key file should be privkey.pem
8. certificates file could be fullchain.pem
9. enter the alias "s1as".
10. enter a password to seal the deal.
11. close and save
12. restart Payara domain
13. done.

Interesting.

When attempting to restart the Payara domain, I used asadmin, and it immediately noticed that my keystore.jks file was changed. It showed me the certificate and asked:

Do you trust the above certificate [y|N]-->
Enter admin user name>
Enter admin password>

Restart your listener

Instead of having to restart your entire Payara Domain, apparently it is also possible to just turn your http-listeners on and off.

That's what this little bit here is all about.

The command "asadmin list-http-listeners", shows you the listeners.

$ ./asadmin list-http-listeners
Enter admin user name> admin
Enter admin password for user "admin">
http-listener-1
http-listener-2
admin-listener
Command list-http-listeners executed successfully.

In order to examine the http listeners fully, it is best to list the different properties, like so:

$ ./asadmin list server.network-config.protocols.protocol.*
Enter admin user name> admin
Enter admin password for user "admin">
server.network-config.protocols.protocol.admin-http-redirect
server.network-config.protocols.protocol.admin-http-redirect.http-redirect
server.network-config.protocols.protocol.admin-listener
server.network-config.protocols.protocol.admin-listener.http
server.network-config.protocols.protocol.admin-listener.http.file-cache
server.network-config.protocols.protocol.http-listener-1
server.network-config.protocols.protocol.http-listener-1.http
server.network-config.protocols.protocol.http-listener-1.http.file-cache
server.network-config.protocols.protocol.http-listener-1.ssl
server.network-config.protocols.protocol.http-listener-2
server.network-config.protocols.protocol.http-listener-2.http
server.network-config.protocols.protocol.http-listener-2.http.file-cache
server.network-config.protocols.protocol.http-listener-2.ssl
server.network-config.protocols.protocol.pu-protocol
server.network-config.protocols.protocol.pu-protocol.port-unification
server.network-config.protocols.protocol.pu-protocol.port-unification.protocol-finder.admin-http-redirect
server.network-config.protocols.protocol.pu-protocol.port-unification.protocol-finder.http-finder
server.network-config.protocols.protocol.sec-admin-listener
server.network-config.protocols.protocol.sec-admin-listener.http
server.network-config.protocols.protocol.sec-admin-listener.http.file-cache
server.network-config.protocols.protocol.sec-admin-listener.ssl
Command list executed successfully.

We can see which ones have ssl enabled, by using get:

$ ./asadmin get server.network-config.protocols.protocol.http-listener-2.security-enabled
server.network-config.protocols.protocol.http-listener-2.security-enabled=true
Command get executed successfully.
$ ./asadmin get server.network-config.protocols.protocol.http-listener-1.security-enabled
server.network-config.protocols.protocol.http-listener-1.security-enabled=false
Command get executed successfully.
$ ./asadmin get server.network-config.protocols.protocol.admin-listener.security-enabled
server.network-config.protocols.protocol.admin-listener.security-enabled=false
Command get executed successfully.
$ ./asadmin get server.network-config.protocols.protocol.sec-admin-listener.security-enabled
server.network-config.protocols.protocol.sec-admin-listener.security-enabled=true
Command get executed successfully.

As you've noticed above, the admin-listener has defined a redirect to sec-admin-listener.

Let's turn http-listener-2 off and on again. See if he picks up the new certificate.

./asadmin set server.network-config.network-listeners.network-listener.http-listener-2.enabled=false
Command get executed successfully.
./asadmin set server.network-config.network-listeners.network-listener.http-listener-2.enabled=true
Command get executed successfully.

Yes! We have a green lockicon in my Firefox bar! Success!

For some reason this little thing, also caused the new certificate to be picked up by the admin listener of the administration console.

Automation

Apparently it's possible to automate the whole thing, and there's a Python script in the Payara application server called payara5/bin/letsencrypt.py that let's you do that.

There's more information available on the website of Certbot on how to automate it, see in the references below.

References

[1] Enabling SSL in Glassfish with certificate from Let's Encrypt.

https://randomthoughtsonjavaprogramming.blogspot.com/2019/01/enabling-ssl-in-glassfish-with.html

Payara Blog - Securing Payara Server with Custom SSL Certificate

https://blog.payara.fish/securing-payara-server-with-custom-ssl-certificate

Payara Blog - Configuring SSL/TLS Certificates with Payara Server and Let's Encrypt

https://blog.payara.fish/configuring-ssl/tls-certificates-with-payara-server-and-lets-encrypt

Github.com - PAYARA-1061 LetsEncrypt integration script

https://github.com/payara/Payara/pull/2727

Certbot - User Guide

https://certbot.eff.org/docs/using.html

Keystore Explorer

https://keystore-explorer.org/

Enabling SSL in Glassfish with certificate from Let's Encrypt.

I wished to use the certificates of Let's Encrypt¹ for my website/glassfish.

I installed certbot using the manual found on the certbot website².

Make sure you are not running a website at the time, because the challenge of Let's Encrypt to verify you own the domain, is done by running a small webserver.

Getting certificates

[root@localhost certificates]# certbot certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'
to cancel): www.mrbear.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for www.mrbear.org
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/www.mrbear.org/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/www.mrbear.org/privkey.pem
   Your cert will expire on 2019-01-12. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Converting PEM

So now I have the necessary PEM files. Now to add them to my keystore and cacerts.

[root@mrbear config]# keytool -import -v -trustcacerts -alias letsencrypt -file /etc/letsencrypt/live/www.mrbear.org/fullchain.pem -keystore cacerts.jks -storepass itsasecret
... lots of text...
#9: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: B6 90 77 77 F6 3B DF 0C   C3 29 25 B5 56 29 EB CF  ..ww.;...)%.V)..
0010: 5D FD 3B 07                                        ].;.
]
]

Trust this certificate? [no]:  yes
Certificate was added to keystore
[Storing cacerts.jks]

I received a warning.

Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.jks -deststoretype pkcs12".

I did just that.

[root@mrbear config]# keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.jks -deststoretype pkcs12                                                                                           
Enter source keystore password:  
Entry for alias godaddy successfully imported.
Entry for alias godaddy2 successfully imported.
Entry for alias glassfish-instance successfully imported.
Entry for alias s1as successfully imported.
Import command completed:  4 entries successfully imported, 0 entries failed or cancelled

Warning:
Migrated "keystore.jks" to Non JKS/JCEKS. The JKS keystore is backed up as "keystore.jks.old".
[root@mrbear config]#

I got a java.security.cert.CertificateParsingException: signed fields invalid when trying to import the Let's Encrypt keys.

I didn't know how to resolve it, so I decided to go with KSE - KeyStore Explorer³.

Verify key

openssl rsa -in /etc/letsencrypt/live/www.mrbear.org/privkey.pem -check

Glassfish Admin console and HTTPS

I had some issues with the admin console which is also behind https.

There's a stackoverflow⁴ that helped me.

Running renew

In order to renew my keys with Let's Encrypt, all I need to do is run "cert renew" apparently.

I get the message that he cannot validate my domain. Apparently he needs to spin up a webserver again.

[root@mrbear ~]# certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.mrbear.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.mrbear.org
Waiting for verification...
Cleaning up challenges
Resetting dropped connection: acme-v02.api.letsencrypt.org

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.mrbear.org/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/www.mrbear.org/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Still using KSE, which is awesome, to change keystore.jks.

Steps taken:

1. Deleted the old "s1as" Key Pair
2. Import Key Pair
3. Select format PKCS#8
4. Encrypted private key checkbox should be unchecked
5. PKCS#8 Private key file should be privkey.pem
6. certificates file could be chain.pem
7. enter the alias "s1as".
8. enter a password to seal the deal.
9. close and save
10. restart Glassfish
11. done.

References

[1] Let's Encrypt

https://letsencrypt.org/

[2] Certbot

https://certbot.eff.org/

[3] SourceForge - Keystore-Explorer

http://keystore-explorer.sourceforge.net/

[4] StackOverflow - Glassfish V4 ssl admin no longer works

https://stackoverflow.com/questions/34935725/glassfish-v4-ssl-admin-no-longer-works/34952975