Notes / Domino by Martijn de Jong
A little Domino container story 
By Martijn de Jong | 1/3/25, 7:05 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
If you’re active in the Domino world, it’s unlikely that you missed that we had a little problem 2½ weeks ago....
This blog post is not about this problem itself, but this problem caused many servers with outdated Domino versions to urgently need an update, and this is a little story about one of those servers.
Modern email protocols: DANE, MTA-STS and TLS-RPT
By Martijn de Jong | 11/8/24, 3:47 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
n my recent OpenNTF webinar on modern E-mail Server operations, I covered several SMTP-related protocols like DKIM, SPF, and DMARC. However, with ongoing efforts to enhance the security of SMTP, new protocols have emerged, and these are the focus of this article.
Two weeks after my OpenNTF presentation, my former colleague Erwin Stamer, contacted me regarding the DANE status of my domain as it was yellow instead of green. He was looking at the status of my domain as they were implementing it at his employer (a large Dutch bank) and was looking for an example. I must admit that I initially had no idea what DANE was, but as it was in line with my presentation, I dived into it. DANE, MTA-STS and TLS-RPT all work together, but let’s look at them separately.
Building custom add-ons for your Domino container image – Martijn's Blog
By Martijn de Jong | 10/14/24, 3:18 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
This is a post that I thought I had already written, but I realised today that I hadn’t. It’s about a feature that Daniel Nashed added to the Domino container community project in the past year and that I showed in my presentations on the Domino container project at Engage and OpenNTF. But apparently, apart from that, Daniel and I never documented it. So here it is. The documentation on how to create your own custom add-on packages for your Domino container image.
Installing Domino REST API in an existing Domino container server – Martijn's Blog
By Martijn de Jong | 10/3/24, 1:18 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
The Domino REST API, a.k.a. DRAPI, is a requirement for running HCL Volt MX Go. On a native Domino server, it’s an add-on that you can install. The installation will install files in both a special install directory, the Domino program directory and the Domino data directory.
On a Domino server using Domino container images, you need a Domino image with the REST API included. After all, the Domino program directory is not persistent, which means that any addition to this directory that was added in the container and not in the image, is lost when the Domino container is stopped and restarted. Something that happens whenever you reboot the host machine. Luckily, the Domino container community image build tool includes the Domino REST API in the build menu, so it’s easy to add.
End of Life for CentOS 7 AND CentOS 8 Stream
By Martijn de Jong | 7/2/24, 6:54 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
CentOS 7 was released on the 7th of July 2014. For many years, it has been the operating system for millions of servers. Last Sunday (30th of June) was the day when, after almost 10 years, CentOS 7 became end-of-life. This means that no (security) updates for CentOS 7 will be released any more, and that servers running CentOS 7 are at risk.
I personally know of quite a few servers that are still running on CentOS 7. Even though the EOL date of CentOS 7 has been known for a very long time, many companies waited till the very last moment to phase out these systems and then missed their target. This is a bad situation to be in. I expect that it won’t be long before vulnerabilities in these systems become public, which then can no longer be patched. Migrating these systems to a new operating system should be top priority for these companies!
The conf-file in the Domino Container build script
By Martijn de Jong | 5/2/24, 9:19 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
In my previous post, I showed that the Domino-container build script now has a menu. When Daniel announced his plans to include a menu, I asked him to add the option to output the result of a menu build in the form build.sh domino 12.0.2 FP3 -verse -nomad etc. Why? So you could use this in a script to build the same container image with an updated Linux OS base layer unattended. Daniel listened, but implemented it in a different way.
Building your Domino Container Image in 2024
By Martijn de Jong | 4/18/24, 1:51 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
When you have a child which you see everyday, you don’t really notice how much he/she grew until you compare their current height with the line on the doorpost of the year before. It’s like that with the Domino container community project for me. My last major post on the Domino container project was in July 2022. Daniel Nashed, the main contributor to this project, has been steadily working on and there are many additions to the project. I use Domino containers on a daily basis, so I’ve seen the progress step by step. Only when reading my post from 2022, I realised how far the project has progressed in the past 21 months. Time for an update! The project also got a new status as since Domino 12.0.2, HCL’s official container images, which you can download from FlexNet, are now also based on the community container scripts!
There are 2 new additions which make creating a Domino container image much easier:
The use of the domdownload script
The build menu
Next to that there are a couple of very interesting new options. In this article, I’ll mainly focus on these two items. In another article, I’ll focus on the new options.
Domino Containers – The Next Step
By Martijn de Jong | 4/11/24, 5:12 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
With the Engage conference less than two weeks away, I’m working hard on my presentation. My topic will be “Domino Containers – The Next Step”. It’s a sequel to the presentation that I gave at Engage 2022 (and that same year at CollabSphere and OpenNTF) about the Domino container community project. Two years ago, I showed that Domino containers were ready to be used in production. On HCL’s FlexNet you had been able to download Domino docker images for quite a while already, but HCL never formally announced that those were for production use as well. During my session, I showed that the community images had quite a few benefits over HCL’s image and that Domino containers, based on these images, were a sensible replacement for your native Domino installations.
So this time, we go a step further. Daniel Nashed has been working hard on the build-scripts for the community image and it has become easier than ever to build your own Domino image. I will show this live during my session.
Security bulletin: Passwords of Domino Internet users are vulnerable
By Martijn de Jong | 2/22/24, 1:23 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
The official title of the security bulletin is: “HCL Domino is susceptible to a weak cryptography vulnerability (CVE-2023-37495).” The problem is with person documents that were created using the “Add Person” button in the Domino Directory. For people less savvy in Domino: that’s not the usual way to add users to Domino. In Domino, we register users using a certifier file. The only time we add persons to the Domino Directory using the “Add person” button, is when we know that these users will only ever access a Domino application through a web browser.
The problem with these “internet users” is that the hash in the Domino Directory for the HTTP password uses a cryptographically weak hash algorithm. If an attacker has access to these hashes, he could determine the user’s password through a brute force attack. You can’t see these hashes from a browser, so the attacker needs to have access to the Domino Directory through a Notes or Nomad client. That limits the potential attackers to all users who are registered as Notes users inside the company.
Installing wireguard on CentOS Stream 9
By Martijn de Jong | 1/15/24, 3:37 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
As I do a lot of my research on new Domino versions, Connections versions and HCL DX on my own server at home and as I’m often not at home, I figured I needed a VPN tunnel to my server, so I can work as if I am home. Wireguard has become kind of the de facto standard for these kind of situations, so I looked into installing it on my CentOS Stream 9 host.
Nginx as reverse proxy and SNI
By Martijn de Jong | 11/10/23, 4:43 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
I had some difficulty to find a good title for this article that would really cover the contents. Therefore, let me start with describing the problem I faced which led to this article.
I have a lot of sites running on my home server (this blog being one of them) using different technologies. As I have a single IPv4 address, all these sites are behind a reverse proxy, for which I use Nginx. A couple of those sites are Domino sites and last week I realised there was something wrong in that area. I have several internet site documents on Domino for different urls. However, last week I realised that all my urls that were forwarded to Domino, were being serviced based on the same internet site document. In other words, Domino did not recognise for which internet site a request was meant.
Certificate Store: Submit vs Save
By Martijn de Jong | 3/30/23, 2:39 AM | Infrastructure - Notes / Domino | Added by Oliver Busse
I regularly receive question about the Certificate Store and CertMgr, which made me realise that there’s a lot of confusion around the Submit Request and the Save & Close buttons in the store and when to use what. Time for an article to hopefully solve some of that confusion.
On Domino thread IDs and Linux/Windows process IDs
By Martijn de Jong | 3/1/23, 9:53 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
A short tip on something which many people are probably not aware of, but which can be a huge time saver when you’re troubleshooting a Domino problem.
As an example, see this error message from a Domino log:
[062372:000014-00007F8001776700] 28/02/2023 13:16:20 CertStore: Error opening CertStore database [CN=PROD02/OU=SRV/O=ACME!!certstore.nsf] : The server is not responding. The server may be down or you may be experiencing network or VPN problems. Contact your system administrator if this problem persists.
[062372:000014-00007F8001776700] 28/02/2023 13:16:20 CertStore: Error opening CertStore on [CN=PROD02/OU=SRV/O=ACME] : The server is not responding. The server may be down or you may be experiencing network or VPN problems. Contact your system administrator if this problem persists.
Your first hunch might be that this is an error that’s caused by the CertMgr process. It’s related to the Certificate Store after all. But is this really the case?
Protecting your Domino container with fail2ban
By Martijn de Jong | 11/7/22, 4:25 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
If your Domino server is connected to the Internet, you’ll find that bots (hacked systems running a script) will throw a brute force attack on your Domino server. For me, especially, my SMTP server was under heavy attack.
The reason why it’s interesting for hackers to find a valid login on an SMTP server, is that this will probably allow them to send spam through your mail server. Most mail servers allow sending mail through their servers for other domains for authenticated users only. The chances of them guessing any of the users in my Domino directory right and then also guessing the password correctly are basically zero, but the pollution of my log file is reason enough to stop them.
Fail2ban is a very elegant program for Linux to do just that. You can configure it to scan log files for certain patterns (it uses RegEx to recognise them) and add hosts that match those patterns too often within a defined period of time, to the block list of iptables.
HCL Traveler and error 500
By Martijn de Jong | 7/21/22, 1:31 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
HCL Traveler is one of those addons for Domino that just works. If you have a properly configured HTTPS stack, you install it, start it and you’re basically done. From now on, you can connect your mobile devices to your Domino server to read your mail and calendar.
At least, that has always been my experience until very recently. The other day I was sent to a customer to fix their problem with Traveler. They had upgraded their Domino server and Traveler installation from 8.5.3 FP5 to 12.0.1 FP1. Everything worked (Kudos for Domino!) except Traveler. Though on further discussion with the client it became clear that Traveler actually already broke earlier and hadn’t been working for the past 6 years or so.
Domino containers revisited
By Martijn de Jong | 7/20/22, 1:57 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
As I wrote in my last post about the Engage conference, a lot has happened in the Domino container space since I wrote my articles, as Daniel Nashed did some serious refactoring on all scripts, removing an insane amount of old code lines and adding some new functionality. This article will show the changes to the project compared to the time that I wrote the original 6-part series.
Working with standard Certificate Authorities in Domino 12
By Martijn de Jong | 3/28/22, 1:56 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
In the past weeks, I helped some colleagues with importing certificates in the Certificate Store of Domino 12 and while doing so, I noticed something peculiar. For many years, we haven’t had a proper way of creating certificates in Domino. The pre-12 database to create keys was completely outdated and didn’t allow for creating strong keys. As a result, most administrators got used to creating keys outside Domino, usually through on openssl command in Linux. This way of working found its way into procedures and many admins, instead of using the Certificate Store database, still follow these old procedures and create their keys outside Domino. I therefore decided to create a short article on how to create certificates with Domino 12 which are signed by a certificate authority which doesn’t support the ACME protocol.
Domino-docker explained – Part 5 : Adding add-ons on top of your Domino image
By Martijn de Jong | 11/2/21, 2:34 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
In the previous parts, I explained how to create a Domino image and deploy it. But what if you want to add fix packs to your Domino image? Or Traveler, Volt or Verse? The scripts of the domino-docker project make this super simple. In this part, I’ll show you how to do this.
Domino-docker explained – Part 4 : The domino_container script
By Martijn de Jong | 10/22/21, 7:22 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
In the previous part, I showed how you can simply start and stop and open the Domino console with the domino_container command. This piece of script is responsible for interacting with the Domino container in a way where the average administrator doesn’t even have to realise that Domino is running inside a container. There are many more functions in this script that will help you manage your Domino server and in this part I will discuss them.
Domino-docker explained – Part 3 : Running your first Domino server in a container
By Martijn de Jong | 9/30/21, 10:56 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
In part 2 we created a Domino container image. Now we want to start the image. Of course, we could just use docker run <options> <imagename>, but with the scripts from the Domino Docker project, there’s a much easier option. In this part, I’ll show you what to do to make running, restarting and stopping images super easy.
Domino-docker explained – Part 2 : Creating your first Domino image
By Martijn de Jong | 9/28/21, 1:54 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
n the previous part, I looked at reasons why you might want to run your Domino server inside a container. In this part, I’m going to show how to create your first Domino image. We have to take one step back though, as since a couple of years, HCL provides their own docker image for Domino. So why would you want to create your own image? My experience is that it leads to a better image and it gives options to add your own tooling to the image. Nevertheless, using HCL’s image is an option and the script also provides an option to build on top of the standard HCl image. My advice: create your own.
Domino-docker explained – Part 1: Why run Domino inside a container?
By Martijn de Jong | 9/28/21, 1:51 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
In November 2018, Thomas Hampel (at that time still working for IBM) created the domino-docker github repository as an open source initiative to create scripts that would make it easier to run Domino inside a container. Even though the repository was started by IBM, the work was done by the community with most of the work done by one man in particular: Daniel Nashed. He contributed his Linux start/stop scripts to the project, but also wrote scripts to completely automate the build of the images.
While working with the scripts, I realised two things:
Daniel has built fantastic scripts to both build and run Domino containers
With so much functionality added, the project didn’t manage to document this new functionality in detail
With help from Daniel, I managed to build my own customised container and I experienced in the past months all the benefits from running Domino as a container, combined with the scripts from the Domino Docker project. However, if this project wants to get the attention it deserves, the documentation needs to be fixed and this is exactly what I’ll try to do in a series of 6 articles:
Domino 12 and Borg backup
By Martijn de Jong | 4/20/21, 4:39 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
I must admit that I’m quite excited about Domino 12. I was thinking this morning why actually. The new features in Domino 12 aren’t necessarily groundbreaking. They’re more about fixing things which should have been in the platform already, but were neglected by IBM in the years in which it would have been logical to implement them.
Domino 12 – SSL Performance
By Martijn de Jong | 3/22/21, 2:54 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro
A few weeks ago I wrote about the new Certificate Manager in Domino 12, which enabled Domino 12 to request and automatically update LetsEncrypt certificates and implemented a better way of Server Name Indication (previously introduced in Domino 11.0.1), so you can use different SSL certificates for different websites without needing multiple IP addresses. The Certificate Manager also allows you to use the most recent (ECDSA) ciphers. The lack of this functionality in previous versions of Domino was an important reason why, in many Domino installations, an Nginx, Apache or IHS server is placed in front of the Domino HTTP task as a reverse proxy. There was however another reason: Domino used a lot of cpu power for and was rather slow to decrypt and encrypt SSL traffic. Letting Nginx/Apache/IHS offload the SSL de-/encryption task, reduced total load on the server and sped up performance. I therefore wondered if HCL also managed to solve this problem.
Domino V12 – The Certificate Manager
By Martijn de Jong | 2/28/21, 4:49 AM | Infrastructure - Notes / Domino | Added by Oliver Busse
HCL Domino V12 is in beta, and we currently have beta 2 to work with. One of the interesting new features of Domino V12 is the Certificate Manager task (certmgr). I’ve been playing around with this task and in this post I’ll tell about my experiences.
Decrypting a stash (.sth) file – Martijn's Blog
By Martijn de Jong | 3/1/20, 6:15 AM | Infrastructure - Notes / Domino | Added by Oliver Busse
HCL Domino saves it certificates in a .kyr file. IBM WebSphere saves it certificates in a Java Keystore / .jks format. Both formats allow you to save the password for the keystores in a stash file which has the extension .sth. The stash files allow you to do most actions without entering a password.