Recent by Martijn de Jong 

Removing security-auditlog indexes from your OpenSearch cluster  

By Martijn de Jong | 7/18/25, 1:21 AM | Infrastructure - Connections | Added by Roberto Boccadoro

After my previous article on Querying the OpenSearch index, a colleague and I used the script to take a good look at the OpenSearch index in our environment. That environment, is a large Connections installation (45K users) in the Netherlands. We found two disturbing facts: The cluster status was yellow, due to the QuickResults index not having a replica of all its shards There was a huge amount of secruity-auditlog indexes, one per day. There were as many as the days passed since our migration to Connections 8 It turns out these two issues were closely related.

Querying your OpenSearch index  

By Martijn de Jong | 7/2/25, 4:30 PM | Infrastructure - Connections | Added by Roberto Boccadoro

n the old days of Connections, we would check our ElasticSearch indexes through Kibana, but Kibana is no longer part of an HCL Connections Component pack installation. In HCL Connections 8, OpenSearch is a core component powering features like Orient Me, Type-Ahead Search, and Metrics. As an administrator, it’s critical not just to ensure that OpenSearch is running, but to verify that it’s indexing, storing, and serving content correctly.

WebSphere 8.5.5 FP27 & HCL Connections… beware!  

By Martijn de Jong | 5/5/25, 3:49 AM | Infrastructure - Connections | Added by Roberto Boccadoro

HCL Connections 8.0 CR9 currently lists in its system requirements WebSphere 8.5.5 FP26 as its highest supported WebSphere version. However, I suspect that CR10 will try to support FixPack 27, which became available on April 21st. Brave companies might just update their servers with the latest fixpack, and usually that works just fine. This fixpack, however, has introduced changes in outbound TLS certificate hostname validation. What it does is if WebSphere will make a connection to https://example.acme.org, it will check if the certificate that example.acme.org returns contains my server.acme.org in either the hostname or as a subject alternative name (SAN). If it doesn’t, you’ll be greeted with a message:

Creating an IBM HTTP Server key db from a .pfx file   

By Martijn de Jong | 1/7/25, 2:41 AM | Infrastructure - Connections | Added by Roberto Boccadoro

It’s a question that I get quite often. “Can you create a IHS key database for me from this pfx certificate”? Multiple of our customers use IBM HTTP Server. Either for HCL Connections, but also as a reverse proxy for other services. Most of those customers use a wildcard certificate for their IHS server that has to be renewed each year. As I hate this kind of repetitive work, I created a script to do it for me. As it could be of use to others, I decided to create a blog article about it.

A little Domino container story  

By Martijn de Jong | 1/3/25, 7:05 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

If you’re active in the Domino world, it’s unlikely that you missed that we had a little problem 2½ weeks ago.... This blog post is not about this problem itself, but this problem caused many servers with outdated Domino versions to urgently need an update, and this is a little story about one of those servers.

Changing the file size limit for a community  

By Martijn de Jong | 11/30/24, 3:27 AM | Infrastructure - Connections | Added by Roberto Boccadoro

By default, a HCL Connections community can contain 500 MB of files in the Files widget. Depending on what kind of content is posted in the community, that might not be enough. It’s possible to change the file limit, both for communities and for personal files.

Modern email protocols: DANE, MTA-STS and TLS-RPT  

By Martijn de Jong | 11/8/24, 3:47 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

n my recent OpenNTF webinar on modern E-mail Server operations, I covered several SMTP-related protocols like DKIM, SPF, and DMARC. However, with ongoing efforts to enhance the security of SMTP, new protocols have emerged, and these are the focus of this article. Two weeks after my OpenNTF presentation, my former colleague Erwin Stamer, contacted me regarding the DANE status of my domain as it was yellow instead of green. He was looking at the status of my domain as they were implementing it at his employer (a large Dutch bank) and was looking for an example. I must admit that I initially had no idea what DANE was, but as it was in line with my presentation, I dived into it. DANE, MTA-STS and TLS-RPT all work together, but let’s look at them separately.

Building custom add-ons for your Domino container image – Martijn's Blog  

By Martijn de Jong | 10/14/24, 3:18 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This is a post that I thought I had already written, but I realised today that I hadn’t. It’s about a feature that Daniel Nashed added to the Domino container community project in the past year and that I showed in my presentations on the Domino container project at Engage and OpenNTF. But apparently, apart from that, Daniel and I never documented it. So here it is. The documentation on how to create your own custom add-on packages for your Domino container image.

Installing Domino REST API in an existing Domino container server – Martijn's Blog  

By Martijn de Jong | 10/3/24, 1:18 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The Domino REST API, a.k.a. DRAPI, is a requirement for running HCL Volt MX Go. On a native Domino server, it’s an add-on that you can install. The installation will install files in both a special install directory, the Domino program directory and the Domino data directory. On a Domino server using Domino container images, you need a Domino image with the REST API included. After all, the Domino program directory is not persistent, which means that any addition to this directory that was added in the container and not in the image, is lost when the Domino container is stopped and restarted. Something that happens whenever you reboot the host machine. Luckily, the Domino container community image build tool includes the Domino REST API in the build menu, so it’s easy to add.

End of Life for CentOS 7 AND CentOS 8 Stream  

By Martijn de Jong | 7/2/24, 6:54 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

CentOS 7 was released on the 7th of July 2014. For many years, it has been the operating system for millions of servers. Last Sunday (30th of June) was the day when, after almost 10 years, CentOS 7 became end-of-life. This means that no (security) updates for CentOS 7 will be released any more, and that servers running CentOS 7 are at risk. I personally know of quite a few servers that are still running on CentOS 7. Even though the EOL date of CentOS 7 has been known for a very long time, many companies waited till the very last moment to phase out these systems and then missed their target. This is a bad situation to be in. I expect that it won’t be long before vulnerabilities in these systems become public, which then can no longer be patched. Migrating these systems to a new operating system should be top priority for these companies!

Installing Huddo Boards on WebSphere-only Connections   

By Martijn de Jong | 6/24/24, 8:21 AM | Infrastructure - Connections | Added by Roberto Boccadoro

Let me clarify what this article is about. Huddo (also known as ISW) is selling 2 versions of their Boards product: Kudos Boards – The Java application that can be installed on WebSphere. Active development has stopped a few years ago Huddo Boards – The product consisting of several containers, which can be installed on Kubernetes or a container engine like Docker. A version with limited functionality is part of the HCL Connections component pack and known as Activities Plus. This product is actively being developed and also available for other platforms. Although the Connections Component pack was introduced in version 6 already, many companies refrained from installing it due to the serious amount of extra infrastructure needed to run these components, which includes a Kubernetes cluster. We call these the WebSphere-only Connections installations, as they only include the WebSphere stack part of Connections. So what if you would like to use the full power of Huddo Boards without having to install a Kubernetes cluster? That’s what this article is about.

My thoughts on Engage 2024  

By Martijn de Jong | 5/3/24, 6:42 PM | Business - Events / People | Added by Oliver Busse

Last week was the HCL user group event Engage in Antwerp, Belgium. It’s one of my favourite events. It’s meticulously organized by Theo Heselmans, who this year organized Engage and its predecessor, BLUG, for the 15th time. As he announced last year in Amsterdam, it was also the last time that he would organize this event, and it’s therefore logical that this fact played an important role during the event.

The conf-file in the Domino Container build script  

By Martijn de Jong | 5/2/24, 9:19 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In my previous post, I showed that the Domino-container build script now has a menu. When Daniel announced his plans to include a menu, I asked him to add the option to output the result of a menu build in the form build.sh domino 12.0.2 FP3 -verse -nomad etc. Why? So you could use this in a script to build the same container image with an updated Linux OS base layer unattended. Daniel listened, but implemented it in a different way.

Building your Domino Container Image in 2024  

By Martijn de Jong | 4/18/24, 1:51 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

When you have a child which you see everyday, you don’t really notice how much he/she grew until you compare their current height with the line on the doorpost of the year before. It’s like that with the Domino container community project for me. My last major post on the Domino container project was in July 2022. Daniel Nashed, the main contributor to this project, has been steadily working on and there are many additions to the project. I use Domino containers on a daily basis, so I’ve seen the progress step by step. Only when reading my post from 2022, I realised how far the project has progressed in the past 21 months. Time for an update! The project also got a new status as since Domino 12.0.2, HCL’s official container images, which you can download from FlexNet, are now also based on the community container scripts! There are 2 new additions which make creating a Domino container image much easier: The use of the domdownload script The build menu Next to that there are a couple of very interesting new options. In this article, I’ll mainly focus on these two items. In another article, I’ll focus on the new options.

Domino Containers – The Next Step  

By Martijn de Jong | 4/11/24, 5:12 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

With the Engage conference less than two weeks away, I’m working hard on my presentation. My topic will be “Domino Containers – The Next Step”. It’s a sequel to the presentation that I gave at Engage 2022 (and that same year at CollabSphere and OpenNTF) about the Domino container community project. Two years ago, I showed that Domino containers were ready to be used in production. On HCL’s FlexNet you had been able to download Domino docker images for quite a while already, but HCL never formally announced that those were for production use as well. During my session, I showed that the community images had quite a few benefits over HCL’s image and that Domino containers, based on these images, were a sensible replacement for your native Domino installations. So this time, we go a step further. Daniel Nashed has been working hard on the build-scripts for the community image and it has become easier than ever to build your own Domino image. I will show this live during my session.

Security bulletin: Passwords of Domino Internet users are vulnerable  

By Martijn de Jong | 2/22/24, 1:23 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The official title of the security bulletin is: “HCL Domino is susceptible to a weak cryptography vulnerability (CVE-2023-37495).” The problem is with person documents that were created using the “Add Person” button in the Domino Directory. For people less savvy in Domino: that’s not the usual way to add users to Domino. In Domino, we register users using a certifier file. The only time we add persons to the Domino Directory using the “Add person” button, is when we know that these users will only ever access a Domino application through a web browser. The problem with these “internet users” is that the hash in the Domino Directory for the HTTP password uses a cryptographically weak hash algorithm. If an attacker has access to these hashes, he could determine the user’s password through a brute force attack. You can’t see these hashes from a browser, so the attacker needs to have access to the Domino Directory through a Notes or Nomad client. That limits the potential attackers to all users who are registered as Notes users inside the company.

Installing wireguard on CentOS Stream 9   

By Martijn de Jong | 1/15/24, 3:37 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

As I do a lot of my research on new Domino versions, Connections versions and HCL DX on my own server at home and as I’m often not at home, I figured I needed a VPN tunnel to my server, so I can work as if I am home. Wireguard has become kind of the de facto standard for these kind of situations, so I looked into installing it on my CentOS Stream 9 host.

Nginx as reverse proxy and SNI  

By Martijn de Jong | 11/10/23, 4:43 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

I had some difficulty to find a good title for this article that would really cover the contents. Therefore, let me start with describing the problem I faced which led to this article. I have a lot of sites running on my home server (this blog being one of them) using different technologies. As I have a single IPv4 address, all these sites are behind a reverse proxy, for which I use Nginx. A couple of those sites are Domino sites and last week I realised there was something wrong in that area. I have several internet site documents on Domino for different urls. However, last week I realised that all my urls that were forwarded to Domino, were being serviced based on the same internet site document. In other words, Domino did not recognise for which internet site a request was meant.

Certificate Store: Submit vs Save  

By Martijn de Jong | 3/30/23, 2:39 AM | Infrastructure - Notes / Domino | Added by Oliver Busse

I regularly receive question about the Certificate Store and CertMgr, which made me realise that there’s a lot of confusion around the Submit Request and the Save & Close buttons in the store and when to use what. Time for an article to hopefully solve some of that confusion.

On Domino thread IDs and Linux/Windows process IDs  

By Martijn de Jong | 3/1/23, 9:53 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

A short tip on something which many people are probably not aware of, but which can be a huge time saver when you’re troubleshooting a Domino problem. As an example, see this error message from a Domino log: [062372:000014-00007F8001776700] 28/02/2023 13:16:20 CertStore: Error opening CertStore database [CN=PROD02/OU=SRV/O=ACME!!certstore.nsf] : The server is not responding. The server may be down or you may be experiencing network or VPN problems. Contact your system administrator if this problem persists. [062372:000014-00007F8001776700] 28/02/2023 13:16:20 CertStore: Error opening CertStore on [CN=PROD02/OU=SRV/O=ACME] : The server is not responding. The server may be down or you may be experiencing network or VPN problems. Contact your system administrator if this problem persists. Your first hunch might be that this is an error that’s caused by the CertMgr process. It’s related to the Certificate Store after all. But is this really the case?

Huddo Boards & Minio problems – Read before you restart!  

By Martijn de Jong | 11/21/22, 2:06 AM | Infrastructure - Connections | Added by Roberto Boccadoro

Last week I got the unpleasant surprise of a no longer working Huddo Boards for Component pack installation at a customer after I had rebooted my Kubernetes environment. I had to reboot this environment after I updated the Kubernetes certificates. Of course, after a change you immediately think that your problem is related to the change you just made, but in this case the only relation was with the restart, which means that this can happen to everyone running Huddo/Kudos Boards for Component pack or Huddo Boards Docker.

Protecting your Domino container with fail2ban  

By Martijn de Jong | 11/7/22, 4:25 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

If your Domino server is connected to the Internet, you’ll find that bots (hacked systems running a script) will throw a brute force attack on your Domino server. For me, especially, my SMTP server was under heavy attack. The reason why it’s interesting for hackers to find a valid login on an SMTP server, is that this will probably allow them to send spam through your mail server. Most mail servers allow sending mail through their servers for other domains for authenticated users only. The chances of them guessing any of the users in my Domino directory right and then also guessing the password correctly are basically zero, but the pollution of my log file is reason enough to stop them. Fail2ban is a very elegant program for Linux to do just that. You can configure it to scan log files for certain patterns (it uses RegEx to recognise them) and add hosts that match those patterns too often within a defined period of time, to the block list of iptables.

HCL Traveler and error 500  

By Martijn de Jong | 7/21/22, 1:31 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

HCL Traveler is one of those addons for Domino that just works. If you have a properly configured HTTPS stack, you install it, start it and you’re basically done. From now on, you can connect your mobile devices to your Domino server to read your mail and calendar. At least, that has always been my experience until very recently. The other day I was sent to a customer to fix their problem with Traveler. They had upgraded their Domino server and Traveler installation from 8.5.3 FP5 to 12.0.1 FP1. Everything worked (Kudos for Domino!) except Traveler. Though on further discussion with the client it became clear that Traveler actually already broke earlier and hadn’t been working for the past 6 years or so.

Domino containers revisited   

By Martijn de Jong | 7/20/22, 1:57 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

As I wrote in my last post about the Engage conference, a lot has happened in the Domino container space since I wrote my articles, as Daniel Nashed did some serious refactoring on all scripts, removing an insane amount of old code lines and adding some new functionality. This article will show the changes to the project compared to the time that I wrote the original 6-part series.

Working with standard Certificate Authorities in Domino 12  

By Martijn de Jong | 3/28/22, 1:56 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In the past weeks, I helped some colleagues with importing certificates in the Certificate Store of Domino 12 and while doing so, I noticed something peculiar. For many years, we haven’t had a proper way of creating certificates in Domino. The pre-12 database to create keys was completely outdated and didn’t allow for creating strong keys. As a result, most administrators got used to creating keys outside Domino, usually through on openssl command in Linux. This way of working found its way into procedures and many admins, instead of using the Certificate Store database, still follow these old procedures and create their keys outside Domino. I therefore decided to create a short article on how to create certificates with Domino 12 which are signed by a certificate authority which doesn’t support the ACME protocol.

Installing Tivoli/Security Directory Integrator on RHEL 8ì  

By Martijn de Jong | 1/14/22, 11:21 AM | Infrastructure - Connections | Added by Roberto Boccadoro

On a new SDI 7.2 installation (with Java 8 and the latest Connections TDISOL directory for Java 8), I ran into a weird error: CTGDKG023E Error while starting main class.java.lang.reflect.InvocationTargetException .. Caused by: java.lang.UnsatisfiedLinkError: i4clntjni (Not found in java.library.path) Luckily, Google could help me on this one. This technote shows that if there are missing libraries, SDI doesn’t properly install and you will have to uninstall SDI, install the missing libraries, and reinstall SDI.

Expired certificate on your Kubernetes environment  

By Martijn de Jong | 12/7/21, 2:36 AM | Infrastructure - Connections | Added by Roberto Boccadoro

Normally a Kubernetes environment is well maintained and regularly updated with the most recent versions of Kubernetes. However, with a Kubernetes environment that is just used a an HCL Connections Component pack installation, this might not be on your radar and it’s easy to let it just run attended. If you do that for too long though, like longer than one year, you’ll get into trouble

Domino-docker explained – Part 5 : Adding add-ons on top of your Domino image  

By Martijn de Jong | 11/2/21, 2:34 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In the previous parts, I explained how to create a Domino image and deploy it. But what if you want to add fix packs to your Domino image? Or Traveler, Volt or Verse? The scripts of the domino-docker project make this super simple. In this part, I’ll show you how to do this.

Domino-docker explained – Part 4 : The domino_container script  

By Martijn de Jong | 10/22/21, 7:22 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In the previous part, I showed how you can simply start and stop and open the Domino console with the domino_container command. This piece of script is responsible for interacting with the Domino container in a way where the average administrator doesn’t even have to realise that Domino is running inside a container. There are many more functions in this script that will help you manage your Domino server and in this part I will discuss them.

Domino-docker explained – Part 3 : Running your first Domino server in a container  

By Martijn de Jong | 9/30/21, 10:56 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In part 2 we created a Domino container image. Now we want to start the image. Of course, we could just use docker run <options> <imagename>, but with the scripts from the Domino Docker project, there’s a much easier option. In this part, I’ll show you what to do to make running, restarting and stopping images super easy.