Introducing Domino Borg Backup Integration V2  

By Daniel Nashed | 3/19/24 2:34 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Borg Backup is an interesting backup option for Linux (https://www.borgbackup.org/) and also works inside a Domino container with a local or remote repository. The first integration with Domino Backup used bash scripts and Borg commands. But this had limitations due to the way Borg handles backups. Each database was stored in a separate repository. I have been looking for direct integration to avoid this overhead and store all backup data into a single backup. There is a newer option to import tar data directly into Borg as a stream -> https://borgbackup.readthedocs.io/en/stable/usage/tar.html.

High Domino Backup performance with native ZFS storage on Proxmox  

By Daniel Nashed | 3/19/24 2:32 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino 12+ default native backup is a very easy to use option, which also works on Docker containers. The resulting backup to a file target is always consistent, because delta information is always applied to the backup file. But a file target raises the challenge that the whole NSF data will be copied to the target file-share or disk. Therefore a de-duplicating target is highly recommended. I took a look into ZFS in detail in my new local setup to test out performance.

Looking into S3 performance numbers for MinIO -- Is this the right target for backup?  

By Daniel Nashed | 3/19/24 2:31 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

I know MinIO for a while and I have been using it for DAOS T2 testing early on. Years later they are now grow up and play in the cloud native storage league. Still the devil is in the detail and for using it in production environment customers hopefully use the enterprise subscription to get tuning support. Paying for support this doesn't make it a cheap storage any more if you look at their price tag.

First look at openSUSE Leap 15.6 Beta with Domino 14  

By Daniel Nashed | 3/19/24 2:26 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

As some of you know from earlier discussions, the latest currently available SUSE Enterprise and openSUSE Leap 15.5 ships with a too old glibc to work out of the box with Domino 14. You could still run it on a Docker(or Podman) host, because the container image brings the glibc run-time with it and only uses the kernel from the Docker host. openSUSE Leap and SUSE Enterprise (SLES) share the repositories and are technically more or less the same. SUSE Linux 15.6 is scheduled for mid 2024 I have been looking into openSUSE Leap earlier with their Alpha version. Now the official beta is available for download As expected Domino 14 works natively with the updated glibc. The requirement is glibc 2.34+. This Linux version will introduce glibc 2.38. But SUSE also switched again to a new major kernel version with a Service Pack. This means HCL will have to re-rest SUSE Linux once the final version is released. It will take some time to have fully tested and support SUSE supported for Domino 14.0.

Important: For Domino SMTP with ECDSA keys for STARTTLS inbound  

By Daniel Nashed | 3/19/24 2:23 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The short version of you don't want to know all the technical details: If you choose a ECDSA key for your web server, make sure you have also a RSA key for SMTP inbound connections In case you are interested in the technical details, read on ...

Introducing the Domino native Linux installer and Domino Linux Menu  

By Daniel Nashed | 3/19/24 2:21 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

When I ask a question like "why admins are not moving to Domino on Linux" I might have a plan in my head already. I cannot solve all the challenges for you at once. But I am helping over years with my Domino Start Script to get Domino on Linux easier to run. The start script already helps to perform standard operations. Istallation is and some other operations might be still more complicated at first glance on Linux. I introduced a build menu into the HCL Domino Community image process recently. And I took that logic and I am making it available for native installations as well. This new option also offers automated downloads via the recently released Domino Download script

Domino 14.0 on Linux does not work on very old CPUs  

By Daniel Nashed | 3/6/24 4:55 PM | Infrastructure - Notes / Domino | Added by Oliver Busse

Domino 14.0 requires glibc 2.34 because it is build on Redhat Linux 9.1. Glibc is the Linux C run-time which is a core building blog of Linux and can't be changed. This means you can't run it on Linux versions which don't have at least glibc 2.34. Glibc 2.34 itself doesn't work on old CPUs not supporting at least a CPU with microarchitecture level x86-64-v2

Auth0 ODIC OpenID with Domino & Some other interesting findings  

By Daniel Nashed | 2/22/24 1:21 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

We are working on a ODIC setup with a German business partner for a larger German customer. Auth0 is one of the major providers. We got it working but only with some tricks for now. It turned out the Auth0 OIDC endpoint has a cache expiration for 15 seconds. This looks like a setting that can't be changed. The Domino OIDC cache uses the expiration header to invalidate the cache. So our cache on the Domino side was constantly reloading and invalid in some cases. You really have to have an expiration that is at least a couple of minutes. Better at least 1 hour. Faking the cache expiration This has been reported to HCL and the team is working on an enhancement. Meanwhile I came up with a work-around setting up a Fake provider on a NGINX server to forward the requests.

Domino Backup/Restore with multiple configurations and targets  

By Daniel Nashed | 2/22/24 1:14 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino Back/Restore is a flexible framework for native Domino backup. The dominobackup.nsf plays an important role for backup and restore operation. It contains the following type of content. Backup/restore/prune configuration Inventory documents for restore operations Restore requests Backup logs You could run backup with different excludes defined on command-line. Or just backup selected databases or incremental backups. But there cannot be different active configurations nor different backup retention in one dominobackup.nsf

Domino autoupdate.nsf for fast internal software downloads  

By Daniel Nashed | 2/22/24 1:13 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino Autoupdate has been introduced in Domino 14.0. It offers automatic downloads from My HCLSoftware download, which has been on early access in parallel and has been released at the same time. My Engage session will go into detail about the functionality with tips and tricks and additional information round both features and the new Domino Download script (https://nashcom.github.io/domino-startscript/domdownload/). But I want already provide some details about options available today with simple integrations.

Easy to use container image providing ICAP support for ClamAV for Domino CScan  

By Daniel Nashed | 2/22/24 1:11 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

My friend and fellow Ambassador Roberto Boccadoro submitted an Engage session about Domino CScan with ICAP. Sadly his session did not make it into the agenda. But he is part of two OpenNTF sessions. This session idea lead to a new OpenSource project I initiated to help with ICAP support. Thanks Roberto for pushing me to get this implemented! :-) The new project provides a simple to build container image, which natively offers ICAP services over TLS with a ClamAV container in the back-end. The container is ready to be consumed with Domino CScan/ICAP (https://help.hcltechsw.com/domino/14.0.0/admin/conf_scanningattachmentsforviruses.html). It comes with a docker-compose file which glues the official ClamAV container with this new image.

Running Domino in LXC containers on Proxmox requires a trick  

By Daniel Nashed | 2/22/24 1:10 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Now that VMware might not be everyone's darling any more because of it's new mother ship, I took another look at Proxmox. I know them for quite a while and they are doing a great job. I rebuilt a Intel NUC with 2 TB NVMe disk with the current version of Proxmox. Proxmox supports full VMs and also LXC -- which is an interesting option for testing in lab environments. You can setup a new Linux test machine in minutes from a template. And there are ready to use templates for all major Linux distributions. I had a post long time ago about Proxmox automation on command-line.

Certificate ASN.1 Decoding online  

By Daniel Nashed | 2/15/24 2:35 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Now that I posted the TLS 1.2 interactive information side today, some of you might also want to get details out of certificates. Certificates are usually public information. So it should be OK to paste them into the website https://asn1js.eu/. But there is a GitHub project referenced and you could run it also locally. The inner guts of certificates are presented in ASN.1. When you ever has looked at OpenSSL C code, you will recognize the structures.

The Illustrated TLS 1.2 Connection -- Every byte explained  

By Daniel Nashed | 2/15/24 2:32 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

While debugging a TLS connection issue, I ran into this website -->https://tls12.xargs.org/ It provides more details then most admins ever want to know. But it is a great resource understanding a TLS connection.

Domino Backup customized and centralized logging  

By Daniel Nashed | 2/12/24 2:25 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This question came up last week in a business partner workshop. The partner wanted to centralize the logging of all Domino backup instances. dominobackup.nsf intended per server. You could configure a global configuration database and local instances for the backup inventory. In theory it could be one database for multiple could be also replicated in smaller environments. This would not be recommended. But there is an easier way for a centralized overview of all your Domino backups.

Domino adding Trusted Roots for Java applications  

By Daniel Nashed | 2/12/24 2:24 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino has different places to store trusted roots depending on the part of the application. Beginning with Domino 12.0.2 HCL started to consolidate root certificates into the new domain wide certstore.nsf. But it will take some time to have all parts of Domino to use the new trusted roots back-end. New callers like OIDC or CScan/ICAP and the certificate URL heath check already use the new back-end including UI integration. JVM trusted roots cacerts overwritten by Domino update Java still uses it's own cacerts file, which is part of the JVM directory. The file is only admin/root writable. Domino release installers replace the cacerts file with the latest cacerts available. But this overwrites custom certificates imported into cacarts. This is a common problem I ran into twice in the last two weeks.

Enable DKIM for Domino  

By Daniel Nashed | 2/12/24 2:22 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Enable DKIM for Domino - DKIM inbound is supported starting with Domino 12.0.1 - DKIM outbound is supported starting with Domino 12.0.2 Now the first providers raise the bar for sending mails. This might not only be relevant for mass mail. Here is a short write up enabling DKIM for RSA and Ed25519 keys.

New default Let’s Encrypt certificate chain with ISRG Root X1 root  

By Daniel Nashed | 2/12/24 2:18 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Let's Encrypt finally changed their default root certificate from DST Root CA X3 to ISRG Root X1. The old root expired already 2 1/2 years ago, but was cross signed with the new chain. Now finally Let's Encrypt uses the new root by default, which results in a shorter chain. They have been using the older, longer chain to specially support older Android devices, which didn't have the X1 root in their trust store. When you are using Let's Encrypt ACME and did not specify an alternate chain, there is nothing to change. The new shorter certificate chain will be automatically used the next time the certificate is renewed. But in case you set specific settings, you might now have to remove those settings, because they flipped the certificate chains. The alternate chain is now the older longer certificate chain. For Domino CertMgr the custom setting is "ACME Alternate Chain Suffix".

The Art of Troubleshooting   

By Daniel Nashed | 1/15/24 3:38 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

In all the years I am involved in troubleshooting, I still see the same patterns. I am planning to start a new initiative this year. To start with, I wrote a short abstract end of the year when looking into this. See this as a beginning of a change from my side to better help on community level and also provide better services as a HCL business partner. It will also include troubleshooting steps for different kind of problem types like crashes, hangs, memory leaks, performance problems. Not all of it can be described in howto material. But raising the awareness on all parts of the support process, can significantly help to solve problems faster. I am in the troubleshooting business for over 25 years, read NSD before breakfast and wrote my own troubleshooting tools.

Important to know about Domino 14 Inbound Mail Disclaimer  

By Daniel Nashed | 1/12/24 2:40 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

A partner ask in OpenNTF channel about an issue with the new inbound mail disclaimer in Domino 14. It turned out I was able to reproduce the problem on one of my lab servers and this is critical to know if you enable disclaimers written into the body. There are two different ways to setup the feature. 1. Tag the subject line 2. Write HTML to the body of the message with fall back to subject for signed or encrypted messages. My personal recommendation is always to tag the subject, because modification of the body can lead to all kind of side effects with add-on software and also issues that could happen with incorrect formatted messages.

Running Traveler 14 on Windows 11 for testing  

By Daniel Nashed | 1/8/24 3:57 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The Windows desktop versions are completely untested and unsupported for all Domino related products. It is not recommended to run any Domino software in production on Windows desktop! There are still a couple of scenarios where Windows desktop can be relevant. For example a local notebook installation, a training environment or the Windows Sandbox -- which is also a Windows desktop environment. Domino 14.0 changed the installer to prompt to confirm the unsupported Windows version. But the installer continues to work with Windows 11 after you confirmed the message. As a German partner discovered last week, Traveler 14.0 doesn't install on Windows 11. InstallAnywhere runs into an error: Windows DLL failed to load during Installation. Windows 10 works unchanged. The issue starts with Windows 11.

Automatically Updating Ubuntu Linux including mail notification  

By Daniel Nashed | 1/8/24 3:51 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This was on my list for a while. I looked into it, but the mail part wasn't what I wanted. Most admins will just install BSD mailx. But this had way too much dependencies for me. I just wrote a simple tool to replace mailx -> https://github.com/nashcom/nsh-tools/tree/main/nshmailx. But you could just continue to use the standard package. It wasn't just an option for me, because I try to not install other mail server components on a Domino server (like sendmail). The setup for automatic updates are pretty easy and base on scripting.

How to send mails native on Linux and MacOS from command-line  

By Daniel Nashed | 1/8/24 3:49 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

End of last year SMTP Smuggling was a hot topic. It turned out Domino is behaving well as you can read in Thomas Hampel's blog post who is also linking an interesting presentation about this topic. (See details here --> https://blog.thomashampel.com/blog/tomcat2000.nsf/dx/is-hcl-notes-domino-affected-by-smtp-smuggling.htm). Because I didn't find any good tool to craft emails required to check, I wrote a simple command-line tool in C. I used the OpenSSL versions of the network communication to abstract the calls from the standard socket operations. It's a nice show case how SMTP works under the covers -- which didn't change for centuries and works the original standards (RFC 821 and 822) are still what is behind all SMTP traffic. Once that worked, I thought adding STARTTLS would be a good addition. After I was done with my smuggling tests and I could sleep well again, I started thinking about to make it a more useful tool. So I added attachment support creating a MIME message with a Base64 encoded attachment. Here is a link to the proejct in my tools repository --> https://github.com/nashcom/nsh-tools/tree/main/nshmailx

Let’s Encrypt new default chain February 8, 2024  

By Daniel Nashed | 1/8/24 3:47 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Let's Encrypt new default chain February 8, 2024 The change has been announced mid last year and there is probably no action for you to take. It was about time for this move and it is well planned ahead of time. If you didn't change anything on Domino CertMgr side, there is very likely no action to take.

Leveraging Domino Autoupdate for company internal downloads  

By Daniel Nashed | 12/18/23 3:33 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Now that we have software automatically downloaded into autoupdate.nsf in Domino 14.0, the next step would be to distribute software inside your organization. The attachment data is stored in autoupdate.nsf. An agent could be used to find the software and redirect to the right URL. I wrote a simple search and redirect agent and a redirect rule. To download a file it can be just referenced by it's file name. The agent takes care of finding the document and generating the right redirect.

Domino 14.0 on Windows important to read before updating!  

By Daniel Nashed | 12/14/23 7:16 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino 14.0 shipped last week and I wanted to keep technical details for my DNUG Deep Dive presentation later this week. However the first customers contacted me with problems installing Domino 14.0 GA on Windows. There is one important change you should be aware of. The change is documented here --> https://help.hcltechsw.com/domino/14.0.0/admin/enabling_domino_nonadmin_user.html But not everyone might read the latest documentation before installing. If you keep the defaults, you will need to adjust your file-system permissions for translog and other external directories not located below the data directory. Else your server will not start!

Domino Download Bash Script leveraging My HCL Software Portal  

By Daniel Nashed | 11/27/23 12:58 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

The My HCL Software Portal is still an early access offering in parallel to Domino 14 early access. It is planned to replace the Flexnet download soon and way easier and much faster to navigate. The website just works and has awesome performance. Domino 14 AutoUpdate leverages a new software download API to automatically download software into autoupdate.nsf. The download just needs a download token, which can be requested if you are log into https://my.hcltechsw.com/. I have been looking for a way to automatically download software for a couple of years. Now with the new portal and this new API it is possible to write a Bash script for full command-line operations including a simple to use menu. There are two different modes. By default the script uses My HCL Software navigation. But alternatively it can also leverage Domino 14 Auto Update software.jwt, which has more granular information and allows a more structured download package browsing experience.

Get your Linux environment ready for Domino V14  

By Daniel Nashed | 10/20/23 3:43 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino V14 is planned to ship end of this year. For Windows the system requirements don't really change, because of the universal run-time. But for Linux a newer compiler brings new OS dependencies. Specially the glibc version, which brings the base run-time support for C and also the C++ standard libs are important. An application build with a newer compiler on a newer Linux version does not run on older versions with lower glibc versions. glibc is the The GNU C Library - https://www.gnu.org/software/libc/ The new version required was released in August 2021 and is part of most current long term release Linux distributions.

Running Domino with SELinux on current REHL/CentOS Stream 9 & Co  

By Daniel Nashed | 10/9/23 2:56 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

Domino 12.0.2 added support for SELinux in enforced mode, which is enabled by default by newer installations. SELinux is a lower level security feature, which can even limit processes running with root permissions. But the application needs to have a SELinux profile. I ran into this week on my own on a RHEL 9.2 machine and I got the same problem from a partner yesterday. It turns out that systemd can't read from /tmp any more. But the Domino service from my Nash!Com start script writes the domino process id into the /tmp folder. With SELinux enabled you get the following error message when looking into your service status (domino statusd). The start and stop operations of your server will also hang, because systemctl will hang. systemd[1]: domino.service: Can't convert PID files /tmp/domino.pid O_PATH file descriptor to proper file descriptor: Permission denied

Required Notes and Domino anti-virus file exclusions   

By Daniel Nashed | 10/9/23 2:54 AM | Infrastructure - Notes / Domino | Added by Roberto Boccadoro

This discussion came up in an OpenNTF Discord channel. The question was if this might be a good idea to keep OS level anti-virus enabled for Notes/Domino files. There is a clear statement from HCL about exclusions. But the technote doesn't explain why those exclusions are important. The exclusion might be different for each anti-virus production in detail. It also depends on customer IT policies how to exclude data. This can be either by path, extension or process. There are also recommendations from some anti-virus vendors stating the same exclusions for their specific product: Guidelines for excluding Notes and Domino directory and files when running an operating system Antivirus https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0093046 ible Ans