Skip to main content

Raymii.org Raymii.org Logo

Quis custodiet ipsos custodes?
Home | About | All pages | Cluster Status | RSS Feed

OpenVMS 9.2 for x86, Installing HAProxy and troubleshooting UNIX file paths

Published: 19-04-2023 22:30 | Author: Remy van Elst | Text only version of this article


❗ This post is over one years old. It may no longer be up to date. Opinions may have changed.

openvms logo

This article shows you how to install HAProxy on OpenVMS 9.2 for x86. I've often used HAProxy in my career as a sysadmin and find it a very useful tool. HAProxy is an open source, fast, reliable load balancer for TCP and HTTP-based applications. This guide assumes you've set up your OpenVMS system via my guide and the second part of my guide, that will give you a fully licensed OpenVMS installation with networking and SSH access. Since I've used HAProxy so very often to set up high-available clusters and load balancers, I was surprised but happy to see it ported to OpenVMS. This guide shows the setup but also a few OpenVMS specific quirks, like file paths and troubleshooting error messages / logs.

You can read all my OpenVMS articles here.

Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:

I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!

Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.

You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $200 credit for 60 days. Spend $25 after your credit expires and I'll get $25!

HAProxy is an open source, fast, reliable load balancer for TCP and HTTP-based applications. It is particularly suited for high traffic web sites and powers quite a number of the world's most popular sites. It is arguably the de-facto standard Open Source load balancer, shipping with most mainstream Linux distributions, and often deployed by default in cloud platforms.

This OpenVMS port of the HAProxy includes all core functionality provided by the Open Source release, including SSL/TLS support. SSL/TLS support is statically linked into the HAProxy image and uses OpenSSL 1.0.2u

The current release of HAProxy for OpenVMS is based on the HAProxy 1.7.9 distribution.

This guide covers installation and a few OpenVMS specific quirks regarding files and line endings.

This is a screenshot of HAProxy running on OpenVMS, showing the built-in statistics page:

haproxy stats

Installing HAProxy on OpenVMS

Make sure you've set up your OpenVMS installation as I described in my guides, part 1 regarding installation and part 2 regarding networking.

Please also read part 3 which sets up unzip and shows software installation.

Login to the Software Portal and download HAProxy, filename is VSI-X86VMS-HAPROXY-V0107-9A-1.ZIP.

Create a folder on your OpenVMS system where packages will be stored:

CREATE /DIRECTORY /VERSION_LIMIT=2 DKA0:[SW]

Go into that folder:

SET DEFAULT DKA0:[SW]

Copy over the VSI-X86VMS-HAPROXY-V0107-9A-1.ZIP file to your OpenVMS system in the (newly created) sw folder:

$ scp VSI-X86VMS-HAPROXY-V0107-9A-1.ZIP system@192.168.1.23:/SW/
system@192.168.1.23's password:
VSI-X86VMS-HAPROXY-V0107-9A-1.ZIP       100% 6538KB   1.6MB/s   00:03

Read part 3 of my guide to learn more about the "foreign command" for unzip. If you haven't read it, execute this line otherwise unzip won't work:

unzip :== $SYS$COMMON:[SYSHLP.UNSUPPORTED.UNZIP]UNZIP.EXE

Unzip the installation package:

unzip VSI-X86VMS-HAPROXY-V0107-9A-1.ZIP

Output:

Archive:  DKA0:[SW]VSI-X86VMS-HAPROXY-V0107-9A-1.ZIP;1
[...]
**********************************************
* VSI HAProxy Version 1.7-9A-1 for VSI x86_64*
* Version E9.2  Release Notes are included in*
* the ZIP file.                              *
**********************************************
  inflating: HAPROXY-1_7_9A-X86-RELEASE-NOTES.PDF
  inflating: VSI-X86VMS-HAPROXY-V0107-9A-1.PCSI$COMPRESSED
 extracting: VSI-X86VMS-HAPROXY-V0107-9A-1.PCSI$COMPRESSED_VNC
  inflating: Manifest.txt

Start the installation: 

product install haproxy

Output:

Performing product kit validation of signed kits ...
%PCSI-I-VSIVALPASSED, validation of DKA0:[SW]VSI-X86VMS-HAPROXY-V0107-9A-1.PCSI$COMPRESSED;1 succeeded

The following product has been selected:
    VSI X86VMS HAPROXY V1.7-9A             Layered Product

Do you want to continue? [YES]

Confirm with ENTER. Installation starts:

Configuration phase starting ...

You will be asked to choose options, if any, for each selected product and for
any products that may be installed to satisfy software dependency requirements.

Configuring VSI X86VMS HAPROXY V1.7-9A: HAProxy for OpenVMS is based on HAProxy Version 1.7.9

     Copyright 2022 VMS Software Inc.

    VSI Software Inc.

* This product does not have any configuration options.

Execution phase starting ...

The following product will be installed to destination:
    VSI X86VMS HAPROXY V1.7-9A             DISK$X86SYS:[VMS$COMMON.]

Portion done: 0%...90%

When the installation is finished you will see the following output:

The following product has been installed:
    VSI X86VMS HAPROXY V1.7-9A             Layered Product

VSI X86VMS HAPROXY V1.7-9A: HAProxy for OpenVMS is based on HAProxy Version 1.7.9

    Post-installation tasks are required.

    To start HAProxy at system boot time, modify the HAProxy
    configuration file as necessary and add the following lines
    to SYS$MANAGER:SYSTARTUP_VMS.COM:

        $ file := SYS$STARTUP:HAPROXY$STARTUP.COM
        $ if f$search("''file'") .nes. "" then @'file'

    To shutdown HAProxy at system shutdown, add the following lines
    to SYS$MANAGER:SYSHUTDWN.COM:

        $ file := SYS$STARTUP:HAPROXY$SHUTDOWN.COM
        $ if f$search("''file'") .nes. "" then @'file'

To make HAProxy run at startup, use the editor to add those lines to the file as stated:

EDIT SYS$MANAGER:SYSTARTUP_VMS.COM

Almost at the bottom of the file, before the EXIT line, insert the following:

$ file := SYS$STARTUP:HAPROXY$STARTUP.COM
$ if f$search("''file'") .nes. "" then @'file'

Save with CTRL+Z.

Configuring HAProxy on OpenVMS

The configuation file is a regular HAProxy configuration file. The only thing that is a bit different is the file paths, but we'll get to that. Here is a basic example showing an example of HAProxy in TCP mode, load balancing between two MQTT servers, on port 1883 and the statistics page, accessible on port 8100, url /stats. Normally you'd put those statistics on a private IP behind a password because it allows you to turn off servers and change settings, but for a hobbyist setup that is not required. The IP of the OpenVMS server is 192.168.1.23.

Edit the configuration file:

EDIT  sys$startup:haproxy.cfg

Contents:

global
  log 192.168.1.23 local0
  log 192.168.1.23 local1 notice
  maxconn 256

defaults
  log               global
  retries           3
  maxconn           256
  timeout connect   5s
  timeout client    50s
  timeout server    50s

listen mqtt
        bind :1883
        balance
        mode tcp
        log global
        retries 3
        server mqtt1 192.168.1.210:1883 check
        server mqtt2 192.168.1.220:1883 check


listen statictics
        bind :8100
        mode http
        option httplog
        stats enable
        stats uri /stats
        stats refresh 5s

To test the configuration before (re)starting the service, execute the following commands. The first makes haproxy a "forgein command" allowing it to execute with arguments, the seconds starts haproxy with our config file:

haproxy :== $sys$system:haproxy.exe
haproxy "-f" "/sys$startup/haproxy.cfg"

You should be able to visit the statistics page now, or even, if you have the backend servers running, use them:

haproxy stats

HTTP forwarding is possible as well:

listen site1
    bind :80
    mode http
    server http1 192.168.1.110:80 maxconn 32
    server http2 192.168.1.210:80 maxconn 32

If you want certificates or HTTPS, read on.

SSL, OpenVMS file paths, unix style and line endings

One of the things I like to do is have custom error pages in the case of haproxy errors. This is supported by creating a file with the entire HTTP response. Not just HTML, but also headers and stuff like HTTP/1.1 404 Not Found.

This turned out to be a nice learning experience regarding OpenVMS, log files, ported UNIX software and logical names. Join me in the experience.

I added a listen block forwarding port 80 to another server with a errorfile directive:

listen site1
    bind :80
    mode http
    errorfile 404 /dka0/haproxy/404.http
    server http1 192.168.1.110:80 maxconn 32

I also created a folder for the files:

create /dir DKA0:[HAPROXY]
set def DKA0:[HAPROXY]

I then added the following contents to the file:

EDIT 404.http

Contents:

HTTP/1.1 404 Not Found
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="utf-8" />
        <title>404 Not Found</title>
    </head>
    <body>
        <main>
            <h1>404 Not Found</h1>

            This is my custom 404 Not Found page, for HAProxy on OpenVMS! <br>
            Visit <a href="https://raymii.org>Raymii.org</a>.
        </main>
    </body>
</html>

Remember the file naming scheme is different on OpenVMS:

node$device:[root.][directory]file-name.file-type;version

The file separator is the period (.), not the slash (/).

However, starting HAProxy failed with the following error:

$ haproxy "-f" "/sys$startup/haproxy.cfg"
    [ALERT] 105/222650 (1069) : parsing [/sys$startup/haproxy.cfg:35] : error reading file </dka0/haproxy/404.http> for custom error message <404>.
    [ALERT] 105/222650 (1069) : Error(s) found in configuration file : /sys$startup/haproxy.cfg
    [ALERT] 105/222650 (1069) : Fatal errors found in configuration.

At first I thought that it couldn't find the file. File paths on OpenVMS are way different than on UNIX / Linux systems. I looked at a few CWSD (Apache for OpenVMS) manuals and config files and that seemed to be the correct syntax.

I also didn't start haproxy with the command above, I just stopped and started the service, then refreshed the web browser. To figure out the correct command, I took a look at the startup script:

type SYS$STARTUP:HAPROXY$Startup.com

Output:

$ ! HAPROXY$STARTUP.COM
$ !+
$ ! 24-Mar-2022
$ !-
$
$ set noon
$
$ run/detach -
/input=sys$startup:haproxy$run.com/output=sys$manager:haproxy.log -
/process_name="HAProxy" -
/authorize sys$system:loginout.exe

That points to the following file for startup: sys$startup:haproxy$run.com and this file for logs sys$manager:haproxy.log. The startup file contains the haproxy command I used above:

type sys$startup:haproxy$run.com

Output:

$ set verify
$
$ haproxy :== $sys$system:haproxy.exe
$ haproxy "-f" "/sys$startup/haproxy.cfg"
$
$ exit

The log file contaied the following:

$ Set NoOn
$ VERIFY = F$VERIFY(F$TRNLNM("SYLOGIN_VERIFY"))
$
$ haproxy :== $sys$system:haproxy.exe
$ haproxy "-f" "/sys$startup/haproxy.cfg"
[ALERT] 108/193349 (1060) : parsing [/sys$startup/haproxy.cfg:35] : error opening file </dka0/haproxy/404.http> for custom error message <404>.
[ALERT] 108/193349 (1060) : Error(s) found in configuration file : /sys$startup/haproxy.cfg
[ALERT] 108/193349 (1060) : Fatal errors found in configuration.    
$
$ exit
  SYSTEM       job terminated at 16-APR-2023 20:55:20.95

  Accounting information:
  Buffered I/O count:                581      Peak working set size:      16416
  Direct I/O count:                   29      Peak virtual size:         249088
  Page faults:                       736      Mounted volumes:                0
  Charged CPU time:        0 00:00:00.26      Elapsed time:       0 00:00:00.28

I tried all different kinds of syntaxes for the error file:

  • /sys$sysdisk/haproxy/404.http
  • /dka0/haproxy/404.http
  • /sys$disk/haproxy/404.http
  • dka0:[haproxy]404.http (didn't work at all)

I even defined a few Logical Names, sort of symlinks but then for an entire filesystem, specific to OpenVMS:

 define haproxy DKA0:[HAPROXY]

Then checking that new logical name:

dir haproxy

Output:

Directory DKA0:[HAPROXY]

404.HTTP;1          MAINTENANCE.HTML;1

Total of 2 files.

Using /haproxy/404.http in the haproxy configuration gave the same error, no matter what logical name I used.

I found the following command on stackoverflow, which shows all the file-related stuff an OpenVMS program does. Reminds me of strace on Linux.

set watch file/class=(all,nodump)

It outputs a lot of logs when running a program:

haproxy "-f" "/sys$startup/haproxy.cfg"
%XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, File protection (13,1,0): Access requested: 00000004, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, Read only directory access (13,1,0)
%XQP, Thread #0, Directory scan for: HAPROXY.EXE;0, Status: 00000000
%XQP, Thread #0, Access  (0,0,0) Status: 00000910
%XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, File protection (16,1,0): Access requested: 00000004, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, Read only directory access (16,1,0)
%XQP, Thread #0, Directory scan for: HAPROXY.EXE;0, Status: 00000001
%XQP, Thread #0, Alternate access requested: 00000001, Required: 0
%XQP, Thread #0, File protection (11728,39373,0): Access requested: 00000005, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, Read attributes: Access mode haproxy.exe;1 (11728,39373,0)
%XQP, Thread #0, Read attributes: Owner UIC haproxy.exe;1 (11728,39373,0)
%XQP, Thread #0, Read attributes: Header 1 accessibility haproxy.exe;1 (11728,39373,0)
%XQP, Thread #0, Read attributes: File protection haproxy.exe;1 (11728,39373,0)
[...]
%XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, File protection (4581,1,0): Access requested: 00000004, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, Read only directory access (4581,1,0)
%XQP, Thread #0, Directory scan for: MESSAGES.;0, Status: 00000000
%XQP, Thread #0, Directory scan for: MESSAGES.DIR;1, Status: 00000000
%XQP, Thread #0, Lookup  (0,0,0) Status: 00000910
[...]
%XQP, Thread #0, Directory scan for: HAPROXY.CFG;0, Status: 00000000
%XQP, Thread #0, Lookup  (0,0,0) Status: 00000910
%XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, File protection (20,1,0): Access requested: 00000004, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, Read only directory access (20,1,0)
%XQP, Thread #0, Directory scan for: HAPROXY.CFG;0, Status: 00000001
%XQP, Thread #0, Lookup  (11744,2,0) Status: 00000001
%XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, File protection (11744,2,0): Access requested: 00000001, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, Read attributes: Creation date haproxy.cfg;17 (11744,2,0)
%XQP, Thread #0, Read attributes: Revision date haproxy.cfg;17 (11744,2,0)
%XQP, Thread #0, Read attributes: Record attributes haproxy.cfg;17 (11744,2,0)
%XQP, Thread #0, Read attributes: Owner UIC haproxy.cfg;17 (11744,2,0)
%XQP, Thread #0, Read attributes: File protection haproxy.cfg;17 (11744,2,0)
%XQP, Thread #0, Read attributes: User file characteristics haproxy.cfg;17 (11744,2,0)
%XQP, Thread #0, Read attributes: hardlink count haproxy.cfg;17 (11744,2,0)
%XQP, Thread #0, Lookup haproxy.cfg;17 (11744,2,0) Status: 00000001

As you can see, it does a directory scan for haproxy.cfg. It does not list the directory is searches in however. In the output I can see that it does try to access my error file, 404.http:

%XQP, Thread #0, Control function  (11744,2,0) Status: 00000001
%XQP, Thread #0, Final status: 1C000870
%XQP, Thread #0, Volume protection: Access requested: 00000001, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, File protection (11566,4,0): Access requested: 00000004, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, Read only directory access (11566,4,0)
%XQP, Thread #0, Directory scan for: 404.HTTP;0, Status: 00000001
%XQP, Thread #0, File protection (11570,2,0): Access requested: 00000001, Status: 00000001, PrvUsd: 00000000
%XQP, Thread #0, Read attributes: Access mode 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Creation date 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Expiration date 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Backup date 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Last access date/time 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Last attribute update date/time 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Data modification date/time 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Revision date 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: ASCII dates 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Access mode 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Journal flags 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: RU active 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Statistics block 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Find ACE by type 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Record attributes 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: User file characteristics 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: File length hint field 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Read attributes: Symlink meta-data 404.HTTP;1 (11570,2,0)
%XQP, Thread #0, Access 404.HTTP;1 (11570,2,0) Status: 00000001

So it could access the file (I used the filename /dka0/haproxy/404.http).

Debugging further got me nowhere, permissions or other rabbit holes. But then I thought of one thing, which are SSL certificates.

SSL Certificates in HAProxy on OpenVMS

I do know of one other way to get HAProxy to read files, which is by using SSL certificated. I generated a standard HAProxy SSL certificate (which is a private key and a cert plus a chain contatenated in PEM format). I copied over that file and placed the following in my config:

listen site1
    bind :443 ssl crt /dka0/haproxy/mydomain.pem
    mode http    
    server http1 192.168.1.120:443 maxconn 32

Starting HAProxy with this config file gave me no errors whatsoever, in a browser I could visit the site and get the correct certificate. So then I started thinking, why can it read an SSL certificate but not an HTTP Error file, in the same folder? It wasn't the file path syntax that was incorrect, it must be something else...

I went back to the haproxy manual on errorfile for this version and one thing stood out to me:

For better HTTP compliance, it is recommended that all header lines end with CR-LF and not LF alone.

Back in 2018 when I was involved with the AXPBox Alpha emulator I wrote a few articles on OpenVMS and one specific article involved line endings.
That article links to the OpenVMS wizard which explains more than I can do here. I tried the following command to convert the line endings: 

SET FILE/ATTRIBUTE=(RFM=STMLF) 404.http

After which I started HAProxy again and to my big surprise, this time, no error message regarding the errorfile! It was line endings all along. The error message it logged wasn't error ACCESSING the file, it clearly said, error READING the file. The path variable was correct all along, it was the file contents that were wrong!

I hope you enjoyed this trip down the rabbit hole of debugging in OpenVMS. We looked into the startup script to find the actual executable and the location of the log files. We looked at different ways to define a file path in a config file that expects UNIX style file paths and we debugged file access with a hidden undocumented command. In the end it wasn't an error in the file path syntax, but in the file contents, namely the line endings.

Tags: alpha , blog , dec , haproxy , itanium , openvms , pdp , vax , vms , vsi , x86