A load balancer to standardize access to heterogeneous web services

Published the Sunday, 28 January 2024 at 07:40

The primary need is to set up a single entry point for different web services including Apache and a few others on exotic ports.

Like any new project, implementation, as usual, took longer than initially expected.

To this end, I packaged the HAProxy service in the Mageia Linux distribution with a transparent proxy configuration by default.

HAProxy does not support the certificate path layout used on Red Hat-derived distributions, where public and private keys are separated into two separate files in two different directories.

HAProxy suffers from a critical bug when using standard output for logging which resumes at the beginning of the file after a restart.

HAProxy is penalized by disabling the buffer on its standard output when used for logging.

Three fixes were made by me for this purpose to resolve these problems:

New features are being developed for the next version of HAProxy that could make my choices and developments on logging obsolete.

To make support of the HTTP/3 protocol possible by HAProxy, it was necessary to integrate the quictls library, a branch of OpenSSL with support for the QUIC protocol.

To install the server:

# urpmi haproxy haproxy-quic

Activate the service:

# systemctl enable haproxy.service

Start the service:

# systemctl start haproxy.service

In order to avoid a waltz of ports when integrating into an existing architecture, port indirection in Shorewall is proposed:

# Redirect tcp traffic from net on port 80 to 8000
REDIRECT net 8000 tcp 80
# Redirect tcp traffic from net on port 443 to 8000
REDIRECT net 8000 tcp 443
# Redirect udp traffic from net on port 443 to 8443
REDIRECT net 8443 udp 443

This allows HAProxy to capture HTTP and HTTPS traffic on port 8000/TCP and QUIC on port 8443/UDP.

The default configuration captures HTTP and HTTPS traffic on the tcp_default front end, then switches it to the tcp_http and tcp_https back ends, the latter will send the flows to the http_default and https_default front ends, taking care to add a proxy header to transmit the original IP address and port. This configuration allows other TCP services (SSH, VPN, etc.) to be captured and redistributed as needed.

The traffic is then received by the http_default and https_default front-ends respectively, the original IP address and port is preserved via the proxy header received from the TCP back-ends.

They can advertise HTTP/3 support, clean up some headers, deny access to bots, forward original IP addresses and ports via a Forwarded header, and distribute to different backends based on hosts and requested paths.

Configuring Apache to receive the proxy header is problematic, it's simpler to pass the Forwarded header than to configure the freshly released mod_remoteip.

Support for the HTTP/3 protocol is announced by sending an alt-svc: h3=":"; ma= where host can be empty, port=443 and timeout=3600 for example. The certificate served must match the initial host name and not the offload server for security reasons.

Configuring the different http backends will come down to declaring the offload servers, the additional headers to add, the compression according to the content types and the method to test their health.

Without further ado, the configuration to adapt to your needs:

# HAProxy configuration file

# Global config
global
    # Log to systemd
    log stdout format short daemon
    # Number of threads
    nbthread 8
    # Pid file
    pidfile /run/haproxy/haproxy.pid

    # Stat socket
    stats socket /run/haproxy/haproxy.sock mode 0660 level admin
    # Stat timeout
    stats timeout 10s
    # Max connections
    stats maxconn 10

    # Certificate base dir
    crt-base /etc/pki/tls/certs
    # Private key base dir
    key-base /etc/pki/tls/private
    # Don't load extra files
    ssl-load-extra-files none
    # Do not verify certificate
    ssl-server-verify none
    # Supported bind ciphersuites
    #XXX: https://wiki.mozilla.org/Security/Server_Side_TLS#Recommended-configurations
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    # Disable SSL-v3 TLSv1.0 TLSv1.1 and TLSv1.2 without TLS tickets
    ssl-default-bind-options ssl-min-ver TLSv1.3

    # SSL/TLS session cache size
    tune.ssl.cachesize 20000
    # SSL/TLS session life time in cache
    tune.ssl.lifetime 300
    # SSL/TLS layer maximum passed bytes at a time
    tune.ssl.maxrecord 0
    # Diffie-Hellman ephemeral keys max size
    tune.ssl.default-dh-param 2048
    # Buffer size
    tune.bufsize 16384
    # Reserved buffer size
    tune.maxrewrite 1024
    # Max number of headers
    tune.http.maxhdr 101

# Default config
defaults
    # Use global log
    log global

    # Set balance mode
    balance random
    # Set http mode
    mode http
    # Set http keep alive mode
    #XXX: https://cbonte.github.io/haproxy-dconv/2.3/configuration.html#4
    option http-keep-alive
    # Dont log empty line
    option dontlognull
    # Dissociate client from dead server
    option redispatch

    # Number of retries on connection failure
    retries 3
    # Max concurrent connections
    maxconn 30000
    # Max pending connections
    backlog 30000

    # Max time for a connection attempt
    timeout connect 5s
    # Max inactivitiy time on client side
    timeout client 60s
    # Max inactivitiy time on server side
    timeout server 60s
    # Max inactivitiy time on client and server for tunnels
    timeout tunnel 3600s
    # Max time for new request
    timeout http-keep-alive 1s
    # Max time for a complete http request
    timeout http-request 15s
    # Max time for a free slot
    timeout queue 5s
    # Duration for tarpitted connections
    timeout tarpit 5s

    # Error documents
    errorfile 400 /usr/share/doc/haproxy/error/400.http
    errorfile 403 /usr/share/doc/haproxy/error/403.http
    errorfile 408 /usr/share/doc/haproxy/error/408.http
    errorfile 500 /usr/share/doc/haproxy/error/500.http
    errorfile 502 /usr/share/doc/haproxy/error/502.http
    errorfile 503 /usr/share/doc/haproxy/error/503.http
    errorfile 504 /usr/share/doc/haproxy/error/504.http

# Default tcp frontend
frontend tcp_default
    # Set tcp mode
    mode tcp
    # Bind to 8000 port
    bind :::8000
    # Log disabled
    no log
    # Set tcp log
    #option tcplog
    # Set inspect delay
    tcp-request inspect-delay 5s
    # Wait for extension detection
    #tcp-request content accept if { req.proto_http } or { req.ssl_hello_type 1 } or { req.ssl_ec_ext 1 }
    tcp-request content accept if { req.proto_http } or { req.ssl_hello_type 1 }

    # Send to https tcp backend
    use_backend tcp_https if { req.ssl_hello_type 1 }

    # Send to ec tcp backend
    #use_backend tcp_ec if { req.ssl_ec_ext 1 }

    # Send to OpenVPN backend
    #acl openvpn payload(0,2) -m bin 003c
    #tcp-request content accept if openvpn
    #use_backend tcp_openvpn if openvpn

    # Send to OpenSSH backend
    #XXX: https://jonnyzzz.com/blog/2017/05/24/ssh-haproxy/
    #XXX: https://issues.apache.org/jira/browse/SSHD-656
    #acl ssh payload(0,7) -m str SSH-2.0
    #tcp-request content accept if ssh
    #use_backend tcp_ssh if ssh

    # Send to http tcp backend (if { req.proto_http })
    default_backend tcp_http

# Http tcp backend
backend tcp_http
    # Set tcp mode
    mode tcp
    # Send to localhost without ssl with v2 proxy header
    server haproxy 127.0.0.1:8080 no-ssl verify none send-proxy-v2

# Https tcp backend
backend tcp_https
    # Set tcp mode
    mode tcp
    # Send to localhost without ssl with v2 proxy header
    server haproxy 127.0.0.1:8443 no-ssl verify none send-proxy-v2-ssl

# Default http frontend
frontend http_default
    # Bind to 8080 port
    bind :::8080 accept-proxy
    # Insert X-Forwarded-For header
    option forwardfor
    # Set http log format
    option httplog
    # Log enabled
    log global

    # Check if acme challenge
    acl acme_challenge path_beg /.well-known/acme-challenge/

    # Add X-Backend header
    #http-response add-header X-Backend %[haproxy.backend_name]

    # Advertise QUIC
    #http-response add-header alt-svc 'h3=":443"; ma=3600'
    #http-after-response add-header alt-svc 'h3=":443"; ma=3600'

    # Remove server and x-powered-by headers
    #http-after-response del-header server
    #http-after-response del-header x-powered-by

    # Redirect to https scheme when unsecure and not acme challenge
    http-request redirect scheme https code 302 unless { ssl_fc } || acme_challenge

    # Check if denied path
    #XXX: use ,url_dec like in https://serverfault.com/questions/754752/block-specific-url-in-haproxy-url-encoding
    #acl denied_path path_reg ^/(\.env|login|admin/|wp-login\.php|\.git/config)$
    # Deny access on denied path
    #http-request deny if denied_path

    # Check if protected path
    #acl protected_path path_reg ^/(contact|register)$
    # Deny access on protected path
    #http-request deny deny_status 503 if protected_path { method 'POST' } { req.ver '1.0' }

    # Store origin variable as txn
    http-request set-var(txn.origin) req.hdr(Origin)
    # Store host variable as txn
    http-request set-var(txn.host) req.hdr(Host),field(1,:),lower
    # Store proto variable as txn
    http-request set-var(txn.proto) ssl_fc,iif(https,http)

    # Set forwarded proto
    http-request set-header X-Forwarded-Proto %[var(txn.proto)]
    # Set forwarded port
    http-request set-header X-Forwarded-Port %[dst_port]
    # Set forwarded for
    #http-request set-header X-Forwarded-For %[src]
    # Set forwarded by
    http-request set-header X-Forwarded-By %[dst]

    # Set forwarded
    #http-request set-header Forwarded by=%[dst]:%[dst_port];for=%[src]:%[src_port];host=%[var(txn.host)];proto=%[var(txn.proto)]
    http-request set-header Forwarded by=%[dst]:%[dst_port];for=%[src]:%[src_port];proto=%[var(txn.proto)]

    # Check if host is cdn.example.com
    acl cdn var(txn.host) -m str cdn.example.com
    # Check if cdn css path
    acl cdn_css path_beg /css
    # Check if cdn js path
    acl cdn_js path_beg /js
    # Check if haproxy status path
    acl haproxy_status path_beg /haproxy-status
    # Check if debug path
    acl debug path_beg /debug

    # Send to css backend if path start with /css
    use_backend http_css if cdn cdn_css
    # Send to js backend if path start with /js
    use_backend http_js if cdn cdn_js
    # Send to status backend if path start with /haproxy-status
    use_backend http_status if haproxy_status
    # Send to debug backend if path start with /debug
    use_backend http_debug if debug

    # Send to https backend
    use_backend https_default if { ssl_fc }

    # Send to default backend
    default_backend http_default

# Default https frontend
#XXX: copy of upper one, just done to skip logs here
frontend https_default
    # Bind to 8443 tcp port as ssl
    bind :::8443 ssl crt haproxy.pem alpn h2,http/1.1,http/1.0 accept-proxy
    # Bind to 8443 udp port as ssl
    #bind quic6@:::8443 ssl crt haproxy.pem alpn h3
    # Insert X-Forwarded-For header
    option forwardfor
    # Set http log format
    option httplog
    # Log enabled
    log global

    # Check if acme challenge
    acl acme_challenge path_beg /.well-known/acme-challenge/

    # Add X-Backend header
    #http-response add-header X-Backend %[haproxy.backend_name]

    # Advertise QUIC
    #http-response add-header alt-svc 'h3=":443"; ma=3600'
    #http-after-response add-header alt-svc 'h3=":443"; ma=3600'

    # Remove server and x-powered-by headers
    #http-after-response del-header server
    #http-after-response del-header x-powered-by

    # Redirect to https scheme when unsecure and not acme challenge
    http-request redirect scheme https code 302 unless { ssl_fc } || acme_challenge

    # Check if denied path
    #XXX: use ,url_dec like in https://serverfault.com/questions/754752/block-specific-url-in-haproxy-url-encoding
    #acl denied_path path_reg ^/(\.env|login|admin/|wp-login\.php|\.git/config)$
    # Deny access on denied path
    #http-request deny if denied_path

    # Check if protected path
    #acl protected_path path_reg ^/(contact|register)$
    # Deny access on protected path
    #http-request deny deny_status 503 if protected_path { method 'POST' } { req.ver '1.0' }

    # Store origin variable as txn
    http-request set-var(txn.origin) req.hdr(Origin)
    # Store host variable as txn
    http-request set-var(txn.host) req.hdr(Host),field(1,:),lower
    # Store proto variable as txn
    http-request set-var(txn.proto) ssl_fc,iif(https,http)

    # Set forwarded proto
    http-request set-header X-Forwarded-Proto %[var(txn.proto)]
    # Set forwarded port
    http-request set-header X-Forwarded-Port %[dst_port]
    # Set forwarded for
    #http-request set-header X-Forwarded-For %[src]
    # Set forwarded by
    http-request set-header X-Forwarded-By %[dst]

    # Set forwarded
    #http-request set-header Forwarded by=%[dst]:%[dst_port];for=%[src]:%[src_port];host=%[var(txn.host)];proto=%[var(txn.proto)]
    http-request set-header Forwarded by=%[dst]:%[dst_port];for=%[src]:%[src_port];proto=%[var(txn.proto)]

    # Check if host is cdn.example.com
    acl cdn var(txn.host) -m str cdn.example.com
    # Check if cdn css path
    acl cdn_css path_beg /css
    # Check if cdn js path
    acl cdn_js path_beg /js
    # Check if haproxy status path
    acl haproxy_status path_beg /haproxy-status
    # Check if debug path
    acl debug path_beg /debug

    # Send to css backend if path start with /css
    use_backend http_css if cdn cdn_css
    # Send to js backend if path start with /js
    use_backend http_js if cdn cdn_js
    # Send to status backend if path start with /haproxy-status
    use_backend http_status if haproxy_status
    # Send to debug backend if path start with /debug
    use_backend http_debug if debug

    # Send to https backend
    use_backend https_default if { ssl_fc }

    # Send to default backend
    default_backend http_default

# Debug http backend
backend http_debug
    # Check if trusted
    acl trusted src 127.0.0.0/8 ::1
    # Allow access from trusted only
    http-request deny unless trusted
    # Server without ssl or check
    server debug 127.0.0.1:8090 no-ssl verify none

# Default http backend
backend http_default
    # Enable check
    option httpchk
    # User server default
    http-check connect default
    # Send HEAD on / with protocol HTTP/1.1 for host example.com
    http-check send meth HEAD uri / ver HTTP/1.1 hdr Host example.com
    # Expect return code between 200 and 399
    http-check expect status 200-399

    # Insert header X-Server: apache
    #http-response add-header X-Server apache

    # Set compression algorithm
    #compression algo gzip
    # Enable compression for html, plain and css text types
    #compression type text/html text/plain text/css

    # Server with ssl and check without certificate verification
    server apache 127.0.0.1:80 no-ssl verify none check #cookie apache

# Default https backend
backend https_default
    # Enable check
    option httpchk
    # User server default
    http-check connect default
    # Send HEAD on / with protocol HTTP/1.1 for host example.com
    http-check send meth HEAD uri / ver HTTP/1.1 hdr Host example.com
    # Expect return code between 200 and 399
    http-check expect status 200-399

    # Insert header X-Server: apache
    #http-response add-header X-Server apache

    # Force HSTS for 5 minutes on domain and all subdomains
    #http-response set-header Strict-Transport-Security max-age=300#;\ includeSubDomains#;\ preload

    # Set compression algorithm
    #compression algo gzip
    # Enable compression for html, plain and css text types
    #compression type text/html text/plain text/css

    # Server with ssl and check without certificate verification
    server apache 127.0.0.1:443 ssl verify none check #cookie apache

# Css http backend
backend http_css
    # Enable check
    option httpchk
    # User server default
    http-check connect default
    # Send GET on /css/empty.css with protocol HTTP/1.1 for host cdn.example.com
    http-check send meth GET uri /css/empty.css ver HTTP/1.1 hdr Host cdn.example.com
    # Expect return code between 200 and 399
    http-check expect status 200-399

    # Server with check without ssl and certificate verification
    server css 127.0.0.1:80 no-ssl verify none check

# Js http backend
backend http_js
    # Enable check
    option httpchk
    # User server default
    http-check connect default
    # Send HEAD on /js/missing.js with protocol HTTP/1.1 for host cdn.example.com
    http-check send meth HEAD uri /js/missing.js ver HTTP/1.1 hdr Host cdn.example.com
    # Expect return code 404
    http-check expect status 404

    # Check if txn.origin start with https://cdn.example.com
    acl cdn_origin var(txn.origin) -m beg https://cdn.example.com
    # Send origin as ACAO
    http-response set-header Access-Control-Allow-Origin %[var(txn.origin)] if cdn_origin
    # Set ACMA for one day
    http-response set-header Access-Control-Max-Age 86400 if cdn_origin

    # Server with check without ssl and certificate verification
    server js 127.0.0.1:80 no-ssl verify none check

# Status user list
userlist status
    # Add user admin
    user admin insecure-password ADMINPASSWORD
    # Add user operator
    user operator insecure-password OPERATORPASSWORD
    # Assign admin in admin group
    group admin users admin
    # Assign operator and admin in operator group
    group operator users operator,admin

# Status http backend
backend http_status
    # Add operator acl
    acl is_operator http_auth(status)
    # Add admin acl
    acl is_admin http_auth_group(status) admin
    # Check if trusted
    acl trusted src 127.0.0.0/8 ::1
    # Enable stats
    stats enable
    # Set stats hook on /haproxy-status
    stats uri /haproxy-status
    # Set refresh time
    stats refresh 10s
    # Display legends
    stats show-legends
    # Display node
    stats show-node
    # Allow access from trusted or authentified operator only
    #stats http-request auth unless trusted or is_operator
    stats http-request auth unless trusted
    # Activate admin interface from trusted or authentified admin only
    #stats admin if is_admin

Hope this article was helpful to you.