HAProxy in front of Kubernetes Workloads: 503 Service Unavailable Errors

I’ve been working to get Resgrid stack running on Docker utilizing Kubernetes. My networks utilize a PFSense firewall running HAProxy that front connections for multiple domain names. Although I have some small static IP blocks dedicating 1 IP in a straight firewall passthrough is not ideal. For all my other setups HAProxy has been working great, even the Rancher UI install has had no issues with running through HAProxy. But I noticed that for workloads using NodePort or HostPort or exposed services via Traefik I was constantly getting the 503 Service Unavailable Error.

I was able to get HAProxy to pass traffic properly to the workloads when I used the TCP mode instead of HTTP/HTTPS (offloading). But utilizing that mode I’m still in the situation where I would need to dedicated 1 IP per Docker workload that I need to expose.

After my trial and error I found that the “Add ACL for certificate Subject Alternative Names” was causing the 503 Service Unavailable error.

When I unchecked both those boxes in the HAProxy Frontend in pfsense and applied the configuration I was able to reach my exposed Traefik service load balancer. I’m also pretty sure this was causing my exposed NodePort\HostPort to get the 503 error as well.

In the Traefik and HAProxy config I won’t be using HAProxy as a load balancer, but instead using it to direct traffic to the correct LB on the backend. I implemented Traefik because of issues I was seeing with rebalancing Pods in the cluster getting ports that HAProxy wasn’t routing to, statically assigning ports also limits the rebalancing the cluster can do.

Hopefully this helps you out! I think I spent 3 solid days trying to get this to work.

I’m the Founder of Resgrid an open source computer aided dispatch (CAD) solution for First Responders, Industrial and Business environments. If you or someone you know is part of a first responder organization like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety or disaster relief organizations check Resgrid out!