HAProxy in front of Kubernetes Workloads: 503 Service Unavailable Errors

I’ve been working to get Resgrid stack running on Docker utilizing Kubernetes. My networks utilize a PFSense firewall running HAProxy that front connections for multiple domain names. Although I have some small static IP blocks dedicating 1 IP in a straight firewall passthrough is not ideal. For all my other setups HAProxy has been working great, even the Rancher UI install has had no issues with running through HAProxy. But I noticed that for workloads using NodePort or HostPort or exposed services via Traefik I was constantly getting the 503 Service Unavailable Error.

What is a 503 Service Unavailable Error? [+How to Fix It]I was able to get HAProxy to pass traffic properly to the workloads when I used the TCP mode instead of HTTP/HTTPS (offloading). But utilizing that mode I’m still in the situation where I would need to dedicated 1 IP per Docker workload that I need to expose.

After my trial and error I found that the “Add ACL for certificate Subject Alternative Names” was causing the 503 Service Unavailable error.

image

When I unchecked both those boxes in the HAProxy Frontend in pfsense and applied the configuration I was able to reach my exposed Traefik service load balancer. I’m also pretty sure this was causing my exposed NodePort\HostPort to get the 503 error as well.

In the Traefik and HAProxy config I won’t be using HAProxy as a load balancer, but instead using it to direct traffic to the correct LB on the backend. I implemented Traefik because of issues I was seeing with rebalancing Pods in the cluster getting ports that HAProxy wasn’t routing to, statically assigning ports also limits the rebalancing the cluster can do.

Hopefully this helps you out! I think I spent 3 solid days trying to get this to work.

I’m the Founder of Resgrid an open source computer aided dispatch (CAD) solution for First Responders, Industrial and Business environments. If you or someone you know is part of a first responder organization like volunteer fire departments, career fire departments, EMS, search and rescue, CERT, public safety or disaster relief organizations check Resgrid out!

About: Shawn Jackson

I’ve spent the last 18 years in the world of Information Technology on both the IT and Development sides of the aisle. I’m currently a Software Engineer for Paylocity. In addition to working at Paylocity, I’m also the Founder of Resgrid, a cloud services company dedicated to providing logistics and management solutions to first responder organizations, volunteer and career fire departments, EMS, ambulance services, search and rescue, public safety, HAZMAT and others.