Skip to content
Logo-black
  • Native Commerce Core™
    • IDC-marketscape-q4-2024-resources-hub-cover-562x562@2x

      Constructor is a leader in the IDC marketscape

      Constructor is a Leader in the IDC MarketScape on Worldwide Knowledge Discovery Software for External-Facing Use Cases, 2024.

        • Search & Autosuggest

          Hyper-personalized ecommerce search driven by AI and real-time shopper behavior.

        • Browse

          Personalize your ecommerce category pages for maximum conversions.

        • Recommendations

          Drive upsells and increase RPV with hyper-personalized product recommendations.

        • Collections

          Dynamic, personalized ecommerce collection pages that learn from every search, click, and purchase.

        • AI Shopping Agent

          Help your online customers discover great products faster.

        • Merchant Controls & Intelligence

          Get the ecommerce merchandising controls and insights you need to make high-impact changes that drive revenue.

        • Attribute Enrichment

          Automatically enrich product attributes and categories to give shoppers the best possible experience.

        • Quizzes

          Create smart product recommendation quizzes that drive shopper engagement.

        • Retail Media

          Constructor Sponsored Listings complement (instead of compete with) your organic results.

        • Cross-Channel & Offsite Discovery

          Create and launch personalized, data-driven marketing campaigns across digital channels.

        • B2B Use-Cases

          Product discovery technology that’s purpose-built to handle the complex needs of B2B ecommerce.

  • Customers
    • A_B Testing Navigation Image

      The Ultimate Guide to A/B Testing for Search & Discovery

      Discover what your ecommerce team needs to know about fixing broken testing strategies, avoiding wasted effort, and unlocking the full value of A/B testing.

        • Upcoming Events

          Visit us in person at ecommerce industry events.

        • Reports & Guides

          New research, guidance, and insights for the future of retail.

        • Webinars

          Educational webinars, conversations with Constructor partners, and more.

        • Blog

          The latest ideas and news from the frontier of commerce search and discovery.

        • Split/Test Digest

          The most interesting learnings from A/B testing at Constructor.

        • News

          Read Constructor's latest news coverage and press announcements.

        • Documentation

          Learn about the components of Constructor's advanced, AI-powered product discovery suite.

        • About Us

          With AI in our DNA and by keeping ecommerce as our core focus, we’ve built the best product search and discovery solution specifically for the unique needs of retailers.

        • Partners

          Constructor partners with the world’s top retail consultants, systems integrators, and technology platforms.

        • Careers

          Constructor is building the next generation of search solutions.

Login
Book a Demo
  • Native Commerce Core™
    • IDC-marketscape-q4-2024-resources-hub-cover-562x562@2x

      Constructor is a leader in the IDC marketscape

      Constructor is a Leader in the IDC MarketScape on Worldwide Knowledge Discovery Software for External-Facing Use Cases, 2024.

        • Search & Autosuggest

          Hyper-personalized ecommerce search driven by AI and real-time shopper behavior.

        • Browse

          Personalize your ecommerce category pages for maximum conversions.

        • Recommendations

          Drive upsells and increase RPV with hyper-personalized product recommendations.

        • Collections

          Dynamic, personalized ecommerce collection pages that learn from every search, click, and purchase.

        • AI Shopping Agent

          Help your online customers discover great products faster.

        • Merchant Controls & Intelligence

          Get the ecommerce merchandising controls and insights you need to make high-impact changes that drive revenue.

        • Attribute Enrichment

          Automatically enrich product attributes and categories to give shoppers the best possible experience.

        • Quizzes

          Create smart product recommendation quizzes that drive shopper engagement.

        • Retail Media

          Constructor Sponsored Listings complement (instead of compete with) your organic results.

        • Cross-Channel & Offsite Discovery

          Create and launch personalized, data-driven marketing campaigns across digital channels.

        • B2B Use-Cases

          Product discovery technology that’s purpose-built to handle the complex needs of B2B ecommerce.

  • Customers
    • A_B Testing Navigation Image

      The Ultimate Guide to A/B Testing for Search & Discovery

      Discover what your ecommerce team needs to know about fixing broken testing strategies, avoiding wasted effort, and unlocking the full value of A/B testing.

        • Upcoming Events

          Visit us in person at ecommerce industry events.

        • Reports & Guides

          New research, guidance, and insights for the future of retail.

        • Webinars

          Educational webinars, conversations with Constructor partners, and more.

        • Blog

          The latest ideas and news from the frontier of commerce search and discovery.

        • Split/Test Digest

          The most interesting learnings from A/B testing at Constructor.

        • News

          Read Constructor's latest news coverage and press announcements.

        • Documentation

          Learn about the components of Constructor's advanced, AI-powered product discovery suite.

        • About Us

          With AI in our DNA and by keeping ecommerce as our core focus, we’ve built the best product search and discovery solution specifically for the unique needs of retailers.

        • Partners

          Constructor partners with the world’s top retail consultants, systems integrators, and technology platforms.

        • Careers

          Constructor is building the next generation of search solutions.

Login
Book a Demo
Back to Blog

High Priority Request Queue with HAProxy

Posted on:
July 29, 2020
Author:
Constructor Team
Table of Contents:

Load balancers help disperse traffic across a fleet of servers using different algorithms. Often we use load balancers to protect our servers from being overwhelmed by limiting the number of concurrent requests to each server. Load balancers are built to manage a huge number of connections while backend servers might not be that capable which can lead to degradation and even downtime. When there are more requests than the server can take, a requests queue starts to form up. In case of a surge, some requests can wait in the queue longer than usual and even timeout with 503 while waiting to be processed. But, what if we want to prioritize some requests so that they don’t wait in the same queue with all other requests?

The use case for request prioritization

In our case, we have a CPU-bound service. Unlike I/O-bound services, like APIs reading from some external DB, CPU-bound services are almost entirely limited to CPU capacity of the machine which hosts the service. Service like that can serve in parallel as many requests as there are CPU cores. Technically, it can process more than that in parallel, but requests will take longer since there is just no available CPU time. If the server hosting this kind of service has 2 CPU cores, we can limit to 2 parallel requests in the load balancer. This will form the queue as soon as more than 2 requests are being processed at the same time.

Image for post
Current setup with ALB and HAProxy

Next, we have AWS ALB accepting all requests from the outside. Unfortunately, ALB doesn’t support request queueing so it just routes all requests to servers (targets). At each server we have HAProxy accepting connections, balancing requests across workers in the same server (exact count of workers depends on the number of cores) and making queue if there are more requests than we can serve in parallel. In case of request surge, request queue forms up at HAProxy level. In that same queue there are health check requests from ALB, waiting to be processed (ALB periodically sends health check requests to ensure targets are healthy). If service doesn’t respond to those health checks on time, after several tries, the instance will be marked as unhealthy. An unhealthy instance will be selected for recreation by Autoscaling Group (ASG). One instance less in rotation means more load during surge for other instances. Following this principle, in a short time, all instances of such service can be kicked out of rotation, scheduled for recreation which will left region with 0 instances serving traffic (until ASG provisions new instances).

Request serving priority comes to help

We solved this challenge by adding higher priority for the processing of health checks. This means that regular traffic can wait in the queue while we process all health checks as soon as they arrive. That makes sure that instance responds to health checks even during request surge by going around of all other requests. Instead of kicking instance out of rotation, by not responding to health checks, the request spike will just cause some requests to timeout (in the worst-case scenario).

Image for post
HAProxy with 2 incoming queues

For this purpose, we use 2 HAProxy backends, regular backend and one for the health checks. In the frontend section we select backend for each request like in this example:

use_backend backend_1 if req_regular use_backend backend_1_health_check if req_health_check

 

We defined two backends like this (some settings are omitted for brevity), both routing to the same set of workers within 1 instance:

backend backend_1 timeout queue 5s default-server weight 1 maxconn 2backend_1_health_check timeout queue 20s default-server weight 1 maxconn 10
Image for post
Final configuration with 3 queues and request flow

Instead of increasing priority for health checks (because that’s not possible as such), we defined 2 queues. One for all traffic and one for health checks traffic. HAProxy will let requests from this queue downstream respecting maxconn to service which has another queue. This ensures that, in the worst-case scenario, we’ll always have a health check request 3rd in line to be processed (check chart above).

Other use cases

Although not many services out there are entirely CPU-bound with the low number of workers so that health checks need to be prioritized, this technique can be used for other use cases. By controlling maxconn in the backend section you can control the ratio of specific requests and implement “soft” throttling where some type of requests are determined to wait longer compared to others.

Some extra notes:

  • low maxconn is not good for I/O-bound services. It’s always good to understand what kind of workload is being balanced and configure accordingly
  • queue has some low overhead in terms of latency. In our case, we measured ≤ 1ms overhead which was completely acceptable
  • nginx supports similar queueing mechanism but only in Plus version
  • prioritization/QoS can be also implemented by having separate backends for high and low priority requests. For example, a higher number of faster servers serving high priority requests and a smaller number of slower servers serving low priority requests. In that case, HAProxy should be placed in front of the fleet and not in an instance as in the example above.
  • keeping the queue at backend (HAProxy backend, just before services), helps with better utilization of workers in the node.
  • Another advantage of this setup, not covered here, is the ability to retry requests on other workers in case of failure of some sort
  • Some other examples of priority queues.

Update: Alternative method for request prioritization

An alternative method, and probably preferred, to prioritize requests is to use a set-priority-class directive which is available from version 1.9 (Thanks to Daniel for pointing it out).

Usage:

http-request set-priority-class <expr> [ { if | unless } <condition> ]

This is used to set the queue priority class of the current request.
The value must be a sample expression which converts to an integer in the
range -2047..2047. Results outside this range will be truncated.
The priority class determines the order in which queued requests are
processed. Lower values have higher priority.

Using this directive, the solution to prioritizing health checks would like something like this:

http-request set-priority-class int(1) if req_health_check http-request set-priority-class int(10) if req_regular

Want to learn more about optimizing on-site search to drive real business metrics?

For years, retailers have optimized search solely on “relevance,” hoping the results that users want to see (and the results that drive important business metrics) appear at the top.

This is exactly wrong — and it’s not how companies like Google and Amazon optimize their search.

Watch our webinar discussing the limitations of optimizing search results for “relevance,” and the 3 attributes retailers should focus on to drive real business results from search. Check it out here: 

Search Relevance is Dead webinar promo banner

 
 

Subscribe to our Blog

Get conversion-driving insights delivered straight to your inbox.

What's next?

Uncover lost revenue opportunities with a complimentary Search Experience Audit.

According to the 2024 State of Ecommerce Search & Product Discovery Survey, nearly 70% of shoppers think the search function on retail websites needs an upgrade. Our team has run over 1000 A/B tests to identify easy-to-implement algorithmic and UX improvements that get results. Use their research to your advantage with a complimentary Search Experience Audit — no strings attached.

Request an Audit
  • Analyze your search results quality
  • Identify "no results" pages
  • Pinpoint irrelevant results for long-form queries
  • Uncover UX opportunities for Search, Autocomplete, and PLPs

According to the 2024 State of Ecommerce Search & Product Discovery Survey, nearly 70% of shoppers think the search function on retail websites needs an upgrade. Our team has run over 1000 A/B tests to identify easy-to-implement algorithmic and UX improvements that get results. Use their research to your advantage with a complimentary Search Experience Audit — no strings attached.

Request an Audit
  • MACH Certified
  • 21972-312_SOC_NonCPA-150x150
  • Cartification-mark_04-09-300x253-1-150x150
Constructor.io Logo
  • Technology
    • Native Commerce Core™
    • Request a Demo
    • The Proof Schedule®
  • Solutions
    • Search & Autosuggest
    • Browse
    • Recommendations
    • Collections
    • AI Shopping Agent
    • Merchant Controls & Intelligence
    • Attribute Enrichment
    • Quizzes
    • B2B Solutions
  • Resources Hub
    • Events
    • Reports, Guides & Webinars
    • Blog
    • Experiments
    • News
    • Constructor vs Alternatives
    • Documentation
  • About
    • About Us
    • Partners
    • Security & Compliance
    • Careers
  • Technology
    • Native Commerce Core™
    • Request a Demo
    • The Proof Schedule®
  • Solutions
    • Search & Autosuggest
    • Browse
    • Recommendations
    • Collections
    • AI Shopping Agent
    • Merchant Controls & Intelligence
    • Attribute Enrichment
    • Quizzes
    • B2B Solutions
  • Resources Hub
    • Events
    • Reports, Guides & Webinars
    • Blog
    • Experiments
    • News
    • Constructor vs Alternatives
    • Documentation
  • About
    • About Us
    • Partners
    • Security & Compliance
    • Careers
© 2025 Constructor All rights reserved
  • Terms of Service
  • Privacy Policy