Request has been rejected, as system is overloaded

[Bernd Welter]

Hi there,

from time to time customers ask me about the meaning of Request has been rejected, as system is overloaded and what we recommend to do if this error occurs.

Well, the meaning of the error is simply a temporary condition of an overloaded queue:

Default queuesize configured in xserver.properties

Another setting which is important to understand in this context is the poolsize:

By default the poolsize of an xServer is 1. If the machine has some unused ressources (both cores and memory) you can easily activate further backend modules. Just check the modules properties file, e.g. xlocate.properties and increase poolSize. Sometimes it is necessary to evaluate the best poolsize for a given environment, especially if you are running several modules (or software servers) at the same machine.

So if an incoming request can't be handeled immediately because all available backend modules are already in use (remember: each backend module can handle a single transaction at the time) it will be forwarded to the queue but if the queue is already full (i.e. queuesize == number of waiting requests) the server returns the message Request has been rejected, as system is overloaded.

Now you might have several options:

• Increase the poolsize: usually servers are preconfigured with a poolsize of 1. If the local ressources are not used by 100% you could increase the poolsize and enable parallel handling of transactions which has a positive and releasing impact on the queue. We call this mechanism Inner Load Balancing because it is handeled within the xServer..
• Increase the queuesize: if the queueing error occurs only once in a while and you don't have more ressources (neither local nor separate server) you might at least increase the poolsize which could prevent you from facing the error message. On the other hand this could lead to longer response times in case (instead of the known message) as a transaction could get into queue rank 100, 200, ... and will be forwarded to a module after getting through the queue.
• Add more server instances: if the available ressources (cores and memory) are no longer able to deal with the incoming workload it is necessary to add further server instances. In this case you also have to add another component into the design: a LOAD BALANCER which could receive incoming xServer transactions and forward them to the available backend instances. This is what we call Outer Load Balancing because the software servers themselves are not taking care of this level of balancing.

I hope this little overview helps you to understand the principles behind the error message and to get rid of them in the future.

Feedback is always welcome,

Best regards,
Bernd