java - get NoHttpResponseException for load testing

Question

I'm running load tests for my application. I have two servers: one with my application and a dummy-server that is responsible to get me responses.

In my dummy server I have the following jsp code:

<%@ page import="java.util.Random" %>
<%@ page language="java" %>
<%@ page session="false" %>
<%
   String retVal = "some json string";
   Thread.sleep(50);
%>

I'm running the application with tomcat7. My server.xml connection pool (in both servers) looks like:

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-" maxThreads="1500" minSpareThreads="1000" prestartminSpareThreads="true" /> 
<Connector port="9031" protocol="HTTP/1.1" 
           connectionTimeout="20000" 
           maxConnections="4000"
           executor="tomcatThreadPool"
           redirectPort="8443" />

The java code I'm running from the servers is:

 HttpPost post = new HttpPost(bidderUrl);
 post.setHeader("Content-Type", "application/json");    
 // I'm using http client with ThreadSafeClientConnManager
 // total conn = 500,  max conn per route = 100, timeout=500millis
 HttpClient httpClient = httpClientFactory.getHttpClient();
    try {
        post.setEntity(new StringEntity(jsobBidRequest));
        HttpResponse response = httpClient.execute(post);
        ...
    catch (NoHttpResponseException e){
        log.error(e);
    }

I'm running Jmetter with 50 concurrent threads (without a loop) and get a lot of exceptions like this:

org.apache.http.NoHttpResponseException The target server failed to respond 

While I'm running just 5 or 10 concurrent threads everything works ok.

Could you please advice me what could be wrong in my setup? For my understanding, I don't see any errors for the 50 concurrent thread requests.

Answer 1

I've found the root cause of the problem.

For some reason the connection becomes invalid and pool is not aware of it.

In this case the NoHttpResponseException is thrown and the request simply fails. I thought that such issues should be resolved in HTTP client pool layer and be transparent to my code, but this is not as it acts.

To solve this problem the HttpRequestRetryHandler in HTTP client should be overridden:

ThreadSafeClientConnManager cm = new ThreadSafeClientConnManager(schemeRegistry);
...
DefaultHttpClient httpClient = new DefaultHttpClient(cm, params);
httpClient.setHttpRequestRetryHandler(new HttpRequestRetryHandler() {
    @Override
    public boolean retryRequest(IOException exception, int executionCount, 
                                HttpContext context) {
        if (executionCount > 3) {
           LOGGER.warn("Maximum tries reached for client http pool ");
                return false;
        }
        if (exception instanceof org.apache.http.NoHttpResponseException) {
            LOGGER.warn("No response from server on " + executionCount + " call");
            return true;
        }
        return false;
      }
   });