Timing Out Waiting For An HTTP Connections

HTTP Connection Timeout

As software engineers we always run into some interesting and frustrating issues. I ran into this one just a few days ago while load testing an application.

org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool

The issue doesn't occur during function tests. Right around 30 to 40 minutes into load tests we start seeing this error and from that point on all request to the target service fails.

Environment/setup

• Java 1.8
• Dropwizard 1.1.1
• Apache httpclient 4.5.3
• javax.ws.rs 2.0.1

We looked at the usual suspects:

1. Determine the size of the connection pool and increase it. 
2. Determine the current timeout setting and increase it. 
3. Find out what state the current connections are in and see if they can be returned to the pool

Connection Pool  size:

We don't set this value. It turns out that when the property

jersey.config.client.async.threadPoolSize

 is not set in the ClientProperties  according to the docs:

If the property is absent then thread pool used for async requests will be initialized as default cached thread pool, which creates new thread for every new request, see Executors. When a value > 0 is provided, the created cached thread pool limited to that number of threads will be utilized.

A default value is not set.

We are using the CachedThreadPool, it will continue to create new connections for each request. Naturally this can't go on forever unless connections are closed, as there are limited resources on the client and on the server.

Connection Timeout setting:

This was set to 50000 ms (50 seconds) which is very generous. Note that this is not the actual response time from the request. This timeout is just to establish the connection with the target server. There is no point in increasing this limit, we needed to locate the real cause of the problem. So this was ruled out as a possible solution.

State of the current connections

A quick check with "netstat" command below showed that there are 1023 connections

netstat -a  | grep CLOSE_WAIT | wc -l

These connections are just awaiting close as CLOSE_WAIT is a terminal state and this connection can not be used to data transfer. Now I see the problem,

Notes on Software Design

The Four Elements of Simple Design

Oracle Merge statement - Upsert

Sometimes you need a way to insert and/or update rows in a table. Idea is if the row exists update it with the given data, if not insert the new row.

This is how it's accomplished in Oracle.

drop table foobar;
create table foobar (
  PUBSTYLE_XREF_PK number primary key, 
  PUBLICATION_ID number, 
  PUBSTYLE_ID  number
);

insert into foobar values (1, 123, 4);
insert into foobar values (2, 124, 5);
insert into foobar values (3, 125, 4);


-- this will do the update
merge into foobar a 
  using foobar b 
    ON (a.PUBSTYLE_XREF_PK = 3)
  WHEN MATCHED THEN
      UPDATE SET PPUBSTYLE_ID = 14
      WHERE a.PUBSTYLE_XREF_PK = b.PUBSTYLE_XREF_PK;

-- this will do the insert or update
merge into foobar a 
  using (select 1 from dual) b ON (a.PUBSTYLE_XREF_PK = 3)
  WHEN MATCHED THEN
      UPDATE SET PUBSTYLE_ID = 14
  WHEN NOT MATCHED THEN
      INSERT (A.PUBSTYLE_XREF_PK, A.PUBLICATION_ID, a.PUBSTYLE_ID) VALUES (3, 126, 24);

Validating XML files against a schama or DTD

Using xsi:noNamespaceSchemaLocation

 

JBoss 5.x Configurations

JBoss Logging with Log4j

Pass to the JVM following param 
-Djboss.server.log.threshold=DEBUG 
 

Controlling Access to an Object

I ran into this issue while doing XSLT programming. From XSL you can only call the public static methods of a Java class. Now this presents a problem if multiple threads access your XSLT at the same time and your Java class has some state information e.g. a counter that you don't want to share with or overwritten by other threads. The ideal implementation would be to have a singleton Java object the access to which is controlled by a queue. So the singleton instance is created and destroyed between each use.

Singletons are easy to create [1], and since Java 5.0, its become even easier to create one using enum [2] . The issue is about controlling access to the data values of the Java object in a multi-threaded env. If the same singleton instance is used by multiple threads you could still get corrupted data. So how do you ensure that only one thread uses the critical resources of the instance. The java synchronize comes to the rescue. Create a single public method in your singleton and synchronize the method. That's it. Now you can rest assured that only one thread is using your singleton.

References:
1. www.javacoffeebreak.com/articles/designpatterns/index.html
2. stackoverflow.com/questions/70689/efficient-way-to-implement-singleton-pattern-in-java

Reading a data file as a Resource

Put the file to be read in the src/main/java/resource folder as shown on the right: eg. I put in the log4j.xml here, and I will read/output the file line by line.

Then you can access this resource from your code as following, remember to put the leading slash on the filename

            InputStream fstream = this.getClass().getResourceAsStream("/log4j.xml");

            // Get the object of DataInputStream

            DataInputStream in = new DataInputStream(fstream);

            BufferedReader br = new BufferedReader(new InputStreamReader(in));

            String strLine;

            //Read File Line By Line

            while ((strLine = br.readLine()) != null) {                

                System.out.println(strLine);

            }

            in.close();