java - JVM unable to create a thread in Cloud Foundry

Question

I'm using Spring boot for the application. There is a particular controller when called, makes a couple of independent requests using OkHttpClient. I am using a thread pool of 200 threads to make these calls. It works absolutely fine in the local environment. However, when deployed to cloud foundry, it shows up an error as follows.

I have reduced the number of threads to 100 but the problem still remains. These are the memory settings in my CF instance -

Instance Memory Limit  8192 MB
Memory Quota  3072 MB (available memory 1024 MB)
Disk Quota  1024 MB

This is the error -

 2019-05-08T19:13:35.68+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.686  INFO 6 --- [ool-2-thread-65] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:35.76+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.769  INFO 6 --- [ool-2-thread-62] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:35.81+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.813  INFO 6 --- [ool-2-thread-54] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:35.95+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.959  INFO 6 --- [ool-2-thread-73] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:35.96+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.967  INFO 6 --- [ool-2-thread-60] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:35.96+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:35.967  INFO 6 --- [ool-2-thread-79] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.03+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.034  INFO 6 --- [ool-2-thread-77] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.05+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.054  INFO 6 --- [ool-2-thread-75] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.06+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.066  INFO 6 --- [ool-2-thread-18] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.11+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.119  INFO 6 --- [ool-2-thread-12] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.121  INFO 6 --- [ool-2-thread-32] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.123  INFO 6 --- [ool-2-thread-71] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] ERR Resource exhaustion event: the JVM was unable to create a thread.
   2019-05-08T19:13:36.12+0530 [APP/PROC/WEB/0] ERR ResourceExhausted! (1/0)
   2019-05-08T19:13:36.13+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.136  INFO 6 --- [ool-2-thread-44] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | Instance Count | Total Bytes | Class Name                                                                                                       |
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583  INFO 6 --- [ool-2-thread-92] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583  INFO 6 --- [ool-2-thread-59] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.583  INFO 6 --- [ool-2-thread-38] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.584  INFO 6 --- [ool-2-thread-25] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 100796         | 11914496    | [C                                                                                                               |
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.584  INFO 6 --- [ool-2-thread-78] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585  INFO 6 --- [ool-2-thread-67] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585  INFO 6 --- [ol-2-thread-100] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 99630          | 2391120     | Ljava/lang/String;                                                                                               |
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.585  INFO 6 --- [ool-2-thread-80] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT 2019-05-08 13:43:36.586  INFO 6 --- [ool-2-thread-10] ContainerTrustManagerFactory$PKIXFactory : Adding System Trust Manager
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 6940           | 2320424     | [B                                                                                                               |
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 41544          | 1661760     | Ljava/util/LinkedHashMap$Entry;                                                                                  |
   2019-05-08T19:13:36.58+0530 [APP/PROC/WEB/0] OUT | 50549          | 1617568     | Ljava/util/HashMap$Node;                                                                                         |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 13832          | 1535128     | Ljava/lang/Class;                                                                                                |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 42807          | 1369824     | Ljava/util/concurrent/ConcurrentHashMap$Node;                                                                    |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 22775          | 1279152     | [Ljava/util/HashMap$Node;                                                                                        |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 21141          | 1183896     | Ljava/util/LinkedHashMap;                                                                                        |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 10387          | 914056      | Ljava/lang/reflect/Method;                                                                                       |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 11925          | 913696      | [Ljava/lang/Object;                                                                                              |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 2202           | 618392      | [I                                                                                                               |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 36548          | 584768      | Ljava/lang/Object;                                                                                               |
   2019-05-08T19:13:36.59+0530 [APP/PROC/WEB/0] OUT | 1168           | 485888      | Lcom/sap/LMChatbot/services/metricSearch;                                                                        |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 992            | 372992      | Ljava/lang/Thread;                                                                                               |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 200            | 370440      | [Ljava/util/concurrent/ConcurrentHashMap$Node;                                                                   |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 18205          | 291280      | Ljava/util/LinkedHashSet;                                                                                        |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 1              | 262160      | [Lorg/h2/mvstore/MVMap;                                                                                          |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 8144           | 195456      | Ljava/util/ArrayList;                                                                                            |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2277           | 182544      | [Ljava/lang/ThreadLocal$ThreadLocalMap$Entry;                                                                    |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 7532           | 172640      | [Ljava/lang/Class;                                                                                               |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2964           | 142272      | Lorg/json/simple/JSONObject;                                                                                     |
   2019-05-08T19:13:36.60+0530 [APP/PROC/WEB/0] OUT | 2177           | 121576      | [Ljava/security/ProtectionDomain;                                                                                |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2406           | 114776      | [Ljava/lang/String;                                                                                              |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 4127           | 99048       | Lorg/springframework/core/MethodClassKey;                                                                        |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 1986           | 95328       | Ljava/util/HashMap;                                                                                              |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 845            | 94640       | Ljava/net/SocksSocketImpl;                                                                                       |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2319           | 92760       | Ljava/util/TreeMap$Entry;                                                                                        |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 1648           | 92288       | Lokio/Okio$4;                                                                                                    |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2185           | 87400       | Ljava/security/AccessControlContext;                                                                             |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 3042           | 73008       | Ljava/util/Collections$UnmodifiableRandomAccessList;                                                             |
   2019-05-08T19:13:36.61+0530 [APP/PROC/WEB/0] OUT | 2242           | 71744       | Ljava/util/LinkedList;                                                                                           |
   2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 2612           | 62688       | Ljava/util/LinkedList$Node;                                                                                      |
   2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 826            | 59472       | Lokhttp3/internal/connection/RealConnection;                                                                     |
   2019-05-08T19:13:36.62+0530 [APP/PROC/WEB/0] OUT | 1846           | 59072       | Ljava/util/concurrent/locks/ReentrantLock$NonfairSync;                                           

Answer 1

So, each of these threads make a REST API call. I was creating a new instance of OkHttpClient for each thread. As it turns out, each client instance would spawn many threads of its own internally. So the total number of threads were more than what I specified.

After sharing the same instance for the OkHttpClient client with all the threads, the issue seems to be resolved.