OutOfMemoryError: unable to create new native thread

Issue

Sometimes when java tries to create a new thread, the operating system won't let it. In this case then you get the following error:

java.lang.OutOfMemoryError: unable to create a new native thread

Environment

Any java application

Cause

The underlying cause here is that the OS is denying the java application to create a thread for some reason. It is usually not an OutOfMemory situation, which can be misleading.

Resolution

There are multiple things you might want to check here. The first old-fashioned way is to check your ulimits.

cmd> ulimit -a 

Output of what you want to key in on:

max user processes              (-u) 1024

If this limit is set high then you probably are not hitting this limit if you are doing this for the right user. You must be the same user as the processes that are running. Another more definitive way of checking this value is to look for:

cmd> cat /proc/<pid>/limits 
......
Max processes 1024 1024 processes
......

Next, you might wonder if your stack size in java is set too high. You check this in your -Xss<size>. If this is set too high, every thread needs to create that space. Usually just using the default or 256k works fine. 

There are also cgroups on linux which is kind of another way of doing the same thing listed above. Here we need to check: 

cmd> systemctl status user-1000.slice
● user-1000.slice - User Slice of xxxxxx
  Loaded: loaded
  Active: active since Wed 2018-03-07 16:36:20 PST; 2 years 7 months ago
    Tasks: 10840 (limit: 12288)
  CGroup: /user.slice/user-1000.slice

There are two ways of updating your max task limits depending on how it's configured. 
First - go to the following file and find:

/lib/systemd/system/user-.slice.d/10-defaults.conf
TasksMax=<...> 

Set the new value there. 

If not, to go:

/etc/systemd/system.conf and look for:
DefaultTasksMax=<.....>

Set the new value there

By eliminating the values set there in both places, you can actually remove the limit

Then to get the changes to take effect you:

cmd> sudo systemctl daemon-reexec

To make sure your changes took place you:

cmd> systemctl status user-1000.slice