Ulimits Usage-Why and How to set them

Ulimits Usage:

Ulimits are a set of resource limits that can be set for processes running on the system. These limits help control and manage system resources such as CPU time, memory, file descriptors, and more for individual users or processes. Setting Ulimits can be important for applications like Apache Solr & Fusion to ensure they do not consume excessive resources and negatively impact the overall system performance.

Why set ulimits?

In production scenarios, we often encounter issues and errors such as:

1) Exceeding the Operating System's thread limit, resulting in errors like:

java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)

2) Hitting the Operating System's file descriptor limit, leading to errors like:

java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

OR

Caused by: java.nio.file.FileSystemException: /opt/cvsearch/4.2.5/data/solr/mddocs_signals_shard1_replica_n2/data/index.properties: Too many open files
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)

Environment:

Fusion 4, Solr

Resolution:

There is a shell command available to set the ulimit for processes initiated within that particular shell session.

To view the limits associated with your Linux server, you can utilize the following command:

ulimit -a.

$ ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 32767
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 50
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

How to set ulimits?

1. In a production environment, it is typically recommended to set ulimits to "unlimited." This can be accomplished using the following command:
   ulimit -u unlimited
2. If you need to modify a specific value individually, you can navigate to the following location on your server and edit the file:
   /etc/security/limits.conf 

Solution for above mentioned errors:

1. In response to the error where the server is unable to create any more threads, we have increased the value of "max user processes" from 50 to unlimited.

2. To address the second error mentioned above, where open files reach a limit, we have set the value of "open files" from 1024 to unlimited.

Note: After making these changes please login with a new session. Do not exit the current session as we have seen instances where servers getting locked out while making these changes

For reference:
https://solr.apache.org/guide/solr/latest/deployment-guide/taking-solr-to-production.html#ulimit-settings-nix-operating-systems