Issue
Sometimes the operating system is putting limits on users/processes and depending on how you started the process it may have different ways of diagnosing the problem and fixing the problem. CGroups are started under systemd, will have error messages in dmesg and impose their own separate limits from /etc/security/limits.conf. Resetting limits.conf will not affect a systemd started process. Ulimits are normally started processes with a user given certain limits and error messages will be in either stdout or stderr. Because ulimit -a may not exactly capture whether or not the limits given back are the true limits of the process (you need to among other things ensure you are testing the correct user), the best way to check this is to check on the limits of the process itself. You would check:
cmd> cat /proc/<PID>/limits.
This shows the actual limits put on the process in every circumstance.
Environment
Fusion 4.x, Solr, Linux
Resolution
CGroups
The way to check/set limits for no file in CGroups:
cmd> systemctl show -a | grep -i nofile (or nproc, whatever).
In general NPROC and NOFILE are the most common limits someone's system is hitting. In CGroups you could either reset the property using:
cmd> systemctl set-property <myservice> LimitNOFILE=<aLot>
You can also set the limit in the service script itself under:
[Service]
LimitNOFILE=<aLot>
CGroups ignores ulimits, so setting this in limits.conf will not matter. Errors are found by looking at cmd> dmesg -T
Ulimits
If you want to set a ulimit you do so in /etc/security/limits.conf. Soft limit just means that a non-root user can set a different value up to the hard limit. But it's still a limit. Errors will be found in stdout/stderr logs