Solaris Tuning

Few of the things that I learned while performing load tests on a Solaris box were quite interesting and informative. Though these might be archaic for some, it was refreshing to view the performance limitations of an application from a different angle, rather than the code related issues (inefficient queries, synchronization blocks that are used haphazardly, memory hoggers, etc).
I'll try to list some of them that I used on the Solaris box to get additional information. First one was pretty obvious, since when the load test was run, weblogic server starting coughing up IOException (too many open files).
Running ulimit command showed that the open files settings on the box was too low for a high load. 
> ulimit -a
...
file size (blocks)      unlimited

open files                256

stack size (kbytes)    8192
...
This number should be adjusted based on the load (# of users) and other processes running on the system.
> ulimit -n 1024
This number is applicable only for the current session. If it needs to be set permanently, then rlim_fd_max (default hard limit) needs to be set to that number and system will need to be rebooted to make this effective.
To view the current list of file descriptors used by a specific process, use:
> ls /proc//fd | wc -l

Another helpful command is prstat or top (if available) that displays the top most processes running on the server that utilizes the most CPU.
> prstat -a (displays the CPU intensive processes grouped by user)
> prstat -n 3 -c (limits to top 3 processes and prints below the previous line)

Next comes sar (System Activity Reporter). This command lets you view the system activity and the most interesting one for me was
> sar -u (which displays the CPU utilization activity)

Time   %usr(user)  %sys(system)  %wio(waiting for I/O)  %idle(inactive)
If you want to watch the CPU utilization for every minute for the next 10 minutes, use
> sar -u 60 10
If idle time is consistently 0 or very low, it indicates that the CPU is running short on resources. Also if the wait time is consistently high, it indicates that there could be some stuck threads or blocking threads.

vmstat is another command that provides virtual memory, disk, page and CPU information
It displays the run, blocked & swapped processes and typically you should not see a high number under 'blocked' queue for consecutive reads.

iostat displays the input/output statistics for each disk. If the r/s and w/s is consistently high along with %b (% of time spent on transactions), then the application needs to be tuned to use the io processes more effectively.

To summarize, following are the commands that helped/guided me to gain additional insight into my app:

• ulimit
• prstat
• sar
• vmstat
• iostat

Remote Monitoring Using jconsole

Recently, we made an architectural change to our app which would store the search results in HttpSession for post-sorting options. To test the impact of this additional memory usage by the app, I had to run a load-test with few hundred users who'd perform random searches and their search results would be saved to their sessions. Locally I was easily able to set up jmx monitoring by adding the following to server start script:

@REM JConsole
set JAVA_OPTIONS=%JAVA_OPTIONS% -Dcom.sun.management.jmxremote -Xmanagement

But when I tried to do the same for the cluster (running on unix server) where I was going to perform the load test, I ran into minor hitch as the server was running on secure layer (https). After few tries..and reading a little bit more from sun site about jconsole setup, I got it working. Here's what I had to include in my managed server startup script:

# JMX Remote Monitoring Settings
JMX_PROPERTIES=" -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=${JMX_PORT} -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
export JMX_PROPERTIES

JMX_PORT - port for each managed server where jconsole should connect for jmx agent.

And on my laptop, I just had to fire of jconsole and enter the following under "Advanced" tab:

service:jmx:rmi:///jndi/rmi://<hostname>:<managed_server_port>/jmxrmi

.. and you are presented with a beautiful sight (or horrendous) based on how well your vm performs. :-)