Solaris Tuning

Few of the things that I learned while performing load tests on a Solaris box were quite interesting and informative. Though these might be archaic for some, it was refreshing to view the performance limitations of an application from a different angle, rather than the code related issues (inefficient queries, synchronization blocks that are used haphazardly, memory hoggers, etc).
I'll try to list some of them that I used on the Solaris box to get additional information. First one was pretty obvious, since when the load test was run, weblogic server starting coughing up IOException (too many open files).
Running ulimit command showed that the open files settings on the box was too low for a high load. 
> ulimit -a
...
file size (blocks)      unlimited

open files                256

stack size (kbytes)    8192
...
This number should be adjusted based on the load (# of users) and other processes running on the system.
> ulimit -n 1024
This number is applicable only for the current session. If it needs to be set permanently, then rlim_fd_max (default hard limit) needs to be set to that number and system will need to be rebooted to make this effective.
To view the current list of file descriptors used by a specific process, use:
> ls /proc//fd | wc -l

Another helpful command is prstat or top (if available) that displays the top most processes running on the server that utilizes the most CPU.
> prstat -a (displays the CPU intensive processes grouped by user)
> prstat -n 3 -c (limits to top 3 processes and prints below the previous line)

Next comes sar (System Activity Reporter). This command lets you view the system activity and the most interesting one for me was
> sar -u (which displays the CPU utilization activity)

Time   %usr(user)  %sys(system)  %wio(waiting for I/O)  %idle(inactive)
If you want to watch the CPU utilization for every minute for the next 10 minutes, use
> sar -u 60 10
If idle time is consistently 0 or very low, it indicates that the CPU is running short on resources. Also if the wait time is consistently high, it indicates that there could be some stuck threads or blocking threads.

vmstat is another command that provides virtual memory, disk, page and CPU information
It displays the run, blocked & swapped processes and typically you should not see a high number under 'blocked' queue for consecutive reads.

iostat displays the input/output statistics for each disk. If the r/s and w/s is consistently high along with %b (% of time spent on transactions), then the application needs to be tuned to use the io processes more effectively.

To summarize, following are the commands that helped/guided me to gain additional insight into my app:

• ulimit
• prstat
• sar
• vmstat
• iostat

Remote Monitoring Using jconsole

Recently, we made an architectural change to our app which would store the search results in HttpSession for post-sorting options. To test the impact of this additional memory usage by the app, I had to run a load-test with few hundred users who'd perform random searches and their search results would be saved to their sessions. Locally I was easily able to set up jmx monitoring by adding the following to server start script:

@REM JConsole
set JAVA_OPTIONS=%JAVA_OPTIONS% -Dcom.sun.management.jmxremote -Xmanagement

But when I tried to do the same for the cluster (running on unix server) where I was going to perform the load test, I ran into minor hitch as the server was running on secure layer (https). After few tries..and reading a little bit more from sun site about jconsole setup, I got it working. Here's what I had to include in my managed server startup script:

# JMX Remote Monitoring Settings
JMX_PROPERTIES=" -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=${JMX_PORT} -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
export JMX_PROPERTIES

JMX_PORT - port for each managed server where jconsole should connect for jmx agent.

And on my laptop, I just had to fire of jconsole and enter the following under "Advanced" tab:

service:jmx:rmi:///jndi/rmi://<hostname>:<managed_server_port>/jmxrmi

.. and you are presented with a beautiful sight (or horrendous) based on how well your vm performs. :-)

Elapsed Time in ksh

Being a novice on shell programming (ksh & bash), I've been looking for something short and sweet that would tell me how much time was taken up for my build (ant) to complete. Printing out the time(s) at the beginning of the script and at the end is very simple.. but what I'd wanted to was to print out something like.. " application build took and was completed successfully at ". I tried googling (of course) and found various scripts involving grep/awk and some perl scripts.. Damn.. quite a lot to do for a simple stuff.. But then my stars aligned properly and I hit upon a forum which gave me EXACTLY what I was looking (something short and sweet)! Just a simple $SECONDS did it! Here's how it was used:

START_TIME=$SECONDS
.. do what you need here..
END_TIME=$SECONDS
ELAPSED_TIME=$(($END_TIME-$START_TIME))

echo "A took $ELAPSED_TIME seconds to complete.."

Mylyn is awesome

When I was trying to install an update for an eclipse plugin, I noticed that there were couple of mylyn related stuff present and I had no idea where they came from.. but later realized that it was bundled along with eclipse europa and later versions. Didnt bother much that time.. but couple of days later, ran into a developerworks article that was talking about how mylyn makes you even more productive. That sprouted my curiosity and I took a dig into what it was about. Played around with it a little bit.. and Boy, was I glad to have run into mylyn! It REALLY DOES make you MUCH MORE productive and more focussed.
And its very simple to use as well.
All you need to do is create a new local task (if you dont have a repository setup) and give it a due date, schedule date (based on ur schedule) and a priority. Activate the task and thats pretty much it! From there on, any file that you open to work on are automatically associated with the currently active mylyn task.. and it also makes it easy on you by removing the files that were closed and that havent been edited, but opened for a long time. Another beauty is that it even associates (displays in package explorer) only the methods that were accessed, which means you dont have to really get lost with sorting through the various classes that are present in your project and even the numerous methods in a class that have no relevance to the task you're working on currently! Finally, since mylyn integrates with CVS, if you have eclipse hooked up to CVS, you'll see that when its time to checkin the changes, all the classes impacted by the task you worked on are automatically grouped for you, making it much easier to group check in the files that were updated for a specific fix or for a new change. Just awesome! No more hassle of trying to figure out which files were changed for which task.. especially when you're working on multiple tasks in a huge project. It also tracks the amount of time spent on each task after its activated.. again making it easy for you to provide an estimate or provide better estimates of time spent on each task (for micro-managing bosses). Im yet to play around with the bug tracking feature.. but I already know that Im gonna love that as well.. even if I dont, Im gonna continue using mylyn from now on for creating my personal tasks.. Please give it a try and fall in love with it.
Here's the link that got me onto mylyn:
http://www.ibm.com/developerworks/java/library/j-mylyn1/

Adios.
Arun

Agile Practices

While I was reading "Pragmatic Programmers" by Venkat Subramaniam and Andy Hunt, I ran into lots of good quips and quotes. BTW, I really enjoyed reading the book. Very well written and doesnt get into a boring lecture, instead keeps it very short and succinct just like what they preach.
I thought it'd be a good idea to summarize some of them for nailing it further into my head.. and also to remind me whenever I feel like Im getting away from the agile practices.. So here they are in no particular order:

• Fix the problem, not the symptom.
• Quick fixes become quick sand.
• You dont have to be great to get started, but you have to get started to be great.
• Learn iteratively and incrementally; Read voraciously.
• Strech beyond purely technical books and topics (project estimation, communication skills etc)
• Learn the new; unlearn the old (which holds you back)
• Only difference between a rut and grave is its dimensions - Keeping old habits past their expiration date is hazardous to your career.
• Small reachable goals keep you moving forward. (Break down your task into smaller and manageable units of code rather than write one do-all function only to realize later that you misunderstood a key point or the client wants to change the process in few areas)
  
• Keep records of decisions and the reasoning behind them. (log, wiki, issue tracking db, etc)
• One test is worth a thousand expert opinions.
• When writing code, choose readability over convenience.
• Instead of being too opaque or clever, follow the PIE (Program Intentively and Expressively) principle.
• Write code to be clear, not clever.
• Dont complicate the design for the sake of perceived performance or elegance.
• Premature optimization is the root of all evil.
• Elegance is easy to understand and recognize, but much harder to create.
• Develop the simplest solution that works - Incorporate patterns, principles and technology only if you have a compelling reason to use them.
• Keeping balance - A box of cotton fibers isnt helpful when you need a sock. (In the name of breaking down functionality into smaller units, do not fragmentize to such an extent that it becomes ridiculously convoluted to invoke a simple function.)
  
• Maintain a log of problems and their solutions.
• Point your colleagues in the right direction instead of handing them the solution.
  
• Code reviews are useless unless you follow up on the recommendations quickly.
• Knowledge grows when given - if you use your candle to light mine, I get light without darkening you.