Friday, January 6, 2012

Should You Trade Forex Or Stocks?





Today's investors and active traders have access to a growing number of trading instruments, from tried-and-true blue chips and industrials, to the fast-paced futures and forex markets. Deciding which of these markets to trade can be complicated, and many factors need to be considered in order to make the best choice.

The most important element may be the trader's or investor's risk tolerance and trading style. For example, buy-and-hold investors are often more suited to participating in the stock market, while short-term traders, including swing, day and scalp traders, may prefer markets where price volatility is more pronounced. In this article, we'll compare investing in the forex market to buying into blue chip stocks, indexes and industrials. (Learn about the forex market and get to know some beginner trading strategies; check out Forex Trading: A Beginner's Guide.)

TUTORIAL: The Ultimate Forex Guide

Forex Vs. Blue Chips
The foreign exchange market is the world's largest financial market, accounting for more than $4 trillion in average traded value each day as of 2011. Many traders are attracted to the forex market because of its high liquidity, around-the-clock trading and the amount of leverage that is afforded to participants.

Blue chips, on the other hand, are stocks from well-established and financially sound companies. These stocks are generally able to operate profitably during challenging economic conditions, and have a history of paying dividends. Blue chips are generally considered to be less volatile than many other investments, and are often used to provide steady growth potential to investors' portfolios.

Volatility
Volatility is a measure of short-term price fluctuations. While some traders, particularly short-term and day traders, rely on volatility in order to profit from quick price swings in the market, other traders are more comfortable with less volatile and less risky investments. As such, many short-term traders are attracted to the forex markets, while buy-and-hold investors may prefer the stability offered by blue chips.

Leverage
Leverage is another consideration. In the United States, investors generally have access to 2:1 leverage for stocks. The forex market offers a substantially higher leverage of up to 50:1, and in parts of the world even higher leverage is available. Is all this leverage a good thing? Not necessarily. While it certainly provides the springboard to build equity with a very small investment - forex accounts can be opened with as little as $100 - leverage can just as easily destroy a trading account. (For more insight, see Forex Leverage: A Double-Edged Sword.)

Trading Hours
Another consideration in choosing a trading instrument is the time period that each is traded. Trading sessions for stocks are limited to exchange hours, generally 9:30am to 4pm Eastern Standard Time, Monday through Friday with the exception of market holidays. The forex market, on the other hand, remains active round-the-clock from 5pm EST Sunday, through 5pm EST Friday, opening in Sydney, then traveling around the world to Tokyo, London and New York. The flexibility to trade during U.S. Asian and European markets, with good liquidity virtually any time of day, is an added bonus to traders whose schedules would otherwise limit their trading activity. (Just because the forex market trades 24 hours a day doesn't mean you have to. See How To Set A Forex Trading Schedule.)

Forex Vs. Indexes
Stock market indexes are a combination of similar stocks, which can be used as a benchmark for a particular portfolio or the broad market. In the U.S. financial markets, major indexes include the Dow Jones Industrial Average (DJIA), the Nasdaq Composite Index, the Standard & Poor's 500 Index (S&P 500) and the Russell 2000. The indexes provide traders and investors with an important method of gauging the movement of the overall market.

A range of products provide traders and investors broad market exposure through stock market indexes. Exchange-traded funds (ETFs) based on stock market indexes, such as S&P Depository Receipts (SPY) and the Nasdaq-100 (QQQQ), are widely traded. Stock index futures and e-mini index futures are other popular instruments based on the underlying indexes. The e-minis boast strong liquidity and have become favorites among short-term traders because of favorable average daily price ranges. In addition, the contract size is much more affordable than the full-sized stock index futures contracts. The e-minis, including the e-mini S&P 500, the e-mini Nasdaq 100, the e-mini Russell 2000 and the mini-sized Dow Futures are traded around the clock on all-electronic, transparent networks. (To learn more, check out Forex Minis Shrink Risk Exposure.)

Volatility
The volatility and liquidity of the e-mini contracts is enjoyed by the many short-term traders who participate in stock market indexes. The major equity index futures trade at an average daily notional value of $145 billion, exceeding the combined traded dollar volume of the underlying 500 stocks. The average daily range in price movement of the e-mini contracts affords great opportunity for profiting from short-term market moves.

While the average daily traded value pales in comparison to that of the forex markets, the e-minis provide many of the same perks that are available to forex traders, including reliable liquidity, daily average price movement quotes that are conducive to short-term profits, and trading outside of regular U.S. market hours.

Leverage
Futures traders can use large amounts of leverage similar to that available to forex traders. With futures, the leverage is referred to as margin, a mandatory deposit that can be used by a broker to cover account losses. Minimum margin requirements are set by the exchanges where the contracts are traded, and can be as little as 5% of the contract's value. Brokers may choose to require higher margin amounts. Like forex, then, futures traders have the ability to trade in large position sizes with a small investment, creating the opportunity to enjoy huge gains - or suffer devastating losses.

Trading hours
While trading does exist nearly around the clock for the electronically traded e-minis (trading ceases for about an hour a day to enable institutional investors to value their positions), the volume may be lower than the forex market, and liquidity during off-market hours could be a concern depending on the particular contract and time of day.

Tax Treatment
While outside the scope of this article, it should be noted that various trading instruments are treated differently at tax time. Short-term gains on futures contracts, for example, may be eligible for lower tax rates than short-term gains on stocks. In addition, active traders may be eligible to choose the mark-to-market (MTM) status for IRS purposes, which allows deductions for trading-related expenses, such as platform fees or education. In order to claim MTM status, the IRS expects trading to be the individual's primary business; IRS Publication 550 and Revenue Procedure 99-17 cover the basic guidelines on how to properly qualify as a trader for tax purposes. It is strongly recommended that traders and investors seek the advice and expertise of a qualified accountant or other tax specialist to most favorably manage investment activities and related tax liabilities. (Trading forex can make for a confusing time organizing your taxes. These simple steps will keep everything straight. Check out Forex Taxation Basics.)

The Bottom Line
The internet and electronic trading have opened the doors to active traders and investors around the world to participate in a growing variety of markets. The decision to trade stocks, forex or futures contracts is often based on risk tolerance, account size and convenience. If an active trader is not available during regular market hours to enter, exit or properly manage trades, stocks are not the best option. However, if an investor's market strategy is to buy and hold for the long term, generating steady growth and earning dividends, stocks are a practical choice. Regardless of which instrument(s) a trader or investor selects, the decision should be based on which is the best fit.

How Forex and Stock Trading Differ

There are many similarities between the stock market and the forex market, but there are also some differences. In this article, I will discuss some of the key differences between the two markets.

Forex Market

The forex market has many names and can be referred to as the retail off-exchange, foreign exchange, forex or just simply FX market. It is extremely liquid, with over four trillion dollars in volume traded every single day. The forex market is open 24 hours a day, 5.5 days a week.

Since forex is a world trading market, different countries' markets will open and close at different times and thus provide needed liquidity and allow for 24 hour a day trading. The forex markets begin with Japan and Singapore, then Europe, then the United States, Canada and Mexico. As one region's market day ends, the next region's market day is just beginning.

If news hits the wire or an event has taken place, you do not need to wait for the market to open, you can trade at any time 5.5 days a week. This is very appealing if you like to trade on the news and want to be able to get in and out anytime you please.

Another advantage or perhaps disadvantage is leverage. Forex allows you to trade from 50:1 up to 400:1 on your initial margin deposit. On the high side, that means that for a margin deposit of $1,000, you can control $400,000 worth of currency. A margin deposit is not the same as what is considered a stock margin account. In forex, the margin deposit can be likened to a good faith deposit used to bind the contract on the purchase of a house. However, in stocks, the margin can be likened to the down payment used to buy the house. Without appropriate use of risk management, a high degree of leverage can lead to large losses as well as large gains.

Stock Market

The stock market in the United States is open from 9:30 AM to 4:00 PM EST. Some pre-market trading can be done between 7:00 AM and 9:30 AM, as well as some after hours trading from 4:00 PM to 8:00 PM. The stock market is open 5 days a week. It is the most liquid during regular market hours. Pre-market or after hours trading has nominal liquidity. If you wish to buy/sell stocks in other countries, you must see if those stocks are available on your online trading platform or if you have to call you broker to make the trade. Most U.S. online brokers or brokers that operate online platforms only allow you to trade stocks that are listed on the NYSE, NASDAQ, AMEX or OTCBB. All stocks are bought and sold on the exchange that hosts that particular company's stock, so if you do not see that exchange as an option on your drop-down menu, you should call your broker to see if they have the ability to purchase a particular stock on an International Exchange.

News trading can be done during market hours and is difficult to time right. Most major banks and investment banks know when something is happening with the company, since their analysts get the most current information and can post a buy/sell/hold on that particular stock. They will most likely open, close or block trade a position based on their Analyst's and Portfolio Manager's recommendation, which is usually before the average person can get in or out of that particular security.

Stock leverage can mostly be utilized when your account value is above $2,000 and comes with a high interest rate against the leverage. It is possible that you would receive a margin call if your stocks dip below a certain percentage of their value, which must be made up or closed out. Margin requirements can be as high as 50% of your available capital to take a position, but should probably never be more than 10% of your available equity.

Let's take a closer look at both types of markets in the chart below:

It is often easier for a self-trader to start with forex as opposed to stocks, if you have limited funds available. This is due to the higher leverage and the ease of only having to track five major currencies. The stock market is finicky and you would really need to analyze each company individually to determine whether or not you should invest in it (cash flow, balance sheet, profit and loss, products offered, products in the pipeline, years in operation, value stocks, growth stocks, emerging market stocks, etc.). A currency is affected by five main things that I like to call “PEPSC” (pepscee):

  1. Political Events – instability, turmoil or an economically friendly new government
  2. Economic Factors – deficits and surpluses, inflation/deflation, economic reports
  3. Psychology of the Market – flight to quality or anticipation of something happening
  4. Speculation – investing in currencies with higher risk to profit from anticipated price change; can have a negative impact on a country's economy
  5. Currency's Supply and Demand (based on a country's exporting of goods)

If you can understand these events and how they affect a country's currency, you can trade forex. Whether you are an experienced trader or just starting out, it would behoove you to learn everything you can about the markets (forex or stock), so you can feel confident in your trading decisions.

How to analyse web logic thread dump?

I notice that there are some stuck threads.

I like to check what could be the cause of the stuck thread just base on the thread dump logs below. Any advise anyone? And also what is the difference between a fat lock and a thin lock?

        "[STUCK] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" id=87495 idx=0x274 tid=15308 prio=1 alive, in native, blocked, daemon              -- Blocked trying to get lock: com/jnn/testController@0x135a26c0[thin lock] 

ANSWER:

One set of thread dumps alone wont be too helpful to get to the root cause. Take 4 or 5 sets of thread dumps at an interval of 5 seconds between each. so at the end you will have a single log file which has around 20 - 25 seconds worth of action on the app server.

Then you should look for you want to check is a stuck thread or long running transaction is happening, all the thread dumps will show a certain thread id is at the same line in your java stack trace. In simpler terms, the transaction (say in an EJB or database) is spanning across multiple thread dumps and hence needs more investigation.

Now when you run these through Samurai or TDA (I havent used TDA myself), it will highlight these in Red colour so you can quickly click on it and get to the lines showing issues.

See an example of this here. Look at the Samurai output image in that link. Green is fine. Red and grey need looking at.

In your case, thread 25 is blocked trying to get the lock on this object

com/jnn/testController@0x135a26c0  

Search the rest of the lock to see what is holding a lock on the same object, and see why it is not releasing the lock - this will be visible in the stack trace


Obtaining and analyzing thread dumps

Most of us run into bugs where tests "hang". Here are some nice tools and tips I found to obtain and analyze thread dumps. I am sure there may be other tools so if you know of some good ones feel free to add.

Jstack
jstack prints Java stack traces of Java threads for a given Java process or core file or a remote debug server. However jstack is not available for Windows platforms or on the Linux platform.

Stacktrace
Stacktrace has great features which include
1. Thread dump for Java processes running as a Windows service (like Tomcat, for example), started with javaw.exe or embedded inside another process.
2. Thread dump for any applet running inside any browser (Apple, IBM and Sun JDKs for Windows and Mac OS X). StackTrace is known to work with IE, Firefox, Safari and Mozilla.
and many other features..

I usually get the java process id using jps

Control Break/ kill options

On UNIX platforms you can send a signal to a program by using the kill command. This is the quit signal, which is handled by the JVM. For example, on Solaris you can use the command kill -QUIT process_id,
where process_id is the process number of your Java program.

Alternatively you can enter the key sequence \ in the window where the Java program was started. Sending this signal instructs a signal handler in the JVM, to recursively print out all the information on the threads
and monitors inside the JVM.
To generate a stack trace on Windows platforms, enter the key sequence in the window where the Java program is running, or click the Close button on the window.

Byron Nevins has also pointed in his blog how to obtain thread dumps in Glassfish.

TDA
This is a great utility I found for analyzing thread dumps.

I especially liked the ability to filter the threads display to be able to ignore e.g. idle threads.. Also as you can see in Fig 1 The three pane view is really helpful

tda.jpg Fig 1 : Using TDA to analyze thread dumps

Useful Unix commands

CPU Usage
The prstat command displays information about active processes on the system and resource consumption. By default, prstat displays information about all processes sorted by CPU usage.

The prstat -s cpu -n 5 command is used to list the five processes that are consuming the most CPU resources. The -s cpu flag tells prstat to sort the output by CPU usage (This is a default flag set, so can be skipped). The -n 5 flag tells prstat to restrict the output to the top five processes.

Adding the -a option to any prstat command will identify how many processes each user is using, what percent of the CPUs, and how much memory, they are using on a system
prstat -a 

The ps command displays information about currently running processes. Pipe the output to search for specific processes
 ps -ef  | grep java

/usr/ucb/ps -auxwww | grep {pid}

This command gives the complete classpath and start parameters of the process mentioned in pid

Example:

/usr/ucb/ps -auxwww | grep 19138 

user 19138 0.4 2.31272184732136 ? S Jun 01 342:11 /resolve/j2sdk1.4.2_03/bin/java -Xmx1024m -Xms1024m -XX:NewSize=341m -XX:MaxNewSize=341m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:+DisableExplicitGC -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -Xloggc:/resolve/CPEE3112/composer/log/pa3.gc -Drmiport=61050 -Dg2config.home=/resolve/CPEE3112/composer/conf -Dg2config.baseipport=61020 -Dg2config.instance=Engine14 -Dg2.processid=3 -cp /resolve/lib/any_custom_libs.jar:/resolve/lib/G2.jar:/resolve/lib/oracle_10_2_0_1_0jdbc.jar com.xyz.platform.Main -recover &
prstat -L -p {pid}

The above cmd gives the summary of lightweight processes which make up the process. More is at an example I gave here

The top command provides an overview of CPU and memory utilization, and a list of the top consumers of CPU, it updates it's display every few seconds so you can monitor continuously. Use http://www.groupsys.com/top/display/ to view details of the stats shown

top

sar -s

Memory Usage
vmstat 5 vmstat reports virtual memory statistics regarding kernel, thread, virtual memory, and disk, trap, and CPU activity. vmstat also helps calculate the average page scan rate.

Below is the vmstat output
vmstat 5

kthr memory page disk faults cpu

r b w swap free re mf pi p fr de sr s0 s1 s2 s3 in sy cs us sy id

0 0 0 11456 4120 1 41 19 1 3 0 2 0 4 0 0 48 112 130 4 14 82

0 0 1 10132 4280 0 4 44 0 0 0 0 0 23 0 0 211 230 144 3 35 62

0 0 1 10132 4616 0 0 20 0 0 0 0 0 19 0 0 150 172 146 3 33 64

0 0 1 10132 5292 0 0 9 0 0 0 0 0 21 0 0 165 105 130 1 21 78

Note: The first line of vmstat shows a cumulative value and must be ignored. sr is the pages scanned by clock algorithm.

On Unix, to find out how much RAM is present on a machine, use
prtconf grep 'Memory size:'

On Linux, the same is done using
free -m

Disk Space
df -ek

du -h


A selection of useful find commands

=== Find User ===

fuser displays the PIDs of processes using the specified files or file systems.
fuser command is very useful in getting rid of the .nfs files created when processes are killed. The .nfs* names refer to files that have been deleted but are still being held open by a running process.

You can find .nfs files using ls -la

rm does not work in removing .nfs files - use fuser to find the process that's locking the file and then kill the process

/usr/sbin/fuser {filename}

kill -9 {pid}

If you stop the processes that have them open, the files will be tidied-up and removed by the file server.


If you're interested a bit of general background follows:

Most operating systems, including UNIX, operate a policy of not actually removing a deleted file (and freeing up it's data blocks) until the last process that has the file open closes it. So, if a running process has a file open and you use the rm(1) command to delete the file the data blocks on the disk will not be freed up by the OS until the process that has the file open closes it.

On a UNIX host using local disk store this behaviour can manifest itself it some seemingly confusing situations. For example, you may wish to free up some space on a file system that's used 900 MB of it's 1GB quota. You have a large file, 200MB say, named myjava.jar that you believe is no longer required, but is actually currently open in your WebLogic server. Not knowing this you delete myjava.jar and do an ls(1) command to see that the file is no longer listed. However, when you use the df(1) command it still reports that 900 MB of it's 1GB quota is used because your WebLogic server still has the file open. When you shutdown the WebLogic server or the server closes the file the disk space will be released and df(1) will report 700MB of it's 1GB is used.

If the file that is removed is on NFS mounted store then it is possible for a file to be deleted on one client whilst still being open on another client. In this situation the same rule of not actually deleting the file until the last process with it open closes it still applies. However, in order for the NFS file handle used by the client that still has the file open not to be broken a filename reference must be maintained. In order to achieve this and remove the files name from the directory ( so it doesn't show up in an ls command output) the NFS file server renames the deleted file to a name beginning '.nfs'. These are the files you are seeing. When the last process with these files open dies or closes them they will be tidied up and removed. Trying to delete them before they are closed will only result in the file being renamed again.

=== Find Class within Jar file ===

To find a class within a binary jar file

for i in `ls *.jar`; do (jar tf $i grep '{classname}' ) && echo $i;done

If you want to search within all subdirectories use this

for i in `find . -name ‘*.jar’`; do (jar tf $i grep '{CLASSNAME}' ) && echo $i;done

=== Find string within file of name ===

find . -name *.xml -exec grep {} \;

example

find . -name web.xml -exec grep -i servlet {} \;

=== Grep within zip file without unzipping it ===


gzip -c -d {file}.gz | grep {string}

Example

gzip -c -d admin_access.log0001_4Dec08_0147.gz | grep -i adq | wc -l


=== Find process using the port ===

The easy way to do this is using netstat passing the port number

netstat -a | grep 61014

If that does not help getting a pid, run this line below

for i in `ls /proc`; do pfiles $i | grep AF_INET | grep 61014 ; done

Output:

pfiles: permission denied: 12363
sockname: AF_INET 0.0.0.0 port: 61014
pfiles: permission denied: 12384

this shows the process appearing in between pids 12363 and 12384 uses that port.

ls /proc
gives the PIDs in order as .. 12363, 12369, 12384 ..


ps -ef grep 12369
wlsuser 12369 7852 0 01:07:40 ? 4:42 /opt/bea/jdk142_05/bin/java -server -DresKBD -Xms512m -Xmx512m -XX:MaxPermSize=

gave the process details which was using the port

Weblogic - Socket Muxers in Thread Dumps

What are these weblogic.socket.Muxer threads seen in thread dumps ?

Note: for a basic primer on taking thread dumps and analyzing them, see this earlier article

Socket Reader Threads accept the incoming request from the Listen Thread Queue and put it on the Execute Thread Queue.

In WL 8.1, there are 3 socket reader threads by default.
In WL 9 and 10, WebLogic allocates 33% of server threads to act as socket readers by default. This need not be changed usually.

One socket reader thread is usually in the poll function, while the others are available to process requests.
The polling thread is highlighted in the thread dump below.
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x016b2148 nid=0x42 waiting for monitor entry [5997f000..5997fc28]
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
- waiting to lock <0x94846b40> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00683c28 nid=0x41 waiting for monitor entry [59a7f000..59a7fc28]
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
- waiting to lock <0x94846b40> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0079e5b0 nid=0x40 runnable [59b7f000..59b7fc28]
at weblogic.socket.PosixSocketMuxer.poll(Native Method)
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
- locked <0x94846b40> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

In an earlier support case on Stuck Threads, we asked BEA:

Should we worry about the Weblogic.socket.Muxer threads which always show 2 threads waiting for lock and 3rd thread locking the same object?

The Muxer TD is attached. This shows same behaviour on all our Weblogic servers.
Full thread dump Java HotSpot(TM) Server VM (1.4.2_05-b04 mixed mode):

"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0151
c588 nid=0x1b4 waiting for monitor entry [ad57f000..ad57fc28]
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
- waiting to lock <0xd9331760> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x0161
d608 nid=0x1b3 runnable [ad67f000..ad67fc28]
at weblogic.socket.PosixSocketMuxer.poll(Native Method)
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:100)
- locked <0xd9331760> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x01bb
6730 nid=0x1b2 waiting for monitor entry [ad77f000..ad77fc28]
at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:91)
- waiting to lock <0xd9331760> (a java.lang.String)
at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:32)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:219)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:178)

The reply from BEA Support was that the above pattern of weblogic.socket.Muxer threads are not a cause of stuck threads.

Why do they mostly show as being Stuck in Samurai TD analyzer ?




As the image shows, when you analyze thread dumps using Samurai, the muxer threads are shown as being Stuck since they're all locked on the same object. This is probably treated as a deadlock condition.
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'"
- waiting to lock <0xd9b61098> (a java.lang.String)


"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'"
- waiting to lock <0xd9b61098> (a java.lang.String)


"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'"
- locked <0xd9b61098> (a java.lang.String)


But you will see the same in any Thread dump even on a development instance with no requests.
The locks mentioned do show up as red in Samurai - but they aren't deadlocks just regular locks.

A thread gains an exclusive lock on an object to perform some action, then frees it allowing the next thread to gain access.

Additionally, if you look at the thread dumps over time, you'll see that these specific locks are not always present - they are moving between the threads which is indicative of their transitory nature.

I want to know more details on Muxers

The socket Muxer manages the server’s existing socket connections.
It first determines which sockets have incoming requests waiting to be processed. It then reads enough data to determine the protocol and dispatches the socket to an appropriate runtime layer based on the protocol.
In the runtime layer, the socket muxer threads determine which execute thread queue to be used and delegates the request accordingly.

From the documentation on http://edocs.bea.com/wls/docs100/perform/WLSTuning.html#wp1152246 ,
Weblogic has two versions of the socket muxer, one is the Java version and the other uses a native library which makes better use of operating system calls. The Enable Native IO checkbox on the server’s configuration settings tells the server which version to use. This is ON by default for most platforms.

Native muxers provide superior scalability because they implement a non-blocking thread model. When a native muxer is used, the server creates a fixed number of threads dedicated to reading incoming requests. Oracle recommends using the default setting of true for the Enable Native IO parameter which allows the server to automatically select the appropriate muxer to use.

You must ensure that to use Native I/O, the native library must be present in the server’s shared library path . This is set up with the default scripts.
When the server does not find the native library, it throws an error
java.lang.UnsatisfiedLinkError: no muxer in java.library.path
and then loads the Java version of the muxer.

Confirm the LD library path is okay and pointing to the Solaris LD path. Check the startup log when starting a managed server. What is the value of java.library.path?
This is where the JVM actually get's the library from.

http://m-button.blogspot.com/2008/08/how-does-weblogic-handle-socket-muxers.html has a good example of how to identify Native vs Java muxer in a thread dump.

The Thread Dump I’ve used in my examples above uses the Native muxer (weblogic.socket.PosixSocketMuxer) on Solaris.

Solaris has another Native muxer called the weblogic.socket.DevPollSocketMuxer
An example TD using this muxer is shown below.
"ExecuteThread: '4' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" RUNNABLE native
weblogic.socket.DevPollSocketMuxer.doPoll(Native Method)
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:96)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" waiting for lock java.lang.String@4edf4f BLOCKED
weblogic.socket.DevPollSocketMuxer.processSockets(DevPollSocketMuxer.java:95)
weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

To change the number of Muxers from the default, follow the instructions given at http://e-docs.bea.com/wls/docs92/ConsoleHelp/taskhelp/tuning/TuningSocketReaders.html

See http://jojovedder.blogspot.com/2009/07/more-on-weblogic-muxers.htmlfor an update on Muxers

Additionally on Oracle JRockit JVMs - there are some information in the thread dumps which point out the same problem in a different manner.

After the normal stack dumps, BEA JRockit performs a deadlock detection. This is done by finding "lock chains" in the Java application. If a lock chain is found to be circular, the application is considered caught in a deadlock.

A detailed explanation of the 3 types of lock chains in JRockit is given here

What is relevant for us is the example of Muxers which are shown as:


Blocked lock chains
===================
Chain 2:
"ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=129 idx=0x218 tid=4079 waiting for java/lang/String@0x37804000 held by:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=127 idx=0x210 tid=4077 in chain 1

Open lock chains
================
Chain 1:
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=128 idx=0x214 tid=4078 waiting for java/lang/String@0x37804000 held by:
"ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=127 idx=0x210 tid=4077 (active)

As per the explanation, the Open lock chain depicts Thread 1 waiting for Thread 0. This is not a deadlock, only a straight dependency.

Since Thread 0 is already part of the Open lock chain, the fact that Thread 2 is also waiting on the same Thread 0 is treated as a "Blocked lock chain".
In this case this is not a problem.

JVM Heap Analysis using GCViewer

Basics

In an earlier article, I had listed the review of JVM memory parameters as one of the important checks for tuning the JEE server platform.

The basic primer for JDK 1.4 is at http://java.sun.com/docs/hotspot/gc1.4.2/

The key points you need to know are:
Total JVM Heap = Young + Tenured(also called Old)

Young = Eden + From (SS1) + To (SS2)

In the diagram below [taken from the Sun website], "From" and "To" are the names of the two Survivor Spaces (SS) within the "Young".

Perm Space (and code cache): stores JVM’s own stuff. This is outside the Heap you assign using Xms and Xmx. A good explanation of this is available here

The JVM Heap is at default initial 2Mb and max 64Mb (for JDK 1.4 on Solaris).
Default Perm Size is 16MB (for JDK 1.4 on Solaris)
The defaults change for each JDK and are different on each OS - so look up the values on the respective websites.


The ratios are as shown below



Now the object life cycle and garbage collection occurs like this:

1. Objects when created are always first allocated to Eden.
2. When Eden fills up, a fast but not comprehensive GC (minor collection) is run over the young generation only.
3. All surviving objects are moved from Eden into one Survivor Space.
4. In consequent minor collections, new objects move from Eden into the other Survivor Space, plus everything from the first Survivor Space (survivors from the previous minor collection) is also moved into the second Survivor Space. Thus one survivor should be empty at that time.
5. When objects in Survivor Space are old enough (or survivor fills up), they are moved to Tenured. By default the long-lived objects may be copied up to 31 times between the Survivor Spaces before they are finally promoted to the Old generation.
6. When tenured fills up, a Full GC collection is run that is comprehensive: the entire heap is analyzed, all objects that can be destroyed are killed and memory is reclaimed.

Note: the above lifecycle changes slightly when advanced options such as ConcurrentMarkSweep etc are enabled.

Look Closer

What do these values mean ?

A full list of options available at http://java.sun.com/docs/hotspot/VMOptions.html and http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp


The absolute basic ones are listed in the table below. Note: This is for JDK 1.4
Some of these have changed in JDK 1.6