Last month we wrote a post titled “Gimmicky ETFs” in which we discussed the launch of two new ETFs designed to track indices covering companies doing business in Oklahoma and Texas.  We don’t normally look at individual ETFs, but thought this would be an interesting case study.

One of our main concerns at the time was the elevated expense ratio of the Oklahoma Exchange-Traded Fund (OOK).  The fund’s adviser Geary Advisors, LLC reports that they have reduced the fund’s fees to 0.20% which is highly competitive with other specialized ETFs.

Michael Johnston at ETF Database takes an in-depth look at the fund in light of the results of a backtest of the fund’s underlying index.  He comes away generally impressed with the fund’s (theoretical) past performance and finds it a cheap way to play the energy sector.  Johnston writes:

Finally, OOK may be worth a look for anyone seeking to generate alpha. Obviously the past performance of the index doesn’t predict its future returns, but the benchmark’s impressive track record is hard to ignore.

Johnston is correct that the fund has outperformed industry benchmarks like the S&P 500.  You can see this in a graph provided by the fund’s investment manager.  However a quick graph created from the SPADE Indexes site tells another story as well.

Over the past five years the SPADE Oklahoma index has indeed beaten the S&P 500, but has it has roughly matched the performance of the Energy Select SPDR (XLE) with a relatively high (visual) correlation.  This performance should not be all that surprising.  As of October 1st, 2009 the fund’s three biggest industry concentrations were Drilling & Exploration (27%) , Pipelines (22%) and Oil &Gas (19%).

Now that the Oklahoma ETF has a competitive expense ratio investors interested in investing in the fund have a more interesting question to ask.  What is the appropriate benchmark for this fund?  If it is a broad-based index like the S&P 500 then there is a great deal of alpha in the backtest.

Of course the typical caveat that past performance is not indicative of future performance remains true.  Indeed most new index-based ETFs have demonstrated some amount of alpha prior to launch.

However if the fund’s benchmark is something energy-related that alpha becomes less pronounced.  The fund’s industry concentration may change (albeit slowly) over time, but for the moment it is heavily weighted towards the energy sector.  The question then becomes whether the Oklahoma ETF is an efficient play on the energy sector itself.  That question we will leave up to each investor to decide for him or herself.

The lesson in all this is that investors need to be careful consumers of fund performance data.  In short, benchmarks matters.  This is especially important in light of the role that the anchoring effect, which the Psy-Fi Blog calls the “the mother of all behavioral biases” can play on our decision making.  Every investor needs to make sure they are not comparing investment apples to oranges.

*No positions in either OOK or XLE.