Measuring The Immeasurable: Market Lucidity Index

Alexandre Pestov Eugene Bragin May 2010
Alexandre Pestov Eugene Bragin
Measuring the Immeasurable: Market Lucidity Index
MAY 2010
Measuring the Immeasurable: Market Lucidity Index Page 0 of 51

ABSTRACT
This research was conducted to develop the ―Market Lucidity Index‖, an index that offers a relatively quantitative
assessment of the market‘s inclination with respect to interpreting news bearing significant impact on equity
markets. This seemingly unachievable goal was accomplished through establishing a rigorous quantitative
framework for evaluating market expected reaction relative to actual.
The Market Lucidity Index proposed in this paper carries no forecasting value. It is instead a reflection of the
dominant market sentiment with respect to the interpretation of significant economic events. Unlike the P/E ratio
or purely technical indicators like Bollinger Bands and RSI, the index detailed in this paper acts as a ―litmus test‖ of
the behavioural discrepancy between the market‘s expected reaction and its actual. In conjunction with other
indicators, however, it can be used a tool to help identify possible issues with the market (e.g. herding behaviour,
severe optimism and pessimism periods) from the behavioural perspective.
By evaluating the research results, we concluded that in 2009 the equity markets exhibited extreme levels of
optimism, signalling possible disconnect of the market participants‘ reaction from the underlying economic
developments. The index suggests that the fine signals, or ―green shoots‖ of a typical recovery, did not exist to
support the rally the way it progressed. Instead, the market went up on sheer optimism and hope, fuelled by
unprecedented liquidity measures developed by governments and central banks around the world. A stock market
action based on emotions alone possesses tremendous opportunities and danger to the investors, depending on
the preceding market action. As the markets rallied on overwhelming optimism in 2009-2010, it is our view that
retail investors should maintain minimal exposure to equities until both market‘s behaviour and value come in sync
with the long-term norms.

TABLE OF CONTENTS
Abstract ..................................................................................................................................................................................................... 1
1 Introduction .......................................................................................................................................................................................... 3
2. About The Project ............................................................................................................................................................................. 6
3. Methodology ........................................................................................................................................................................................ 7
3.1 Inputs.............................................................................................................................................................................................. 7
3.2 Assumptions and General Approach ..................................................................................................................................... 8
4. Analysis................................................................................................................................................................................................10
4.1 Defining Relevant Economic Indicators ...............................................................................................................................10
4.2 Filters on the S&P 500 Index..................................................................................................................................................10
4.3 Newscast Content ....................................................................................................................................................................11
4.4 Step 1: Calculating Component Changes............................................................................................................................11
4.5 Step 2: Calculating Index Sensitivity .....................................................................................................................................12
4.6 Step 3: Measuring Bias Strength ............................................................................................................................................12
4.7 Step 4: Finalizing Individual Components ............................................................................................................................13
4.8 Step 5: Finalizing The Index ....................................................................................................................................................17
5. Parting Words ...................................................................................................................................................................................19
Acknowledgements ..............................................................................................................................................................................22

1 INTRODUCTION
An optimist believes we live in the best of all possible worlds.

A pessimist fears this is true.
Murphy's Law
The financial and economic areas of human activities, now fully reliant on instant and global communication
exchanges, are amongst the most striking phenomena distinguishing the contemporary society from the previous
eras. They started as frequent human interactions, but they now moved almost entirely into the virtual world.
Myriads of digits symbolizing wealth and capital shift up and down, reflecting ever-changing expectations of
individuals, companies and entire countries that collectively form what we know as ―the market‖. These constant
movements occupy a worthy place in our daily lives. Despite their digital nature, they remain tremendously
important and strangely captivating to human humans—similarly to the powerful natural forces that cause
hurricanes, earthquakes and other significant events.
As it is the case with dealing with significant natural forces, sophisticated observers have developed gauges to
measure movements of the markets known as ―indexes‖. Market indexes are many and show different facets of a
specific asset class in a specific space. Typically, each respective index employs a number of unique methodologies
for calculating index values. They can be simple or sophisticated. However, they all are used to consolidate and
simplify complex realities in order to facilitate their understanding.
Indexes are part of human-made notional arrangements. They combine collective behaviour exhibited by myriad of
market participants. Gauges invented to measure natural forces provide necessary degree of certainty for making
decision. By looking at thermometer we can say whether it is cold or hot, which will help to determine the optimal
course of action, choosing appropriate cloth to wear in this case. Readings of a market index generally don‘t carry
the same suggestive powers. Nor they directly impact most of those who actively monitor them. As a rule, most
index watchers don‘t get richer when index rises or poorer when it falls. Rare exceptions to this rule include
successful day traders and sophisticated HFT algos. Yet, there is no shortage of those who obsessively monitor the
various indexes and not just in the world‘s financial hubs. Presently, even in the most remote towns of Siberia,
where residents don‘t have a slightest idea of who or what Dow Jones is and why it should remain above the
psychologically important level of 10,000, TV broadcast take it as their sworn duty to deliver intra-day updates on
DOW‘s movements. It is important for the markets, then, to establish secondary indicators based on the
underlying mood or condition changes, which help to measure the strength of prevailing forces and aim to
establish future direction. Some examples of it include TED spread and VIX index.
Since the beginning of the financial crisis in 2007 the market has gone through the phases of remarkable volatility.
The massive global sell-off of 2007-2009 was followed by a vigorous comeback staged by the global markets in
2009. And while the events of 2008 could be expected due to the deteriorating fundamentals and liquidity freeze,
the briskness of the market reversal was quite surprising. What was even more surprising is the swiftness by which
market gains outpaced improvements, or lack of thereof, in underlying economic fundamentals. Starting with the
―green shoots‖ that Ben Bernanke observed in mid-March of 2009, the ―recovery‖ somehow has never felt
genuine. Even as the illusionary ―green shoots‖ spread roots in minds of market participants, the situation
continued to feel surreal, even as the long-awaited market recovery ensued. Every ―green‖ day on the market
seemingly went against all odds, as if the market was a mirror copy, a complete opposite, of what should have
occurred. And while it felt like a make-believe game at the beginning, with time it exceeded even wildest
expectations. The market seemed to be bullish, no matter what (from The Daily Bell).

It's BULLISH No Matter What! ...
The price of oil is rising - BULLISH! More profits for the energy companies, and more investments in "clean
energy."
Most of the new jobs created in March were part-time or temporary - BULLISH! Since the economy has turned
the corner full-time job offers are practically a sure thing.
But didn't wages go down too? - BULLISH! Revenues - Costs = Profits!
41 states have revenue shortfalls - BULLISH! Various states have always complained about shortfalls. It's another
sign that things are getting back to normal.
8 million people are still unemployed - BULLISH! That's 8 million spenders, not savers.
Interest rates are rising - BULLISH! Yet another sign that the economy is getting stronger.
Stocks may be going up but on very low volume - BULLISH! That means the "dumb money" hasn't even bought
into this rally yet.
People have a lot of concerns and uncertainty about the future - BULLISH! Not until the "wall of worry" ends will
this party be over.
So much new liquidity will cause inflation - BULLISH! Stocks are one of the best hedges against inflation.
The wars in Iraq and Afghanistan are bankrupting us - BULLISH! Don't get mad, get even. Debit the Treasury
and Credit the defense companies.
Inflation in China is picking up - BULLISH! That should dampen any bubbles that some people worry about.
Gold is going up in price - BULLISH! This is a broad-based rally.
Wait, maybe gold is going down - BULLISH! That means economic fears are dissipating.
Actually the gold price seems to be consolidating and moving sideways - BULLISH! A sell off or rally would mean
things are overheating.
Iran seems determined to develop it's nuclear program - BULLISH! More nuclear power plants means less
demand on oil which means lower energy costs which means more profits.
Israel may be forced to handle Iran themselves militarily - BULLISH! That will kick-start the construction industry
when we rebuild both sides.
The Health Insurance Reform bill is an abomination - BULLISH! If insurance premiums rise there will be
subsidies; if doctors check out they'll be replaced with cheap foreign ones; if care is rationed then costs will be
controlled and profits ensured.
And now the student loan programs are nationalized - BULLISH! Good riddance for the banks. Now the
government can garnish wages and lower the deficit.
The markets are being purposely manipulated with government money - BULLISH! What's not to like? That
means the market ain't going down no matter what.

Big Media is spewing propaganda about the economy - BULLISH! Perception is reality. People only know what
they're taught. Advertising works.
Greece may default - BULLISH! Greek bond holders will make up their loses in the stock market.
Japan is a bug in search of a windshield - BULLISH! Just imagine how much more deficit spending we need to do
to beat them.
The Euro is getting weaker - BULLISH! King dollar is back.
Source: http://www.thedailybell.com/949/Happy-Days-Are-Here-Again.html
We therefore found it useful to develop a framework for validating market lucidity - extreme bullishness or
bearishness - in quantifiable terms. The end objective of our project was to examine the market and identify any
possible interference that must be considered by market participants. We established our hypothesis to test as:
Actual market movements can be evaluated in their correlation to the

expected market direction to identify positive or negative bias dominating
the markets at a given point in time.

2. ABOUT THE PROJECT
To test the hypothesis it was imperative to restrain our natural scepticism towards stock market efficiency. As the
first step, we had to eliminate our personal doubts about the adequacy of market‘s behaviour and questions about
rationality of their existence. Instead, the focus of our research was on finding a rational basis that guides market
participants.
Financial news and data releases form the basis of knowledge that influences decisions of market participants.
Again, it is necessary to define stringent criteria to reduce accidental information perception, excessive amount of
irrelevant news events that form the ―white noise‖, which is a direct result of subjective news interpretations and
occasionally direct market manipulations. In the end, speculators in the market do not engage in a profound socio-
economic analysis. Instead they rush to the surface of the conventional market-moving factors, trying to guess the
intentions of other market players who, by and large, are driven by the same instincts and considerations.
Therefore, it seems quite acceptable and warranted to consider news – information releases that carry market-
influencing powers - as a key source of pulses for market fluctuations. It must be mentioned that these news
releases are sketchy at the best: : They exist only a day or two at most – and are soon deleted, discarded, and
replaced by the next urgent headline.
From this reasoning was born the basic idea behind this paper: Measure unpredictable market movements by
evaluating equally unpredictable news releases. At first glance, the idea of using news to measure the market
sounds as highly complex. However, the task of establishing correlations between news and the market is
simplified by these four essential assumptions:
 Macroeconomic or major financial events, such as earnings releases, unemployment and GDP changes, are
expected to have the greatest consistent influence over the behaviour of speculators;
 Generally, news releases can be defined by a set of three critical numbers: values before and after, and
expected value (consensus);
 Index performance is well documented, and so are the news releases, which allows for evaluating possible
correlations;
 Finally, the original conviction in rational perception of news releases allows restricting the field of study
to the two data series: market performance and corresponding consolidated index of news releases.
And so we defined the objective for this paper: To construct an index that offers a relatively independent and
unbiased assessment of the market lucidity.

3. METHODOLOGY
3.1 INPUTS
The scope of this paper is confined to examining the lucidity of US stock markets participants. Although numerous
stock indexes can be used to track market activities, we selected S&P 500 as a sufficiently representative and the
most appropriate indicator of the broader US stock market. Additionally, the S&P 500 index contains a complete
array of the most significant companies listed on the U.S. exchanges, and consolidated weighted earnings release
evaluation for them can be included as a complimentary component to enhance the ―Market Lucidity Index‖.
From the news analysis perspective, we limited the extent of our research to the news range that can be
quantified. News feeds are delivered to the worldwide consumer marketplace continuously, streaming down from
countless TV screens, web portals, newspapers and other media outlets. Collectively, news is one of the most
persuasive influences on the public opinion, with the power to reinforce or significantly challenge peoples‘ beliefs.
Even the smallest changes to the emotional ―tone‖ of newscasts have the potential to affect behaviour of individual
market participants. If the daily news flow is sufficiently voluminous and convincing, these individuals begin ‗voting
with their feet and wallets‘ collectively. In the most meaningful phase, these market participants begin to network
among themselves, creating a market culture that becomes self-determinant and mutually reinforcing, materially
impacting the broader market. For the purposes of this research, however, we assumed that myriads of newscasts
create generally net-neutral ―white noise‖ that should be filtered out from, rather than included in, our analysis.
Furthermore, the sheer volume of newscasts and unfeasibility to accurately quantify their individual market impact
dictates elimination of this ―white noise‖ from the news impact calculation. As a result, we focused our attention
on the generally accepted economic indicators that have known, and thus possibly quantifiable, effects on the
market performance. Table 1 lists the indicators evaluated for the purposes of this research.
Table 1: Key Economic Indicators
Economic Indicator
Auto and Truck Sales
Business Inventories
Construction Spending
Consumer Confidence
Consumer Credit
CPI : Consumer Price Index
Durable Goods Orders
Employment Cost Index
Existing Home Sales
Factory Orders
GDP : Gross Domestic Product
Housing Starts and Building Permits
Industrial Production
Initial Claims
International Trade
Leading Indicators

M2
NAPM: National Association of Purchasing
Managers
New Home Sales
Personal Income and Consumption
PPI: Producer Price Index
Productivity and Costs
Regional Manufacturing Surveys
Retail Sales
The Employment Report
Treasury Budget
Weekly Chain Store Sales
Wholesale Trade
Additionally, we included a supplementary model that was adjusted for an accumulative S&P 500 companies‘
earnings impact as described later in this document.
News releases for each of the economic indicators were evaluated using the three key components:
 Actual value of the component as presented in the newscast;
 Previous value of the same component;
 Estimates or expected value of the component (market consensus).
These three values were linked using the newscast dates to the S&P 500 index performance to create the four
pivot points of our analysis. Since the objective of our research included measurement of the long-term market
fluctuations, the time range of our study was limited only by the availability of reliable information and data series.
This analysis is made on the basis of the data collected for the range of January 1, 1998 to May 15, 2010.
3.2 ASSUMPTIONS AND GENERAL APPROACH
As noted in the introduction section of this document, we assumed that the market is driven by a Brownian
motion of pseudo-rational actions. These actions are numerous, sporadic and frequent. Although collectively they
form the broader market shifts, individually they exist only on a very short time line, and small sporadic
movements of individual market participants effectively cancel each other out, having net-zero effect on a broader
market. With this in mind, as well as the assumption of that the main driving force behind the market shifts is the
economic health at the moment in time as perceived by the market participants, we established that an unbiased
market will move up in reaction to positive economic news or shift down in response to negative. The schedule
for each news release tied to the corresponding market movements allowed isolating individual components in
order to assess the market impact of each newscast.
Certainly, limiting market impact evaluation to a set of scheduled economic indicators and earning releases does
not accurately reflect all sentiments prevailing in the market. There are plenty of other factors that affect the
minds of speculators. Since our objective was to create a relative evaluation of market lucidity, the presence of
other news can be combined and encapsulated into a notion of ―market bias‖. Then a background informational
(or noise) will be expressed as a disagreement with the expected rational market behaviour, and thus allow
identifying the nature of the bias. For example, if the market moves up on negative news, we can conclude that a

positive bias exists within the system. Similarly, if the market falls on positive news, we determine that the market
bias is negative. On a long enough timeline this approach will allow to indirectly integrate the analysis of the sum of
all expectations to determine the prevailing mood of the market and identify lapses in market ability to interpret
news.
To simplify the analysis, in situations when the expected market reactions matches the actual, the mismatch in
reaction sensitivity (e.g. on a massively positive news the market reaction is positive, although fairly muted) is
ignored. For example, the market reaction to an unemployment rise of 0.5% was as follows:
 6 June 2008: -1.8%

 January 9, 2009: - 4%
 March 6, 2009: - 2.5%
 June 5, 2009: - 0.004%
Although in all cases, rising unemployment resulted in falling markets, the reaction was much stronger on January
9, 2009 than in June 5, 2009, when it showed only symbolic move down. Such sensitivity analysis requires more
thorough investigation and modeling, and it will be reserved for the future research. However, for the purposes of
this paper, only correlations of news and the S&P 500 index changes direction were considered.
To smooth out fluctuations and reduce the impact of outliers, moving averages were used where applicable. To
factor in actual component changes as well as market expectation, a combination of value change (current vs.
previous value) and missed/exceeded expectations (current vs. expected value) was employed in the research.
The value for the earnings releases was calculated on the basis of individual weighting of a particular stock within
the S&P 500 index, ensuring that all companies received relative merit proportional to their significance for the
equities markets.
The final index presented in this paper is a combination of weighted individual component indexes. This index is a
measurement of the bias prevailing on the market at any point in time. Comparison of the reaction of the market
in different periods of time allows seeing the dynamics of its optimism and pessimism, which is essentially
measuring the immeasurable.

4. ANALYSIS
4.1 DEFINING RELEVANT EC ONOMIC INDICATORS
The first step of our analysis was to separate relevant economic indicators from those that have no impact on the
market. After running correlation analysis between S&P 500 index and all components listed in Table 1, we
determined that the following indicators (Table 2) exhibited a good degree of correlation with the index on the
basis of reliable historical data. The other economic indicators were found to be uncorrelated, derivative or
redundant. Thus, the economic indicators no listed in Table 2 were excluded from the further analysis.
Table 2: Economic Indicators Exhibiting Correlation with S&P 500 Index
Component
Building Permits
Chicago PMI
Durable Goods Orders
Existing Home Sales
GDP
Housing Starts
Industrial Production
Initial Claims
ISM Index
New Home Sales
Personal Income
Retail Sales
Unemployment Rate
University of Michigan Consumer Sentiment
Index
It must be noted that news releases for GDP usually follow in the following sequence: preliminary, initial and final.
Similar situation applies to the University of Michigan Consumer Sentiment Index, which is comprised of
preliminary and revised releases. It was assumed that the market should react just as rationally to revisions as it
should to the actual net-new changes. Therefore, each of the preliminary, initial and final releases was treated as
discrete news pieces expected to impact the market.
The analysis of individual components listed in Table 2 was carried out concurrently, and the results were merged
to form the final index. The calculations of the other components, including earnings, followed exactly the same
path. The complete analysis results will be presented in the section 4.8.
4.2 FILTERS ON THE S&P 500 INDEX
In order to smooth out random index fluctuations in the days preceding and following news releases, we used
closing averages as follows:

 Before_Close – the index value before news release, which is the average of the S&P 500 index close
for the two trading days prior to the news release;
 After_Close – the index value after news release, which is the average of the S&P 500 index close of
two days, including the actual date of the release and the following trading day. The rational for using
two days is the influence of a particular release is statistically greatest on the day of the release, while
effects of the release can be felt on the second day as well, once the ―dumb money‖ reacts to the
release.
The changes in S&P 500 Index were calculated as:
∆Index =
4.3 NEWSCAST CONTENT
For the purposes of this analysis, we used only the actual values for each news release without offering an
accompanying analytical explanation for the results. As mentioned before, the analysis included value from the
previous news release (PrevValue), the consensus estimates for the current release (ConsValue) and the actual
current release (CurrentValue).
4.4 STEP 1: CALCULATING COMPONENT CHANGES
In the first step, the component changes are calculated. For components expressed as index or percentage, a
direct subtraction was used:
ΔValue = CurrentValue - PrevValue
For components expressed in volume terms (e.g. existing home sales), a percentage changes was used:
–
ΔValue =
Similarly, the calculations of the expectations vs. actuals were done using the unit of measurement for each
component:
ΔCons_vs_Value = CurrentValue - ConsValue
–
ΔCons_vs_Value =
To ensure consistency in the movement direction, both ΔValue and ΔCons_vs_Value were selectively adjusted
by ―-1‖ to indicate the direction of the news components. For example, rising unemployment is perceived as a
negative development, and thus ΔValue and ΔCons_vs_Value were multiplied by ―-1‖, whether rising home sales
is deemed positive, thus ΔValue and ΔCons_vs_Value are taken as is. The total list of adjustments is provided in
Table 3.
Table 3: Economic Indicator Component and Adjustments for “-1”

Component Adjusted?
Building Permits No
Chicago PMI No
Durable Goods Orders No
Existing Home Sales No
GDP No
Housing Starts No
Industrial Production No
Initial Claims Yes
ISM Index No
New Home Sales No
Personal Income No
Retail Sales No
Unemployment Rate Yes
University of Michigan Consumer Sentiment Index No
4.5 STEP 2: CALCULATING INDEX SENSITIVITY
The formula used for calculating index reaction to changes in the component was:
Δ
Reaction_to_Value = (-1) *
Δ
The ―-1‖ multiplier was used for highlighting bias as a positive number. Any index and underlying component
movement in opposite directions will create a positive value signalling either positive or negative bias, whether
movements in the same direction will create a negative number indicating no bias. The larger the positive number,
the stronger the bias. For example, if unemployment rises (note: in the previous step in ΔValue obtain a negative
value) and the market falls, then multiplying their ratio by (-1), we obtain a negative value (no bias). However, if
unemployment goes up and so does the market, the value Reaction_to_Value be positive, indicating the presence
of a bias. At this point the nature of the bias (positive or negative) is not yet established.
The second formula, which runs in a parallel to the first one, is the calculation of the market reaction when
comparing news release relative to expectations:
Δ
Reaction_to_Cons = (-1) *
Δ
After completing Step 2, the market bias for each component becomes visible, identified by the positive
Reaction_to_Value and Reaction_to_Cons values.
4.6 STEP 3: MEASURING BIAS STRENGTH
Reaction_to_Value and Reaction_to_Cons show the presence and degree of bias of the market in its reaction to
news releases.

To calculate bias leaning (ValueBiasStrength) we applied the following formula:
ValueBiasStrength =
If Reaction_to_Value <0, then the bias doesn't exist, so output is 0;
If Reaction_to_Value > 0 (bias), and the index moved up (meaning that the value change was
negative) the bias is positive, then ValueBiasStrength = ΔIndex * ΔValue * (-10,000);
If Reaction_to_Value > 0 (bias), and the index moved down (meaning that the value change was
positive) the bias is negative, then the formula takes the following form: ValueBiasStrength = Δ
Index * ΔValue * 10,000.
The 10,000 multiplier is used to improve the readability of the ValueBiasStrength, as ΔIndex * ΔValue produce
small fractions under 0.001. Positive or negative sign of the 10,000 multiplier is assigned to show the leaning of the
bias. A positive value of this indicator signals positive bias (i.e. negative news are ignored or downplayed), and the
negative value indicates that positive economic news are perceived negatively (negative bias).
Similarly, the expectations bias (ConsBiasStrength) is calculated:
ConsBiasStrength =
If Reaction_to_Cons <0, then no bias, output = 0;
If Reaction_to_Cons > 0 (bias), and the index is up (meaning that release missed expectations)
the bias is positive, then ConsBiasStrength = ΔIndex * ΔCons_vs_Value * (-10,000);
If Reaction_to_Cons > 0 (bias), and the index moved down (meaning that component exceeded
market expectations) the bias is negative, then the formula takes the following form:
ConsBiasStrength = ΔIndex * ΔCons_vs_Value * 10,000.
Upon completing the Step 3, the market bias direction (positive or negative) becomes apparent, as identified by the
positive and negative ValueBiasStrength and ConsBiasStrength values.
4.7 STEP 4: FINALIZING INDIVID UAL COMPONENTS
In Step 3 the positive and negative market bias was identified. Step 4 will help to recognize bias consistency to
eliminate sporadic fluctuations that indicate hesitancy of the market participants.
To smooth out sharp oscillations for ValueBiasStrength and ConsBiasStrength a 12-period simple moving average
was applied. The 12-period interval was selected through a calibration process. Application of the moving averages
allows determining leaning of the market bias over time: it shows whether markets have been mostly positive in
their view of the world, negative or neutral. Calculations output was stored as MA_12_ValueBiasStrength and
MA_12_ConsBiasStrength.
After applying moving average, we determined standard deviation for ValueBiasStrength and ConsBiasStrength
using the same interval of 12 values. Use of standard deviation permits to gauge consistency of the market leaning:
the smaller the value for standard deviation, the more consistent the market is in its bias, while large standard
deviation value will signal more sporadic movements, indicating the lack of consistency in the market mood. The
output was recorded as STD_ValueBiasStrength and STD_ConsBiasStrength.

Having identified market leaning on the optimism-pessimism scale and consistency of the bias, it is feasible to
construct a consolidated index for each indicator. Logic suggests that the wider the standard deviation, the weaker
the trend, as values are scattered over a greater range. Therefore, adjusting bias for standard deviation will aid
identifying the magnitude of the bias. The adjustment is done using the following formulas:
ValueBias =
ConsBias =
Finally, we assumed that changes in each economic indicator value carry the same importance as market
expectations for the same indicator. For example, the fact that GDP numbers went up is just as significant as the
fact that the GPD expectations were missed. In this example, the net effect will be zero, as positive news release
will be canceled by the failure to meet or exceed market expectations. The preliminary component index value
was defined as:
Pre_Index = ValueBias + ConsBias
In some cases we observed large spikes caused by unexpected events. To reduce volatility, we applied logarithmic
scale to soften extreme values. The final index for each component was created using the formula:
Index = * (Ln(ABS(Pre_Index)+Exp(1))-1)
The following are the result for individual components that will comprise the final index (the dotted line
represents index values smoothed by a moving average):
Building Permits Chicago PMI

6 15
4 10
Lucidity Index
Lucidity Index
2 5
0 0
-2 -5
-4 -10
-6 -15
Consumer Confidence Durable Goods Orders

20 4
15 3
Lucidity Index
Lucidity Index
10 2
5 1
0
0
-1
-5 -2
-10 -3
-15 -4

Existing Home Sales GDP

4 2.5
3 2.0
2 1.5
Lucidity Index
Lucidity Index
1 1.0
0 0.5
-1 0.0
-2 -0.5
-3 -1.0
-4 -1.5
-5 -2.0
Housing Starts Industrial Production

6 1.0
4
0.5
Lucidity Index
Lucidity Index
2
0.0
0
-0.5
-2
-1.0
-4
-6 -1.5
Initial Claims ISM Index

6 15
4 10
2
Lucidity Index
Lucidity Index
5
0
0
-2
-5
-4
-6 -10
-8 -15

New Home Sales Personal Income

5 0.4
4 0.3
3
0.2
2
Lucidity Index
Lucidity Index
1 0.1
0 0.0
-1 -0.1
-2
-0.2
-3
-4 -0.3
-5 -0.4
Retail Sales Unemployment Rate

2 0.2
1 0.2
0.1
Lucidity Index
Lucidity Index
1
0.1
0
0.0
-1
-0.1
-1 -0.1
-2 -0.2
University of Michigan Consumer Earnings

20 10
Sentiment
15
5
10
Lucidity Index
Lucidity Index
5 0
0 -5
-5
-10
-10
-15 -15

4.8 STEP 5: FINALIZI NG THE INDEX
Having defined individual indexes for each component, a single index combining information from all key economic
indicators can be contracted. The limited information and volatility introduced by the earnings report dictates
creating two indexes for the purposing of selecting the most accurate sentiment indicator.
To combine separate components together we leveraged the A-F scale of economic merit assigned to each
indicator. Weights were assigned based on the indicator importance: A received weighting of 4, B of 3, C of 2 and
D of 1. Shades of the same merit, i.e. ―+― and ―-‖ signs such as A- and B+, were awarded +/-.30 points added to
the anchor weight. Through a calibration process Earnings received the importance of ―A‖. It must be noted that
despite the subjective nature of the weighting assignment done through a calibration process, different weighting
schemes did not result in significant deviations in the final function shape. It strongly suggests the validity and
consistency of evaluating the market‘s rationality using the proposed approach.
Individual component charts presented above illustrate a slight compatibility issue. Each component showed
different oscillation ranges, and using component as is would cause the one with larger swing amplitude to dwarf
the importance of other components and their contribution to the final index. To level the field we measured each
component in terms of its standard deviation, rather than actual values calculated through the steps outlined
earlier in this paper. It must be noted that for the purposes of this research standard deviations were derived from
the available data spanning over 12 years. Going forward it is imperative to establish the fixed multipliers for each
component to preserve the historical Lucidity Index value, as new data appended to the history of each indicator
will alter the standard deviation calculations. Table 4 lists the weights used for calculating the final index.
Table 4: Economic Indicators’ Weight in the Final Index
Component Importance Weight

Building Permits B- 2.7
Chicago PMI B 3
Consumer Confidence B- 2.7
Durable Goods Orders B 3
Earnings A 4
Existing Home Sales C 2
GDP B 3
Industrial Production B- 2.7
Initial Claims C+ 2.3
ISM Index A- 3.7
New Home Sales C+ 2.3
Personal Income C+ 2.3
Retail Sales A- 3.7
Unemployment Rate A 4
University of Michigan Consumer Sentiment Index B- 2.7
The final index is a combined total of weighted individual components grouped by months. Negative and positive
biases are expected to cancel each other out, giving way to the dominant trend. Over time, the trend is expected
to highlight periods of extreme optimism and pessimism.

Exhibit 1 and Exhibit 2 show the variations of Market Lucidity Index. Exhibit 1 includes the Earnings component,
while the chart on Exhibit 2 is calculated without it. The limited availability of accurate earnings data - the earnings
data for S&P 500 companies collected at the time of the research dates back to 2003 rather than 1998 as other
data series do – necessitated confirmation of the final output with a chart that has earnings component removed.
Both charts include a band of acceptable values established at one standard deviation from the mean. Periods with
the index moving above the band mark the times of extreme optimism. During these episodes market downplayed
or ignored incoming negative news, reacting positively to the situation that otherwise are deemed negative.
Similarly, deviations below the bottom band indicate periods of strong pessimism when the market went down on
positive news.
Exhibit 1: Market Lucidity Index (with the Earnings component)
Market Lucidity Index (Earnings Included)

120
100
80
60
40
Index
20
0
-20
-40
-60
-80
Exhibit 2: Market Lucidity Index (without the Earnings component)
Market Lucidity Index (Earnings Excluded)

60
40
20
Index
-20
-40
-60

5. PARTING WORDS
Financial markets, while in theory a function of rational behaviour based on unbiased calculations, in reality are
inefficient and frequently irrational. Herding behaviour and tunnel vision of market players remain the key and
inseparable components of the natural and harmonic market functioning. Quite frequently, these behavioural
distortions overshadow sensible valuations, leading to the asset bubbles and subsequent busts. The Market Lucidity
Index proposed in this paper is designed for the purpose of measuring the market‘s reaction to news, which is
indicative of the market bias. Strong bias is expected to signal potential herding behaviours and irrationally fixated
views of the investing crowd.
By itself the Market Lucidity Index carries no forecasting or explanatory value. It is merely a reflection of the
dominant market sentiment with respect to the interpretation of significant economic events. Unlike the P/E ratio,
for instance, which can reflects relative affordability or overvaluation of the markets, or purely technical indicators
like Bollinger Bands and RSI, the index proposed in this paper acts as a ―litmus test‖ of the behavioural discrepancy
between the market‘s expected reaction and its actual. In conjunction with other indicators, however, it can be
used a tool to help identify possible issues with the market from the behavioural perspective.
In the ideal world governed by the rational behaviour the index function should oscillate around the 0-mark. Small
index fluctuations will be expected due to the ―white noise‖ created by the constant stream of global
developments reflected in news releases. However, extreme index deviations from the mean would signal
potential issues that investors must pay close attention to. Let‘s review some examples of the market behaviour
embedded in the index. Exhibit 3 shows an overlay of the S&P 500 index performance and the Market Lucidity
Index. In late 1999 and early 2000 the market was dominated by the pessimism. Despite seemingly solid
macroeconomic signals, the market reacted negatively to the news releases. It could potentially indicate two
notions: the performance of the economic indicators consistently came below expectations or the market sensed
the trouble ahead and did not fully buy into the positive news.
Exhibit 3: Market Lucidity Index (without the Earnings component) and S&P 500 Performance
60 1600
1500
40
1400
1300
20
1200
0 1100
1000
-20
900
800
-40
700
-60 600
Index S&P

From the mid-2000 to early 2008, the index fluctuation range narrowed clustering around the 0-level. It strongly
suggests that during this period the market interpreted the news more or less in line with expectations, rising with
positive economic data and falling in line with negative. In March and April of 2008, the market was engulfed by the
very strong optimism, pushing the index up, only see a breath-taking plunge spanning over the next several months.
Commenting on the 2009-2010 rally, the Market Lucidity Index signals a severe disconnect of the market
performance from the underlying fundamentals. The Market Lucidity Index factors both actual changes to the
economic indicators as well as market expectations set for the news releases. The strong positive bias during that
time suggests that market was possessed by unprecedented optimism (unprecedented based on the available data)
and rallied up not just in situations when economic data showed worsening conditions, but also when economic
readings came below market expectations. The expectancy theory is repeatedly used to explain abnormal market
shifts. Too frequently, when reason and logic fail to explain a particular market reaction, the phase ―market already
priced it in‖ is used. This, however, is not applicable to the market action exhibited during the 2009-2010 period:
according to the index, the economic readings came consistently below what‘s was already priced in, as expressed
by the market consensus factored in the Market Lucidity Index.
There are many theories explaining the overly optimistic and irrational market performance in 2009-2010. Often,
the lack of true market volume substituted with the determined one-way HFT algos and FEDs fiddling on the
futures market are singled out as possible drivers behind the rally. Whatever the distortions might be they or the
true market participants pushed the stock indexes higher despite the declines in fundamentals and missed
expectations. This introduces a potentially serious issue for retail investors.
The world of irrational shifts from massive sell-off to extreme rallies is a dangerous place for retail investors to be
in. This paper does not evaluate the market fundamentals not it provides any views on the immediate forecasts for
various markets. We, however, wanted to caution 401K and RRSP holders to stay cautious and conservative. Just
like an emotionally-imbalanced person experiencing sharp mood swings from deepest depression to maniacal
laughs cannot be trusted with helping post-traumatic stress disorder victims; this unpredictable equity market
cannot be trusted with your retirement savings. As the hollow and vigorous optimism replaced the tremendous
market fear, it can easily slide back to the fear again if the public suddenly discovers that it has no ground to stand
on. Until the equity market regains its emotional balance backed by fundamentals, it should be avoided. After all, as
Joe Saluzzi of Themis Trading puts it, optimism (which was the driving force behind the rally) is not a strategy.
Certainly, an argument can be made in defence of the market as the leading economic indicator. However, the
leading nature of the market spawns from its ability to read fine signals of improvement or deterioration of the
economic situation. What is commonly felt on the intuitive level by experienced market professionals, the Market
Lucidity Index identifies in quantitative terms. This is the reason for the 2003-2007 market rally not being marked
by any significant changes in the Market Lucidity Index. During that period the market merely reacted to the fine
signals generated by the underlying fundamentals, and index fluctuated around the 0-mark. On the contrary, the
2009-2010 period is associated with widespread optimism. The index suggests that the fine signals, or ―green
shoots‖ of a typical recovery, did not exist to fuel the rally the way it progressed. It is true that at the initial stages
the recovery was driven by the extremely oversold conditions that developed by March of 2009. However, after
the initial bounce, the rally appears to be supported by optimism only.
If history carries any predictive value, the similar sharp declines and tremendous rebounds of the market sentiment
(business confidence) were observed in 1954-1955, 1958-1959, 1971-1972 and 1982-1983. In all cases it led to the
corrections with the strong optimism of 1971-1972 giving way to the pessimism of the 1973-1975 that brought the
stock indexes down below their pre-rally levels. It is our view that investors should maintain minimal exposure to
equities until the market reaches balance in emotional and fundamental terms.

If you are interested in receiving further information regarding this research or have any questions for its authors,
please direct your inquiries to alec.pestov@yahoo.com. Thank you for reading.

ACKNOWLEDGEMENTS
We are grateful to Jeff Stuart for his editing assistance.
Jeff Stuart is the President of King James Capital, a private corporation that provides comprehensive services to
publicly listed companies in a professional and cost effective manner. Services include corporate finance, investor
relations and corporate communications. In addition, Jeff provides a range of editing, sales and marketing services
for select newsletter writers in the mining sector. A frequent speaker at mining investment conferences, Jeff‘s
years of experience as an insider and background as a licensed investment advisor provide him with unique and
balanced insights into the successful development of junior resource companies. More information on Jeff‘s
business development process can be found at kingjamescapital.com

Measuring The Immeasurable: Market Lucidity Index

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Measuring The Immeasurable: Market Lucidity Index

Uploaded by

Copyright:

Available Formats

Alexandre Pestov Eugene Bragin May 2010

Alexandre Pestov Eugene Bragin

Measuring the Immeasurable: Market Lucidity Index

Measuring the Immeasurable: Market Lucidity Index Page 0 of 51

Measuring the Immeasurable: Market Lucidity Index Page 1 of 22

2. About The Project ............................................................................................................................................................................. 6

3.2 Assumptions and General Approach ..................................................................................................................................... 8

4.1 Defining Relevant Economic Indicators ...............................................................................................................................10

4.2 Filters on the S&P 500 Index..................................................................................................................................................10

4.3 Newscast Content ....................................................................................................................................................................11

4.4 Step 1: Calculating Component Changes............................................................................................................................11

4.5 Step 2: Calculating Index Sensitivity .....................................................................................................................................12

4.6 Step 3: Measuring Bias Strength ............................................................................................................................................12

4.7 Step 4: Finalizing Individual Components ............................................................................................................................13

4.8 Step 5: Finalizing The Index ....................................................................................................................................................17

5. Parting Words ...................................................................................................................................................................................19

Measuring the Immeasurable: Market Lucidity Index Page 2 of 22

An optimist believes we live in the best of all possible worlds.

Measuring the Immeasurable: Market Lucidity Index Page 3 of 22

It's BULLISH No Matter What! ...

But didn't wages go down too? - BULLISH! Revenues - Costs = Profits!

Gold is going up in price - BULLISH! This is a broad-based rally.

Measuring the Immeasurable: Market Lucidity Index Page 4 of 22

The Euro is getting weaker - BULLISH! King dollar is back.

Actual market movements can be evaluated in their correlation to the

Measuring the Immeasurable: Market Lucidity Index Page 5 of 22

2. ABOUT THE PROJECT

Measuring the Immeasurable: Market Lucidity Index Page 6 of 22

Table 1: Key Economic Indicators

Measuring the Immeasurable: Market Lucidity Index Page 7 of 22

 Actual value of the component as presented in the newscast;

 Previous value of the same component;

 Estimates or expected value of the component (market consensus).

3.2 ASSUMPTIONS AND GENERAL APPROACH

Measuring the Immeasurable: Market Lucidity Index Page 8 of 22

 6 June 2008: -1.8%

Measuring the Immeasurable: Market Lucidity Index Page 9 of 22

4.1 DEFINING RELEVANT EC ONOMIC INDICATORS

Table 2: Economic Indicators Exhibiting Correlation with S&P 500 Index

4.2 FILTERS ON THE S&P 500 INDEX

Measuring the Immeasurable: Market Lucidity Index Page 10 of 22

The changes in S&P 500 Index were calculated as:

4.3 NEWSCAST CONTENT

4.4 STEP 1: CALCULATING COMPONENT CHANGES

ΔValue = CurrentValue - PrevValue

ΔCons_vs_Value = CurrentValue - ConsValue

Table 3: Economic Indicator Component and Adjustments for “-1”

Measuring the Immeasurable: Market Lucidity Index Page 11 of 22

4.5 STEP 2: CALCULATING INDEX SENSITIVITY

4.6 STEP 3: MEASURING BIAS STRENGTH

Measuring the Immeasurable: Market Lucidity Index Page 12 of 22

To calculate bias leaning (ValueBiasStrength) we applied the following formula:

If Reaction_to_Value <0, then the bias doesn't exist, so output is 0;

Similarly, the expectations bias (ConsBiasStrength) is calculated:

If Reaction_to_Cons <0, then no bias, output = 0;

4.7 STEP 4: FINALIZING INDIVID UAL COMPONENTS

Measuring the Immeasurable: Market Lucidity Index Page 13 of 22

Building Permits Chicago PMI

Consumer Confidence Durable Goods Orders

Measuring the Immeasurable: Market Lucidity Index Page 14 of 22

Existing Home Sales GDP

Housing Starts Industrial Production

Initial Claims ISM Index