by Jeff Miller, A Dash of Insight
[The following is a work of fiction. It is intended as educational, illustrating why some research methods look great but have poor results. Those who grasp the problems illustrated can figure out where to apply the conclusions. It is also intended to be fun!]
The setting: The research lab of a well-known fund company.
The participants: Dr. B (the boss), Dr. Z (the research director), Mr. S (a staff member), and the Rookie (well-educated, but new to the team).
B: I need some fresh material. How about a new syndrome?
S: But we have so many already…..
Z: People love to read about new syndromes. Our regular articles top the lists in popularity.
Rookie: What’s a syndrome?
Z: That is where we show why the current market conditions are strongly tied to a market crash, ten years of pestilence, an imminent recession, or something equally bad.
Rookie: If we have created these before, why do we need a new one?
Z: Some of the former predictions did not work out.
Rookie: Why not?
Z: The standard reasons. The Fed and other central banks flooded the market with liquidity.
Rookie: I read that most of the Fed expansion stayed on bank balance sheets. Hasn’t the economy gotten better?
Z: Let’s focus on syndromes. We explain past performance in terms that everyone will accept. They all hate the Fed. That is our playbook. And kid – it is OK to ask questions, but keep an open mind. Focus on learning our system.
Rookie: OK, how do we discover a syndrome?
S: We have an established method. We look for a bad former period and ask what that time had in common with current conditions.
Z: Any two time periods share many characteristics. If the fit is not as good as we want, we can do some tweaking?
Rookie: What do you mean by tweaking?
Z: We might need to specify that a variable has a specific value before the effect takes place. Or that two elements occur at the same time.
Rookie: There are not very many recessions and market crashes. If you do too much of this tweaking, don’t you risk over-fitting the – er – syndrome? One of my classes included something about “degrees of freedom” and not using too many variables.
S: That is the beauty of our method. Since we use all of the data on every test, no one can prove that we are wrong. There is no evidence to provide refutation.
Rookie: Don’t we keep some out-of-sample data as verification? That was recommended in one of my classes.
Z: Wasting data that way would not give us enough cases to prove the point. There are too few relevant business cycles already.
B: Enough of the basic education. The kid can learn more as we go along. I want to call the new syndrome Grandma Gertrude. It will show that the current market rally is at extremes of valuation, stretched in time, and indicating the most dangerous conditions except for the last two market crashes.
S: Why do we always name the syndrome after a female relative. Shouldn’t we be like the hurricane center? Mix in a few guys’ names.
B: You need to learn about symbolism. Everyone loves female relatives and feels protective. We sympathize with their frailties and worry about them. Who would care about a market syndrome called “Uncle Harold?”
Z: OK, we’ll get started. I assume that we are starting with “old reliable?”
B: Absolutely! The Shiller CAPE ratio always confirms bad times and has earned tremendous credibility. It is the foundation of every syndrome.
Rookie: I read that Dr. Shiller does not use it for market timing – just for choosing sectors.
B: No one knows that, so who cares?
Z: We can mix in some other variables that show recent weakness, but none of them indicate a recession by themselves.
B: No problem. That is why we have a syndrome. We can explain that the effects occur only when several things happen at the same time. Then we can use the magic words….
Z: You mean “ever and always?”
B: Yes! We want to say that whenever the syndrome has occurred disaster has come as well. It is a powerful statement.
Rookie: In one of my classes we learned that you were supposed to begin with a hypothesis and then see whether the data supported it.
Z: We already know what is going to happen. We are just looking for evidence for our readers and investors.
Rookie: I am curious. Suppose we were to reverse the process. What if we took the very best times to invest – lowest risk or something – and looked for variables correlated to current times? Couldn’t we prove the exact opposite of the new syndrome?
B: Kid, you ask too many questions. If you want to work here, you need to get with the program.
