DOH! Stupid Statistical Paradoxes anyway!

A Late Carnival Arrival

This is a guest post from Jeremy Shapiro aka HRMetricsguy, from Hodes iQ.  He initially submitted to run as part of the Carnival, but it was way past the deadline, and I had some guest post opportunities coming up this week, so I am running it today.  I hope you enjoy it!

I enjoy reading “The Numbers Guy” column in the WSJ, but today, guest writer Cari Tuna really hit it out of the park with her piece on a problem in statistical analysis called Simpson’s Paradox. Cari reports that the paradox is under-stating the severity of unemployment data.

A few weeks ago, I wrote a piece of three ways to look at the DOL employment data. Now, a fourth view emerges because of Simpson’s Paradox.

First, what is the paradox? Here’s Wikipedia’s version:

Simpson’s paradox (or the Yule-Simpson effect) is an apparent paradox in which the successes of groups seem reversed when the groups are combined. This result is often encountered in social and medical science statistics, and occurs when frequency data are hastily given causal interpretation; the paradox disappears when causal relations are derived systematically, through formal analysis.

Or, more simply (my version): Sometimes, because one subgroup of data can be much larger than another group, the total average looks better (or worse) than what’s really going on.

For unemployment, what Cari reported is that for college graduates and high school graduates, employment is really worse than the 80′s, but the total average doesn’t reflect this reality on the ground. Consider this:

Total Unemployment 25 and Older
1983: 8.5%
2009: 8.2%

For College Grads
1983: 3.6%
2009: 4.9%

For High School Dropouts:
1983: 13.6%
2009: 14.9%

And there’s the paradox at work, both College grads and High School dropouts are worse off, but the total unemployment data does not reflect the reality. Why? There are more college grads today (1/3rd of the working population) vs. 1983 (1/4 of the working population), which skews comparisons between the 2 recessions!

The WSJ article is good reading. It should be noted that the analysis was done by Henry Farber , an economist from Princeton found here.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Jeremy Shapiro, Senior Vice President, Hodes iQ

email           jshapiro@hodesiq.com

web             www.hodesiq.com or www.smartpost.com

Twitter        hrmetricsguy

LinkedIn    www.linkedin.com/in/jeremyashapiro

Snail Mail   220 E. 42nd Street, 14th floor New York,  NY 10017

Phone         212 999 9722

Join our LinkedIn Group for Hodes iQ Users.

http://www.linkedin.com/groupRegistration?gid=52788

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

Tagged as: High School, Jeremy Shapiro, LinkedIn, New York, New York City, Statistics, Wikipedia