Statistics is a tricky subject. To some, it seems like the ultimate in waffling – why can’t you just say what the right answer is after all? To others, it’s the ultimate in risk mitigation – but what if these 15 things were to happen. With the recent election forecasting, renewed debate over climate change data, economic modeling and any number of difficult topics, statistics has been taking a very public beating.
Here’s the thing about statistics and data modeling: it’s simply a tool. Like a gun, it can be used for good, evil or any range of grey areas in between. Like people holding guns, some know to use the safety while others wave it loaded around in the air like it’s a toy. The problem with statistics is you typically only see the outcome of a process and not what went into getting it. Essentially you get the target with holes in it but don’t see how the shooter put them there.
To carry the analogy forward, sometimes shooters get lucky and cluster 3 shots right in the middle of a target even though they would never be able to do it again. Sometimes a previously used target is placed up and the shooter only pretends to have accomplished the feat. Sometimes you have a skilled shooter that actually knows what they are doing but has a bad day and misses. Things happen.
Judging performance of statistics in hindsight is useful to be able to understand the tolerances and error associated with a given model. But even then, past performance is not always an indicator of future performance. It’s important to understand the role that statistics and data modeling should play in your decision making process but throwing them out completely is taking a very powerful tool (often perfect for the task) and choosing to never use it because it wasn’t used properly by amateurs in the past or a customer was unhappy even though the product was technically correct.
All tools that you used should be evaluated regularly to understand their best roles and uses, where they work and where they don’t, what can go right and what can go wrong. Statistics is powerful but with that power comes inherent risks that you should always keep in mind.
Trust but verify. It’s even true with math.
Great post and great analogy you’ve used there! I’ve seen it said recently that we don’t have a lack of information anymore, we have a lack of understanding/interpreting that information. Statistics are one the things thrown up to hit home a point, but as you said, we don’t know how those statistics were made. People would be wise to question things, but not to dismiss things entirely.