Analytics and Life

Human beings bestowed by the intelligence, sometimes tend to overuse it. We overestimate causality to that extent that we sometimes tend to fit patterns or try to find reason for something that is complete random. The world is so overwhelmed with data and data analytics tools that usually we succeed in finding the reasons for random event as well. Due to advent of technology and enormous computing power at our disposal, we can search pattern even in the complete random event. It seems like that humans are so arrogant of their reason that they are not able to accept that there might be some event that may have no reason at all. This is something call as fooled by randomness these days. The real challenge is not to be blinded by Socratian Logic. I guess Logic is bit overrated.

We in data analytics call it ‘noise’. One of the major challenges of modern day data analytics is separation of information from the noise. For any robust model data sanity is of paramount importance. If the data selected for the modeling is wrong the model is bound to fail. Usually analyst defines the scope of the project that how much the model can predict and for what time horizon. There we tend to be very cautious and predicts with disclaimer. The funny thing is in business problem we are very aware that we are doing prediction based on historic data, which might go wrong if data pattern changes or say situation changes.

But in real life we are so confident of ourselves that we do prediction with conviction that too on the basis of insufficient data/experiences. We are somehow so blinded or arrogant that we extrapolate our prediction on entirely different situations. We don’t bother to see whether our experiences are even worth using or not .In technical term we don’t care that the sample being used for analysis is information or noise.

The whole point is to understand that everything in universe needs not to have reason behind it. Something might be accidental but yet have profound impact on our lives. I am not saying that we cannot predict at all. Of course, we can but with some probability of failure as well and under the constrain that situation will remain similar to what we have experienced in past. I guess this much humbleness is not an unjustified expectation from the predictors of humanity.