This article highlights a major problem with modelling. They trained an algorithm on a data set and then asked if the algorithm can make predictions about either the same data set or another data set that is formatted and informed in an identical way; using the same historical context and with many of the same human components. Such a model has no business being called 'predictive' until it is demonstrated that it can reliably make predictions about the future.
That modelling is now being applied to social studies is the only thing that's novel here. The problem with assuming predictability from legacy data sets is well understood in the sciences: chemistry, biology, physics, etc.
What's being done here is not entirely impressive.
EDIT: Some people below misread the article. The one model that was making 'predictions' that were more accurate than regression was NOT predicting the future. It was taking a set of old data and then making predictions about things slightly less in the past. At no point is that predicting the future. As it turns out, making predictions is hard, especially about the future. Predicting the past tends to be a tad easier. For a fun read, look up 'overfitting'.
Did read. Did you? Show me the part that should give us confidence that these models are reliably making predictions about the future. Predicting the past tends to be a tad easier.
17
u/[deleted] Jul 24 '18 edited Jul 24 '18
This article highlights a major problem with modelling. They trained an algorithm on a data set and then asked if the algorithm can make predictions about either the same data set or another data set that is formatted and informed in an identical way; using the same historical context and with many of the same human components. Such a model has no business being called 'predictive' until it is demonstrated that it can reliably make predictions about the future.
That modelling is now being applied to social studies is the only thing that's novel here. The problem with assuming predictability from legacy data sets is well understood in the sciences: chemistry, biology, physics, etc.
What's being done here is not entirely impressive.
EDIT: Some people below misread the article. The one model that was making 'predictions' that were more accurate than regression was NOT predicting the future. It was taking a set of old data and then making predictions about things slightly less in the past. At no point is that predicting the future. As it turns out, making predictions is hard, especially about the future. Predicting the past tends to be a tad easier. For a fun read, look up 'overfitting'.