The most common problem is that all these new systems — metrics, algorithms, automated decisionmaking processes — result in humans gaming the system in rational but often unpredictable ways. Sociologist Donald T. Campbell noted this dynamic back in the ’70s, when he articulated what’s come to be known as Campbell’s law: “The more any quantitative social indicator is used for social decision-making,” he wrote, “the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
On a managerial level, once the quants come into an industry and disrupt it, they often don’t know when to stop. They tend not to have decades of institutional knowledge about the field in which they have found themselves. And once they’re empowered, quants tend to create systems that favor something pretty close to cheating. As soon as managers pick a numerical metric as a way to measure whether they’re achieving their desired outcome, everybody starts maximizing that metric rather than doing the rest of their job — just as Campbell’s law predicts.
Felix Salmon, “Why Quants Don’t Know Everything”, Wired, 2014-01-14
February 21, 2015
QotD: Campbell’s Law
Comments Off on QotD: Campbell’s Law
No Comments
No comments yet.
RSS feed for comments on this post.
Sorry, the comment form is closed at this time.
