On Complexity, ‘fog and moonlight’, prediction, and politics VII: why social science is so bad at prediction & what is to be done

Introduction

‘Ideas thus made up of several simple ones put together, I call Complex; such as are Beauty, Gratitude, a Man, an Army, the Universe.’ Locke.

‘I can calculate the motion of heavenly bodies but not the madness of people.’ Newton, after the South Sea Bubble ‘Ponzi scheme’. 

‘Everything in war is very simple, but the simplest thing is difficult. The difficulties accumulate and end by producing a kind of friction that is inconceivable unless one has experienced war… Countless minor incidents – the kind you can never really foresee – combine to lower the general level of performance, so that one always falls short of the intended goal.  Iron will-power can overcome this friction … but of course it wears down the machine as well… Friction is the only concept that … corresponds to the factors that distinguish real war from war on paper.  The … army and everything else related to it is basically very simple and therefore seems easy to manage. But … each part is composed of individuals, every one of whom retains his potential of friction… This tremendous friction … is everywhere in contact with chance, and brings about effects that cannot be measured… Friction … is the force that makes the apparently easy so difficult… Finally … all action takes place … in a kind of twilight, which like fog or moonlight, often tends to make things seem grotesque and larger than they really are.  Whatever is hidden from full view in this feeble light has to be guessed at by talent, or simply left to chance.’ Clausewitz.

‘It is a wonderful feeling to recognise the unity of complex phenomena that to direct observation appear to be quite separate things.’ Einstein to Grossman, 1901.

‘All stable processes we shall predict. All unstable processes we shall control.’  Von Neumann.

‘Imagine how much harder physics would be if electrons had feelings.’ Richard Feynman.

---

[Sorry for lack of posts, been ill… Makes sense to do US update after midterm results next Tuesday… Will also look at Sunak regime ASAP… I’ll do a AMA 1900, 21/11.]

Previous blogs in this series covered: general problems of how people in politics cope with nonlinear systems that are inherently impossible to predict in detail for long, the history of computing, von Neumann and the use of maths for economics (see HERE).

Below is a summary of two very interesting pieces, one in Nature and one in Science, by a physicist turned computational social scientist, Duncan Watts. He considers why social science is so dysfunctional, why it doesn’t even pay proper attention to the conflicts between key theories, why SS theories rise and fall more like fiction and fashion than like physics theories, the incentive problems of academia, and what could be done.

One of the many curious things about politics and government is how amazingly little effort is made to think about how improvements in prediction create huge value. Perhaps the most obvious political example is the dire standards of politicians for polling and election models where they fail to focus effort even on a problem they are apparently highly incentivised to focus on. Remember CCHQ, CTF and pundits across SW1 mocking MRP in 2017 as some absurd ‘new-fangled’ gizmo?! (Part of the explanation for the mystery, apart from the low calibre of people, is that the roughly 4-5 year feedback scale of elections is too long to be a good incentive to focus politicians rationally.)

An obvious government example was predicting cases, hospitalisations, intensive care beds and so on in 2020. Pre-covid the core institutions at the apex of the British state in Downing Street / HMT / 70 Whitehall did not even have basic modern technical infrastructure to work on such problems. No10 did not even have a secure cloud system for file sharing when we arrived, everything was done by emailing attachments like it was 1999. Obviously there was nothing like a data science team to provide modern infrastructure, never mind apply state-of-the-art tools to practical prediction problems. (A reason we organised the polling on covid (to provide data on symptoms, footfall etc for SAGE) the way we did was to use the cloud dashboards we’d built for the election campaign weeks earlier — not political data but the infrastructure which No10/70WH did not have — so we could share information across Whitehall using modern tools. Before this, polls were shared by emailing PDFs which remains the standard method in Westminster. This obviously sensible simple innovation was attacked as ‘illegal’ and millions wasted on court cases, in keeping with SW1’s general response to 2020.)

A fortiori, such a system could not work properly on a problem like covid. We started changing this before covid. Work accelerated very rapidly March-April when a mix of officials (e.g Nin Pandjit at NHS) and outside experts (e.g Faculty AI) built state-of-the-art systems including dashboards for senior decision-makers that transformed the situation. Part of what these tools did was provide predictions of critical data so that problems like shortages could be easily seen and quickly fixed, with predictions constantly tested against reality. Covid persuaded almost everyone of the importance of the new No10 data science team and the value of applying modern tools to practical prediction problems. Both Cabinet Secretaries, Sedwill and Case, supported the shift. Yet Boris-Normal-Version 2021-22 lost interest in all this as he switched back to ‘government as steering by daily newspaper babble’ then Truss, as part of her spectacular Day 1 machine-gun-my-own-feet, removed the data science team from the PM’s office, thus undermining her own power and ensuring worse decisions from future PMs, unless this is reversed. (NB. People, ideas, machines — in that order!)

Watts’ arguments should be influential in social science departments but I assume they won’t be short-term because of exactly the incentives problems he describes. Academic economists, sociologists, ‘political scientists’ etc are incentivised to keep doing what they do, to focus on papers as the key metric, to ignore the methodological problems and conflicting theories, to underrate building reliable tools to solve practical prediction problems and so on. And of course academia has inherent problems in competing with technology companies for key people and in building technology.

Watts’ papers should also be considered by what was the No10 data science team, if it has not been entirely vandalised by the Tories, and by other parts of government considering which social science research to fund and not fund, including UKRI and ARIA.

They also suggest ideas for practical political projects I’ll touch on in the future.

Fields such as market research are about to be upended — the process has already started though it is entirely invisible if you depend for political news on the old media. This process will not be mainly driven by academia or by >99% of existing marketing and market research companies. It will be driven by organisations building new tools to solve practical prediction problems. From this process will come new theoretical insights and there will be fast feedback between new theories <> new tools.

Ps. Watts is also one of the experts who, after Trump’s 2016 victory, tried to explain that most of what media like the New York Times said about ‘fake news’ and social media was false. So of course he was ignored. The NYT, Guardian et al had no interest in the facts — particularly given the NYT’s coverage of Hillary’s emails was orders of magnitude more important than ‘Twitter-bots’. An interesting aspect of how the post-graduate classes themselves were conned en masse by misinformation about misinformation on Putin/Trump/Brexit is visible all around us today. Part of why the Left, in the form of AOC and Bernie Sanders, argue as they do on Ukraine is because they see the war against Putin as part of the war against Trump. It’s tragi-comic that such a large fraction of our post-graduate elite Idiocracy that tells itself over dinner parties that ‘we have a big problem with non-college whites falling for misinformation’ are themselves the biggest suckers for misinformation! But as I’ve remarked many times, this has usually been the case historically. It wasn’t the deplorables who fell for Stalin’s propaganda over the Ukraine famine and the Great Terror in the 1930s, it was the NYT and graduate-elites. Thucydides wrote about the same thing in the context of civil wars in Greece, in one of his most striking and immortal passages (Bk 3).

Pps. Please don’t comment on the lines of ‘please don’t put these ideas on the internet, your political enemies will read them!’ You couldn’t force such new ideas on SW1 if you put a gun to their heads. In 2019 I wrote about how COBR would fail in the next crisis like a pandemic and how state-of-the-art technology could improve crisis management. Not only was it ignored at the time, it has been ignored even after covid demonstrated on a grand scale all the arguments and showed how to do it much better. Is there even one team of five people in all British politics/government — outside Vote Leave — working on bret Victor’s ideas? Per above, No10 has even removed the data science team that helped transform performance in 2020 — in keeping with other vandalism such as ending sewage monitoring for bio-threats, dismantling the vaccine taskforce and so on.

It’s impossible to exaggerate how bad the UK political world is at thinking about these things and how hard it is for those officials who do understand these things to change how government works, given Whitehall’s incentives. I could post a detailed description of how edge-of-the-art tools will be used to improve election models and polling 2023-4 and I guarantee the old media would either ignore it or laugh. Don’t believe me? Look at how the same characters — characters who’ve never built anything in their lives — spend all day on Twitter giving very confident management assessments on the guy who built SpaceX…! There is no end to Pundit World’s overconfident ignorance.

Ppps. My blog on von Neumann is relevant to the issues below but I strongly encourage people not to read it but instead to read the introduction von Neumann wrote to his classic Theory of Games and Economic Behavior, written while working on the Manhattan Project and other wartime projects. It gives a remarkable explanation of the role of maths in physics and other natural sciences and describes how it can be, and should not be, used in economics. Like much of vN’s writing, e.g on nuclear war and existential risks, it’s as relevant today as it was 70 years ago. Much of what very senior academics say today about maths and social science is shown to be wrong by von Neumann. Everyone doing a social science PhD should have to read this introduction.

(NB. many media articles were written in 2019-20 claiming that ‘Cummings is using game theory for the Brexit negotiations’. Some articles had elaborate theories of what I was doing (‘nuclear madman!’) and ‘why it’s so stupid’. All such articles were nonsense. ‘Game theory’ played no part in calculations over Brexit negotiations, by me or any other senior person. I think my blog on vN was partly the source of such media fairy tales, plus pundit desire to believe I was operating with some mad theory.)