Ben Whitelaw

Newsrooms need more data if they are to become profitable again. But what will editors do when the machines take over?

Picture the scene. A question flashes up on a news editor’s smartphone: “What story do you want to tell today?” They are at home – getting up to speed on the morning’s news while making their small child’s breakfast – but they could be anywhere. Since Covid-19, newsrooms have been operating remotely, thrown into 21st century working practices by the pandemic. Some miss the buzz of the office, but most don’t.

Our editor thinks for a moment and then responds to the prompt with a list line and a short summary before proceeding to answer a dozen questions about the story. What type of article is this? What is the tone of the article? What is the headline type? Once, these details were discussed in a phone call with a reporter or in a hastily scribbled email. Now they are dropdowns in the system, baked into the structured commissioning of every story. It was annoying at first, but became less tedious when it became clear that this metadata was crucial to understanding which stories pulled in paying readers.

On the next screen, a series of red dots – the newsroom has come to call them flags – appears like unfilled form fields in an online questionnaire. These are powered by machine learning algorithms that can compare the performance of past stories in real time to the soon-to-be commissioned story. The dots show that a few of the characteristics our editor has inputted didn’t work very well in the past for the topic in question. The flags are just recommendations – helpful hints that can be easily dismissed – but our editor tweaks the brief and moves on.

Next, a list of staffers and freelancers appears, just as some cornflakes fly out of a bowl nearby. This list is based on the beat that the journalists cover, but also the geography of the story and their availability (the rota is baked into the system so being bugged on your day off is less common than before). The editor hits one, skips the option to add notes (these are also captured in the system for analysis later) and suggests a deadline and wordcount. Seconds later, the reporter in question gets a notification, accepts the job and gets to work.

To many editors, this is still a scene some dystopian fiction. The idea of artificial intelligence informing what stories are covered or algorithms recommending the tone that a reporter should strive for makes traditional types come out in a cold sweat. Most would side with Richard Waters, the Financial Times west coast editor, when he said in 2018 – surrounded by Silicon Valley start-ups and apps of all shapes and sizes – that it “is in danger of being overrated”. Nothing, they say, trumps an editor’s instinct.

Overrated or not, it is here. All over the world, journalists are working in tandem with machine learning and data processing to produce journalism for the 21st century news consumer. Categorising photos, tagging names and places in articles, and profiling people to serve subscription offers to: these things happen every day in most newsrooms, with the help of artificial intelligence, or AI.

Last year, JournalismAI, a Google News Initiative-funded project at Polis, the journalism think tank at the London School of Economics and Political Science, surveyed 71 news outlets from 32 countries and found half used, or were about to use, AI in editorial and half planned to.

So far, algorithms have tended to fall into two categories: either they have sought to refine parts of the well-worn editorial process – for example, The Guardian’s Typerighter subbing bot – or, at Reuters, The Washington Post and countless others, they have been used to produce content wholesale, using raw data to write sports reports and crime stories without human intervention.

The role played by editors – commissioning stories, editing copy, and managing the placement of those stories – has mostly been untouched by AI. In many newsrooms, editors – chained to their desks, dedicated to the flow of stories, always working towards a deadline – have been less exposed to the potential of this technological innovation. If you ask them about it, they shrug, point to one of their more digitally minded colleagues and say it’s not for them. They won’t be able to do that much longer.

Over the past decade, the process and personnel involved in commissioning stories have changed. Data journalists, visual specialists and multimedia experts are augmenting editors’ ideas for screens, while those with expertise in tracking audiences online use data from search engines and social media to find new readers or listeners. Teams of product experts, data scientists and marketing professionals help test alternative subscription offers and optimise page load times to get more people to the stories. You could call it the optimisation of everything before, and after, the part done by the editors.

David Caswell, executive product manager at BBC News Labs and founder of the computational journalism project Structured Stories, calls these changes “ordinary innovation”: a business’s typical response to shifts in technology, competition and user behaviour. The process is important – news outlets should clearly seek to tell better stories. But it is not enough in the face of a host of threats – the rise of dis and misinformation, huge competition for readers’ attention, and the collapse of the digital advertising market to name three – that journalism as an industry faces. These require, he says, “existential innovation”.

The coronavirus pandemic has added to that need to be bold and think big. According to a recent report by the Reuters Institute for the Study of Journalism (RISJ), more than three-quarters of respondents from 43 countries said Covid-19 has accelerated plans for digital transformation.

More and more outlets are moving swiftly to subscription and membership models, which place fresh importance on journalism’s main product; its stories. As soon as the reader revenue switch is flicked, an organisation’s foundations – the beats it covers, how it covers them, the types of stories it writes and the style and tone it adopts – suddenly become ripe for questioning. Which means that the role and the effect of an editor come under the spotlight too.

In Stockholm, editors at the public broadcaster Swedish Radio (SR) perform a task not dissimilar to the scene set out at the start of this article. Every story filed by a reporter is rated on the desk for three attributes: its magnitude, its public service value, and its lifespan. The primary job of editors is not to commission, but to review.

Each story receives a score that is used to determine where it is placed on Swedish Radio’s app and websites. Stories with a longer lifespan or a higher public service score get more visibility. This means editors no longer manually curate their digital products. This, according to Olle Zachrison, SR’s head of digital news development, means editors have more time for “strengthening the storytelling, improving the headlines or finding better related content”.

The Wall Street Journal is also rethinking how editors work. In a 142-page report leaked to Buzzfeed News last year, its newly assembled “digital experience and strategy” team made a case for reassessing “the very way we tell stories, the topics we cover, the voices we feature and the messages we send about who we are”. The report laid out an argument for “focus(ing) more on the groups of people who read us but who aren’t our ‘heavy’ readers” and recommended new beats, a focus on listening to readers and a smarter use of data for commissioning stories – clear examples of Caswell’s “ordinary innovation”.

But it also went as far as to question the role that editors play, remarking that “we are coming out of an era in journalism where editors decided everything that was the news”. For some, this is akin to blasphemy. But, according to Buzzfeed News, it amounted to a “come-to-Jesus” moment for some in the business-focused daily. And a clear example of the existential thinking that the industry needs.

Will the Netflix model work for traditional media?

We’ve seen this kind of self-reckoning before, in a media company whose name is now famous. Back in 2006, Netflix was a US DVD business that asked users to rate movies out of five stars and tried to predict what they would like to be sent in the post, based on their review. It was successful to a point, but then launched a streaming service that gave it access to more data with which to predict what people would like to watch. So it did something crazy: it tried essentially to reverse-engineer every movie. Todd Yellin, Netflix’s vice-president of product and the man who came up with the idea, said his goal was to “tear apart content”.

What followed would probably enrage any Hollywood director or producer who believed in their craft. Netflix paid large teams of people to watch films and tag them with all kinds of metadata: lead characters’ personality traits, plot lines, locations, ratings for goriness, romance levels, sexual suggestiveness, and more. The process was wildly successful; not only did it yield the famous recommendation system that makes Netflix so addictive, but its data model gave the company a way of spotting trends and commissioning original content that its userbase would enjoy.

Having recently passed 200million subscribers, with $25billion of  revenue in 2020 alone, there’s a case to make that the “tearing apart of content” was pivotal to its success. Which raises the question: how fast can news organisations do the same?

That’s not easy to answer, for most traditional newspapers and broadcasters continue to work with a mass-media mindset. It doesn’t help that many fail to pay attention to newsroom working culture, which can make large scale changes like AI adoption difficult and demanding.

But time is running out. We are into the “existential innovation” phase, which raises tough questions: what are editors for? What does their job fundamentally entail? How can we arm them with data and insight to make the stories they commission have an impact on the reader and thus the business? As the automated world gains pace, the answers might not be as scary as they think.