To shed light on how large amounts of evergreens can be successfully leveraged, we talked with travel startup Culture Trip. The business is based on a vast catalogue of articles about places to visit around the world, categorised in a taxonomy that makes it easy for travellers to find the content that is relevant for them. 

The entire site is built on evergreen content, in the sense that it prioritises articles of highly-relevant value in a given context. Culture Trip articles are not news, but they offer great information in a very specific context. The taxonomy is a very well-defined set of tags that indicate the main features and geographic location of the article.

Articles are indexed through auto-tagging. Culture Trip has indexed around 80,000 articles using a machine learning algorithm called StarSpace, developed by Facebook Research. StarSpace is an algorithm that creates text embeddings but also includes the extension to create tag embeddings simultaneously in the same vector space. A text embedding is an alternative machine-readable representation of a text that aims to describe its semantic properties in an abstract way. According to Culture Trip, this method is very reliable, with 90% precision, but it also requires a very strict taxonomy and a carefully trained algorithm.

But even evergreen content doesn't really stay green forever. Deep links expire, details get outdated and pictures get old. To be able to still use that content effectively, it's important to have a method to maintain it. This is especially true for a business like Culture Trip, that aims to guide users around the world.

Culture Trip put significant effort into identifying the lifecycle of a piece of content, which helps them to understand and handle the biology of the content. Certain stages of the lifecycle call for a review, while others might suggest when it's time to "retire" the content. They also realised further advantages of updating an article: the accuracy of the content they produced is crucial for their business model but an updated timestamp also triggers better performance in search engines.

To manage to keep 80,000 articles constantly up-to-date, Culture Trip once again made use of technology, and established what they call a “Maintenance Machine”. It is basically a system that monitors a ranked set of parameters in the article database. That includes dead links, low-resolution images, or outdated information, for example about whether a business is closed – information that is available from the Google Places API. The Maintenance Machine displays a list of articles that need review, ordered by how urgent that same review is.

The approach of Culture Trip offers interesting insights for newsrooms, in spite of the clear differences in the objectives of a news organisation. In the context of news, taxonomies and tagging structures are more complex and less strict, and the definition of evergreen content varies a lot. However, the kind of content that Culture Trip offers is often relatable to at least some of the content types in the news business, such as guides, reviews, and recommendations. Newsrooms can surely learn from the Culture Trip methodology if they want to explore innovative ways to leverage the content in their archives.  

Thanks to Roop Gill Axelsen, Product Manager, and Ana Jakimovska, Chief Product Officer at Culture Trip – and Collab coach – for sharing their knowledge and experience.

Some media houses are experimenting with introducing evergreen tags – the Wall Street Journal and Nice-Matin in France, among others. In the case of Nice-Matin, an 'evergreen' tick-box was created in the CMS back in 2016, to allow the journalists to catalogue an evergreen article. We spoke with Damien Allemand, Head of Digital Content, to find out more about that:

What was behind the decision to add tags to all Nice-Matin articles?

We realised that we were too often rewriting articles that we had already published. And when we did, the new article was just as good as the version that was already published.  

So we wanted to build a database of all these published articles that a journalist could retrieve with a simple keyword-based filter. For example, a lifestyle article on "The 30 reasons to never go to the Côte d'Azur" had 100,000 views when it was first published; three months later, we republished the same piece and recorded 200,000 views. Since then, it has been republished about ten times, each time driving good traffic to the website.

At the same time, this strategy allows us to save significant time to our journalists. When they are about to write a new article, they can easily find if a similar story was already published and, if it was, maybe they will just spend their time better writing about something else. 

What would you say is the proportion of evergreen articles in Nice-Matin 's archive?

In our estimation, the content that can be republished without the need for much editing represents around 10% of the archive. If we add to that other articles that only need to be slightly updated, we reach around 20%. These are articles such as "What is a red weather warning?", or "Why can we see Corsica from Nice?", which can be republished pretty much every year around the same time.

What are the challenges you have encountered with this manual-tagging system?

What doesn't work at times is that journalists do not always remember to tick the tagging box. It is hard to build the habit to do it with the consistency that is required for the system to bear fruits. Some reporters maybe do it for a few days and then they stop – maybe because it's hard to see immediate benefits.

What would it take to make the system work?

To optimise the use of evergreen content and build an efficient recommendation engine for your journalists, you need a detailed plan, good organisation, and buy-in from the newsroom. The tagging of evergreen articles is a task that might also be assigned to someone in the archive department. But that takes resources and willingness to invest them, which is often a problem for local and regional outlets like we are at Nice-Matin.

Thanks to Daniel Allemand, Head of Digital Content at Nice-Matin, for sharing his insights.

With so many headlines, stories, photos, videos and graphics published at Reuters – as many as 9,000 individual items per day – accurate metadata application, the use of topics to describe a piece of news, will always be ripe for automation. The challenge is also to recapture the total number of hours our journalists spend manually applying this metadata, a significant savings in time that can be turned toward newsgathering.

Feedback from our clients, especially other news organisations, tells us that they like stories that summarise newsworthy subjects with relevant, timely facts, what we refer to as evergreens. That has raised yet another challenge: how to use metadata to quickly surface evergreen story candidates, and possibly assemble a large portion of an evergreen story.

There is already some automation that has proved useful. For nearly every story, an algorithm applies topics whenever a story is transferred to an editor or between editors; the use of the ‘Add Topics’ button in the user interface is also encouraged; and some topics are applied based on trading symbols, or white-listed geographies. Applying this to thousands of stories per day helps, but it still feels incomplete.

In analysing how we apply metadata internally, we assumed the gold standard in assigning topics was going to be published-content that had passed review or insertion by a human editor. But that wasn’t necessarily the case. The number of topics is so great, with updated versions every few weeks, that no one person knew the codes and their definitions in detail, and codes often went missing or were over-applied.

Reuters has several story-types that are de facto evergreen items, with codes and associated names such as Explainer, Factbox and Take-A-Look. As long as a story was deemed evergreen from the start, the metadata suggestion approach worked well. It was less successful in applying metadata in determining what potentially could be an evergreen story.

We still do not have a quick way to say, ”This Factbox is relevant again this week, what edits does it need in order to be updated and republished?” It's a human task that could use an automated or semi-automated solution. Factboxes about politics are especially prone to age rapidly. Ensuring proper context and updating dates, times and facts is critical to these items, and make up the majority of editing time.

We have other automated efforts with structured data, around sports, corporate results and markets coverage. We found machine-produced text to be useful, if not sparkling prose. Most of the feedback internally was on the sometimes stilted nature of the writing and the punctuation. Moving from sentences using structured data to suggestions based on unstructured text is worth experimenting. 

We may find in experimenting that certain facts, as complete sentences, are evergreens, and can be suggested wholesale as the story writing progresses. For example: “This is the fourth story today, or this week, that can be grouped together. May I suggest these facts?” Or: “Based on previously published stories about this topic, you are writing on a subject that would make a popular Explainer.” Or perhaps an evergreen “score” that could help guide journalists.

Thanks to the Editor for Automation and News Technology at Reuters, Padraic Cassidy, for sharing this case study.

One way to apply machine learning, the most common form of AI nowadays, is to split its application into two steps: 

1)  In a first technical step, the objects of interest – for instance, news articles for media houses – are processed into a structured format;

2)  While in a second heuristic step, this structured format is exploited for various applications. A natural application is to search for similar news articles.

We illustrate this process in Figure 1:

A popular approach to give structure to news articles is to automatically assign tags. Tags can be broadly classified as follows: 

1)  Tags that are directly contained in the text, for example the name of a person – say, 'Joe Biden';

2)  Abstract tags that are not necessarily explicit in the text. For example, a topic like 'Politics';

3)  Meta-information like a manually-assigned 'Evergreen' tag.

The tags of the first type can be dealt with by classic Named Entity Recognition (NER) algorithms. Common subclasses are “People”, “Organisations”, and “Locations”. On the other hand, the second class of tags detailed above requires some specifically-trained classification algorithms. Finally, the last group (which is not used at TX Group yet) would require an additional automatic or semi-automatic process that will depend on the type of meta-information. For instance, we have seen in earlier sections that there are many ways to define evergreen content.

Searching for articles

The main benefit of merging combining different tags in the same solution lies in the possibility to use them in a unified search procedure. At TX Group, we are currently using the following logic in a beta test:

1)  Given a specific article, we score all other articles published within a reasonable time window according to their similarity to the first one, with respect to tag overlap. That is, we assign some points for each joint tag, and then compute the score by simply summing up the points.

2)  A major improvement is to assign points according to the frequency of a tag. For instance, if a tag is very common like 'Politics', it should correspond to fewer points than a more specific one like the name of an individual. To deal with this, we use a so-called Term-Inverse Frequency strategy: If a tag appears in only one article every 100, it receives 1/100 points on the similarity score. So the more frequent a tag is, the less important it is for the score.

3)  Another positive adjustment is to linearly scale the points according to the number of occurrences of a tag, which applies only to the first group of tags described above. For instance, if two articles contain the name 'Joe Biden' several times, then the assigned points are scaled accordingly.

Figure 2 shows a screenshot of our current search GUI (graphical user interface). Our goal is to integrate this more tightly into the journalistic production process.

Technical Implementation

To power this process, a NER-tagger is required to create the tags of the first class. As already mentioned, NER stands for Named Entity Recognition, an NLP technique to extract keywords from a text that has a special meaning, like names of people. There are many paid APIs on the market that cover this niche. The ones we tested provide quite good results: Rosette is a good example. On the other hand, there are also reasonably good Python packages that provide this functionality. Arguably the most popular ones are spaCy and Flair:

spaCy 

• PRO: most popular NLP package, runs very fast 
• CON: limited quality for NER in our test cases

Flair

• PRO: acceptable quality and good multi-language support
• CON: quite new and far slower

In the end, since we favoured quality over speed, we settled on Flair for our NER tagging process.

On the other hand, to classify articles by creating tags of the second class, we use a standard neural network which encodes articles using word vectors. As written above, the integration of class-3 tags is still an open challenge at TX-Group – which our journalists are asking us to solve.

Thanks to Tim Nonner, Chief Data Scientist at TX Group, for contributing to the report with this case study.

There are various technical approaches that an AI solution for recommending archived or evergreen content can take. Recommendations could be based upon existing tags, content categories, seasonal search traffic, social media trends, article headlines. The greater the accuracy and comprehensiveness in which article data is structured, the better chance a programme can be built to recommend the most suitable articles from the archive.

Unfortunately for the AI, there's usually no consistent method in which a newsroom structures its article performance and data, which can cause problems when trying to highlight the right article. There are certain 'hygiene' steps that all newsrooms can take to ensure their archive is accessible and enriched with usable data. The first is unlocking raw CMS data, which isn’t always picked up by traditional analytics software such as Google Analytics, Adobe Analytics, or Chartbeat. The core information needed is the headline, published date, last modified date, word count, content categories, content tags, and the unique ID/URL of an article. Also relevant is the author, what additional sites it appeared on (if any) and, if possible, the complete body text of the article. Whilst the latter would improve the accuracy versus performing Natural Language Processing (NLP) on just the headline, this would increase the storage cost and may be challenging to access for newsrooms.

The next step is to join the raw CMS data with article performance data. As a minimum, this would require a connection with a single analytics source via a unique article ID or URL, including page views, unique users and dwell time. Calculated metrics, powered by segments, can be built to identify the referral traffic breakdown per story, such as Search Page Views, Social Page Views, and Direct Page Views, which can be helpful to understand the evergreen nature of an article. Additional third-party data sources can be connected to aid understanding of how the article was engaged with off-platform. For example, the average position in the Google SERPS via Google Search Console, and the volume of comments, likes, and shares from Facebook and Twitter Analytics. 

Performance metrics do not even need to be exclusively digital, as recommending an article that appeared on the front page of a newspaper should be greater in priority than one that appeared in page 35, regardless of how well it performed on Facebook! 

Building a meaningful overview

Structuring of the data in such a manner will mean that the AI solution does not recommend articles, for example, which are fewer than 50-words long, which are published to irrelevant categories or with irrelevant tags, which only had a dwell time of 15 seconds, or which never ranked on Google. This should also be newsroom-configurable in order to best link with strategy. 

An additional benefit is that it can be linked to Business Intelligence (BI) and Data Visualisation software, such as Tableau, Qlik, and PowerBI. This provides the newsroom with the ability to integrate their archives by looking at the average performance over time, seeing if certain content categories perform better when published in the evenings or whether individual authors need support in their SEO due to under-indexing in search referrals. 

At Archant, we segment all articles into three easily understandable performance buckets, allowing journalists and editors to know what behaviours need changing to write fewer ‘red’ stories, tying in with business performance expectations. The three buckets can be set completely by the newsroom, as what they define the targets for an article. So it can be good, OK, or bad. We have used Page Views in the past to correct behaviour at Archant.

Thanks to Nick Cameron, Head of Performance at Archant, for this case study.

Although headline generation is not directly related to the archive, some tools and methods are the same, and the content in the archive is necessary as training data for the headline-generator. In this section, you can read of Axel Springer's experience with auto-generated SEO headlines.

Methodology

The SEO title – which is the title displayed in the Google Search results – is one of the main factors that influence how high the article will rank among the search results. The higher the position in the result list, the higher the click-through rate (CTR) for each search result. The SEO-title creation is a simple but rather time-consuming task for editors. Our experiment at Axel Springer is focused on automatically generating an SEO title for each article, which then the editor just have to review and approve. 

An SEO title should be descriptive, intriguing, and contain keywords that are specific to the context of the article, as well as frequently sought for. We decided to split the problem of automatic generation into two separate strands: 

First: Generate a title that is basically a very short summary of the article

This is essentially an extreme form of text summarisation. In recent years, summarisation problems were usually tackled by using neural networks with an encoder-decoder architecture. To train such a model, one needs a large amount of article data. Therefore, we extracted 500,000 WELT – one of the main news brands of Axel Springer – news articles from the archives to familiarise the model with the structure of the texts and titles. 

Second: Identify keywords that are important & frequently sought on Google Search

When composing the SEO-title, the SEO experts usually come up with context-specific keywords and then pick those with the highest expected search volume. To mimic this workflow we decided to take a two-step approach: First, we extracted the article’s keywords and their context-specific relevance score by applying named entity recognition (NER). Next, we used the PyTrends API to retrieve the expected search volume of the keywords. We sorted the keywords based on an XGBoost ranking trained on articles with good SEO-titles. In a last step, we tweaked the output of the neuronal network to incorporate the keywords. More information on the technical part can be found here.

The editors can now access the generated SEO-title along with relevant keywords via a browser plugin, which allowed us to work with the browser-based content management system (CMS) that’s used by journalists at WELT. Once the editor clicks on the SEO-title text field, the plugin pops up and generates a title prediction as well as a list of ranked keywords that could be included in the SEO-title. If the title prediction works fine, the editor can insert it into the CMS with just one click. If it’s of no use at all – which definitely happens from time to time – the editors can use the keyword suggestions as a supporting tool to create a title of their own.

Lessons learned

For the success of the experiment, it was essential to involve the editors in every single step of the project. We found out that the newsroom was mostly sceptical about AI tools. Therefore, we saw the need to first develop a clear understanding of what the product can and cannot do. Our team communicated a lot with the key stakeholders in the newsroom to collect feedback on each iteration and develop trust step-by-step. Finally, instead of having a fully automated SEO-title generator, we created a “human-in-the-loop” AI product that assists the editors to perform more efficiently a complex task they usually dislike. With this solution, the editors are still in control of how the article eventually appears on the web and are more willing to use the tool. 

Thanks to Sebastian Maulbeck, Senior Product Owner, Content Intelligence at Axel Springer, for sharing this case study.

On the topic of automating the creation of SEO titles, we also talked with Agnes Stenbom, Responsible Data & AI Specialist at Schibsted, and fellow Collab participant. Here is what we learned from her:

Intentions 

We had two key goals in experimenting with generating SEO-headlines through Machine Learning (ML). One the one hand, we wanted to create SEO-headlines for historical articles that didn’t already have an SEO headline (and thereby gain traffic and value from already performed work). On the other, we wanted to assist journalists in writing SEO-headlines for future articles. We saw this as a quite clear example of humans and AI working together to create value: journalists would bring their unique skills in creating journalistically relevant headlines, and machines would generate headlines that would be ranked high on search engines. An important note here is that the ambition was never to create and publish the headlines without human intervention. What we sought to do was to create a model that could generate suggestions that our journalists would review directly in the CMS.

Approach

After considering a number of different ML-approaches we landed on a 'stepwise' training process utilising transfer learning. Roughly speaking, our approach was to first teach the model to generate regular headlines and then add training data to make it able to also generate SEO-friendly headlines.  

The output of the model was put in a collaborative sheet where an editorial team member with knowledge about SEO would assess each of the generated headlines, either accepting or rejecting them. The results were mixed, with a majority of headlines being accepted. This was exciting – not least as we are working in Swedish, in which NLP/NLG is still at a relatively early stage. However, we had set a very ambitious goal of 80% acceptance in order to take the model further and the goal was not reached.

Lessons learned

We learned a lot from this experiment, and while the specific application(s) we had in mind at the start did not go into production, we still view this process as an important contributor to organisational learning about AI's potential. One of the biggest lessons we learned was the need to include different skillsets early on in the process, preferably already from the very start. Technical and editorial teams should set goals together and then work towards those same goals collaboratively.