Misleading, maliciously edited and other fake videos are on the rise around the world.
To help, the Duke Reporters’ Lab is leading a drive to create MediaReview, a new tagging system that will enable fact-checkers to be more consistent when they debunk false videos and images. It should help search engines and social media companies identify fakes more quickly and take prompt action to slow or stop them.
MediaReview is a schema similar to ClaimReview, a tagging system developed by the Reporters’ Lab, Google and Jigsaw that enables fact-checkers to better identify their articles for search engines and social media. MediaReview is based on a video-labeling vocabulary that Washington Post journalists created to describe misleading videos.
The new tagging system will allow fact-checkers to quickly label fake and manipulated videos and images with standardized tags such as “missing context,” “transformed,” “edited,” etc.
Bill Adair and Joel Luther are leading this project at the Reporters’ Lab. You can read about their work in a recent NeimanLab article and in their writing describing the project and why it’s needed:
With the U.S. election now less than a year away, at least four-dozen American fact-checking projects plan to keep tabs on claims by candidates and their supporters – and a majority of those fact-checkers won’t be focused on the presidential campaign.
The News & Observer has not abandoned fact-checking. It launched a new statewide initiative of its own — this time without PolitiFact’s trademarked Truth-O-Meter or a similar rating system for the statements it checks. “We’ll provide a highly informed assessment about the relative truth of the claims, rather than a static rating or ranking,” The N&O’s editors said in an article announcing its new project.
Among the 20 U.S. state and local fact-checkers that are not PolitiFact partners, at least 13 use some kind of rating system.
Of all the state and local fact-checkers, 11 are affiliated with TV stations — like WRAL, which had its own fact-checking service before it joined forces with PolitiFact this month. Another 11 are affiliated with newspapers or magazines. Five are local digital media startups and two are public radio stations. There are also a handful of projects based in academic journalism programs.
One example of a local digital startup is Mississippi Today, a non-profit state news service that launched a fact-checking page for last year’s election. It is among the projects we have added to our database over the past month.
We should note that some of these fact-checkers hibernate between election cycles. These seasonal fact-checkers that have long track records over multiple election cycles remain active in our database. Some have done this kind of reporting for years. For instance, WISC-TV in Madison, Wisconsin, has been fact-checking since 2004 — three years before PolitiFact, The Washington Post and AP got into the business.
One of the hardest fact-checking efforts for us to quantify is run by corporate media giant TEGNA Inc. which operates nearly 50 stations across the country. Its “Verify” segments began as a pilot project at WFAA-TV in the Dallas area in 2016. Now each station produces its own versions for its local TV and online audience. The topics are usually suggested by viewers, with local reporters often fact-checking political statements or debunking local hoaxes and rumors.
A reporter at WCNC-TV in Charlotte, North Carolina, also produces national segments that are distributed for use by any of the company’s other stations. We’ve added TEGNA’s “Verify” to our database as a single entry, but we may also add individual stations as we determine which ones do the kind of fact-checking we are trying to count. (Here’s how we decide which fact-checkers to include.)
A Global Movement
As for the global picture, the Reporters’ Lab is now up to 226 active fact-checking projects around the world — up from 210 in October, when our count went over 200 for the first time. That is more than five times the number we first counted in 2014. It’s also more than double a retroactive count for that same year –- a number that was based on the actual start dates of all the fact-checking projects we’ve added to the database over the past five years (see footnote to our most recent annual census for details).
Other recent additions to the database involved several established fact-checkers, including PesaCheck, which launched in Kenya in 2016. Since then it’s added bureaus in Tanzania in 2017 and Uganda in 2018 — both of which are now in our database. We added Da Begad, a volunteer effort based in Egypt that has focused on social media hoaxes and misinformation since 2013. And there’s a relative newcomer too: Re:Check, a Latvian project that’s affiliated with a non-profit investigative center called Re:Baltica. It launched over the summer.
Peru’s OjoBiónico is back on our active list. It resumed fact-checking last year after a two-year hiatus. OjoBiónico is a section of OjoPúblico, a digital news service that focuses on an investigative reporting service.
We already have other fact-checkers we plan to add to our database over the coming weeks. If there’s a fact-checker you know about that we need to update or add to our map, please contact Joel Luther at the Reporters’ Lab.
The Reporters’ Lab added 21 fact-checkers to our database of reporting projects that regularly debunk political misinformation and viral hoaxes, pushing our global count over 200.
The database now lists 210 active fact-checkers in 68 countries. That nearly quintupled the number the Reporters’ Lab first counted in 2014. It also more than doubled a retroactive count for that same year – a number that was based on the actual start dates of all the fact-checking projects we’ve added to the database over the past five years (see footnote to our most recent annual census).
The database now lists several other recent additions that also launched in 2019, mainly to focus on upcoming elections. Bolivia Verifica launched in June, four months before this past weekend’s vote, which may be headed for a December runoff. Reverso in Argentina also launched in June, followed by Verificado Uruguay in July. The general elections in those two countries are this coming Sunday.
Other 2019 launches include Namibia FactCheck, GhanaFact and, in the United States, local TV station KCRG-TV’s in Cedar Rapids, Iowa. KCRG is a bit of a special case, since it’s hardly a newbie. The TV station was previously owned by a local newspaper, The Cedar Rapids Gazette. Even after the sale, the two newsrooms collaborated on fact-checking for several years through last year’s U.S. midterm elections. But now they have gone separate ways. Starting in March, the investigative reporting team at KCRG began doing its own fact-checking segments.
At least six other fact-checkers that launched in 2019 were already in our database before this month’s update, several of which were intentionally short-term projects that focused on specific elections. We’re checking on the status of those now. At least one, Global Edmonton’s Alberta Election Fact Check, is already on our inactive list. For that reason, we expect our count might not grow much more before the end of 2019 and might even drop slightly.
In addition to the projects that began in 2019, we also added three established fact-checkers to our database that were already in operation before this year: Local TV station KRIS-TV in Corpus Christi, Texas, has been on the fact-checking beat since 2017. The journalists who do fact-checking for Syria-focused Verify-Sy have worked from locations in Turkey, Europe and within that war-torn country since 2016. And Belgium’s Knack magazine has provided a fact-checking feature to its readers since 2012.
We weren’t sure we would cross the 200 fact-checkers milestone in October, since we also moved seven dormant projects to our separate count of inactive fact-checkers this month. Our count in September was 195 before we made this month’s updates.
If there’s a fact-checker you know about that we need to update or add to our database, please contact Joel Luther at the Reporters’ Lab. (Here’s how we decide which fact-checkers to include.)
The Duke Reporters’ Lab is adding seven fact-checkers from three continents to our global database. That puts our ongoing count of reporting projects that regularly debunk political misinformation and viral hoaxes close to 200.
With this month’s additions, the Lab’s database now counts 195 projects in 62 countries, including every project the International Fact-Checking Network has verified as signatories of its code of principles.
One new addition uses a name that’s inspired many others in the fact-checking community: the polygraph machine, also known as the lie detector. DELFI’s Melo Detektorius (“Lie Detector”) launched last November. It’s the fact-checker for the Lithuanian outlet of a commercial media company that operates digital news channels in the Baltic states and across Eastern Europe.
Many others have used variations of the name before, including the Columbian news site La Silla Vacía’s Detector de Mentiras and the Danish Broadcasting Corporation’s weekly political fact-checking TV program Detektor. There are versions of polygraph too, such as Polígrafo in Portugal and El Poligrafo, a fact-checker for the print edition of the Chilean newspaper El Mercurio. At least three inactive entries in our database used similar names.
The fact-checkers at Spondeo Media in Mexico City avoided the wording, but apparently liked the idea. Instead, they deploy a cartoon polygraph machine with emoji-like facial expressions to rate the accuracy of statements.
Two news sites associated with the TV Today Network in New Delhi and its corporate parent India Today are also recent additions to our database. In addition to the work that appears on India Today Fact Check, the company’s fact-checkers produce reports for the Hindi-language news channel Aaj Tak and the Bangla-language news and opinion portal DailyO. When claims circulate in multiple languages, fact-checks are translated and published across platforms.
“Broadly, the guiding principle for deciding the language of our fact- check story is the language in which the claim was made,” explained Balkrishna, who leads the Fact Check Team at the India Today Group. “If the claim is Hindi, we would write the fact check story in Hindi first. If the same claim appears in more than one language, we translate the stories and publish it on the respective websites.”
While it’s relatively common for fact-checkers in some countries to present their work in multiple languages on one site, it’s less common for one media company to produce fact-checks for multiple outlets in multiple languages.
As we approach a Canadian national election slated for Oct. 21, we are adding two fact-checkers from that part of the world. One is Décrypteurs from CBC/Radio-Canada in Montréal. It launched in May to focus on digital misinformation, particularly significant claims and posts that are flagged by its audience. But the format is not entirely new to the network, where reporter Jeff Yates had produced occasional fact-checks under the label “inspecteur viral.”
The Walrus magazine in Toronto is also focusing on digital misinformation on its fact-checking site, which launched in October 2018.
We have added two other well-established fact-checkers that have a similar focus. The first is the Thai News Agency’s Sure and Share Center in Bangkok. The Thai News Agency is the journalism arm of Mass Communication Organization of Thailand, a publicly traded state enterprise that was founded in 1952 and privatized in 2004.
The other is Fatabyyano, an independent fact-checker based in Amman, Jordan. It covers a wide range of misinformation and hoaxes throughout the Arab world, including nearly two dozen countries in the Middle East and North and East Africa. Applied Science Private University and the Zedni Education Network are among its supporters.
We learned that Fatabyyano’s name is a reference to a holy command from the Quran meaning “to investigate” from an article by former Reporter’s Lab student researcher Daniela Flamini. She wrote about that site and other fact-checking projects in the Arab world for the Poynter Institute’s International Fact-Checking Network.
Several of the sites Flamini mentioned are among a list of others we plan to add to our database when we post another of these updates in October.
The Duke Reporters’ Lab has launched a global effort to expand the use of ClaimReview, a standardized method of identifying fact-check articles for search engines and apps.
Funded by a grant from the Google News Initiative, The ClaimReview Project provides training and instructional materials about the use of ClaimReview for fact-checkers around the world.
ClaimReview was developed through a partnership of the Reporters’ Lab, Google, Jigsaw, and Schema.org. It provides a standard way for publishers of fact-checks to identify the claim being checked, the person or entity that made the claim, and the conclusion of the article. This standardization enables search engines and other platforms to highlight fact-checks, and can power automated products such as the FactStream and Squash apps being developed in the Reporters’ Lab.
“ClaimReview is the secret sauce of the future,” said Bill Adair, director of the Duke Reporters’ Lab. “It enables us to build apps and automate fact-checking in new and powerful ways.”
Slightly less than half of the 188 organizations included in our fact-checking database use ClaimReview.
At the Global Fact 6 conference in Cape Town, the Lab led two sessions designed to recruit and train new users. During a featured talk titled The Future of ClaimReview, the Lab introduced Google’s Fact Check Markup Tool, which makes it easier for journalists to create ClaimReview. They no longer have to embed code in their articles and can instead create ClaimReview by submitting a simple web form.
In an Intro to ClaimReview workshop later in the day, the Lab provided step-by-step assistance to fact-checkers using the tool for the first time.
The Lab also launched a website with a user guide and best practices, and will continue to work to expand the number of publishers using the tool.
My opening remarks from Global Fact 6, Cape Town, South Africa, on June 19, 2019.
It’s wonderful to be here and see so many familiar faces. It’s particularly cool to see our new team from the IFCN, not just Baybars and Cris, but also Daniela Flamini, one of our journalism students from Duke who graduated last month and is now working for the IFCN.
And it warms my heart to see my old friend Stephen Buckley here. When Stephen was dean of the faculty at Poynter, the two of us organized the first Global Fact meeting in London in 2014. That wasn’t easy. We had difficulty raising enough money. But Stephen was determined to make it happen, so he found some money from a few different accounts at Poynter. Global Fact – and our important journalistic movement – would not have happened if it weren’t for him.
I’m impressed by this turnout – more than 250 attendees this year! I confess that when I saw the headline on Daniela’s story last week that said this was “the largest fact-checking event in history”… I wanted a fact-check. But I did one, and as PolitiFact would say, I rate that statement True!
I want to start today with a quick reminder of the importance of holding people accountable for what they say — in this case…me.
You will recall that last year at Global Fact, I promised that I would get a tattoo. And after some discussion, I decided it would be a tattoo of my beloved Truth-O-Meter. But a year went by and a funny thing happened: I decided I didn’t want a tattoo.
Now, as fact-checkers, we all know the importance of holding people accountable for what they say. We did that at PolitiFact with the Obameter and other campaign promise meters. PolitiFact has a special meter for a broken promise that usually features the politician with a big frown. We have fun choosing that photo, which has the person looking really miserable.
So I’ve created one to rate myself on the tattoo promise: The Bill-O-Meter. Promise broken!
My message today to open Global Fact is also about accountability. It’s about the need to make sure we fact-check all players in our political discourse.
Julianna Rennie and I recently wrote a piece for Poynter that looked at a new trend in the United States we call “embedded fact-checking.” It’s the growing practice of reporters including fact-checks in their news articles, when they drop in a paragraph or two that exposes a falsehood. For example, they may write that someone “falsely claimed that vaccines cause autism.”
We were glad to find a growing trend of embedded fact-checking in news and analysis articles in the New York Times, the Washington Post, and the AP over the past four years. But we also found the subject was nearly always the same: Donald Trump. It was wonderful to see the trend, but it was lopsided.
Trump is a prime target for fact-checking because his volume of falsehoods is unprecedented in American history — and probably in world history, too. Journalists rightly should question everything he says. And you may have similar figures in your own countries who deserve similar scrutiny.
But we shouldn’t focus so much on Trump that we neglect other politicians and other parties. That’s true not just in the United States but everywhere. Indeed, when we put together the IFCN code of principles three years ago, we said that fact-checkers “do not concentrate their fact-checking on any one side.”
In the United States and around the world, we need to make sure that we check all the important players in the political discourse, whether it is for news stories or our fact-checking sites.
So my message for you today is a simple one: check everybody. Hold everyone accountable.
The number of fact-checking outlets around the world has grown to 188 in more than 60 countries amid global concerns about the spread of misinformation, according to the latest tally by the Duke Reporters’ Lab.
Since the last annual fact-checking census in February 2018, we’ve added 39 more outlets that actively assess claims from politicians and social media, a 26% increase. The new total is alsomore than four-times the 44 fact-checkers we counted when we launched our global database and map in 2014.
Globally, the largest growth came in Asia, which went from 22 to 35 outlets in the past year. Nine of the 27 fact-checking outlets that launched since the start of 2018 were in Asia, including six in India. Latin American fact-checking also saw a growth spurt in that same period, with two new outlets in Costa Rica, and others in Mexico, Panama and Venezuela.
The actual worldwide total is likely much higher than our current tally. That’s because more than a half-dozen of the fact-checkers we’ve added to the database since the start of 2018 began as election-related partnerships that involved the collaboration of multiple organizations. And some those election partners are discussing ways to continue or reactivate that work – either together or on their own.
Over the past 12 months, five separate multimedia partnerships enlisted more than 60 different fact-checking organizations and other news companies to help debunk claims and verify information for voters in Mexico, Brazil, Sweden, Nigeria and Philippines. And the Poynter Institute’s International Fact Checking Network assembled a separate team of 19 media outlets from 13 countries to consolidate and share their reporting during the run-up to last month’s elections for the European Parliament. Our database includes each of these partnerships, along with several others – but not each of the individual partners. And because they were intentionally short-run projects, three of these big partnerships appear among the 74 inactive projects we also document in our database.
Politics isn’t the only driver for fact-checkers. Many outlets in our database are concentrating efforts on viral hoaxes and other forms of online misinformation – often in coordination with the big digital platforms on which that misinformation spreads.
We also continue to see new topic-specific fact-checkers such as Metafact in Australia and Health Feedback in France — both of which launched in 2018 to focus on claims about health and medicine for a worldwide audience.
(Here’s how we decide which fact-checkers to include in the Reporters’ Lab database.)
Fact-Checkers by Continent Since Feb. 2018
Africa:4 to 9 Asia: 22 to 35 Australia: 3 to 5 Europe: 52 to 61 North America: 53 to 60 South America: 15 to 18
TRACKING THE GROWTH
As we’ve noted, elections are not the only draw for aspiring fact-checkers. Many outlets in our database are concentrating their efforts on viral hoaxes and other forms of online misinformation – often in coordination with the big digital platforms on which that misinformation spreads. And those platforms are also providing incentives.
In one such effort, the Reporters’ Lab worked with Google and Schema.org to develop ClaimReview, an open-source tagging system for fact-checks. Google, Microsoft’s BING, Facebook and YouTube use this system to help identify and showcase fact-checkers’ work in their news feeds and search results – a process that generates traffic and attention for the fact-checkers. It also provides data that is powering experiments in live, real time fact-checks that can be delivered to users automatically. (Disclosure: Google and Facebook are among the funders of the Reporters’ Lab.)
Another driver: Facebook. It has recruited independent fact-checking partners around the world to help identify misinformation on its platforms. The social network began that effort in late 2016 with help from the Poynter’s Institute’s IFCN. (Poynter is a journalism training and research center in St. Petersburg, Florida, that also is home to the U.S. fact-checking site PolitiFact.)
Meanwhile, YouTube gave fact-checking a boost in India when it started putting fact-checks at the top of YouTube search results, which helped contribute to a surge of new outlets in that country. Now India has 11 entries in our database, six of which launched since our February 2018 census. And it’s likely there are others to add in the next few weeks.
KINDS OF FACT-CHECKERS
A bit more than half of fact-checkers are part of a media company (106 of 188, or 56%). That percentage has been dropping over the past few years, mostly because of the changing business landscape for media companies in the United States. In our 2018 census, 87% of the U.S. fact-checkers were connected to a media company (41 out of 47). Now it’s 65% (39 out of 60). In other words, as the number of fact-checker in the U.S. has grown, fewer of them have ties to those companies.
Among fact-checkers in the rest of the world, the media mix remains about half and half (67 out of 128, or 52% — very close to the 54% we saw in 2018).
The fact-checkers that are not part of a larger media organization include independent, standalone organizations, both for-profit and non-profit (the definitions of these legal and economic entities vary greatly from country to country). Some of these fact-checkers are subsidiary projects of bigger organizations that focus on civil society and political accountability. Others are affiliated with think tanks and academic institutions.
Among the recent additions is the journalism department at the University of the Philippines’ College of Mass Communication, which was the coordinator of Tsek.ph, a political fact-checking partnership mentioned earlier that also involves two other academic partners.
About 70% of the fact-checkers (131 of 188) have well-defined rating systems for categorizing the claims they investigate — similar to what we’ve seen in past years.
As usual, we found many of the rating systems to be entertaining. One of our new favorites comes from Spondeo Media in Mexico, which launched in December. It supplements a basic, four-point, true-to-false scale with a mascot – NETO, a cartoon lie-detector who smiles and jumps for joy with true claims but gets steamed with false ones. Another, India Today Fact Check, rated claims using a scale of one-to-three animated crows, along with a slogan in Hindi: “When you lie, the crow bites” (also the title of a popular movie: “Jhooth bole kauva kaate”).
We decided to time this year’s fact-checking census to correspond with the sixth annual GlobalFact Summit, which begins next week in Cape Town, South Africa. About 250 attendees from nearly nearly 60 countries are expected at this year’s gathering — which is yet another measure of fact-checking’s continued growth: That’s five times the number from the first GlobalFact in London in 2014.
Joel Luther, Share the Facts Research and Outreach Coordinator at the Duke Reporters’ Lab, and former student researcher Daniela Flamini (now an intern at the Poynter Institute’s International Fact Checking Network) contributed to this report.
FOOTNOTE: ANOTHER WAY TO COUNT FACT-CHECKERS?
A challenge we have each time the Duke Reporters’ Lab conducts our annual fact-checking censuses is that our final tally depends so much on when we happen to discover these outlets. Our counting also depends on when fact-checkers come and go — especially short-term, election-focused projects that last several months. If a fact-checker was hard at work most of the year covering a campaign, but then closed up shop before we did our census, they’ll still be counted — but in our list of inactive projects.
That inactive list is an interesting trove of good ideas for other fact-checkers to mine. It also provides an entirely different way for us to tally fact-checkers: by counting all the projects that were active at some point during the year — not just the ones that make it to winter.
This approach might better showcase the year in fact-checking. And it also would show that fact-checking was in fact growing faster than we even thought it was.
Here’s chart that compares the number of fact-checkers that we know were active in certain years — even the ones that ultimately closed down — with the subsequent census number for that year….
There are reasons why the Reporters’ Lab would still need to keep counting fact-checkers the way we have since 2014. For one, we need current lists and counts of serious fact-checking projects for all kinds of reasons, including academic research and the experiments that we and others want to try out with real-world fact-checkers.
And yet it’s still great to see how fast fact-checking is growing — even more than we sometimes thought.
(The small print for anyone who’s fact-checking me: The adjusted numbers shown here combine any fact-checker in our database that was active at some point during that given year. Most of our census reports were meant to count the previous year’s activity. For example our February 2018 census appears in this chart as our count of 2017 fact-checkers, even if some of those 2017 fact-checkers were only counted in last year’s census as inactive by the time the census was published. The number shown for 2018 is the 16-month 2018-19 number we are releasing in this report. You also might note that some other numbers here are slightly off from data we’ve previously shared. The main reason is that this proposed form of counting depends on having the dates that each project began and ended. In a handful of cases, we do not.)
It’s now much easier for fact-checkers to use ClaimReview, a tagging tool that logs fact-checks published around the world into one database. The tool helps search engines — and readers — find non-partisan fact-checks published globally. It also organizes fact-check content into structured data that automated fact-checking will require.
Currently, only half of the roughly 160 fact-checking organizations that the Duke Reporters’ Lab tracks globally use ClaimReview. In response, Google and the Duke Reporters’ Lab have developed an easier method of labelling the articles to help both recruit more users and expand a vital fact-check data set.
ClaimReview was created in 2015 after a conversation between staff at Google and Glenn Kessler, the Washington Post fact-checker. Kessler wanted Google to highlight fact-checks in its search results. Bill Adair, director of the Duke Reporters’ Lab, was soon brought in to help.
Dan Brickley from Schema.org, Justin Kosslyn from Google and Adair developed a tagging systembased on the schemas maintained by Schema.org, an organization that develops structured ways of organizing information. They created a universal system for fact-checkers to label their articles to include the claim checked, who said it and a ruling on its accuracy. “It’s the infrastructure that provides the atomic unit of fact-checking to search engines,” Adair said.
Initially, ClaimReview produced a piece of code that fact-checkers copy and pasted into their online content management system. Google and other search engines look for the code when crawling content. Next, Chris Guess of Adair’s team developed a ClaimReview widget called Share the Facts, a content box summarizing fact-checks that PolitiFact, FactCheck.org and the Washington Post can publish online and share on social media.
The latest version of ClaimReview no longer requires users to copy and paste the code, which can behave inconsistently on different content management systems. Instead, fact-checkers only have to fill out Google form fields similar to what they used previously to produce the code.
While the concept of ClaimReview is simple, it opens to the door to more innovation in fact-checking. It organizes data in ways that can be reused. By “structuring journalism, we can present content in more valuable ways to people,” said Adair.
By labeling fact-checks, the creators effectively created a searchable database of fact-checks, numbering about 24,000 today. The main products under development at the Reporters’ Lab, from FactStream to Squash, rely on fact-check databases. Automated fact-checking especially requires a robust database to quickly match untrue claims to previously published fact-checks.
The database ClaimReview builds offers even more possibilities. Adair hopes to tweak the fields fact-checkers fill in to provide better summaries of the fact-checks and provide more information to readers. In addition, Adair envisions ClaimReview being used to tag types of misinformation, as well as authors and publishers of false content. It could also tag websites that have a history of publishing false or misleading articles.
The tagging already is already benefiting some fact-check publishers. “ClaimReview helps to highlight and surface our fact-checks on Google, more than the best SEO skills or organic search would be able to achieve,” said Laura Kapelari, a journalist with Africa Check. ClaimReview has increased traffic on Africa Check’s website and helped the smaller Africa Check compete with larger media houses, she said. It also helps fact-checkers know which facts have already been investigated, which reduces redundant checks.
Joel Luther, the ClaimReview project manager in the Reporters’ Lab, expects this new ClaimReview format will save fact-checkers time and decrease errors when labeling fact-checks. However, there is still room to grow. Kapelari wishes there was a way for the tool to automatically grab key fields such as names in order to save time.
The Reporters’ Lab has a plan to promote ClaimReview globally. Adair is already busy on that front. Early this month, a group of international fact-checkers and technologists met in Durham for Tech & Check 2019, an annual conference where people on this quest share progress on automated fact-checking projects intended to fight misinformation. Adair, an organizer of Tech & Check, emphasized new developments with ClaimReview, as well as its promise for automating fact-checking.
Not much would be possible without this tool, he stressed. “It’s the secret sauce.”
We tested two fact-checking products during the State of the Union address. One failed, the other showed great promise.
The failure was FactStream, our iPhone app. It worked fine for the first 10 minutes of the speech. Users received two timely “quick takes” from Washington Post Fact Checker Glenn Kessler, but then the app crashed under an unusual surge of heavy traffic that we’re still investigating. We never recovered.
The other product is a previously secret project we’ve code-named Squash. It’s our first attempt at fully automated fact-checking. It converts speech to text and then searches our database of fact-checks from the Post, FactCheck.org and PolitiFact. When it finds a match, a summary of the fact-check pops onto the screen.
We’ve been testing Squash for the last few weeks with mixed results. Sometimes it finds exactly the right fact-checks. Other times the results are hilariously bad. But that’s what progress looks like.
We went into last night’s speech with very modest expectations. I said before the speech I’d be happy if the speech simply triggered fact-checks to pop up, even if it was a poor match.
But Squash actually performed pretty well. It had 20 pop-ups and six of them were in the ballpark.
Overall, the results were stunning. It gave us a glimpse of how good automated fact-checking can be.
We’ll have more to share once we’ve reviewed the results, so stay tuned.
As for FactStream, it now has lots of timely fact-checks from the State of the Union on the main home screen, which continues to function well. We will fix any problems we identify with the live event feature and plan to be back in action for real-time coverage for campaign events later this year.
UPDATE, Feb. 5, 11 p.m.: Our FactStream app failed during the State of the Union address. We apologize for the problems. We are still sorting out what happened, but it appears we got hit with an unexpected surge of traffic that overwhelmed our servers and our architecture.
As we noted at the bottom of this post, this was a test – only our second of the app. We’ll fix the problems and be better next time.
The Reporters’ Lab is teaming up with the Washington Post, PolitiFact and FactCheck.org to offer live fact-checking of the State of the Union address on Tuesday night on our new FactStream app.
Journalists from the Post, PolitiFact, and FactCheck.org will provide real-time updates throughout the speech in two forms:
Ratings – Links to previously published fact-checks with ratings when the president repeats a claim that has been checked before.
Quick takes – Instant updates about a statement’s accuracy. They will be labeled red, yellow and green to indicate their truthfulness.
Tuesday’s speech will be the second test of FactStream. The first test, conducted during last year’s State of the Union address, provided users with 32 updates. We got valuable feedback and have made several improvements to the app.
FactStream is part of the Duke Tech & Check Cooperative, a project to automate fact-checking that is funded by Knight Foundation, the Facebook Journalism Project and the Craig Newmark Foundation. Additional support has been provided by Google.
FactStream is available for iPhone and iPad (sorry, no Android version yet!) and is a free download from the App Store.
The app has two streams. One, shown by the home symbol in the lower left of the screen, provides a constant stream of the latest fact-checks published every day throughout the year. The live event feature for the State of the Union address is marked by an icon of a calendar with a check mark.
Because this is a test, users could encounter a few glitches. We’d love to hear about any bugs you encounter and get your feedback at email@example.com.