Category: Tech & Check Cooperative
Tech & Check Cooperative
What we learned during our experiment with live fact-checking
We got some nice feedback and helpful suggestions about FactStream, our new app
By Bill Adair - February 1, 2018
Except for the moment when we almost published an article about comedian Kevin Hart’s plans for his wedding anniversary, the first test of FactStream, our live fact-checking app, went remarkably smoothly.
FactStream is the first in a series of apps we’re building as part of our Tech & Check Cooperative. We conducted a beta test during Tuesday’s State of the Union address that provided instant analysis from FactCheck.org, PolitiFact and Glenn Kessler, the Washington Post Fact Checker.
Overall, the app functioned quite well. Our users got 32 fact-checks during the speech and the Democratic response. Some were links to previously published checks while others were “quick takes” that briefly explained the relative accuracy of Trump’s claim.
When President Trump said “we enacted the biggest tax cuts and reforms in American history,” users got nearly instant assessments from FactCheck and PolitiFact.
“It is not the biggest tax cut,” said the quick take from FactCheck.org. “It is the 8th largest cut since 1918 as a percentage of gross domestic product and the 4th largest in inflation-adjusted dollars.”
PolitiFact’s post showed a “False” Truth-O-Meter and linked to an October fact-check of a nearly identical claim by Trump. Users of the app could click through to read the October check.
Many of the checks appeared on FactStream seconds after Trump made a statement. That was possible because fact-checkers had an advance copy of the speech and could compose their checks ahead of time.
We had two technical glitches – and unfortunately both affected Glenn. One was a mismatch of the URLs for published Washington Post fact-checks that were in our database, which made it difficult for him to post links to his previous work. We understand the problem and will fix it.
The other glitch was bizarre. Last year we had a hiccup in our Share the Facts database that affected only a handful of our fact-checks. But during Tuesday’s speech we happened to hit one when Glenn got an inadvertent match with an article from the Hollywood rumor site Gossip Cop, another Share the Facts partner. So when he entered the correct URL for his own article about Trump’s tax cut, a fact-check showed up on his screen that said “Kevin Hart and Eniko Parrish’s anniversary plans were made up to exploit the rumors he cheated.”
Oops!
Fortunately Glenn noticed the problem and didn’t publish. (Needless to say, we’re fixing that bug, too.)
This version of FactStream is the first of several we’ll be building for mobile devices and televisions. This one relies on the fact-checkers to listen for claims and then write short updates or post links to previous work. We plan to develop future versions that will be automated with voice detection and high-speed matching to previous checks.
We had about 3,100 people open FactStream over the course of the evening. At the high point we had 1,035 concurrently connected users.
Our team had finished our bug testing and submitted a final version to Apple less than 48 hours before the speech, so we were nervous about the possibility of big crashes. But we watched our dashboard, which monitored the app like a patient in the ICU, and saw that it performed well.
Our goal for our State of the Union test was simple. We wanted to let fact-checkers compose their own checks and see how users liked the app. We invited users to fill out a short form or email us with their feedback.
The response was quite positive. “I loved it — it was timely in getting ‘facts’ out, easy to use, and informative!” Also: “I loved FactStream! I was impressed by how many fact-checks appeared and that all of them were relevant.”
We also got some helpful complaints and suggestions:
Was the app powered by people or an algorithm? We didn’t tell our users who was choosing the claims and writing the “quick takes,” so some people mistakenly thought it was fully automated. We’ll probably add an “About” page in the next version.
More detail for Quick Takes. Users liked when fact-checkers displayed a rating or conclusion on our main “stream” page, which happened when they had a link to a previous article. But when the fact-checkers chose instead to write a quick take, we showed nothing on the stream page except the quote being checked. Several people said they’d like some indication about whether the statement was true, false or somewhere in between. So we’ll explore putting a short headline or some other signal about what the quick take says.
Better notifications. Several users said they would like the option of getting notifications of new fact-checks when they weren’t using the app or had navigated to a different app or website. We’re going to explore how we might do that, recognizing that some people may not want 32 notifications for a single speech.
An indication the app is still live. There were lulls in the speech when there were no factual claims, so the fact-checkers didn’t have anything new to put on the app. But that left some users wondering if the app was still working. We’ll explore ways we can indicate that the app is functioning properly.
What to expect tonight from FactStream, our live fact-checking app
It’s an early step toward automated fact-checking. What could go wrong?
By Bill Adair - January 30, 2018
Tonight we’re conducting a big test of automated fact-checking. Users around the world will be able to get live fact-checks from the Washington Post, PolitiFact and FactCheck.org on our new FactStream app.
It’s an ambitious experiment that was assembled with unusual speed. Our team – lead developer Christopher Guess, project manager Erica Ryan and the designers from the Durham firm Registered Creative – built the app in just three months. We were still testing the app for bugs as recently as Sunday night (we found a couple and have fixed them!).
FactStream, part of the Duke Tech & Check Cooperative, is our name for apps that provide live fact-checking. This first version will rely on the fact-checkers to identify claims and then push out notifications. Future versions will be more automated.
We’re calling tonight’s effort a beta test because it will be the first time we’ve used the app for a live event. We’ve tested it thoroughly over the past month, but it’s possible (likely?) we could have some glitches. Some things that might happen:
- President Trump might make only a few factual claims in the speech. That could mean you see relatively few fact-checks.
- Technical problems with the app. We’ve spent many hours debugging the app, fixing problems that ranged from a scrolling glitch on the iPhone SE to a problem we called “the sleepy bug” that caused the app to stop refreshing. We think we’ve fixed them all. But we can’t be sure.
- Time zone problems. If you set an alert for tonight’s speech before we fixed a time zone bug this morning, you got a notification at 3 p.m. Eastern time today that said “2018 State of the Union Address will begin in fifteen minutes.” Um, no, it’s at 9 p.m. Eastern tonight. But we believe we’ve fixed the bug!
(I’m writing this at the suggestion of Reporters’ Lab co-director Mark Stencel, who notes that Elon Musk has highlighted video of his rockets exploding to make the point that tests can fail.)
Long road to reusabity of Falcon 9 primary boost stage…When upper stage & fairing also reusable, costs will drop by a factor >100. pic.twitter.com/WyTAQ3T9EP
— Elon Musk (@elonmusk) September 14, 2017
The future of fact-checking is here. Our goal tonight is to test the app and explore the future of automated journalism. We’re excited to try – even if we encounter a few problems along the way.
I hope you’ll try the app and let us know what you think. You can email us at team@sharethefacts.org or use this feedback form.
Want to help us test our fact-checking app during the State of the Union?
The FactStream app provides live fact-checking during political events. We’d like your help testing it during the speech.
By Rebecca Iannucci - January 26, 2018
The Duke Reporters’ Lab is seeking beta testers for FactStream, our new second-screen app that will provide live fact-checking during political events.
On Tuesday, Jan. 30, the Reporters’ Lab will partner with PolitiFact, The Washington Post and FactCheck.org, which will provide FactStream users with live fact-checking of President Trump’s State of the Union address.
Throughout the speech, FactStream users will see pop-ups on their screen, alerting them to previously published fact-checks or real-time analyses of President Trump’s claims. By pressing on a pop-up, users can read the full text of a fact-check, share the fact-check on various social media platforms or simply receive additional context about Trump’s statements.
FactStream is a product of the Duke Tech & Check Cooperative, a $1.2 million effort that uses automation to help fact-checkers do their work and broaden their audience. Launched in September 2017, Tech & Check also serves as a hub to connect journalists, researchers and computer scientists who are doing similar work.
The first iteration of FactStream is a manual app that requires the work of human fact-checkers behind the scenes. It is an important first step toward the “holy grail” of fact-checking — automated detection of a claim that is instantly matched to a published fact-check.
If you are an iPhone or iPad user and would like to test FactStream during the State of the Union, here’s how:
(1) Download FactStream from the App Store.
(2) Open and use the app during President Trump’s speech (Jan. 30 at 9 p.m. ET), making sure to test the app’s various screens and shared fact-checks.
(3) After the speech is over, send us feedback about the app with this Google Form.
New Tech & Check projects will provide pop-up fact-checking
With advances in artificial intelligence and the growing use of the ClaimReview schema, Reporters' Lab researchers are developing a new family of apps that will make pop-up fact-checking a reality
By Julianna Rennie - January 16, 2018
For years, fact-checkers have been working to develop automated “pop-up” fact-checking. The technology would enable users to watch a political speech or a campaign debate while fact-checks pop onto their screens in real time.
That has always seemed like a distant dream. A 2015 report on “The Quest to Automate Fact-Checking” called that innovation “the Holy Grail” but said it “may remain far beyond our reach for many, many years to come.”
Since then, computer scientists and journalists have made tremendous progress and are inching closer to the Holy Grail. Here in the Reporters’ Lab, we’ve received $1.2 million in grants to make automated fact-checking a reality.
The Duke Tech & Check Cooperative, funded by Knight Foundation, the Facebook Journalism Project and the Craig Newmark Foundation, is an effort to use automation to help fact-checkers research factual claims and broaden the audience for their work. The project will include about a half-dozen pop-up apps that will provide fact-checking on smartphones, tablets and televisions.
One key to the pop-up apps is a uniform format for fact-checks called the ClaimReview schema. Developed through a partnership of Schema.org, the Reporters’ Lab, Jigsaw and Google, it provides a standard tagging system for fact-checking articles that makes it easier for search engines and apps to identify the details of a fact-check. ClaimReview, which can be created using the Share the Facts widget developed by the Reporters’ Lab, will enable future apps to quickly find relevant fact-checking articles.
“Now, I don’t need to scrape 10 different sources and try to wrangle permission because there’s this database that will be growing increasingly,” says Dan Schultz, senior creative technologist at the Internet Archive.
This works because politicians repeat themselves. For example, many politicians and analysts have claimed that the United States has the highest corporate tax rate.
The Reporters’ Lab is developing several pop-up apps that will deliver fact-checking in real time. The apps will include:
- FactStream, which will display relevant fact-checks on mobile devices during a live event. The first version, to be tested this month during the State of the Union address Jan. 30, will be a “manual” version that will rely on fact-checkers. When they hear a claim that they’ve checked before, the fact-checkers will compose a message containing the URL of the fact-check or a brief note about the claim. That message will appear in the FactStream app on a phone or tablet.
- FactStream TV, which will use platforms such as Chromecast or Apple TV for similar pop-up apps on television. The initial versions will also be manual, enabling fact-checkers to trigger the notifications.
Another project, Truth Goggles, will be a plug-in for a web browser that will automatically scan a page for content that users should think about more carefully. Schultz, who developed a prototype of Truth Goggles as a grad student at the MIT Media Lab, will use the app to experiment with different ways to present accurate information and help determine which methods are most valuable for readers.
The second phase of the pop-up apps will take the human fact-checker out of the equation. For live events, the apps will rely on voice-to-text software and then match with the database of articles marked with ClaimReview.
The future apps will also need natural language processing (NLP) abilities. This is perhaps the biggest challenge because NLP is necessary to reflect the complexities of the English language.
“Human brains are very good at [NLP], and we’re pretty much the only ones,” says Chris Guess, the Reporters’ Lab’s chief technologist for Share the Facts and the Tech & Check Co-op. Programming a computer to understand negation or doublespeak, for instance, is extremely difficult.
Another challenge comes from the fact that there are few published fact-checks relative to all of the claims made in conversation or articles. “The likelihood of getting a match to the 10,000 or so stored fact-checks will be low,” says Bill Adair, director of the Reporters’ Lab.
Ideally, computers will eventually research and write the fact checks, too. “The ultimate goal would be that it could pull various pieces of information out, use that context awareness to do its own research into various data pools across the world, and create unique and new fact-checks,” Guess says.
The Reporters’ Lab is also developing tools that can help human fact-checkers. The first such tool uses ClaimBuster, an algorithm that can find claims fact-checkers might want to examine, to scan transcripts of newscasts and public events and identify checkable claims.
“These are really hard challenges,” Schultz says. “But there are ways to come up with creative ways around them.”
Knight Foundation, Facebook and Craig Newmark provide funding to launch Duke Tech & Check Cooperative
New automated fact-checking project will build apps and coordinate with other researchers around the world
By Bill Adair - September 25, 2017
The John S. and James L. Knight Foundation, the Facebook Journalism Project and the Craig Newmark Foundation are awarding grants to the Duke University Reporters’ Lab for a $1.2 million project to automate fact-checking.
The Duke Tech & Check Cooperative will bring together teams from universities and the Internet Archive to develop new ways to automate fact-checking and broaden the audience for this important new form of journalism.
During the two-year project, computer scientists and journalism faculty from Duke, the University of Texas at Arlington and Cal Poly-San Luis Obispo will build a variety of new tools and apps. Some will help journalists with time-consuming reporting tasks, such as mining transcripts, media streams and social feeds for the most important factual claims. Others will provide instant pop-up fact-checking during live events.
The Reporters’ Lab will also coordinate and share its automation efforts with journalists and computer scientists across the country and around the world. The Tech & Check Cooperative will connect the leaders of similar projects through its relationships with the International Fact-Checking Network, the global association of fact-checkers, and awardees of Knight Prototype Fund grants to address misinformation. The Lab will host an annual meeting and will hold regular video conferences.
Knight has provided $800,000 for the project and the Facebook Journalism Project has contributed $200,000. The Newmark Foundation has pledged $200,000.
“A multitude of people and solutions are required to tackle the problem of misinformation in the digital age. The Reporters’ Lab is tackling the issue through an effective, multi-pronged approach, bringing together a network of journalists and technologists to build new tools that will promote the flow of accurate news, while strengthening their connections with major technology companies,” said Jennifer Preston, the vice president for journalism at Knight Foundation.
“The Duke Tech & Check Cooperative will tap into the power of technology to improve and expand fact-checking on a global scale,” said Campbell Brown, head of news partnerships at Facebook. “This important initiative will bring together some of the most respected experts in the industry along with new digital innovations to create practical and efficient tools for journalists and newsrooms.”
“News consumers like me want the truth, which requires more and better fact-checking,” said Newmark, founder of craigslist and the Craig Newmark Foundation. “The Duke University Tech & Check Cooperative will soon become a vital part of the fact-checking network, and I’m excited to work with them to help build a system of information we can trust.”
The Tech & Check Cooperative will incorporate technology and content developed in Share the Facts, a Duke Reporters’ Lab partnership with the Google News Lab and Jigsaw. Share the Facts provides a way for the world’s fact-checkers to identify their articles for search engines and apps.
“Automated fact-checking is no longer just a dream,” said Bill Adair, the Knight Professor of the Practice of Journalism and Public Policy at Duke and the leader of the Tech & Check Cooperative. “Advances in artificial intelligence will soon make it possible to provide people with real-time information about what’s true and what’s not.”
Partners in the Tech & Check Cooperative include:
● The University of Texas at Arlington, which has developed ClaimBuster, a tool that can mine lengthy transcripts for claims that fact-checkers might want to examine.
● The Internet Archive, which will help develop a “Talking Point Tracker” that will identify factual claims that are used repeatedly by politicians and pundits.
● Truth Goggles, a project created by developer Dan Schultz and the Bad Idea Factory to provide pop-up fact-checking for articles on the web.
● Digital Democracy, an initiative of the Institute for Advanced Technology and Public Policy at Cal Poly-San Luis Obispo, which will develop ways to identify factual claims from video of legislative proceedings in California.
About the John S. and James L. Knight Foundation
Knight Foundation is a national foundation with strong local roots. We invest in journalism, in the arts, and in the success of cities where brothers John S. and James L. Knight once published newspapers. Our goal is to foster informed and engaged communities, which we believe are essential for a healthy democracy. For more, visit knightfoundation.org.
About Facebook
Founded in 2004, Facebook’s mission is to give people the power to build community and bring the world closer together. People use Facebook to stay connected with friends and family, to discover what’s going on in the world, and to share and express what matters to them.
The Facebook Journalism Project was created in January 2017 to establish stronger ties between Facebook and the news industry. FJP focuses on three pillars: collaborative development of new products; tools and trainings for journalists; and tools and trainings for people.
About Craig Newmark
Craig Newmark is a Web pioneer, philanthropist, and leading advocate on behalf of trustworthy journalism, voting rights, veterans and military families, and other civic and social justice causes. In 2017, he became a founding funder and executive committee member of the News Integrity Initiative, administered by the CUNY Graduate School of Journalism, which seeks to advance news literacy and increase trust in journalism.
About the Reporters’ Lab
The Duke Reporters’ Lab is a project of the DeWitt Wallace Center for Media & Democracy at the Sanford School of Public Policy. The Lab conducts research into fact-checking and explores how automation can be used to help journalists and broaden audiences for their work.