Using artificial intelligence to expand fact-checking

Reporters' Lab projects are harnessing machine learning to assist fact-checking journalism

By Andrew Donohue – September 16, 2019 | Print this article

As news organizations adapt to the digital age, they’re turning to artificial intelligence to help human journalists produce the content consumers need. This is especially true in fact-checking.

Because politicians often repeat claims – even after they have been debunked – AI can help hold the politicians accountable by quickly finding relevant fact-checks. This technology can also search through vast amounts of content for fact-checkable claims, saving journalists time. 

“Fact-checking is uniquely suited to the use of AI,” said Bill Adair, director of the Duke Reporters’ Lab. 

The Reporters’ Lab uses AI to build tools like Squash, a system under development that fact-checks video of politicians as they speak. The goal is to display related fact-checks on viewers’ screens in a matter of seconds.

Squash listens to what politicians say and transcribes their words, making them searchable text. It then compares that text to previously published fact-checks to look for matches. 

 “We’ve made some huge advancements in the past three years,” Adair said. “Squash has improved in accuracy since we demo’ed it at the State of the Union back in February.”

A screenshot of Squash, a fully automated fact-checking tool under development at The Reporters’ Lab.

This fall, the Squash team is refining its claim-matching technology. Its performance is inconsistent because people can make similar claims using different language.

Reporters’ Lab researchers hope to use more advanced machine learning techniques to help Squash become smarter at recognizing similar meaning even when the words don’t match. That will take time.

 “We’re dependent on technological processes improving,” Adair said. “Voice-to-text and matching algorithms are two big things we’re reliant on and those are continually improving, but still have a long way to go.” 

The Reporters’ Lab is also running user experience testing with Squash this fall to learn more about the most effective ways to display fact-checks on screens. Media researcher Jessica Mahone recently joined the lab to help develop a more effective user experience.   

 Squash could be the first step to a future where instant fact-checking is broadly available on broadcast TV, cable news and even web browsers, all thanks to the power of AI. Eventually viewers of all live political speeches and debates could benefit from Squash.

All of this is part of a larger movement within journalism starting to take advantage of AI’s possibilities. Outlets such as the Associated Press publish stories about sports and earning reports entirely written by computers. Xinhua, the Chinese’ state news agency, is experimenting with producing news broadcasts with virtual news anchors.

The Reporters’ Lab is one of the leading organizations in the world applying AI to fact-checking,  along with outlets FullFact in England and Chequeado in Argentina. The Lab’s Tech & Check Alerts, for instance, use AI to find and share checkable claims for fact-checking journalists around the country, so they do not have to spend time looking themselves. The Alerts have often shared claims that journalists have fact-checked.

It works like this: bots developed by Duke student researchers scrape Twitter posts and CNN transcripts daily to start the hunt for checkable claims. That content is fed to the ClaimBuster algorithm developed at the University of Texas, Arlington, which identifies potentially promising claims for fact-checkers. 

“Reading transcripts and watching TV looking for factual claims takes humans hours, but ClaimBuster can do it in seconds,” Adair said.

The Reporters’ Lab just last week debuted a new alert, The Best of the Bot, intended to flag the best of what the bots dig up. 

“We needed Best of the Bot because our Alerts had become so successful in finding claims that fact-checkers didn’t even have time to read them,” Adair said. “I think of it as a back-to-the-future approach. We now need a human to read the great work of the bot.”

Back to top

John Hechinger

Bloomberg editor discusses Greek life at Duke, new book on the hazards of fraternities

In a lecture at Duke University, author John Hechinger explores the uncertain future of Greek life on college campuses

By Riley Griffin – January 25, 2018 | Print this article

“Insurance companies have rated fraternities just above toxic waste.”

John Hechinger, a senior editor at Bloomberg News, addressed a room of Greek-affiliated and unaffiliated Duke undergraduates on Jan. 23, devoting a portion of his lecture to the issue of liability insurance within fraternities.

“You should know this,” he said solemnly. “Students are taking the liability on themselves. You’re likely to be named if someone dies.”

John Hechinger
John Hechinger (left) and Duke professor John Burness discuss Hechinger’s book, “True Gentlemen: The Broken Pledge of America’s Fraternities.” Photo by Bill Adair.

In September 2017, Hechinger published True Gentlemen: The Broken Pledge of America’s Fraternities, an exposé of American fraternity life. The book offers a deep dive on Sigma Alpha Epsilon, a historically white fraternity that has made headlines for sexual assault, racism and alcohol-induced deaths during hazing.

“There had never been an African-American member of SAE, and I wanted to explore that,” Hechinger said during a discussion provocatively titled, “Can Fraternities Be Saved? Can They Save Themselves?”

“Turns out at the University of Alabama, there are a whole bunch of fraternities… none of them have ever had African-American members,” he continued.

Hechinger said the lack of diversity that exists among historically white fraternities can be seen on Duke’s own campus.

“It’s an extreme example of what the Duke Chronicle is now writing about,” he said, referencing a Jan. 19 article that examined socioeconomic and geographic diversity within Duke fraternities and sororities.

But Hechinger said Duke’s Greek system is still very different from those at other universities. He identified Duke’s efforts to delay rush until the spring semester of each school year and bolster non-Greek social organizations, such as Selected Living Groups, as successful ways to create a safer campus environment.

“I think Duke does a lot of things right,” he said.

One student asked Hechinger how Duke administrators could be more transparent about fraternities. “It takes exposure to force an organization to change,” he responded. “I’d like to see all the reports of sexual assault disclosed and mapped so you can see where they happen… and know the demographics, too.”

Although national fraternities have been thrust into the limelight over scandal and death, Hechinger said fraternities are more popular than ever.

True Gentlemen“They are popular for a reason,” Hechinger said. “People really find value in them. Research shows that people who belong to fraternities believe they’ve had a better college experience and have a better sense of well-being.”

Hechinger also said fraternities provide members with powerful networks upon graduation.

Fraternity men tend to earn higher salaries after college than non-fraternity men with higher GPAs, according to Bloomberg News. They also dominate business and politics. Fraternity members make up about 76 percent of U.S. senators, 85 percent of Supreme Court justices and 85 percent of Fortune 500 executives, according to The Atlantic.

“That’s a testament to the power of networking,” Hechinger said.

For this reason, universities and fraternities have a tenuous relationship. “They infuriate, yet need, each other,” Hechinger writes in his book. “College administrators who try to crack down on fraternity misbehavior often find themselves confronting an influential, well-financed and politically connected adversary.”

Hechinger concluded his lecture by advocating for institutional change.

“If fraternities grapple with these issues, particularly the diversity issue, I think they do have a future,” he said. “I hope they focus more on values of brotherhood.”

Back to top

News labels

Reporters’ Lab study finds poor labeling on news sites

As news executives discuss efforts to improve trust, survey finds most sites don't include labels showing article type

By Rebecca Iannucci & Bill Adair – August 15, 2017 | Print this article

Editor’s note: Today we’re releasing the results of our study on article labels. We’ve published an account of our findings and a few recommendations on Poynter.org. The post below has slightly more detail on our methodology and a link to our data.

To assess how well news organizations are labeling their articles, the Duke Reporters’ Lab examined articles from 49 publications.

The study was prompted by the ongoing public debate about declining trust in the news media. Some journalists and educators have said one way to improve trust is for people to better understand the type of content they’re reading.

Online journalism provides readers with access to thousands of news sources, but readers may not understand the type of article they’re reading. Our hypothesis was that many news organizations do not label article types to indicate whether they are news, analysis, opinion or a review.

Students in the Lab analyzed 49 news organizations — 25 local newspapers and 24 national news and opinion websites. Students collected 25 articles from each organization — five articles from five different sections of the publication. For each piece of content, they note, among other things, if that article had a label, what the label said, if the label was clearly defined anywhere on the page and the size, location and color of the label.

Our findings:

* Of the 49 organizations analyzed, the Reporters’ Lab found that only 20 of them — 40 percent — labeled article type at least once in at least one section of their website.

* Of the 20 organizations that did label article types, 16 of them — 80 percent — only used labels in the opinion section. Those labels included editorial (used on 15 news sites), commentary (seven sites), column or columnist (six sites) and letters (seven sites).

* Our students found none of the publications labeled content well across every section of the website.

* Of the 29 organizations that did not label content, 13 of them were local newspapers and 16 were national organizations.

* These 29 organizations often labeled the section of the website in which an article belonged such as sports or entertainment, but they did not specify the type of article being read.

The raw data of our analysis can be viewed on this spreadsheet.

TAGGED: ,

Back to top

2016 Republican convention

Our share of ‘the take’ on brokered conventions

Headlines speculating about a floor fight when Republicans gather in Ohio next summer showcase the art of media insta-analysis.

By Mark Stencel – December 11, 2015 | Print this article

One of my favorite quadrennial election traditions — other than the once-every-four-years chance to use words like “quadrennial” — is media speculation about brokered conventions. Typically this dribbles out slowly over the course of a long, contentious primary season. But there are moments when it just spwats like a big glob of ketchup from a nearly empty bottle.

Reagan-Ford in 1976
The last time a Republican presidential nomination was still up for grabs as the party’s convention convened was nearly 40 years ago, when incumbent Gerald R. Ford survived Ronald Reagan’s challenge. (Image by White House photographer William Fitz-Patrick via Wikimedia Commons.)

News that Republican party leaders have discussed the possibility of a brokered convention produced a giant ketchup glob in the past 24 hours, offering a helpful guide to the art of the take. The take is when a large media gaggle simultaneously pounces on a news peg, producing a pile of instant analysis that usually reveals less about the subject of observation than the slant of the observers.

Some illustrations from the current feed…

The traditional conservative media establishment take:
“Krauthammer’s Take: Brokered Convention Won’t Happen Because Trump Won’t Win” (The Corner from National Review)

The traditional liberal media establishment take:
“The GOP may have a brokered convention and it’s going to be great.” (New Republic’s The Minute)

The explain-what-everyone-is-really-talking-about media take:
“What If Republicans Can’t Pick A Nominee Before Their Convention?” (FiveThirtyEight)

The New York man-on-the-street, tabloid take:
“Fearful GOP bigs brainstorm Trump alternatives” (New York Post)

The New York man-above-the-street and Washington-is-beneath-us take:
“The Republican Race Keeps Getting Weirder” (New York magazine’s Daily Intelligencer)

The local take:
“Hey Cleveland, here’s what you should know about the prospect of a brokered Republican convention” (Cleveland.com)

The Democrat just-helping-my-Republican-friends take:
“Trump Can Be Stopped: Here’s How” (former DNC executive director Mark Siegel for Huffington Post)

The it’s-all-there, plug-my-new-book take:
“The Secret Plan To Nominate Mitt Romney From The Convention Floor” (BuzzFeed’s McKay Coppins)

The jaded/grizzled “Political reporters have been yearning for a brokered convention for as long as I’ve been alive” take:
“If this news fills you with glee, it doesn’t necessarily make you a bad person…” (Washington Post Plum Line’s Happy Hour Roundup)

The jaded/grizzled, meta-media take:
Oh, wait…. You’re reading it.

TAGGED:

Back to top

Coverage of Global Fact-Checking Summit

Fact checkers from around the world convene in London

By Bill Adair – June 25, 2014 | Print this article

The Duke Reporters’ Lab was one of the co-sponsors of the Poynter Institute’s inaugural Global Fact-Checking Summit, held at the London School of Economics June 9-10. It attracted about 50 fact-checkers and academics from countries ranging from India to Chile.  Here’s some of the coverage received:

Washington Post, The global boom in political fact-checking

ABC Australia, Fact checking around the world: Pioneers Bill Adair and Glenn Kessler speak to ABC Fact Check

Africa Check Director Peter Cunliffe-Jones: Why fact-checking matters

Duke professor Bill Adair: Lessons from the Poynter global fact-checking summit

Tampa Bay Times Editor Neil Brown: 5 essential understandings of the fact-checking movement 

Poynter: Fact-checkers plan international organization

 

TAGGED: ,

Back to top

Goat

Why Digital Tools Stay in the Shed: ‘The Goat Must Be Fed’

Fueling the insatiable beast

By Mark Stencel – May 14, 2014 | Print this article

For all the talk about digital tools and a data reporting revolution in the news business, the hype doesn’t match the reality in most American newsrooms.

That’s what we heard when the Duke Reporters’ Lab set out to understand why so many news staffs have such a difficult time figuring out how to open these digital toolboxes — even when peers at other organizations have shown what even one data-savvy journalist on staff can accomplish.

The resulting report, published today, got its title from an answer we heard in an interview with Jim Farley, the recently retired news leader at WTOP-FM in Washington, D.C., one of the best-staffed and most successful radio news operations in the country.

“We’re live and local, 24/7, 365,” Farley told us. “The goat must be fed.”

It turns out the John S. and James L. Knight Foundation — one of the news industry’s primary funders for digital tools and training — has been asking similar questions. From the point of view of Alberto Ibargüen, Knight’s president and CEO, “the biggest failure we’ve had has been precisely at the point of adoption.”

Based on our interviews with senior editors and producers in more than 20 newsrooms, the Reporters’ Lab found that:

  • Many U.S. newsrooms are not taking advantage of the emerging low-cost digital tools that enable journalists to report and present their work in innovative ways. Editors and producers cling to familiar methods and practices even when they know better, more engaging digital alternatives are available, often for free.
  • Journalism awards and well-attended conferences create a sense that the adoption of data reporting and digital tools is broader than it really is. But there is a still significant gap between the industry’s digital haves and have-nots — particularly between big national organizations, which have been most willing to try data reporting and digital tools, and smaller local ones, which haven’t.
  • Local news leaders often cite budget, time and people as their biggest constraints. But conversations with the editors and producers we spoke to also revealed deeper issues — part infrastructure, part culture. This includes a lack of technical understanding and ability and an unwillingness to break reporting habits that could create time and space to experiment.
  • The local newsrooms that have made smart use of digital tools have leaders who are willing to make difficult trade-offs in their coverage. They prioritize stories that reveal the meaning and implications of the news over an overwhelming focus on chasing incremental developments. They also think of the work they can do with digital tools as ways to tell untold stories — not “bells and whistles.”

Many of the news leaders we spoke to said they and their staffs struggle with the trade-offs this work requires of them — especially when it means cutting back on what were once core elements of their routine news coverage. “We have to be really careful picking our spots — what we’re going to do and not going to do,” said Marty Kaiser, editor and senior vice president of the Milwaukee Journal Sentinel, where he has made data reporting and digital tools a priority.

The report was written by digital journalist Mark Stencel; Bill Adair, the Knight Chair for Computational Journalism at Duke; and Prashanth Kamalakanthan, a student researcher in the Reporters’ Lab.

The full report is available at GoatMustBeFed.com.

TAGGED:

Back to top

What’s next for the Reporters’ Lab

The sign should say "Under New Management." Our first task: figure out whether the apostrophe goes before or after the "s"

By Bill Adair – October 29, 2013 | Print this article

If journalism is in the doldrums, you wouldn’t know it from the Online News Association conference in Atlanta last weekend.

The sold-out conference offered a dizzying array of great panels and a midway that lived up to its name. Vendors ranging from Google to the Knight Foundation showcased a wide range of new digital tools for journalists. Matt Waite flew his drone.

The conference was a reminder that we’re at a moment of reinvention in journalism when we can radically improve how we tell stories and inform people. And that is our mission for the Reporters’ Lab.

I took over the lab when I became the Knight Chair at Duke a few months ago. It’s been dormant while I focused on teaching my fall classes, but now that the semester is well underway, I’ve got several projects underway. You’ll be hearing about them in the next few months.

I inherited the lab from my predecessor Sarah Cohen, a talented colleague I know from our days at the St. Petersburg Times. Sarah created the lab and used it to develop great tools for journalists.

I’ll be continuing that mission and broadening the focus. As the founder of PolitiFact, I’ve long been interested in developing new story forms. In a TED speech last year, I said it was time to blow up the news story and experiment with new forms.

We’ll be doing that in the Reporters’ Lab (although we will make sure the explosions don’t damage the Sanford building).

I’ve got some a veteran journalist and some talented students to help with our new mission:

Mark Stencel, a national leader in digital journalism who ran NPR’s website for the past four years. He’ll be writing occasional articles for our website as he explores what tools are available to journalists and what else they need.

Prashanth Kamalakanthan, a senior political science and film student at Duke who has written for the Nation, Alternet and the Duke Chronicle. Prashanth is researching digital tools and new story forms.

Aaron Krolik, a Duke electrical engineering student who has a talent for writing code and an interest in journalism. Aaron is developing our first digital project, which you’ll be hearing about very soon.

We’ll approach everything we do with a sense of curiosity and experimentation. We’ll try new things. Some will work. Some won’t. We welcome your feedback and suggestions. You can reach me at bill.adair@duke.edu.

Back to top