Lessons from our first experiment with pop-up fact-checking

More than 380 people tried our pop-up fact-checking during the presidential debate.

By Bill Adair – October 20, 2016 | Print this article

Several hundred people who watched the presidential debate last night got a preview of what fact-checking will look like in the future.

More than 380 beta testers of our Reporters’ Lab browser extension saw fact-checks pop onto their screens in near-real time. The pop-ups were actually tweets from PolitiFact that said when statements from Hillary Clinton or Donald Trump were true, false or somewhere in between.

The pop-ups provided instant fact-checks from PolitiFact's Twitter feed.
The pop-ups provided instant fact-checks from PolitiFact’s Twitter feed.

The “FactPopUp” tool, built by Reporters’ Lab student Gautam Hathi, used the notification feature in the Chrome browser to display the PolitiFact tweets. The pop-ups appeared over a livestream of the debate. They were generated by Aaron Sharockman, a PolitiFact editor who was tweeting links to items that had been previously published.

Viewers saw 29 pop-ups with messages such as “Trump said Clinton would double taxes for Americans. False.”

We had modest expectations. We knew that the success of the test depended on Sharockman’s speed finding previous fact-checks and the time delay for the NBC News livestream we were using. It turned out that Sharockman was quite fast and the NBC livestream was slightly delayed, which made the pop-ups appear very quickly for most users.

I was so pleased at one point that I tweeted:

Still, there are many lessons and plenty of things we can improve:

Ease of installation. We tried to make it as easy as possible, but we were relying on Chrome’s notifications and Twitter for the pop-ups, so the sign up and installation was more complicated than we wanted. Users needed a Twitter account to activate the browser extension. Next time, we probably should build it as a standalone tool that doesn’t rely on Chrome or Twitter.

Type size. This was another challenge because of our reliance on Chrome notifications. We’d like to increase the size of the type (we had to rely on the standard font size) and the length of the messages (we were limited to about 80 characters). Building it as a standalone tool would allow us to make the type larger and the fact-checks slightly longer.

Location of the pop-ups. We discovered a difference in the location of the pop-ups for Mac and PC users. On the Mac version, they appeared in the upper right, which sometimes blocked Hillary Clinton’s face when NBC was showing a split screen. On the PC version of Chrome, they appeared in the lower right, which seemed less obtrusive.

Dwell time. We had tinkered with how long the pop-ups stayed on the screen. They need to stay long enough that people can read and digest them, but not too long. When the pop-up is in the lower right, you could have a a slightly longer dwell time because it doesn’t block anyone’s face.

Other functions. It’s worth exploring other functions such as the ability to save a fact-check to read later or immediately share it with others. Two of our testers suggested synchronizing the pop-ups with a video recording so users could go back to them later or see where they occurred in the debate.

Overall, we’re quite pleased with our pioneering effort at pop-up fact-checking. It shows great promise not just for future debates but also for live events such as major speeches. We’ll be making it available in GitHub for others to try and build on.

We’d love to hear your feedback. You can write to me at bill.adair@duke.edu.