FactRank is a free tool that automatically filters out statements from Dutch-language texts worthy of fact-checking.
The Flemish Journalism Fund (FJF) was the innovation incubator of Journalismfund.eu, supported by the government of Flanders (the northern, Dutch speaking part of Belgium). Throughout 2019 and 2020 the FJF supported 11 projects that innovated and diversified journalism in Flanders. This is one of them.
FactRank is a free tool that automatically filters out statements worthy of fact-checking from reports of parliamentary debates and other Dutch-language texts. FactRank currently searches the reports of the Flemish, Belgian and Dutch parliaments, the tweets of Flemish and Dutch members of parliament, and the subtitles of the VRT news programmes Terzake and De Afspraak.
FactRank ranks statements in order of “check-worthiness”. The highest scores go to statements that are both factual and socially relevant.
FactRank also allows you to enter your own text and have it searched for statements that need to be fact-checked. The text is then split up into individual statements whose check-worthiness can be assessed.
FactRank also checks whether a statement can be matched with other fact-checks that have already been published on Knack, Factcheck.Vlaanderen and Nieuwscheckers.
Fact-checking tools like FactRank were already available for texts written in English (e.g. Claimbuster). Now there’s a tool for Dutch-language users as well.
Deep learning and human knowledge
FactRank uses a deep-learning model that has been trained on thousands of statements that were manually labelled by Leiden University students. These students were trained by experts in journalism and fact-checking.
The creators have also ensured that FactRank can continue to improve by collecting feedback from users. This feedback, in combination with periodic re-training of the deep-learning model, ensures continuous improvement.
This project is the result of Flemish–Dutch teamwork. Led by freelance journalist Dr. Jan Jagers (fact-checker for Knack Magazine and lecturer in journalism at the Vrije Universiteit Brussel), the project team included Prof. Peter Burger and Alexander Pleijter (Nieuwscheckers.nl, Leiden University), Prof. Peter Van Aelst (Antwerp University), Prof. Bettina Berendt (KU Leuven and TU Berlin) and machine learning engineer Rafael Hautekiet.
In their own words
Jan Jagers from FactRank explains the project in his own words:
What exactly did the project involve?
With FactRank, we have developed a free tool that automatically filters out statements from Dutch texts that are worth checking.
Thanks to a cross-border collaboration of professionals from various universities and fact-checkers from both Flanders and the Netherlands, we were able to build on an existing prototype, improving its accuracy and broadening its scope. We developed a codebook, manually labelled thousands of statements, trained a machine-learning system to predict the likelihood of check-worthy claims, and devised a system to refine the tool even further by learning from the feedback that users provide.
Journalists and anyone else who wants to scan the public debate for factual and questionable claims can use FactRank free of charge. The tool automatically flags questionable claims, which users can then check.
While the prototype only scanned Belgian parliamentary debates, FactRank now also monitors plenary debates in the Flemish Parliament and the Dutch House of Representatives, tweets from Flemish and Dutch politicians, and subtitles from the VRT news programmes Terzake and De Afspraak. We were able to find a solution for the technical issues and other problems that the addition of these other sources presented, for example, how to show the claims in their direct context.
We also made a start on claim matching, a process by which new statements are checked against a database of previously fact-checked material. If it turns out that the statement in question has already been checked, the results of the previous fact-check appear immediately.
How we created FactRank and what choices and assumptions underlie its development have been meticulously recorded in our scientific article, “FactRank: Developing Automated Claim Detection for Dutch-Language Fact-Checkers”.
What about the project are you the most satisfied with?
The multidisciplinary, cross-border collaboration.
What went less smoothly than expected, or what would you have done differently?
The multidisciplinary, cross-border collaboration. Streamlining the collaboration of a multidisciplinary team whose members are physically far apart and all have busy, complex schedules is anything but easy. It takes time, creativity, and goodwill from all parties. And it leads to results that we couldn’t achieve otherwise.
What will be the impact of this project once the subsidy period has ended? Will it continue to exist, or has it contributed or changed anything in your company or organization?
Now FactRank has to prove its value in journalistic practice. We will continue to use the tool as an efficient means to identify statements worth checking, and we hope many others will do the same. We hope to keep FactRank operational in the future, to improve it, and to add even more data sources, such as subtitles from talk shows and transcriptions of radio programmes, for example.
At the moment, we are working without structural funding and are thus looking for financial support and partners. We welcome any suggestions for collaboration or support.
What do you think is the most important thing we can learn from this project for journalism in Flanders?
Collaborations are worth the effort.