Here's a checklist to help organize your evaluation of a reference.
Annotated checklist
The goal
- What are we trying to do here?
- A succinct description of what we want to find out, e.g. "We want to find out what the actual area of Baltimore is" or whatever.
The material
- What's the material that the ref supports?
- Is it contentious or contended?
- Contentious material is material that people might take a position on for ideological reasons. Contended material is just material that another editor has made a reasonable challenge to or where sources disagree. (Whether a person was born on April 19 or April 20 might be contended but it's not an ideological issue.) If it's not contentious, the source is only going to be wrong because of failure of diligence. If it's contentious, we also have to be aware of the possibility of deliberate bias.
- Does the ref indeed support the material?
- For instance, if it supports a quote, does the quote indeed appear in the ref, and so forth.
The author
- Who is the author?
- Does the author have a Wikipedia article?
- This provides a quick-and-dirty (albeit imperfect) gauge of notability, which is not the same as veracity but is a data point. And the article and its links provide a good start in figuring out who the person is.
- What are the author's academic credentials and professional experience?
- What else has the author published?
- Is the author, or this work, cited in other reliable sources? In academic works?
- This is a rough indicator of post-publication peer review and acceptance. (Pro tips: if there is a DOI link, this will often show "where cited" for scientific publications. If it's a book, searching at Google Books (enclose the book title in quotes) can bring up cites.) WP:NJournals describes some other citation-finding tools and methods.)
- How does the author make a living?
- If he works for a salary, he has an incentive to not get fired. That means if he's a professor at an established university and that's his main source of income, he has an incentive to avoid outright mendacity, since that'll get you fired. Newspaper reporter, same thing. But other entities might encourage mendacity if it supports their mission. If he makes a living writing books or whatnot, his main financial incentive is to increase sales rather than necessarily get his facts right.
- What about reputation? Are there any big character markers?
- If he's been fired for plagiarism or indicted for perjury or successfully sued for libel or whatever, those are data points. It's probably best to be skeptical about other markers. Matters unrelated to his writing such as sex scandals might or might not indicate anything. Awards and accolades might matter some, depending on the source, but it's probably best to avoid giving much importance to man-of-the-year type logrolling. Most everyone has enemies, so a lot of people have been called a liar by somebody. If there's a pattern of disinterested people doing that then it might indicate a problem.
- Does the author have an opinion on the matter? On the continuum running from "utterly disinterested investigator or reporter" to "complete polemicist", where does this person fit?
- Even if he's utterly disinterested, he can still be inaccurate, of course. But if he's well to the right end of this continuum, that's a big red flag. It doesn't mean everything he says is inaccurate, of course, but it's an important data point. You have to be honest here – if he's a polemicist who supports your version of things, he's still a polemicist.
- Anything else?
The publication
- What is it?
- Is it a peer-reviewed scholarly journal, or a magazine (or newspaper) known to have an effective fact-checking operation?
- WP:RS, in its sections WP:SCHOLARSHIP and WP:NEWSORG, strongly (and sensibly) indicates that these are the only sources that are assumed to be reliable. (This doesn't prove that they are reliable in a given case, just that the assumption that they are is your starting point.) Everything else is up for debate. WP:RS is lengthy and there's a lot of hedging, but that's a reasonable summary.
- If not, is there any reason to believe that anyone has checked the author's facts?
- What's their circulation?
- Size doesn't prove anything, but it's a data point. The New England Journal of Medicine and the North Carolina Literary Review are both scholarly journals, but they're not equal. Ditto the New York Times and the Easton (Maryland) Gazette. A bigger operation means more resources for fact-checking, a bigger reputation to uphold, and greater likelihood of employing top-tier people.
- What about the publisher? What kind of outfit are they? What's their reputation?
- Do they have an agenda?
- What's their business incentive for veracity?
- Some magazines and newspapers rely on their reputation for veracity as part of their marketing model. If they don't pay attention to that they're eventually out of business. Others, not so much.
- Anything else?
Other
- Does the source have standing to address the material?
- If the Easton Gazette is used to reference a claim that Easton High School opened in 1989, that's one thing. If it's used to reference a claim that Hitler died in Brazil, that's probably outside their area of competence. The Lancet is a good source, but if used to support material on the origins of American Baseball, maybe not so much.
- Anything else?
Summary
Blank checklist (to copy and use)
The goal
- What are we trying to do here?
The material
- What's the material that the ref supports?
- Is it contentious or contended?
- Does the ref indeed support the material?
The author
- Who is the author?
- Does the author have a Wikipedia article?
- What are the author's academic credentials and professional experience?
- What else has the author published?
- Is the author, or this work, cited in other reliable sources? In academic works?
- How does the author make a living?
- What about reputation? Are there any big character markers?
- Does the author have an opinion on the matter? On the continuum running from "utterly disinterested investigator or reporter" to "complete polemicist", where does this person fit?
- Anything else?
The publication
- What is it?
- Is it a peer-reviewed scholarly journal, or a magazine (or newspaper) known to have an effective fact-checking operation?
- If not, is there any reason to believe that anyone has checked the author's facts?
- What's their circulation?
- What about the publisher? What kind of outfit are they? What's their reputation?
- Do they have an agenda?
- What's their business incentive for veracity?
- Anything else?
Other
- Does the source have standing to address the material?
- Anything else?
Summary
Examples
Example #1
The goal
- What are we trying to do here?
- Determine if Gregor Mendel's work was the foundation of genetics.
The material
- What's the material that the ref supports?
- Mendel demonstrated that the inheritance of certain traits in pea plants follows particular patterns, now referred to as the laws of Mendelian inheritance. Although the significance of Mendel's work was not recognized until the turn of the 20th century, the independent rediscovery of these laws formed the foundation of the modern science of genetics. As with many cites, it's sometimes not clear exactly what's being ref'd – it might be only the second sentence.
- Is it contentious or contended?
- No, neither. We're just using this an example. (If the material described Mendel himself as "the founder of genetics" or something, this could be contended, since Mendel's laws were forgotten and probably not much would be different if he'd never lived. However, the material is careful not to make this claim.)
- Does the ref indeed support the material?
- It seems to. It's a book, and it's not online, and no page ref is given so this makes it hard to judge using the limited preview that Google Books provides. The ref'd material discusses Mendel at length and the material in the article is a severe reduction and simplification of this. It looks to be a reasonable extrapolation from the ref'd material.
The author
- Who is the author?
- Peter J. Bowler. He's a historian of biological science and a professor at Queen's University Belfast.
- Does the author have a Wikipedia article?
- Yes, he does, Peter J. Bowler.
- What are the author's academic credentials and professional experience?
- He has a Ph. D. from the University of Toronto. He's worked as a teacher or professor. He's a past President of the British Society for the History of Science.
- What else has the author published?
- A number of scholarly works. His article lists some of them. He's written a textbook and a few other books. Judging just by the titles these look to be high-end general-interest books for the educated layman.
- Is the author, or this work, cited in other reliable sources? In academic works?
- Yes, some. It's listed as a ref in J. L. Heilbron's The Oxford companion to the history of modern science and some other works.
- How does the author make a living?
- He's a professor at Queens College. This is probably his primary source of income, although he also gets book royalties.
- What about reputation? Are there any big character markers?
- Don't know of any. He's a member of good standing of various scholarly bodies, according to his article.
- Does the author have an opinion on the matter? On the continuum running from "utterly disinterested investigator or reporter" to "complete polemicist", where does this person fit?
- He's an academic. There's no reason to believe that he'd have any incentive to either glorify or deprecate Mendel's role. He is a strong public voice against creationism, but 1) that shouldn't affect his regard of the standing of Mendel in particular as opposed to other figures, and 2) that's not really being polemic, that's just being a biologist, as essentially 100% of reputable biologists are opposed to creationism.
- Anything else?
- No.
The publication
- What is it?
- It's a book: Bowler, Peter J. (2003). Evolution: the history of an idea. Berkeley: University of California Press. ISBN 0-520-23693-9.
- Is it a peer-reviewed scholarly journal, or a magazine (or newspaper) known to have an effective fact-checking operation?
- No.
- If not, is there any reason to believe that anyone has checked the author's facts?
- Not sure. It's University of California Press and that's a reputable outfit. Whether they check Bowler's work or just trust him I don't know. Probably the latter.
- What's their circulation?
- Not applicable. UC Berkely is a big school and their publishing arm is also a big operation.
- What about the publisher? What kind of outfit are they? What's their reputation?
- University of California Press, Berkely. It's a highly reputable academic publisher.
- Do they have an agenda?
- No.
- What's their business incentive for veracity?
- It's high, as far as that goes, since they're an academic publisher.
- Anything else?
- No.
Other
- Does the source have standing to address the material?
- Yes, certainly. He has considerable expertise in the subject.
- Anything else?
- No.
Summary
It's an acceptable ref. It's a book, and we don't like to use books as sources, since books aren't usually fact-checked, so we are basically depending on Bowler's reputation. It appears to be excellent. There are no markers to indicate that he would have any incentive to get this wrong (quite the contrary, he has an academic reputation to uphold) and every indication that he has the competence to get it right.
Supplementary material
About fact-checking
Most large magazines employ fact-checkers. Books publishers and most newspapers don't. They employ copy editors whose main brief is fixing grammar and style. Copy editors may check facts, but only on an ad hoc basis.
When you cite a book, you are relying almost entirely on the author. Book publishers have little incentive to worry about facts since people generally buy books based on the author rather than the publisher. For this reason books are seldom very reliable sources.
When you cite a newspaper, you are also relying on the author but mainly on the publication. Rather than checking facts, newspaper editors will expect reporters to check their own facts and they'll fire them if they don't and reporters know this. Newspapers do have an incentive to worry about facts since people do generally buy newspapers based (partly) on the paper's general reputation for veracity and not on the names of particular reporters. It depends a great deal on the newspaper, of course, and business incentives to get facts right varies a lot among newspapers, and so does editorial rigor.
Journalistic entities known to have good fact-checking operations
- But don't just throw your favorite paper into this list. A good citation describing their fact-checking operation would be helpful.
- Der Spiegel. According to the Columbia Journalism Review (2010) they have about eighty full time fact-checkers – more than any other publication on the planet – and a very rigorous and well-organized operation.[1]
- The New Yorker is famous for the rigor of its editing (including fact checking). According to the Columbia Journalism Review (2010) they employ 16 fact-checkers.[1] That's a lot for small weekly that doesn't specialize in hard news. Here's a New Yorker fact-checker recounting her service: "[A] long piece... received her full-time attention for three or four weeks... Each word in the piece that has even a shred of fact clinging to it is scrutinized... After an error gets into The New Yorker, heat-seeking missiles rise off the earth and home in on the author, the fact-checker, and the editor"[2] Look at this: "Sometimes a phrase would contain hidden facts, as in 'Jane’s youngest son.' You’d have to check maternity and birth order, but you’d also have to confirm that Jane had at least three sons for one to be considered 'youngest.'"[3] (If only two, it would be "younger", get it? These people are serious.) And "[T]he material originally appeared in The New Yorker, which, along with Time magazine, originated the practice of fact checking and has for many years been famous for the reliability of its content." - Ben Yagoda[4]
- The Economist. (The Economist has a good reputation and a strong business incentive to get their facts straight since their core constituency includes high-powered decision-maker types who require reliable data and pay a lot for it, but so far we don't have a cite describing their operation, so... [citation needed]) There have been sporadic complaints about fact-checking at The Economist,[5][6] but as noted in The Atlantic, the Economist seems to employ fact checking in the book review process.[7]
- British Broadcasting Corporation. The BBC has a rigorous set of guidelines including for accuracy, impartiality, fairness, and the reporting of controversial events such as war, terror and emergencies.[8]
See also
Further reading
- Smith, Sarah Harrison (2004). The Fact Checker's Bible. Anchor. ISBN 978-0385721066.
- D'Agata, John; Fingal, Jim (February 27, 2012). The Lifespan of a Fact. W. W. Norton & Company. ISBN 978-0393340730. Retrieved February 2012.
An innovative essayist and his fact-checker do battle about the use of truth and the definition of nonfiction.
- Excerpt: John D'Agata and Jim Fingal (February 2012). "What Happened in Vegas". Harper's. Retrieved February 12, 2012.
- Review: Laura Miller (February 9, 2012). "In Defense of Fact Checking". Slate. Retrieved February 12, 2012.
- Review: Hannah Goldfield (February 9, 2012). "The Art of Fact-Checking". The New Yorker. Retrieved February 12, 2012.
References
- ^ a b Craig Silverman (April 9, 2010). "Inside the World’s Largest Fact Checking Operation". Columbia Journalism Review. Retrieved March 5, 2013.
- ^ John McPhee (February 9, 2009). "Checkpoints (abstract)". The New Yoker. Retrieved September 2, 2011.
- ^ Virginia Heffernan (August 20, 2010). "What 'Fact-Checking' Means Online". New York Times. Retrieved September 2, 2011.
- ^ Ben Yagoda (March 20, 2013). "Fact Checking 'In Cold Blood'". Slate. Retrieved August 21, 2012.
- ^ Portes, Jonathan (15 March 2012). "Not the Treasury view...: The Economist: fact check fail....". National Institute of Economic and Social Research blog. Retrieved March 5, 2013.
- ^ Cox, Wendell (April 20, 2010). "Portland Myths & The Economist’s Need for Fact Checking". Demographia Observations (blog). Retrieved March 5, 2013.
- ^ McArdle, Megan (January 24, 2011). "Why Don't Publishers Check Facts". The Atlantic. Retrieved March 5, 2013.
The Economist pens one of its customarily acerbic book reviews in which it notes an extraordinary number of basic errors
- ^ "Editorial Guidelines". BBC. Retrieved 19 December 2015.
External links
- "Fact-Checking – Fakten finden, Fehler vermeiden" [Fact-Checking - find the facts, avoid mistakes]. Netzwerk Recherche. 2010. Retrieved September 2, 2011. (German)
- Craig Silverman (April 9, 2010). "Top fact checkers and news accuracy experts gather in Germany". Regret the Error. Retrieved September 2, 2011.
- Kevin Drum (January 4, 2011). "The Mystery of Fact Checking". Mother Jones. Retrieved September 2, 2011. It's a blog, so it's not a very reliable source itself... and the comments are completely unreliable of course, but interesting and probably true. We couldn't even think about using those in an article, but this isn't one, so read away.
- Brendan I. Koerner (May 23, 2003). "Who Uses Fact Checkers, Anyway?". Slate. Retrieved September 2, 2011.
- Katherine Goldstein (June 6, 2011). "Orgasm Guaranteed". Slate. Retrieved September 2, 2011. Fact-checking at Cosmopolitan.
- Ta-Nehisi Coates (August 21, 2012). "In Praise of Fact-Checkers". The Atlantic. Retrieved August 21, 2012. Coates comments on the Niall Ferguson affair and the downfall of Newsweek, and other matters.