← Thinking in the Age of AI

⏱ About 15 min✦ 15 XP

Information Audit

An information audit is exactly what it sounds like: a systematic, methodical evaluation of a set of information items. Professional journalists do this before publishing. Researchers do this before drawing conclusions. Policy analysts do this before making recommendations. In this lesson, you will do it too — applying every technique you have learned in this module to a structured set of tasks.

The goal is not just to identify what is true and what is false. The goal is to develop the habit of evaluating automatically — so that when you encounter a new claim in the wild, your instinct is to ask who made it, what evidence supports it, whether independent sources confirm it, and what the source's incentives are. That habit, applied consistently, is one of the most valuable things a person can possess in the information age.

What You Will Be Auditing

During this audit, you will evaluate three categories of information: a social media claim (text or image), a website publishing health or science content, and an AI-generated answer to a factual question. For each one, you will work through a structured evaluation protocol and produce a brief written verdict. The protocol uses the skills from lessons one through eight of this module.

The Evaluation Protocol at a Glance

For every item you audit: (1) Identify the claim. (2) Evaluate the source — who made it, what is their expertise and track record, what are their incentives? (3) Assess the evidence — what type is it, and how strong? (4) Check for misinformation techniques — emotional triggering, misleading headlines, fabricated attribution. (5) Cross-check — find at least two independent sources. (6) Apply lateral reading — search externally for what others say about the source. (7) Write a verdict: credible, credible with caveats, unverified, or misleading.

Full Information Audit — Three Items

  1. ITEM 1 — Social Media Claim
  2. Find a post on any social media platform making a factual claim about health, science, politics, or current events. Screenshot or copy it.
  3. Step 1A: Write out the specific claim being made.
  4. Step 1B: Identify any emotional language or visual elements designed to provoke a reaction.
  5. Step 1C: Search for the claim on at least two fact-checking sites (Snopes, PolitiFact, FactCheck.org, AFP Fact Check).
  6. Step 1D: Search for the claim on two independent news sources that conducted their own reporting.
  7. Step 1E: Write your verdict in one sentence, with one sentence of reasoning.
  8. ITEM 2 — Health or Science Website
  9. Find a website making a claim about a health supplement, diet, or environmental issue.
  10. Step 2A: Do NOT read deeply into the site. Spend 30 seconds maximum on the site itself.
  11. Step 2B: Perform lateral reading: search the organization name plus 'funding,' 'credibility,' and 'criticism.'
  12. Step 2C: Find the Wikipedia article for the organization if one exists and read the references section.
  13. Step 2D: Identify whether the site cites primary research, and if so, look up one of those studies to verify it supports the claim as stated.
  14. Step 2E: Write your verdict in one sentence, with one sentence of reasoning.
  15. ITEM 3 — AI-Generated Answer
  16. Ask an AI chatbot: 'What are the three most important proven health benefits of [a common food or supplement of your choice]?' Record the response exactly.
  17. Step 3A: Identify every specific factual claim in the AI's answer (numbers, percentages, study names, expert names).
  18. Step 3B: Look up each specific factual claim in a non-AI source — a peer-reviewed database, a government health agency website, or a major medical institution's website.
  19. Step 3C: Mark each claim: Verified, Partially accurate, Unverified, or Incorrect.
  20. Step 3D: Write a two-sentence evaluation of the AI's reliability on this specific topic.
  21. FINAL REFLECTION
  22. After completing all three audits, write one paragraph answering: What surprised you most about this process? Which type of information was hardest to evaluate, and why? What habit will you commit to practicing before you share or act on information in the future?

Calibrating Your Confidence

A key outcome of information auditing is not just identifying false claims — it is calibrating your confidence appropriately. After evaluating a claim, you should be able to say something more precise than 'I believe it' or 'I doubt it.' You should be able to say: 'This claim is well-supported by two independent peer-reviewed studies and corroborated by a government health agency, so I have high confidence.' Or: 'This claim appears in one unverified social media post with no independent corroboration, so my confidence is very low.' Calibrated confidence is different from certainty — it is being right about how sure you should be.

The Verdict Ladder

Use these four verdict levels to describe your audit conclusions with precision. Credible: supported by multiple independent, high-quality sources with no major red flags. Credible with caveats: generally supported but with noted limitations such as preliminary evidence or a conflict of interest. Unverified: neither confirmed nor debunked by independent sources — cannot be relied upon without further investigation. Misleading: the claim contains technically true elements arranged to create a false impression, or has been definitively debunked.

During your audit of a health website, you find that all of its research citations trace back to studies funded exclusively by the company selling the product. What verdict is most appropriate at this stage?

What does 'calibrated confidence' mean in the context of information evaluation?

← Back to Thinking in the Age of AI