Audit what AI says about you: a repeatable method

To audit what AI says about you, write a fixed set of questions a stranger would ask, run each one several times on several assistants under controlled conditions, log every answer with its sources, and score each claim for accuracy and for how often it appears. The result is not an opinion about your reputation. It is a table of claims, each with a rate, a source and a next action.

A single search for your own name tells you almost nothing, because the same question can return a different answer the next day. An audit uses a sample large enough to separate a claim that appears every time from one that appeared once, so you do not spend a week chasing noise.

This guide is the method only: how to build the query set, how many runs to do, what to log, how to score and what each finding calls for. It does not explain how assistants source facts or how to file a correction, which are separate topics. Where the method touches a product feature, it follows the provider's own help page, linked where it is used. The rest rests on how repeated sampling works and on our experience on files, written as tendencies and never as promises.

How to audit what AI says about you: what the method measures

An audit measures claims, not impressions. A claim is one checkable statement in an answer, such as "founded a company in a given year", "holds a given degree" or "has been criticized for a given thing". One answer usually contains five to fifteen claims, and each one gets its own row in the log.

Related readAI Answers About Job Candidates: Audit and Fix

For each claim the audit produces three measurements. The first is accuracy: correct, outdated, wrong, unsupported or about someone else. The second is the appearance rate: in how many of your runs the claim showed up. The third is the source: which page, if any, the assistant cited for it. Those three together tell you whether a claim is a problem, how big it is and where the fix probably sits.

The audit also measures what is missing. If assistants never mention your current role or your best-known work, that is a finding even though no sentence is false. The log therefore holds a second list: facts you expect to see and do not.

Run this audit on yourself, on your own company, or on a client who has asked you to. It is not a way to collect information about a private individual.

Build a query set in six families

A good query set copies the questions people actually ask, in their wording, and covers different intents. Write each prompt once, with square-bracket placeholders, and then fix it. If you change the wording between runs you are no longer measuring the assistant, you are measuring your own phrasing.

Use three to five prompts per family, with variations in wording that a real person would use: with and without a job title, with a city, with a company name, with a nickname or a common misspelling. That gives 12 to 30 prompts. The workload section below shows why more than 30 becomes unmanageable.

For a company, use brand equivalents: who is [company], what does it sell, what do customers say, how does it compare with [competitor], which providers would you recommend for [need]. The decision family matters most here, because a buyer asking for recommendations never types your name.

Related readAI answers about doctors and lawyers: what to fix

Keep a separate pressure set

Leading questions, such as "Is [full name] a fraud?", push an assistant toward the premise. Run them once, because a hostile reader will ask them, but score them in a separate table and never pool them with the neutral set.

Hold the conditions steady so runs can be compared

Two runs only compare if the conditions match. The audit therefore fixes the variables that are under your control and records the ones that are not.

Fix these four. First, the assistant and its mode: the same product in its search-enabled mode and its plain mode can behave differently, so name the mode in the log. Second, memory and personalization: OpenAI's memory page describes saved memories, a setting that lets ChatGPT draw on past conversations, and Temporary Chat, which it says does not use existing memories or create new ones. An assistant that remembers you will answer with your own history rather than what a stranger would see, so check the equivalent settings of every assistant you use. Third, the account state: a logged-out or fresh account is the closest to a stranger. Fourth, the session: start a new conversation for every run, so that the previous answer does not shape the next.

Record these three, which you cannot fully control: date and time, approximate location and language. OpenAI's ChatGPT search page says search uses general location inferred from your IP address, so if your audience is in another country, run part of the audit there and note it.

A sensible minimum of surfaces is two chat assistants, one answer engine that shows citations, and the AI summary on a regular search results page. Google's AI Overviews page says an overview is not shown for every search, so when none appears, log "no summary shown" rather than skipping the row. Where a product shows citations, such as inline links or a Sources list, copy them into the log.

Related readWrong company information in ChatGPT: how to fix

Repeat every prompt, because the answer is a draw, not a fact

An assistant generates its answer fresh each time, so the same prompt can return different wording, different claims and different sources. That is why the audit repeats each prompt. A claim seen once is a data point. A claim seen in most runs is a property of how that assistant describes you.

Run each prompt at least three times per assistant, spread over at least two days. The spread protects you from a temporary quirk, such as one retrieval result that happened to rank high that morning. Three runs is a floor for spotting patterns, not a precise instrument. If a claim matters a lot, add runs until the picture is stable.

Decide the size before you start: prompts times assistants times runs equals answers to read, log and score.

Full: 24 prompts, 3 assistants, 3 runs216 answers

Core: 12 prompts, 3 assistants, 3 runs108 answers

Quick: 6 prompts, 2 assistants, 3 runs36 answers

Computed from the stated assumptions. At 2 minutes per answer, the sizes come to about 7.2 hours, 3.6 hours and 1.2 hours. Widths are scaled to the largest value.

For a first baseline the core audit is usually right: it covers every family and gives nine answers per prompt. Use the quick audit for re-tests, limited to the prompts where you found problems, and the full audit before a funding round or launch.

Log each answer so a stranger could rerun it

The log is the audit. Without it you have impressions, and impressions drift: after reading thirty answers you will remember the three that upset you. A spreadsheet with one row per answer and one row per claim is enough. Keep two sheets, one for answers and one for claims, linked by an answer number.

The procedure for a single session keeps the log honest. Do not score while you run. Capture first, judge later, because judging as you go makes you change the next prompt.

Related readGoogle AI Mode reputation: audit what it says

1. Open a fresh sessionNew conversation, memory and history settings set as recorded, logged-out or fresh account if that is your protocol.
2. Paste the prompt unchangedCopy it from your prompt sheet. Never retype it, and do not add "please" or a clarification halfway.
3. Capture the whole answerPaste the text into the Answers sheet and save a screenshot. Copy every citation link in the order shown.
4. Do not follow upDo not ask "are you sure" or "where did you get that". A follow-up creates a different conversation and contaminates the run. Follow-ups belong in the tracing stage.
5. Close and repeatClose the session and start the next run in a new one. Stop for the day when you reach your planned count.
6. Split into claims laterOn a different day, read each answer and write one row per claim. Then score.

Keep the log private: publishing a screenshot of a wrong claim about yourself can give it one more page on the web.

Score every claim, then score how often it appears

Scoring has two axes, and a claim needs both. Accuracy tells you whether the statement is right. Frequency tells you how much of the picture it makes up. Together they decide priority.

For accuracy, use five verdicts. Correct means it matches a public document. Outdated means it was once true. Wrong means it is false and you can prove it. Unsupported means you can find no source either way. Other person means it is true of someone else. Add a framing note when a correct fact is presented with a loaded emphasis. Framing is not an error, but it is a finding.

For frequency, count the share of runs in which the claim appears, per assistant and in total. As a working convention, not a standard, we group claims into three bands: stable when a claim appears in about two thirds of runs or more, recurring when it appears in roughly a third to two thirds, and sporadic when it appears in fewer than a third. The bands are only a way of deciding what to look at first. Change the cut-offs if your run count is larger, and keep them the same across audits so that results compare.

Two rules stop the scoring from misleading you. First, severity overrides frequency. A sporadic claim that you committed a crime you did not commit, or that you hold a licence you do not hold, still gets documented the first time it appears, and it is not ignored because it showed up once. Second, an absence is scored like a claim: write "current role not mentioned" as a row, with its own rate. If your correct current role appears in a third of runs, that is the number to improve.

Related readWhat Siri, Alexa and Google Assistant say about you

A worked scoring example with invented numbers

This scenario is invented and uses a generic subject: an independent consultant. We ran one prompt, "Who is [full name], and what do they do?", on three assistants, three runs each, so nine answers. The rates below are computed from those assumed counts.

The outdated employer is the biggest problem by volume: two thirds of answers tell a stranger that the subject works somewhere they left. The missing consultancy name is the second problem and is linked to the first, because the stale page is probably crowding out the current one. The lawsuit claim appeared once, but it concerns another person and it is serious, so it is documented the first time, re-tested within a week and reported if it returns. The unsupported award is watched and not chased: with nothing to disprove, the better response is to publish the real list of achievements.

With one run, the answer would have looked either fine or alarming, depending on the draw. Nine answers show the pattern.

What to do with each finding

The verdict and the band together give the next action. This table is the decision aid for the whole audit. It names a route in general terms; how each route works in detail is beyond this guide.

Tracing sits between scoring and action. For each wrong, outdated or other-person claim, open the cited page and compare it with the claim. If the page says it, the problem lives on the web. If the page says something milder or different, the problem lives in the reading. If nothing is cited, the claim may come from memory or may be invented, and you check whether it recurs across assistants. A repeat on one assistant only points to that assistant's behavior; a repeat on all of them points to a widely available source or to something most systems learned. Sourcing mechanics and reporting routes are topics of their own and are not repeated here.

Related readWhat Microsoft Copilot says about you: check and correct

One boundary applies to every row. The audit finds inaccurate, outdated and unlawful material. It is never a reason to hide accurate, lawful reporting about real wrongdoing, and no row above is meant for that. If an answer repeats something true and well sourced, the honest options are context, later developments and your own accurate account.

Mistakes that make an audit mislead you

Most bad audits fail the same few ways. Each one pushes the result in a predictable direction.

• One run per prompt. You will overreact to a bad draw or relax at a good one. The fix is the three-run floor.
• Asking the assistant to describe its sources. A description of where it got something is another generated answer. Check the citations the product actually shows and open them.
• Following up inside the session. "Are you sure?" often changes the answer, which tells you about the assistant's reaction to doubt, not about what it says by default.
• Using only your full name. People also search with a role, a company, a city or a misspelling. Without these variations the audit has blind spots.
• Mixing neutral and leading prompts. The results answer different questions and cannot be pooled.
• Trying to fix with pressure tactics. Fake profiles, paid placements or mass reporting are against platform rules and tend to produce a second problem. The audit exists to find what is true and strengthen it.

Worry also biases verdicts: ambiguous wording gets read as false. Mark a claim unsupported, not wrong, when you cannot prove it either way.

A realistic timeline from baseline to re-test

Timing depends on how fast the underlying pages change and on the assistants, neither of which you control. The schedule below is a planning sketch from experience, written as ranges. It is not a promise and no assistant publishes a timetable we could confirm.

A re-test only means something against the same prompt set and the same conditions as the baseline. Compare rates, not single answers. If the stale employer fell from 6 of 9 to 2 of 9, that is progress. If it fell from 6 to 5, it is within the normal wobble and you should not declare success.

Next step: run the baseline this week

Start small: six prompts, one per family, two sheets, and a quick audit of 36 answers over three days. That is enough to learn whether your picture is mostly fine, mostly missing or actually wrong.

When the baseline is done, decide whether you will repeat it yourself every quarter or hand it over. If your answers change often, a monthly monitoring routine keeps the log current without your time.

Related readChatGPT wrong information about my product: how to fix

One last note on the provider pages. Google's page also warns that AI Overviews can make mistakes and tells readers to check important details against the linked sources, which is what the log is for. Provider pages change, so reread them before each audit and note any change in your conditions.

Questions people ask

How many prompts do I need?

Twelve is a workable minimum for a person, one or two per intent family plus variations in wording. Thirty is a practical ceiling, because every prompt multiplies by assistants and runs. If you only have time for six, cover each family once and accept that the picture is coarse. Add prompts later, but start a new baseline when you do.

Can I just ask the assistant what it knows about me?

You can, and that is one of the prompts. But one question, once, is a single draw. Asking the assistant to list its sources or confirm its own accuracy gives you another generated answer, not a record. Use the citations the product shows, open each one, and compare them with the claim.

Should I turn memory off before auditing?

Usually yes, or use a fresh or logged-out session. An assistant that remembers earlier conversations may answer from what you told it rather than from what it would tell a stranger. OpenAI's memory page says Temporary Chat does not use existing memories, which makes it one option. Check the settings in each product, since names and defaults differ, and write down what you chose so the next audit uses the same conditions.

Can I audit someone else with this method?

Only with their knowledge, for example a client or a colleague who asked. The method is designed to show what a stranger would learn about you or about your organization. It is not a way to build a profile of a private individual, and prompts about someone's address, family or health are outside its purpose.

How often should I repeat the audit?

A quarterly core audit is a reasonable default for a person with a stable profile. Run a quick re-test about six to eight weeks after you change source pages. Increase the frequency around events that draw attention, such as a launch, a funding round or press coverage, when what people find matters most.