Build an AI Visibility Monitor for About $10 a Month

To track brand mentions in ChatGPT and Perplexity without a subscription: write 25 prompts a buyer would actually type, run each one three to five times a week against the OpenAI Responses API with web_search enabled and the Perplexity Sonar API, and store five fields per response — was the brand named, was your domain cited, how deep in the answer, how many sources it pulled, what tone. That is about 120 lines of Python and roughly $10 a month in API spend at the light setting.

It gets you a trend line and a citation log. It does not get you prompt volume, Google AI Overviews, or a number you can trust at n=5. All three of those are below, with the arithmetic.

One disclosure first. My Search Console shows 4,553 impressions across 85 days and none of those queries are about AI visibility tracking, so this post is not backed by demand I can prove. It exists because I built the monitor for my own site.

What the paid tools actually measure

I checked before claiming a gap. Most of the feature list is easy to replicate; one item is not.

The part that is genuinely hard to replicate

Prompt volume. Profound's Prompt Volumes page describes processing "billions of near real-time data signals" without stating the collection mechanism. The Register quotes a Profound sales email claiming access to "150+ million real user conversations," described as opt-in clickstream where users automatically share their ChatGPT conversations. Nick Lafferty's breakdown describes a double opt-in panel contributing hundreds of millions of prompts a month across ten regions.

You cannot build that. There is no API for what strangers type into ChatGPT. Every DIY monitor answers "does the brand show up for the prompts I chose," a strictly weaker question than "does the brand show up for the prompts people ask."

Engine coverage you cannot reach

Google AI Overviews and AI Mode have no public API. Per Zapier's tool roundup, Otterly covers AI Overviews, ChatGPT, Perplexity and Copilot; Peec covers ChatGPT, Perplexity and AI Overviews at baseline with Gemini, Claude, Grok and others as paid add-ons; Profound reaches ten engines but gates AI Overviews behind enterprise. The tools get AI Overviews by scraping SERPs at scale. If AI Overviews is where your category lives, stop reading and buy something.

Share of voice is not the moat

I expected a gap here and did not find one. Share of voice, as these tools report it, counts how often each brand in a named competitor set appears across the same response corpus. Once you have the responses, that is a GROUP BY. The hard part was never the panel UI — it is knowing the right competitor set and the right prompts, and you supply both either way. Sentiment is similar: one classification call per mention, cheap to run and expensive to trust.

The prompt set is the whole product

Everything downstream is plumbing. The prompt set decides whether the numbers mean anything, and no script writes it for you.

Where mine came from

Three sources, in descending order of usefulness. Sales calls — the literal sentence a prospect used before they knew my name. Existing ranking queries, where I already have position data and can compare classic search against answer engines directly, which is the comparison I unpack in AI SEO vs traditional SEO. And competitor phrasings: "X vs Y" and "alternatives to X".

What I threw out: anything containing my own brand name. "What does Santosh Paudel do" is a vanity prompt — the model finds me because I am in the prompt. My monitor tracks zero branded prompts.

Audit what is on the site before measuring whether machines quote it, or you will spend three months watching a flat line caused by thin pages rather than by retrieval. That is the order I use in a content audit: fix what exists, then instrument it. I cut this site from 267 published posts to 103 for exactly that reason.

Ten to twenty-five non-branded prompts is enough to start. Every prompt you add multiplies against runs and engines in the cost model below.

The monitor

Stdlib only — urllib, sqlite3, json, math. No SDK, no framework. Set OPENAI_API_KEY and PERPLEXITY_API_KEY, put one prompt per line in prompts.txt, and fire it from cron on three separate days a week.

#!/usr/bin/env python3
"""DIY AI visibility monitor. Python 3.11+, stdlib only.

  python monitor.py run        # one sampling pass -> visibility.db
  python monitor.py report     # mention rate per engine, with Wilson bounds
  python monitor.py selfcheck  # offline, no API keys needed
"""
import json, math, os, sqlite3, sys, urllib.request
from datetime import date

BRAND_DOMAIN = "santoshpaudel.me"
ALIASES = ["santosh paudel", "santoshpaudel"]
RUNS = 1  # samples per pass; three calls seconds apart are not three draws
DB = "visibility.db"

SCHEMA = """
CREATE TABLE IF NOT EXISTS obs (
  day TEXT, engine TEXT, prompt TEXT, run INTEGER,
  mentioned INTEGER, cited INTEGER, depth REAL,
  n_sources INTEGER, tone TEXT,
  PRIMARY KEY (day, engine, prompt, run)
);
"""

def _post(url, payload, key):
    req = urllib.request.Request(
        url, data=json.dumps(payload).encode(),
        headers={"Authorization": "Bearer " + key,
                 "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=180) as r:
        return json.load(r)

def _openai_text(d):
    parts, urls = [], []
    for item in d.get("output", []):
        for c in item.get("content", []):
            if c.get("type") == "output_text":
                parts.append(c.get("text", ""))
                urls += [a["url"] for a in c.get("annotations", [])
                         if a.get("type") == "url_citation"]
    return "".join(parts), urls

def ask_openai(prompt):
    d = _post("https://api.openai.com/v1/responses",
              {"model": "gpt-5-mini", "tools": [{"type": "web_search"}],
               "input": prompt},
              os.environ["OPENAI_API_KEY"])
    return _openai_text(d)

def ask_perplexity(prompt):
    d = _post("https://api.perplexity.ai/chat/completions",
              {"model": "sonar", "messages": [{"role": "user", "content": prompt}]},
              os.environ["PERPLEXITY_API_KEY"])
    # the API has shipped both shapes; accept either
    raw = d.get("search_results") or d.get("citations") or []
    urls = [c if isinstance(c, str) else c.get("url", "") for c in raw]
    return d["choices"][0]["message"]["content"], urls

def tone_of(text):
    """One classification call. See the noise section before believing it."""
    d = _post("https://api.openai.com/v1/responses",
              {"model": "gpt-5-mini",
               "input": "Reply with exactly one word - positive, neutral or "
                        "negative - for the tone toward " + BRAND_DOMAIN +
                        " in this passage:\n\n" + text[:4000]},
              os.environ["OPENAI_API_KEY"])
    return _openai_text(d)[0].strip().lower()[:8]

def score(text, urls):
    low = text.lower()
    hit = next((a for a in ALIASES if a in low), None)
    return {
        "mentioned": int(bool(hit)),
        # 0.0 = named in the first sentence, 1.0 = named at the very end
        "depth": round(low.index(hit) / max(len(low), 1), 3) if hit else None,
        "cited": int(any(BRAND_DOMAIN in u for u in urls)),
        "n_sources": len(urls),
    }

ENGINES = {"openai": ask_openai, "perplexity": ask_perplexity}

def run():
    db = sqlite3.connect(DB); db.executescript(SCHEMA)
    prompts = [p.strip() for p in open("prompts.txt") if p.strip()]
    today = date.today().isoformat()
    for name, fn in ENGINES.items():
        for p in prompts:
            for i in range(RUNS):
                try:
                    text, urls = fn(p)
                except Exception as e:          # one bad call must not kill the pass
                    print("skip", name, p[:40], e, file=sys.stderr); continue
                s = score(text, urls)
                s["tone"] = tone_of(text) if s["mentioned"] else None
                db.execute(
                    "INSERT OR REPLACE INTO obs VALUES (?,?,?,?,?,?,?,?,?)",
                    (today, name, p, i, s["mentioned"], s["cited"],
                     s["depth"], s["n_sources"], s["tone"]))
                db.commit()
    db.close()

def wilson(k, n, z=1.96):
    """95% CI for a proportion. Correct at k=0, where the normal approx is not."""
    if n == 0:
        return (0.0, 1.0)
    p, d = k / n, 1 + z * z / n
    c = p + z * z / (2 * n)
    m = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return ((c - m) / d, (c + m) / d)

REPORT_SQL = """
SELECT day, engine,
       SUM(mentioned) AS hits,
       COUNT(*)       AS n,
       SUM(cited)     AS cites,
       ROUND(AVG(depth), 3) AS avg_depth
FROM obs GROUP BY day, engine ORDER BY day, engine;
"""

def report():
    db = sqlite3.connect(DB); db.executescript(SCHEMA)
    print(f"{'day':<12}{'engine':<12}{'rate':>8}{'95% CI':>18}{'cites':>7}{'depth':>7}")
    for day, eng, hits, n, cites, depth in db.execute(REPORT_SQL):
        lo, hi = wilson(hits, n)
        print(f"{day:<12}{eng:<12}{hits/n:>7.1%}"
              f"{f'{lo:.1%}-{hi:.1%}':>18}{cites:>7}"
              f"{(depth if depth is not None else float('nan')):>7.2f}")
    db.close()

def selfcheck():
    lo, hi = wilson(0, 10)
    assert (round(lo, 4), round(hi, 4)) == (0.0, 0.2775), (lo, hi)
    lo, hi = wilson(3, 20)
    assert (round(lo, 3), round(hi, 3)) == (0.052, 0.360), (lo, hi)
    s = score("Ask Santosh Paudel about it.", ["https://santoshpaudel.me/blog"])
    assert s == {"mentioned": 1, "depth": 0.143, "cited": 1, "n_sources": 1}, s
    assert score("nobody relevant here", [])["mentioned"] == 0
    print("ok")

if __name__ == "__main__":
    {"run": run, "report": report, "selfcheck": selfcheck}[sys.argv[1]]()

Three decisions worth explaining

depth instead of position. LLM answers are prose, not a ranked list, so "position 3" has no meaning. Character offset over response length does: 0.08 means named in the opening sentence, 0.91 means an afterthought before the sign-off. It moves before the mention rate does, which makes it the earliest signal in the table.

INSERT OR REPLACE on a composite key. A cron job that fires twice on the same day overwrites rather than doubling your sample, which would silently narrow every confidence interval on the report.

Failures are skipped, not retried. A rate-limited call that gets retried until it succeeds biases the sample toward whatever the API was willing to answer. A dropped call just makes n smaller, and the interval widens honestly to show it.

The selfcheck command is the only test. It asserts the interval math and the scorer against known values and needs no keys — run it after any edit.

Reading the numbers without fooling yourself

This is where most DIY monitors quietly lie, including my first one. Five samples of a stochastic system is not a measurement.

Wilson bounds on the same 15% observed rate, at four sample sizes:

Read the top row twice. Ten runs with zero mentions is compatible with a true visibility rate of 27%. If you launch a campaign, see 0/10 the next week, and conclude nothing happened, you have concluded nothing.

The practical consequence: 25 prompts across 2 engines, sampled on three separate days, gives n=150 a week, which lands you near the third row for a whole-site rate — direction, monthly, not weekly. The separate days are not cosmetic. Three calls fired seconds apart hit the same index in the same state and behave closer to one draw than to three, which quietly narrows every interval on the report. Per-prompt you are at n=6 forever, so never report a per-prompt rate. Report the corpus rate and use the per-prompt rows only to decide which prompt to read by hand.

Sentiment is the noisiest column

Sentiment fires only on responses that mention you, so its n is your mention count, not your run count. At a 15% mention rate over 150 runs that is about 23 classified passages a week, from a classifier that will flip labels on rewording. I keep the column so I can read the negative rows by hand once a month and see what the model is actually saying. I do not report it as a metric.

What the API cannot see

gpt-5-mini with web_search is not the ChatGPT a customer uses. No memory, no personalization, no account history, no whatever UI experiment they are in this week. Perplexity's sonar is closer to its consumer surface, still not identical. Everything here is a proxy; the honest use of a proxy is direction over time on a fixed instrument, never a claim about what a specific person saw. Attribution for the traffic that does arrive is a separate and worse problem — the dark funnel post covers it.

What it costs, against what the tools cost

Verified list prices, and a labelled worked example for the DIY rows.

Assume 8,000 input tokens and 800 output tokens per answered call — a web-search answer pulls a lot of retrieved content into context. OpenAI's pricing page lists web search at $10.00 per 1,000 calls with retrieved content billed as input tokens, and gpt-5-mini at $0.25 in / $2.00 out per 1M tokens. Perplexity's pricing docs list sonar at $1 in / $1 out per 1M tokens plus a per-request fee of $8 per 1,000 at medium search context ($5 low, $12 high).

25 prompts x 2 engines x 3 runs = 150 calls a week. The OpenAI half (75 calls): $0.75 tool + $0.15 input + $0.12 output = $1.02. The Perplexity half (75 calls): $0.60 request fee + $0.66 tokens = $1.26. Weekly $2.28, monthly about $9.90. Push it to 40 prompts and 5 runs and the same arithmetic gives roughly $26 a month.

Prompt counts, engine counts and prices from Zapier's roundup.

Note the row that undercuts the usual pitch. DIY at 25 prompts costs about what Otterly Lite costs, and Otterly covers four engines including AI Overviews. The DIY build does not win on price at small scale. It wins past 40 prompts, where a marginal prompt costs under a dollar a month and costs a vendor a plan upgrade, and it wins on ownership: the raw responses sit in your SQLite file, so you can ask questions the dashboard has no chart for. It never wins on prompt volume. Either figure is small next to what a badly-chosen model costs on a real content workload.

When paying is the correct answer

Buy the tool if any of these holds. AI Overviews is your category's main surface, because no API will give it to you. You need prompt volume to choose what to write instead of a list drawn from sales calls. Someone other than you reads the dashboard weekly. Or the reporting has to survive procurement, where "I wrote a Python script" is a liability regardless of whether the script is better.

Build it if you are a small team with a short prompt list, you want the raw text rather than a score, and the honest alternative to a $10 script is no measurement at all rather than a $160 subscription. That is where most small brands sit — the argument is in small brands and AI answer visibility, and for engines that weight freshness, cadence moves the number more than instrumentation does: Perplexity and publishing cadence.

I run the light setting and have not bought a tool. When my mention rate gets high enough that the interval stops swallowing the trend, that is when $82.50 starts being worth it.

FAQ

How do I track brand mentions in ChatGPT and Perplexity for free?

Not free, but close to it: about $10 a month in API spend. Run a fixed prompt set against the OpenAI Responses API with web_search and the Perplexity Sonar API on a schedule, and store whether the brand was named and whether your domain was cited. The script above does it in roughly 120 lines with no dependencies. Nothing genuinely free samples the models themselves, because every sample is a paid API call.

Can I track Google AI Overviews with an API?

No. Google publishes no API for AI Overviews or AI Mode. Commercial tools get it by scraping search results at scale, which makes AI Overviews the clearest reason to pay rather than build. The Gemini API is a different surface and says nothing about what appears in an Overview.

How many times should I run each prompt?

Enough that the confidence interval is narrower than the change you want to detect. At three runs per prompt per engine, a 25-prompt set gives 150 observations a week, which supports monthly direction on a site-wide rate. It supports no per-prompt claim at all: six samples puts that 95% interval nearly forty points wide in the best case, and past fifty at any rate near 15%.

Is the OpenAI API the same as what users see in ChatGPT?

No. The API has no chat memory, no personalization, no account history, and does not run the interface experiments the consumer app runs. Treat every number from a DIY monitor as a proxy measured on a fixed instrument: good for direction over time, not a claim about what any individual user saw.

Measuring visibility before fixing what is being measured? I cut this site from 267 published posts to 103 before I cared what a model thought of them, and the trend line got easier to read afterwards. Start with a content audit or get in touch.