Skip to main content

Open in AthenaHQ

app.athenahq.ai/oracle

​
Purpose

Oracle is an automated QA and verification tool for your brand’s AI search presence. It scans the AI-generated responses collected from your tracked prompts and cross-references them against your Knowledge Base. By automatically checking what AI models (like ChatGPT, Claude, and Gemini) say about your brand against your verified facts, Oracle surfaces confirmed inaccuracies, internal contradictions within your Knowledge Base, and disagreements between different AI models. This allows you to quickly triage AI hallucinations and take action to correct the record.

​
What’s on the page

Page Header The top bar includes the page title, a brief description, a clock icon for the Run history menu, and the main Run Oracle (or Cancel) button. Run Status Banner Visible only while an Oracle scan is actively running. It displays the current stage (e.g., “Gathering responses”, “Verifying claims”), the number of responses scanned so far, and an estimated time remaining. Overview Dashboard When a run is complete (or while it’s processing but has finished verifying claims), you will see the main results dashboard:
  • Inaccuracy stat card: Shows the percentage of scanned responses that contained a confirmed inaccuracy, along with the raw count of flagged responses.
  • Responses scanned stat card: Shows the total number of AI responses analyzed in this run. If the system hit a limit and couldn’t scan every eligible response, a “Stream capped” note will appear here.
  • Priority findings: A list of up to 5 high-impact (Critical or Major) inaccuracies found during the run. Each row displays the severity, the false AI claim, and the true Knowledge Base fact. You can click “See all” to view the full queue.
  • Knowledge base issues: A collapsible section showing places where your Knowledge Base contradicts itself.
  • Model disagreements: A collapsible section showing instances where two different AI models provided conflicting answers to the same prompt.
Charts
  • Accuracy by model: A bar chart showing the percentage of sampled responses that had no confirmed findings, broken down by AI model (e.g., ChatGPT, Claude). Tooltips reveal the exact number of flagged vs. scanned responses for that model.
  • Topics AI gets wrong: A bar chart grouping confirmed findings by the model-generated Knowledge Base topic tag on the contradicted fact. (e.g., “Features”, “Pricing”).
  • Prompt topics: A bar chart grouping confirmed findings by the taxonomy topic of the prompt that originally surfaced the bad response.
Finding Review Overlay (Drilldown) Clicking a priority finding or the “See all” button opens a full-screen review flow. It displays the full detail for a single finding, including the false claim, the true fact, and cited evidence. From here, you can step through findings and triage them. AI Response Drawer (Drilldown) Clicking “View response” inside a finding opens a side drawer displaying the complete text of the AI response, with the problematic quote highlighted. Fact Editor (Drilldown) Clicking “Open fact” or “Edit fact” inside a finding opens the Knowledge Base editor, allowing you to update the underlying truth immediately without leaving Oracle.

​
What you can do here

  • Run a scan: Click the Run Oracle button in the top right to start a new analysis of the latest completed prompt run against your current Knowledge Base.
  • Cancel a scan: Click Cancel in the top right if you need to stop an in-progress run.
  • Switch between past runs: Click the clock icon (Run history) to open a dropdown of your past scans. Select any date/time to load that run’s dashboard.
  • Review a finding: Click any row in the “Priority findings” list to open the detailed review overlay.
  • Acknowledge a finding: Inside the review overlay, click Acknowledge to mark the finding as reviewed. Oracle will reconsider it on the next run.
  • Ignore a finding: Click Ignore to dismiss a finding if it is not actionable.
  • Reopen a finding: If you previously acknowledged or ignored a finding, click Reopen to move it back to Pending for review.
  • Edit a Knowledge Base fact: While reviewing a finding, click Edit fact to fix the underlying Knowledge Base entry.

​
Data shown

Oracle data is built from the latest completed prompt run and cross-referencing their claims against the verified facts you’ve added to your Knowledge Base. Findings are generated using AI to detect genuine contradictions between the two datasets.

​
Common workflows

Running a new Oracle scan
  1. Navigate to the Oracle page.
  2. Click the Run Oracle button in the top right corner.
  3. Wait for the scan to progress through its stages (Gathering responses, Comparing, Verifying, etc.).
  4. Once complete, review the Overview Dashboard to see your inaccuracy stats and prioritize findings.
Reviewing and resolving a finding
  1. On a completed run’s dashboard, click any row under Priority findings or click See all.
  2. In the review overlay, read the false AI claim alongside your true Knowledge Base fact and the cited evidence.
  3. If the fact needs updating, click Edit fact.
  4. When you are done investigating, click Acknowledge or Ignore to clear the finding from your pending queue.

​
Empty, loading, and error states

  • Setup Empty State: If your Knowledge Base has fewer than 25 facts, the page shows “Set up your knowledge base first.” You cannot run Oracle until this minimum is met.
  • First-run Explainer: If your Knowledge Base is ready but you haven’t run Oracle yet, you’ll see an explainer reading “Find what AI gets wrong about your brand” with a button to trigger your first run.
  • Loading: While switching runs or waiting for a run to finish, a skeleton placeholder of the dashboard cards is shown.
  • Errors: If a run fails, a red warning box appears explaining why. Specific reasons include:
    • Knowledge base too large/small to audit.
    • No responses to scan (if recent prompt runs yielded no readable responses).
    • This website tracks more than 2,000 live prompts (the supported scope was exceeded; contact AthenaHQ support).
    • The last scan failed (generic failure).
  • Access Error: If the billing service is unreachable, a “Couldn’t verify access” error screen appears with a “Retry” button.
  • Linked from: The main sidebar navigation, and the Knowledge Base page (which directs users here for verification flows).
  • Links to: The Knowledge Base (to fix facts/topics), and Content pages (to view remediation drafts).

​
Common support questions

Why can’t I click “Run Oracle”? The button will be disabled if you do not have enough verified facts in your Knowledge Base (you need at least 25), or if you have just added prompts and the system is still waiting for your first round of AI responses to finish collecting. What does “Stream capped” mean on my responses scanned? To protect performance and manage limits, Oracle can only pull up to a certain maximum number of responses per run (usually 100,000). This limit applies to the latest completed prompt run, not to every response collected since the last Oracle scan. If it is reached, the dashboard warns that the stream was truncated; interpret the results as incomplete and contact support. Why does a past run say “Imported history” and show no accuracy rate? Runs that were backfilled from older versions of Oracle do not have the raw response samples needed to calculate an exact accuracy percentage. They will only show the total count of findings discovered on that date.