Oracle

Analyzes AI model responses to detect and help resolve inaccuracies (discrepancies) about the user’s brand, and verifies Knowledge Base facts against external sources.

Open in AthenaHQ

app.athenahq.ai/oracle

Purpose

The Oracle page analyzes AI model responses to detect and help resolve inaccuracies (discrepancies) about your brand. It also allows you to verify your Knowledge Base facts against external sources.

By checking AI claims against your verified facts, this tool sits directly in your workflow to monitor hallucination rates, pinpoint false claims, and keep the information AI search engines provide to your customers accurate.

What’s on the page

• Empty / First-Run State: Shown before any analysis has been run, prompting you to start your first check.
• First Run In-Progress State: Shown while the very first analysis is running, displaying a spinner and an estimated time message.
• Overview Tab – Brand Search: A search bar that lets you ask what AI is saying about your brand, returning a synthesized summary with inline citations and a sentiment badge.
• Overview Tab – Metric Cards: Two KPI cards showing the current discrepancy rate (with a sparkline trend) and total open discrepancies (with a high-impact count sub-label).
• Overview Tab – High-Impact Discrepancies List: Displays the top unresolved high-impact discrepancies as compact rows. Includes a “See all” link when there are more than five.
• Overview Tab – Coverage Section: Charts showing accuracy by model, topics AI gets wrong, and other coverage analytics.
• Fact Verification Card: Shows the results of the latest fact-verification run including the total contradictions found, the number of sources ranked, a table of the top 3 contradicting sources, and links to view all sources or run a new verification.
• Run Progress Overlay (floating): A floating bottom-right card showing live progress of an in-flight accuracy check or fact-verification run, complete with cancel and dismiss controls.
• Check Responses Dialog: A modal for configuring and starting a new accuracy analysis run. It includes a date range picker, response count, and live progress display when running.
• Oracle Review Modal: A full-screen modal for reviewing and resolving discrepancies. Shows a list rail on the left and a detail card on the right containing Claim A/B selection, explanation, research verdict, and notes.
• Validation Drawer (Verification Sources): A side drawer showing the ranked list of sources that contradict Knowledge Base claims for a selected fact-verification run.
• Source Contradictions Dialog: A modal showing individual contradiction cards for a selected source domain, with previous/next navigation.
• Run History Drawer: A side drawer listing all past accuracy and fact-verification runs with their details and status.
• Verify Facts Dialog: A modal for selecting Knowledge Base pillars to include in a new fact-verification run.
• Responses Drawer: A side drawer showing full details of an individual AI response referenced from a discrepancy or search result.

High-Impact Discrepancies List

Shows the top unresolved high-impact discrepancies on the overview page.

• Index: Sequential number of the discrepancy in the list.
• Severity: Impact level of the discrepancy: High (critical), Medium (major), or Low (minor).
• Claim: The incorrect claim text found in AI responses.
• Truth: The correct claim text from the Knowledge Base.
• Resolution: If resolved, shows the resolution label (e.g. “A correct”, “B correct”, “Both valid”, “Dismissed”), who resolved it, and when.

Source Rank Table (Verification Sources)

Ranks external sources by how many contradictions they have with the Knowledge Base.

• Domain: The root domain of the contradicting source, or “Knowledge base” for internal KB inconsistencies. The info tooltip reads: “Places where your KB topics contradict each other (internal inconsistencies)”.
• Contradictions: Number of contradictions found from this source.
• Arrow: Click affordance to drill into the contradictions for this source.

Run History Table

Lists all past Oracle accuracy and fact-verification runs.

• Date: Date and time the run was created.
• Type: Whether the run was a “Response check” (accuracy) or “Fact verify” (validation).
• Responses / claims: Number of AI responses scanned (for accuracy runs) or KB claims checked (for fact-verify runs).
• Findings: Number of discrepancies or contradictions found.
• Discrepancy rate: Percentage of responses with at least one discrepancy (accuracy runs only; ”—” for fact-verify).
• Status: Current status of the run: Complete, Failed, Cancelled, or Running.

Accuracy by Model Chart Table

Shows accuracy metrics broken down by AI model.

• Model: The AI model name (e.g. ChatGPT, Claude, Gemini).
• Accuracy: Percentage of responses from this model that were accurate.
• Responses: Number of responses scanned for this model.
• Discrepancies: Number of discrepancies found for this model.

Topics AI Gets Wrong

Shows which topic categories have the most wrong claims in AI responses.

• Topic: The Knowledge Base topic/category name.
• Count / % Wrong: Either the raw count of wrong claims or the percentage of claims that are wrong, toggled by the user.

Charts

• Discrepancy Rate Sparkline: A line sparkline (X: run date, Y: discrepancy rate %) showing the trend of the discrepancy rate across recent analysis runs.
• Accuracy by Model Chart: A horizontal bar chart (X: accuracy %, Y: model name). Clicking a model shows a trend line overlay to let users drill into per-model trends over time.
• Topics AI Gets Wrong Chart: A horizontal bar chart (X: count or % wrong, Y: topic category). Surfaces which Knowledge Base topic categories have the highest error rates in AI responses.

What you can do here

• Search: Type a query in the Oracle Search bar and press Enter to search synthesized AI claims about your brand.
• Run first analysis: Click the “Run first analysis” button on the empty state to open the Check Responses dialog to configure and start your first accuracy analysis.
• Run analysis (dropdown trigger): Click the dropdown in the page header to reveal run options.
• Check responses: From the dropdown, select “Check responses” to open the dialog, set a date range, and scan AI responses for discrepancies.
• Verify facts: From the dropdown, select “Verify facts” to open the dialog, select Knowledge Base pillars, and start a verification against external sources.
• Start (Check Responses dialog): Click to start a new accuracy analysis run for the selected date range.
• Cancel (Check Responses dialog – while running): Dismisses the dialog while the run continues in the background.
• Dismiss (Check Responses dialog – while running): Closes the dialog while the run continues in the background.
• Review discrepancies (Check Responses dialog – post-run): Closes the dialog and opens the Oracle Review Modal to show unresolved discrepancies.
• Close (Check Responses dialog – post-run): Closes the dialog.
• Cancel run (Run Progress Overlay): Click “Cancel run” on the overlay to immediately stop an in-flight accuracy analysis run (no confirmation dialog).
• Hide progress (Run Progress Overlay ×): Hides the floating progress overlay for the current run session without cancelling it.
• Cancel run (Validation Run Progress Overlay): Cancels the in-flight fact-verification run.
• See all: Click next to the High-Impact list to open the Oracle Review Modal showing all unresolved discrepancies.
• Claim A / Claim B selection: In the Oracle Review Modal, select which claim is correct to enable the Resolve button.
• Mark as both valid: In the Oracle Review Modal, use the menu to resolve the discrepancy as both claims being valid.
• Dismiss: In the Oracle Review Modal, mark the discrepancy as dismissed.
• Resolve: In the Oracle Review Modal, save your resolution (Claim A, Claim B, Both valid, or Dismissed).
• Edit: On a locked card in the Oracle Review Modal (previously marked both valid or dismissed), click “Edit” to unlock it for re-resolution.
• Mark all as resolved: Use the bulk action in the Oracle Review Modal to resolve all visible discrepancies at once.
• + Add a note: In the Oracle Review Modal, expand a textarea to save a resolution note alongside your choice.
• View sources: On the HeroValidationCard, click to open the Validation Drawer and browse the ranked source list.
• Verify facts (HeroValidationCard): Click to open the Verify Facts dialog and start a new run.
• View past runs →: On the HeroValidationCard, click to open the Run History drawer.
• Verify your facts (tile – no run yet): Opens the Verify Facts dialog.
• Knowledge Base (tile – no run yet): Navigates to the Knowledge Base page.
• Start verification: In the Verify Facts dialog, starts a new run for the selected pillars.
• Cancel: Closes the Verify Facts dialog without starting a run.
• Previous / Next: Navigate between contradiction cards for a selected source in the Source Contradictions dialog.
• View response: On a DiscrepancyCard, open the Responses Drawer to see the full AI response containing the contradicting claim.
• Source URL link: On a DiscrepancyCard, click to open the source URL in a new browser tab.
• Open site: In the Source Contradictions dialog, click to open the source domain in a new tab.
• Select run row: In the Run History table, click a row to open the Oracle Review Modal (accuracy runs) or Validation Drawer (fact-verify runs).
• Run analysis (Run History drawer header): Opens the run analysis dropdown menu directly from the drawer.
• Type filter dropdown: In the Run History drawer, filter the table to show All runs, Response check runs only, or Fact verify runs only.
• Select source domain row: In the Source Rank Table, click a row to open the Source Contradictions dialog for that domain.
• Open discrepancy row: In the High-Impact list, click a row to open the Oracle Review Modal focused on that specific discrepancy.
• Citation pill / Response chip: In Oracle Search results, click to open the Responses Drawer for the referenced AI response.
• Research verdict expand: In the Oracle Review Modal, click to expand the research verdict callout to show the summary and sources.

Data shown

Information on this page is derived from AI responses collected for your tracked website over the selected date range, and Knowledge Base claims (approved facts organized by pillars/topics). Fact-verification runs check KB claims against external sources. Discrepancy resolutions and research verdicts are backed by deep external web research on disputed claims. Run history details are stored securely in the database.

Common workflows

1. Run first accuracy analysis

   1. Navigate to /oracle.
   2. See the empty state and click the “Run first analysis” button.
   3. Select the date range in the Check Responses dialog (or use the default).
   4. Click “Start” and monitor the progress in the floating Run Progress Overlay.
   5. When complete, the dialog reopens. Click “Review discrepancies” to open the Oracle Review Modal.

2. Review and resolve a discrepancy

   1. From the Overview tab, click a discrepancy row or “See all”.
   2. In the Oracle Review Modal, click a discrepancy row in the left rail to view its detail.
   3. Read Claim A and Claim B; optionally expand the Research Verdict.
   4. Select the correct claim (A or B) or choose “Both valid” / “Dismiss”.
   5. Optionally click ”+ Add a note” to record your reasoning.
   6. Click “Resolve” to save the resolution.

3. Run a fact-verification against external sources

   1. Click the “Run analysis” dropdown in the page header and select “Verify facts”.
   2. In the Verify Facts dialog, select the Knowledge Base pillars to check.
   3. Click “Start verification” and monitor progress in the overlay.
   4. When complete, click “View sources” on the HeroValidationCard.
   5. Browse the ranked source list in the Validation Drawer and click a source domain to see individual contradiction cards.

4. Search what AI is saying about the brand

   1. On the Overview tab, locate the “What AI is saying about your brand” search section.
   2. Type a query (e.g. “pricing”, “competitors”, “founders”) and press Enter.
   3. Read the AI-synthesized summary with inline citation pills.
   4. Click a citation pill to open the Responses Drawer for the referenced AI response.

Empty, loading, and error states

• Empty: When no analysis has ever been run, a centered empty state is shown with a shield icon, a “No analysis run yet” heading, and a “Run first analysis” button. When the first run is in progress, a spinner and a “Running your first analysis” message is shown instead.
• Loading: A centered spinner is shown while the latest run data is loading on initial page load.
• Error: If the subscription/entitlement check fails due to an outage, a “Something went wrong” screen with a Retry button is shown instead of locking you out. Individual data fetch errors will surface as small toast notifications.

Linked from / links to

• Linked from:
  • The main sidebar navigation (under Oracle or Recommendations).
  • Deep-link URLs (e.g., ?analysisRunId=, ?reviewRunId=, ?validationRunId=, ?review=, ?runHistory=open).
  • Legacy URLs (like ?tab=runs, ?tab=kb-analysis, ?tab=fact-bank) which redirect to the current structure automatically on load.
• Links to:
  • The Knowledge Base page (/knowledge-base).
  • Knowledge Base Review (/knowledge-base/review).
  • Content Editor Drafts (/content?published=in-progress and /content?contentId=<id>).

Common support questions

• Why is the “Verify facts” button hidden? Fact verification tiles and buttons are hidden until you have successfully completed at least one Oracle accuracy run and your Knowledge Base corpus is ready.

• Where did my dismissed discrepancy go? The Oracle Review Modal filters to “Unresolved” by default. To find a dismissed or resolved discrepancy, change the filter tab at the top of the modal.

• Why does the page say it is locked? Oracle requires a qualifying upgraded subscription plan. If your plan does not include Oracle access, you will see an upgrade prompt screen.

• I clicked a link from an email but my discrepancy isn’t scrolling into view. If you follow a deep link to a specific discrepancy that has already been resolved or dismissed, the page may silently fail to scroll if the modal is still set to the “Unresolved” filter.