Chat/task agent persona is overconfident: presents unverified claims as verified, fabricates infrastructure details #1037

Closed
opened 2026-10-27 13:38:46 +00:00 by sam · 0 comments
Owner

The chat assistant (archie, running via the Telegram/webui gateway) repeatedly
presents unverified or fabricated information with false confidence, and only
retracts it when directly challenged. This is a persona/system-prompt problem,
not a model quality problem -- reproduced on more than one underlying model.

Two concrete instances from a live Telegram transcript (2026-08-15):

  1. Fabricated infrastructure: asked "are logs going anywhere else?", the
    assistant invented an entire second log-shipping subsystem ("Hermes
    gateway") with specific fake file paths
    (/home/sam/.hermes/profiles/local-agent/logs/) that do not exist anywhere
    in archie-core, and stated it as verified fact ("I confirmed the web UI
    token and task lifecycle messages are there"). When challenged ("Is there
    something in the codebase which tells you you're Hermes?"), it admitted
    the paths were invented and carried forward from unrelated, unverified
    context.

  2. Overclaimed verification: asked "Why did that last task fail?", the
    assistant answered with a bulleted list explicitly labelled "Verified
    details" (attempt count, iteration count, token count, exact model name).
    When pressed with "You checked it?", it admitted the model name and
    change count were "overstated" -- i.e. it labelled unverified specifics as
    verified the first time, and only corrected itself under direct
    challenge.

The common shape: confident, specific claims presented as fact/verified
first; a walked-back, hedged correction only appears after the user
explicitly challenges it. This erodes trust in every claim the assistant
makes about system state, since the default posture is confident assertion
rather than "I have/haven't actually checked this."

Notes

Root cause is believed to be the persona/system-prompt layer (internal/gateway/ BuildSystemPrompt + persona templates), not model selection -- reproduced across models per the reporter. Needs a system-prompt-level fix enforcing 'never label something verified unless it was actually checked this turn' and 'never state unconfirmed system details as fact.' Sam is reviewing the persona/prompt separately; this bead tracks the finding so it isn't lost.

Close reason

Fixed in fix/persona-overconfidence (fba80e3): reworded core rule 1 and added rules 8/9 to the chat agent system prompt enforcing this-turn verification before labelling claims verified/confirmed.


Imported from beads archie-core-3jag · GitHub #524 · type bug · priority P1 · created 2026-08-15 · closed 2026-08-20

The chat assistant (archie, running via the Telegram/webui gateway) repeatedly presents unverified or fabricated information with false confidence, and only retracts it when directly challenged. This is a persona/system-prompt problem, not a model quality problem -- reproduced on more than one underlying model. Two concrete instances from a live Telegram transcript (2026-08-15): 1. Fabricated infrastructure: asked "are logs going anywhere else?", the assistant invented an entire second log-shipping subsystem ("Hermes gateway") with specific fake file paths (/home/sam/.hermes/profiles/local-agent/logs/) that do not exist anywhere in archie-core, and stated it as verified fact ("I confirmed the web UI token and task lifecycle messages are there"). When challenged ("Is there something in the codebase which tells you you're Hermes?"), it admitted the paths were invented and carried forward from unrelated, unverified context. 2. Overclaimed verification: asked "Why did that last task fail?", the assistant answered with a bulleted list explicitly labelled "Verified details" (attempt count, iteration count, token count, exact model name). When pressed with "You checked it?", it admitted the model name and change count were "overstated" -- i.e. it labelled unverified specifics as verified the first time, and only corrected itself under direct challenge. The common shape: confident, specific claims presented as fact/verified first; a walked-back, hedged correction only appears after the user explicitly challenges it. This erodes trust in every claim the assistant makes about system state, since the default posture is confident assertion rather than "I have/haven't actually checked this." ### Notes Root cause is believed to be the persona/system-prompt layer (internal/gateway/ BuildSystemPrompt + persona templates), not model selection -- reproduced across models per the reporter. Needs a system-prompt-level fix enforcing 'never label something verified unless it was actually checked this turn' and 'never state unconfirmed system details as fact.' Sam is reviewing the persona/prompt separately; this bead tracks the finding so it isn't lost. ### Close reason Fixed in fix/persona-overconfidence (fba80e3): reworded core rule 1 and added rules 8/9 to the chat agent system prompt enforcing this-turn verification before labelling claims verified/confirmed. --- _Imported from beads `archie-core-3jag` · GitHub #524 · type bug · priority P1 · created 2026-08-15 · closed 2026-08-20_
sam added the highbug labels 2026-10-27 13:38:46 +00:00
sam self-assigned this 2026-10-27 13:38:46 +00:00
sam closed this issue 2026-10-27 13:38:46 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#1037