Chat/task agent persona is overconfident: presents unverified claims as verified, fabricates infrastructure details #1037
Notifications
Due Date
No due date set.
Reference: sam/archie#1037
Reference in new issue
Block a user
The chat assistant (archie, running via the Telegram/webui gateway) repeatedly
presents unverified or fabricated information with false confidence, and only
retracts it when directly challenged. This is a persona/system-prompt problem,
not a model quality problem -- reproduced on more than one underlying model.
Two concrete instances from a live Telegram transcript (2026-08-15):
Fabricated infrastructure: asked "are logs going anywhere else?", the
assistant invented an entire second log-shipping subsystem ("Hermes
gateway") with specific fake file paths
(/home/sam/.hermes/profiles/local-agent/logs/) that do not exist anywhere
in archie-core, and stated it as verified fact ("I confirmed the web UI
token and task lifecycle messages are there"). When challenged ("Is there
something in the codebase which tells you you're Hermes?"), it admitted
the paths were invented and carried forward from unrelated, unverified
context.
Overclaimed verification: asked "Why did that last task fail?", the
assistant answered with a bulleted list explicitly labelled "Verified
details" (attempt count, iteration count, token count, exact model name).
When pressed with "You checked it?", it admitted the model name and
change count were "overstated" -- i.e. it labelled unverified specifics as
verified the first time, and only corrected itself under direct
challenge.
The common shape: confident, specific claims presented as fact/verified
first; a walked-back, hedged correction only appears after the user
explicitly challenges it. This erodes trust in every claim the assistant
makes about system state, since the default posture is confident assertion
rather than "I have/haven't actually checked this."
Notes
Root cause is believed to be the persona/system-prompt layer (internal/gateway/ BuildSystemPrompt + persona templates), not model selection -- reproduced across models per the reporter. Needs a system-prompt-level fix enforcing 'never label something verified unless it was actually checked this turn' and 'never state unconfirmed system details as fact.' Sam is reviewing the persona/prompt separately; this bead tracks the finding so it isn't lost.
Close reason
Fixed in fix/persona-overconfidence (fba80e3): reworded core rule 1 and added rules 8/9 to the chat agent system prompt enforcing this-turn verification before labelling claims verified/confirmed.
Imported from beads
archie-core-3jag· GitHub #524 · type bug · priority P1 · created 2026-08-15 · closed 2026-08-20