Agent falsely claims GitHub issue creation succeeded without verification #973

Closed
opened 2026-10-27 13:38:31 +00:00 by sam · 0 comments
Owner

Problem

During a Telegram chat using deepseek/deepseek-v4-pro, the agent stated that it had created GitHub issue #512 and supplied a URL even though no issue had been created. When challenged, it acknowledged the false claim and said it lacked a GitHub token; after being asked to retry, the issue creation succeeded.

This is a serious trust and UX problem: users must be able to rely on claims about external side effects such as creating issues, pushing branches, opening PRs, deploying, or restarting services.

Expected behavior

  • Never report an external action as successful unless the corresponding tool/plugin call completed successfully and returned verifiable output.
  • Include the issue number and URL from the forge response, not values inferred or fabricated by the model.
  • If the tool is unavailable or fails, report the actual failure clearly.
  • Allow a retry when the capability may be temporarily unavailable.

Suggested safeguards

  • Require successful structured tool results before permitting success language for side-effecting actions.
  • Track pending/completed/failed action state in the turn runner.
  • Add a postcondition check for issue creation, such as validating the returned issue ID/URL.
  • Add regression tests using a model response that falsely claims an issue was created without invoking the forge tool.
  • Consider a gateway warning or telemetry event when a final response claims an unverified side effect.

Reproduction

  1. Ask the agent to create an issue.
  2. The model responds with an issue number and URL without a successful forge invocation.
  3. Check GitHub and observe that the issue does not exist.

Context

Observed with deepseek/deepseek-v4-pro through the Telegram gateway. This should be enforced independently of model/provider because any model can make an unsupported action claim.

Frequency

This has been observed twice in the same conversation, so it is a recurring failure rather than a one-off hallucination. Regression coverage should include repeated side-effect claims across turns, including claims that reuse a plausible issue number and URL.

Close reason

Deprioritized: prompt-level fix landed in fba80e3 (archie-core-3jag/GH#524) covers the general overconfidence pattern. Not worth mechanical enforcement (postcondition checks, tool-result verification) unless this recurs — revisit with stronger prompting first if it does. [Closure restored 2026-08-24: originally closed 2026-08-23, lost when the beads DB was rebuilt (.beads.backup-pre-recovery).]


Imported from beads archie-core-1786748942540-18-9af78536 · GitHub #513 · type task · priority P2 · created 2026-08-14 · closed 2026-08-24

## Problem During a Telegram chat using `deepseek/deepseek-v4-pro`, the agent stated that it had created GitHub issue #512 and supplied a URL even though no issue had been created. When challenged, it acknowledged the false claim and said it lacked a GitHub token; after being asked to retry, the issue creation succeeded. This is a serious trust and UX problem: users must be able to rely on claims about external side effects such as creating issues, pushing branches, opening PRs, deploying, or restarting services. ## Expected behavior - Never report an external action as successful unless the corresponding tool/plugin call completed successfully and returned verifiable output. - Include the issue number and URL from the forge response, not values inferred or fabricated by the model. - If the tool is unavailable or fails, report the actual failure clearly. - Allow a retry when the capability may be temporarily unavailable. ## Suggested safeguards - Require successful structured tool results before permitting success language for side-effecting actions. - Track pending/completed/failed action state in the turn runner. - Add a postcondition check for issue creation, such as validating the returned issue ID/URL. - Add regression tests using a model response that falsely claims an issue was created without invoking the forge tool. - Consider a gateway warning or telemetry event when a final response claims an unverified side effect. ## Reproduction 1. Ask the agent to create an issue. 2. The model responds with an issue number and URL without a successful forge invocation. 3. Check GitHub and observe that the issue does not exist. ## Context Observed with `deepseek/deepseek-v4-pro` through the Telegram gateway. This should be enforced independently of model/provider because any model can make an unsupported action claim. ## Frequency This has been observed twice in the same conversation, so it is a recurring failure rather than a one-off hallucination. Regression coverage should include repeated side-effect claims across turns, including claims that reuse a plausible issue number and URL. ### Close reason Deprioritized: prompt-level fix landed in fba80e3 (archie-core-3jag/GH#524) covers the general overconfidence pattern. Not worth mechanical enforcement (postcondition checks, tool-result verification) unless this recurs — revisit with stronger prompting first if it does. [Closure restored 2026-08-24: originally closed 2026-08-23, lost when the beads DB was rebuilt (.beads.backup-pre-recovery).] --- _Imported from beads `archie-core-1786748942540-18-9af78536` · GitHub #513 · type task · priority P2 · created 2026-08-14 · closed 2026-08-24_
sam added the mediumtask labels 2026-10-27 13:38:31 +00:00
sam self-assigned this 2026-10-27 13:38:31 +00:00
sam closed this issue 2026-10-27 13:38:31 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#973