feat(tools): web_fetch tool with dial-time SSRF protection #705

Closed
opened 2026-10-27 13:37:25 +00:00 by sam · 0 comments
Owner

Archie has no web access of any kind and no MCP server configured to substitute, so it cannot open a URL pasted into chat.

New package internal/tools/webfetch, registered directly in cmd/archied/main.go beside skill_activate.

Schema: url (required), raw (bool, skip HTML extraction).

  • http/https scheme allowlist.
  • SSRF protection in DialContext, NOT as a pre-flight URL check: resolve the host, reject loopback/private/link-local/unspecified/multicast, then dial the vetted IP. A pre-flight URL check leaves a DNS-rebinding hole, and this daemon runs on a host with the Docker socket and NATS reachable on 172.19.0.1.
  • CheckRedirect caps hops and re-applies the scheme check.
  • Body capped by io.LimitReader; result then capped by the per-tool limit.
  • text/html -> readable text via golang.org/x/net/html; text/*, json, xml as-is; anything else an error naming content type and size.
  • Non-2xx returns status plus a short excerpt.

Config [tools.web_fetch]: enabled, timeout, max_bytes, allow_private_networks (default false). Defaults via configuration.applyDefaults.

Close reason

Complete: internal/tools/webfetch (webfetch.go, tool.go) implements every clause. Dial-time SSRF protection via custom DialContext (webfetch.go:66,233,240) that resolves the host and rejects private/loopback before dialing the vetted IP -- not a pre-flight URL check, matching the DNS-rebinding concern the issue calls out explicitly. CheckRedirect present (webfetch.go:74), AllowPrivateNetworks config wired through internal/config/config.go:322-326 (default false) and internal/tools/webfetch/webfetch.go:49-52,256, HTML-to-text via golang.org/x/net/html, and the tool is registered in cmd/archied/main.go:992 beside skill_activate as specified. go test ./internal/tools/webfetch/... -race -count=1 is green.


Imported from beads archie-core-1786637494512-180-38b8f04b · GitHub #201 · type feature · priority P2 · created 2026-08-08 · closed 2026-08-15

Archie has no web access of any kind and no MCP server configured to substitute, so it cannot open a URL pasted into chat. New package internal/tools/webfetch, registered directly in cmd/archied/main.go beside skill_activate. Schema: url (required), raw (bool, skip HTML extraction). - http/https scheme allowlist. - SSRF protection in DialContext, NOT as a pre-flight URL check: resolve the host, reject loopback/private/link-local/unspecified/multicast, then dial the vetted IP. A pre-flight URL check leaves a DNS-rebinding hole, and this daemon runs on a host with the Docker socket and NATS reachable on 172.19.0.1. - CheckRedirect caps hops and re-applies the scheme check. - Body capped by io.LimitReader; result then capped by the per-tool limit. - text/html -> readable text via golang.org/x/net/html; text/*, json, xml as-is; anything else an error naming content type and size. - Non-2xx returns status plus a short excerpt. Config [tools.web_fetch]: enabled, timeout, max_bytes, allow_private_networks (default false). Defaults via configuration.applyDefaults. ### Close reason Complete: internal/tools/webfetch (webfetch.go, tool.go) implements every clause. Dial-time SSRF protection via custom DialContext (webfetch.go:66,233,240) that resolves the host and rejects private/loopback before dialing the vetted IP -- not a pre-flight URL check, matching the DNS-rebinding concern the issue calls out explicitly. CheckRedirect present (webfetch.go:74), AllowPrivateNetworks config wired through internal/config/config.go:322-326 (default false) and internal/tools/webfetch/webfetch.go:49-52,256, HTML-to-text via golang.org/x/net/html, and the tool is registered in cmd/archied/main.go:992 beside skill_activate as specified. go test ./internal/tools/webfetch/... -race -count=1 is green. --- _Imported from beads `archie-core-1786637494512-180-38b8f04b` · GitHub #201 · type feature · priority P2 · created 2026-08-08 · closed 2026-08-15_
sam added the mediumfeature labels 2026-10-27 13:37:25 +00:00
sam self-assigned this 2026-10-27 13:37:25 +00:00
sam closed this issue 2026-10-27 13:37:25 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#705