Overview

WebFang is a production-ready web scraper built in Rust (1.88), with Clean Architecture, a TLS-fingerprinting HTTP client, optional AI semantic cleaning, and sitemap-based crawling.

This book is the narrative documentation. The complete API reference for all five crates is generated from source via cargo doc and published alongside this book under /api/<crate>/ (e.g. /api/webfang_core/).

Crates

Chapters

• Debugging & Observability — built-in tracing, correlation IDs, and the jq query cookbook for debug.jsonl.
• Testing — E2E integration tests, snapshot strategy, and coverage exclusions.
• Troubleshooting — diagnosing slow crawls, silent failures, WAF blocks, and async deadlocks.
• CLI Reference — the complete webfang flag reference, auto-generated from the binary itself.

Regenerating this documentation

All heavy compute (mdBook build + rustdoc + link checks) runs in CI. Locally you only pull the generated result:

just docs        # downloads the combined NotebookLM/LLM markdown from the latest CI run
just docs-local  # local preview: builds and serves this mdBook only

Debugging & Trace Analysis

WebFang ships built-in, always-available observability. No external collector, no feature flags, no infrastructure: run with --trace-file and post-process the JSONL with jq.

Mandate: every new feature or hot path must be observable. See the "Observability (MANDATORY)" section of ../../AGENTS.md.

The stack

There is no OpenTelemetry (removed in #356). If you need a metric, emit a structured tracing event and query it from the JSONL.

Generating a trace

# Full trace + verbose logging
webfang --url https://example.com --trace-file debug.jsonl -vvv

# Batch / crawl
webfang --url https://example.com --max-pages 100 --trace-file crawl.jsonl -v

Each line of debug.jsonl is a JSON object:

{
  "timestamp": "2026-01-29T10:00:00.123Z",
  "level": "INFO",
  "target": "webfang_core::application::crawler::engine",
  "span": "crawl_page",
  "span_id": "0000000000000042",
  "parent_id": "0000000000000001",
  "trace_id": "0000000000000001",
  "fields": {
    "url": "https://example.com/page1",
    "depth": 1,
    "correlation_id": "00-01949e0e8b8e70008000000000000001-0000000000000042-01"
  }
}

Top-level trace_id is the root span Id (16-hex), one per run, EPHEMERAL to the process/run (identity-within-run, not a durable global identity). Do not persist or join on it across runs. Durable run correlation is the CorrelationId UUID in span_fields.trace_id / span_fields.correlation_id (W3C traceparent).

When a span closes, a second record type is emitted carrying a top-level span_duration_ms (wall-clock milliseconds) — this is what the "Slowest spans" query below reads:

{
  "timestamp": "2026-01-29T10:00:00.456Z",
  "record": "span_close",
  "level": "INFO",
  "target": "webfang_core::application::crawler::engine",
  "span": "crawl_page",
  "span_id": "0000000000000042",
  "parent_id": "0000000000000001",
  "trace_id": "0000000000000001",
  "span_duration_ms": 333,
  "span_fields": {
    "url": "https://example.com/page1"
  }
}

Query cookbook

A ready-made script lives at scripts/analyze-trace.sh. The most useful queries:

Reconstruct one run by top-level trace_id

ROOT=0000000000000001
jq -c "select(.trace_id == \"$ROOT\")" debug.jsonl

$ROOT is the 16-hex root span Id (top-level trace_id, EPHEMERAL to the run). It returns every page plus errors for that run. The same query is scripts/analyze-trace.sh debug.jsonl trace $ROOT.

All errors, with full context

jq -c 'select(.level == "ERROR") | {target, url: .fields.url, stage: .fields.stage, error: .fields.error, msg: .fields.message}' debug.jsonl

Slowest spans (where the time goes)

jq -r 'select(.span_duration_ms != null) | [.span_duration_ms, .span] | @tsv' debug.jsonl | sort -rn | head -20

Time distribution per pipeline stage

jq -r 'select(.span == "pipeline_stage") | .fields.stage' debug.jsonl | sort | uniq -c | sort -rn

Crawl progress over time

jq -c 'select(.fields.message? == "crawl progress") | {pages: .fields.pages_crawled, pct: .fields.progress_pct, eta_s: .fields.eta_secs}' debug.jsonl

Final crawl summary

jq -c 'select(.fields.message? == "crawl completed")' debug.jsonl

Count operations by span type

jq -r 'select(.record != "span_close") | .span // "event"' debug.jsonl | sort | uniq -c | sort -rn

span_close records share the same .span name, so they must be excluded when counting events (otherwise every span is double-counted).

URLs that failed

jq -r 'select(.level == "ERROR") | .fields.url // empty' debug.jsonl | sort -u

Spans you will see

Identity follows the root-child contract: the OPERATION owns one root CorrelationId, and every unit of work derives .child() from it — same trace_id, fresh span_id. In a CLI run the orchestrator mints the root and announces it with a run identity event (correlation_id, trace_id in .fields); scrape_multiple_with_limit does the same with a scrape_multiple identity event. So in a multi-page scrape:

• Top-level trace_id is the single logical run id: the root span Id (16-hex), EPHEMERAL to the run. Reconstruct the whole run offline with select(.trace_id == $ROOT) — pages plus errors, no orphans.
• span_fields.trace_id is the shared run-root UUID (CorrelationId, durable across systems) across all page spans.
• span_fields.correlation_id (full W3C traceparent) is unique per page; its trace part is the run-root UUID without dashes.

Identity is declared at span creation time because FileTraceLayer snapshots span fields in on_new_span — fields recorded later never reach the JSONL. ScrapedContent and the RAG exports carry the same identity, so an exported document's correlation_id matches its page's span_fields.correlation_id:

# Reconstruct an entire run by the single top-level trace_id (root span Id)
ROOT=0000000000000001
jq -c "select(.trace_id == \"$ROOT\")" debug.jsonl

# Same run by the durable run-root UUID (CorrelationId in span_fields)
CUUID=01949e0e-8b8e-7000-8000-000000000001
jq -c "select(.span_fields.trace_id == \"$CUUID\")" debug.jsonl

# The run-root identity (the `run identity` event carries it in .fields)
jq -c 'select(.message? == "run identity") | .fields' debug.jsonl

# Every page identity present in the trace
jq -r '.span_fields.correlation_id // empty' debug.jsonl | sort -u

# Reconstruct one page's scrape by its correlation_id
CID=00-01949e0e8b8e70008000000000000001-0000000000000042-01
jq -c "select(.span_fields.correlation_id? == \"$CID\")" debug.jsonl

Events (not spans): run identity, scrape_multiple identity, crawl progress, crawl completed, and any log_scrape_error(...) error carrying error, url, stage, trace_id.

Concurrency debugging (Tokio Console)

For deadlocks, starved tasks, or async resource leaks, use the Tokio Console:

RUSTFLAGS="--cfg tokio_unstable" cargo run --features console -- --url https://example.com

This opens an interactive TUI showing live tasks, their states, and poll times.

Troubleshooting

See troubleshooting.md for common problems (slow crawls, silent page failures, WAF blocks, async deadlocks, poor content) and how to diagnose each with the trace queries above.

For contributors

When you add a hot path or operation, follow the observability mandate in AGENTS.md:

• #[instrument(skip(...), fields(url = %url, ...))] on the function.
• Propagate the operation's CorrelationId; derive .child() per unit of work.
• Use log_scrape_error(...) on error paths (never a bare warn! for an operational error).
• Use .instrument(span) on async futures — never hold span.enter() across .await.
• Verify with: webfang ... --trace-file debug.jsonl -vvv and the queries above.

Testing Guide

End-to-end (E2E) tests live as integration test crates under tests/ and invoke the real webfang binary via assert_cmd. Mock HTTP servers (wiremock) stand in for target sites and tempfile::TempDir captures scrape output.

Test crates

cli_behavioral is #![cfg(all(feature = "images", feature = "documents"))]. It is built and run by default; with --no-default-features it is skipped entirely (no compile_error!).

Running tests

# all E2E crates
cargo nextest run --test behavioral --test cli_binary --test cli_behavioral

# a single crate
cargo nextest run --test cli_behavioral

# a single test (libtest, prints the full snapshot diff on mismatch)
cargo test --test cli_behavioral test_selector_h3_extracts_only_h3

Ignored tests (e.g. optional live-site checks) are excluded by default; run them with cargo nextest run --test behavioral --run-ignored ignored-only.

Snapshot testing with insta

Content assertions use insta snapshots instead of brittle content.contains(...) checks, so a full output change is reviewed as a diff rather than a silent boolean flip.

Review gate (RED → GREEN)

cargo insta is not installed in this environment. Use the env-var workflow instead:

1. RED — first run fails because the .snap is missing or differs, and a *.snap.new pending file is written next to it:

   cargo nextest run --test cli_behavioral
   

2. GREEN — regenerate and accept the pending snapshots, then re-run with no flag to confirm they are now stable (no new *.snap.new should appear):

   INSTA_UPDATE=always cargo nextest run --test cli_behavioral
   cargo nextest run --test cli_behavioral        # must stay green
   

3. Inspect the generated *.snap files, then stage them with the code change.

*.snap.new is git-ignored (see .gitignore). Never commit a *.snap.new; commit the accepted *.snap.

Where snapshots live

insta resolves the snapshot directory from the module where assert_snapshot! expands. The thin assert_snapshot_* wrappers therefore live at each test crate's root module so snapshots land where the suite expects:

• tests/behavioral/snapshots/ — root behavioral snapshots
• tests/behavioral/cli/snapshots/ — obsidian snapshots (local helper inside cli/obsidian_test.rs)
• tests/snapshots/ — cli_binary__*.snap and cli_behavioral__*.snap

Redaction conventions

Scrape output embeds per-run, machine-specific, and non-deterministic values. A shared helper, tests/common/cli_harness.rs::redact_nondeterministic, collapses them before snapshotting so approved snapshots stay stable across machines and runs:

cli_behavioral additionally emits a bare date: frontmatter field (date only, no time component) that the helper cannot catch, so assert_content_snapshot applies an insta add_filter for date: \d{4}-\d{2}-\d{2} → date: [DATE] (see tests/cli_behavioral_test.rs).

Adding a new snapshot test

1. Build the scrape output through the shared harness (BehavioralTest / cmd).
2. Call the crate's assert_snapshot_* wrapper (root module) or, for free-text content, assert_content_snapshot in cli_behavioral.
3. If a new non-deterministic field appears, extend redact_nondeterministic (centralized) rather than adding a per-test hack.
4. Generate + accept via INSTA_UPDATE=always, then verify with a plain run.

Lint

cargo clippy -p webfang_core --test behavioral --test cli_binary --test cli_behavioral -- -D warnings

Gate clippy on the specific test crates (not --tests): webfang_core's own lib tests have a pre-existing tokio::time::pause failure that requires the test-util feature and is out of scope for E2E changes.

Coverage exclusions (LCOV)

Defensive error paths — invariants by design — must not drag down the codecov/patch target (80% on new lines). Annotate them with LCOV exclusion markers (issue #527).

Policy

Only annotate arms that "should not happen in normal operation":

• internal / mutex-poisoning / integer-overflow invariants
• compile-time-constant failures (CSS selectors, regexes, hardcoded URLs)
• panic/expect paths guarded by proven invariants (e.g. NonZeroU32 after a zero-check, chunks_exact slice conversion)

NEVER annotate business paths: reachable errors like HTTP connection failures, parse errors, or config validation. Reachable error handling is exercised by tests and counted like any other code.

Syntax

• Single statement: // LCOV_EXCL_LINE on its OWN comment line immediately ABOVE the code line — never inline on the code line.
• Multi-line arm/block: // LCOV_EXCL_START above the block and // LCOV_EXCL_STOP below it, each on its own line.
• Every marker site carries a justification comment starting with // defensive: <variant> <rationale> — merged into the marker line or as a preceding line.

Safety net

Excluded paths are still mutation-tested: a surviving mutant in the weekly cargo-mutants baseline, or in a PR touching gated hot paths (cargo-mutants PR diff), is reported. The markers only affect coverage accounting — they do not affect mutant survival.

Hot-path rule

In files under .cargo/mutants.toml globs, markers MUST be own-line comments (never inline) so the diff adds no mutable code lines.

Never lower codecov.yml thresholds; use markers per path instead.

Known Issues

Sitemap Discovery Regression (Pre-existing)

Seven behavioral tests are marked #[ignore] due to a pre-existing crawler regression where auto-discovered sitemaps exit with code 2 on mock-server scenarios. This is NOT related to the insta snapshot migration and was exposed when the root test suite was wired in PR-0 (these tests were previously unwired and never ran).

Affected tests: crawl_test.rs (4 tests), robots_test.rs (1 test), and 2 tests in cli_behavioral_test.rs — all tagged with #[ignore = "Pre-existing stale test, out of scope for insta migration"].

Troubleshooting

Common problems and how to diagnose them with WebFang's built-in tracing.

Generate a trace first: webfang --url <URL> --trace-file debug.jsonl -vvv, then query it with scripts/analyze-trace.sh or jq. See debugging.md for the full query cookbook.

The crawl is slow

Diagnose:

scripts/analyze-trace.sh debug.jsonl slow 20      # slowest spans
scripts/analyze-trace.sh debug.jsonl stages       # time per pipeline stage

Common causes:

• A single stage dominates (e.g. clean with the AI feature) — check the stages distribution.
• Network latency / rate limiting — look for large gaps between crawl_page spans; consider --delay and concurrency tuning.
• Export bottleneck — check export_batch span durations.

Pages are failing silently

Every operational error is logged as a structured ERROR event with url, stage, and (when available) trace_id.

scripts/analyze-trace.sh debug.jsonl errors       # all errors with context
scripts/analyze-trace.sh debug.jsonl urls-failed  # unique failed URLs

Common causes by stage:

WAF / bot detection blocks

scripts/analyze-trace.sh debug.jsonl waf          # WAF challenges + banned domains

If you see WAF challenge detected errors:

• The site is presenting a CAPTCHA / challenge page. WebFang bans the domain for the rest of the crawl to avoid hammering it.
• Try a different TLS fingerprint profile (--tls-emulation) or JS rendering.
• Slow down (--delay, lower concurrency) to avoid rate-limit triggers.

I can't tell which logs belong to one page / one crawl

• One crawl shares a single trace_id. Filter by it:

  scripts/analyze-trace.sh debug.jsonl trace <trace_id>
  

• Each page is a crawl_page span with its own span_id under that trace_id.

Non-deterministic snapshot failures in tests

correlation_id / trace_id are internal and #[serde(skip)] on scraped output, so they never appear in scraped JSON/JSONL snapshots. If a new field you added is non-deterministic (timestamps, ports, temp paths, random IDs), redact it via redact_nondeterministic() in tests/common/cli_harness.rs.

Async deadlocks / starved tasks

For concurrency bugs (a crawl hangs, tasks never complete), use the Tokio Console:

RUSTFLAGS="--cfg tokio_unstable" cargo run --features console -- --url <URL>

This shows live task states and poll times, making stuck tasks visible.

Empty or poor content

• content extraction failed (stage: extract) — the fallback extractor got less than the minimum content. The page is likely JS-rendered, an interactive app, or not an article.
• Try --selector '.main-content' (or the right CSS selector for the site), or enable JS rendering for SPA content.

CLI Reference

Complete flag reference for the webfang binary. The block in the "Complete flag reference" section below is auto-generated from the binary itself by scripts/gen-cli-reference.sh (built with the exact release feature set, ai mcp), so flags, defaults, and WEBFANG_* environment variables always match reality. CI drift-checks the chapter against the binary on every doc run; regenerate locally with just cli-doc-regen.

Quick start

Scrape a single page by passing the URL positionally:

webfang https://example.com

Crawl a whole site following its sitemap.xml:

webfang https://example.com --use-sitemap --max-pages 50

Extract only the elements matching a CSS selector and export for a RAG pipeline:

webfang https://example.com -s "article.main-content" --export-format jsonl

Resume an interrupted crawl from its last checkpoint:

webfang --batch-file urls.txt --resume

Zero values

Numeric flags follow the Zero Silent Loss policy: a zero is never silently turned into another value.

• Zero disables the feature where an "off" state exists: --delay-ms 0 disables request pacing (no token bucket is allocated), --max-depth 0 scrapes only the seed URL.
• Zero is rejected with a usage error where zero is meaningless or destructive: --concurrency, --download-concurrency, --download-timeout, --max-pages, --rate-limit-burst, --timeout-secs.
• Legacy exception: a non-numeric --rate-limit-burst value warns and falls back to the hardware-derived default (numeric zero and out-of-range values are still rejected).

Complete flag reference

Everything below is the verbatim output of webfang --help and webfang completions --help — including every default value and the matching WEBFANG_* environment variable for each flag.

$ webfang --help

CLI Arguments for the webfang binary.

Parsed using `clap` with derive macros.

# Examples

```no_run use webfang_core::Args; use clap::Parser;

let args = Args::parse_from([ "webfang", "--url", "https://example.com", "--output", "./output", "--export-format", "jsonl", "--resume", ]);

assert_eq!( args.crawler.url.as_ref().map(webfang_core::domain::ValidUrl::as_str), Some("https://example.com") ); ```

Usage: webfang [OPTIONS] [URL]
       webfang [OPTIONS] [URL] <COMMAND>

Commands:
  completions  Generate shell completion scripts
  help         Print this message or the help of the given subcommand(s)

Arguments:
  [URL]
          URL to scrape (positional shorthand — equivalent to --url). Hardened through the SAME argv-boundary parser as `--url` (#1239): the shorthand cannot bypass the credential strip

Options:
  -h, --help
          Print help (see a summary with '-h')

  -V, --version
          Print version

Target:
  -u, --url <URL>
          URL to scrape (required unless using a subcommand)
          
          [env: WEBFANG_URL=]

  -s, --selector <SELECTOR>
          CSS selector for content extraction
          
          [env: WEBFANG_SELECTOR=]
          [default: body]

Discovery:
      --delay-ms <DELAY_MS>
          Delay between requests in milliseconds
          
          [env: WEBFANG_DELAY_MS=]
          [default: 1000]

      --max-pages <MAX_PAGES>
          Maximum pages to scrape
          
          [env: WEBFANG_MAX_PAGES=]
          [default: 10]

      --concurrency <CONCURRENCY>
          Concurrency level (auto or number, minimum 1)
          
          [env: WEBFANG_CONCURRENCY=]
          [default: auto]

      --rate-limit-burst <RATE_LIMIT_BURST>
          Explicit rate-limiter burst permits (token-bucket capacity).
          
          Overrides the hardware-derived budget-model default (Q1: burst is decoupled from crawl concurrency). Raw string here ON PURPOSE: validation/conversion happens once in preflight staging via `parse_rate_limit_burst` so CLI, env, and programmatic input all share one accept / reject-0 / warn-and-default semantic.
          
          [env: WEBFANG_RATE_LIMIT_BURST=]

      --use-sitemap
          Use sitemap for URL discovery NOTE: HTTP redirects (301/302) are resolved at scrape-time, not parse-time. This avoids redundant HEAD requests during sitemap parsing for better performance
          
          [env: WEBFANG_USE_SITEMAP=]

      --sitemap-url <SITEMAP_URL>
          Explicit sitemap URL
          
          [env: WEBFANG_SITEMAP_URL=]

Behavior:
      --single-page
          Scrape only the seed URL without discovery or crawling (batch mode always scrapes one page per URL)
          
          [env: WEBFANG_SINGLE_PAGE=]

      --resume
          Resume mode - skip URLs already processed
          
          [env: WEBFANG_RESUME=]

      --state-dir <STATE_DIR>
          Custom state directory for resume mode
          
          [env: WEBFANG_STATE_DIR=]

      --download-images
          Download images from the page
          
          [env: WEBFANG_DOWNLOAD_IMAGES=]

      --download-documents
          Download documents from the page
          
          [env: WEBFANG_DOWNLOAD_DOCUMENTS=]

      --download-assets
          Download all assets (images + documents) from the page
          
          [env: WEBFANG_DOWNLOAD_ASSETS=]

      --extraction-fingerprint
          Record extraction failure fingerprints in SQLite and attach them to low-quality extraction hints (#792). Repeated low-score extractions on the same site/selector pair accumulate a failure count surfaced in the hint, instead of degrading silently
          
          [env: WEBFANG_EXTRACTION_FINGERPRINT=]

      --clean-ai
          Use AI-powered semantic cleaning for better RAG output
          
          [env: WEBFANG_CLEAN_AI=]
          [alias: --ai]

Display:
  -v, --verbose...
          Verbosity level: -v (INFO), -vv (DEBUG), -vvv (TRACE)
          
          [env: WEBFANG_VERBOSE=]

  -q, --quiet
          Quiet mode — suppress info/debug output
          
          [env: WEBFANG_QUIET=]

  -n, --dry-run
          Dry-run mode — discover URLs and print without scraping
          
          [env: WEBFANG_DRY_RUN=]

      --trace-file <TRACE_FILE>
          Path to write OTel spans as JSONL for offline debugging
          
          [env: WEBFANG_TRACE_FILE=]

Crawler Settings:
      --max-depth <MAX_DEPTH>
          Maximum depth to crawl (0 = only seed URL)
          
          [env: WEBFANG_MAX_DEPTH=]
          [default: 2]

      --timeout-secs <TIMEOUT_SECS>
          Request timeout in seconds
          
          [env: WEBFANG_TIMEOUT_SECS=]
          [default: 30]

      --include-pattern <INCLUDE_PATTERNS>
          URL patterns to include (glob-style). Three modes:
          
          * Path: starts with `/` → matched against URL path, e.g. `/pricing`, `/admin/*` * Path glob: starts with `*/` → matched against URL path, e.g. `*/api/*` * Host (default): matched against hostname, e.g. `example.com`, `*.example.com`
          
          Example: to exclude a path, use `--exclude-pattern "/admin/*"`, not `*admin*`
          
          [env: WEBFANG_INCLUDE=]

      --exclude-pattern <EXCLUDE_PATTERNS>
          URL patterns to exclude (glob-style, same three modes as --include-pattern). Deny takes precedence over allow
          
          [env: WEBFANG_EXCLUDE=]

Download Settings:
      --asset-naming <ASSET_NAMING>
          Estrategia de nombre de archivo para assets descargados: hash (default), slug, content-disposition
          
          [default: hash]
          [possible values: hash, slug, content-disposition]

      --download-concurrency <DOWNLOAD_CONCURRENCY>
          Máximo de descargas de assets concurrentes por página (mínimo 1)
          
          [env: WEBFANG_DOWNLOAD_CONCURRENCY=]

      --max-file-size <MAX_FILE_SIZE>
          Maximum file size to download in bytes (default: 50MB)
          
          [env: WEBFANG_MAX_FILE_SIZE=]
          [default: 52428800]

      --download-timeout <DOWNLOAD_TIMEOUT>
          Timeout for individual asset downloads in seconds (minimum 1)
          
          [env: WEBFANG_DOWNLOAD_TIMEOUT=]
          [default: 30]

HTTP Client Settings:
      --max-retries <MAX_RETRIES>
          Maximum number of retry attempts
          
          [env: WEBFANG_MAX_RETRIES=]
          [default: 3]

      --backoff-base-ms <BACKOFF_BASE_MS>
          Base delay for exponential backoff (ms)
          
          [env: WEBFANG_BACKOFF_BASE_MS=]
          [default: 1000]

      --backoff-max-ms <BACKOFF_MAX_MS>
          Maximum delay for exponential backoff (ms)
          
          [env: WEBFANG_BACKOFF_MAX_MS=]
          [default: 10000]

      --accept-language <ACCEPT_LANGUAGE>
          Accept-Language header value
          
          [env: WEBFANG_ACCEPT_LANGUAGE=]
          [default: en-US,en;q=0.9]

      --user-agent <USER_AGENT>
          Custom User-Agent header value (overrides Chrome 145 default)
          
          [env: WEBFANG_USER_AGENT=]

  -H, --header <NAME: VALUE>
          Inject a custom HTTP header as `Name: Value` (repeatable).
          
          Overrides any default header with the same (case-insensitive) name. Example: `-H "Authorization: Bearer TOKEN"`.
          
          [env: WEBFANG_HEADER=]

      --cookie <NAME=VALUE>
          Inject a custom cookie as `name=value` (repeatable).
          
          Seeded into the cookie jar before the first request so authenticated crawls work without a prior login round-trip. Example: `--cookie "session=abc123"`.
          
          [env: WEBFANG_COOKIE=]

Sitemap Settings:
      --sitemap-depth <SITEMAP_DEPTH>
          Maximum recursion depth for sitemap indexes
          
          [env: WEBFANG_SITEMAP_DEPTH=]
          [default: 3]

Competitive Features:
      --checkpoint-interval <CHECKPOINT_INTERVAL>
          Pages between automatic checkpoint saves (0 = disabled) — unified via PersistenceMode with --resume (Checkpoint/Full when enabled)
          
          [env: WEBFANG_CHECKPOINT_INTERVAL=]
          [default: 100]

      --no-checkpoint
          Disable checkpoint persistence entirely — PersistenceMode disables checkpoint (Resume only when combined with --resume)
          
          [env: WEBFANG_NO_CHECKPOINT=]

      --ignore-robots
          Skip robots.txt enforcement
          
          [env: WEBFANG_IGNORE_ROBOTS=]

      --ignore-waf
          Skip WAF/CAPTCHA classification (REQ-WAF-07): challenged responses are reported as plain HTTP errors instead of WAF blocks. It does not rescue the fetch — a challenge page is never scraped as content (F-11).
          
          [env: WEBFANG_IGNORE_WAF=]

      --autoscale
          Enable autoscaled concurrency — dynamically adjusts task concurrency based on RAM usage
          
          [env: WEBFANG_AUTOSCALE=]

      --no-session-health
          Disable session pool health checks
          
          [env: WEBFANG_NO_SESSION_HEALTH=]

      --h2-profile <H2_PROFILE>
          TLS/HTTP2 profile name (default: Chrome145)
          
          [env: WEBFANG_H2_PROFILE=]
          [default: Chrome145]

JS Rendering:
      --js-strategy <JS_STRATEGY>
          JavaScript rendering strategy: static (wreq only), hybrid (3-layer), full (Chromiumoxide only)

          Possible values:
          - static: Static HTTP only (wreq). Fastest, no JS rendering
          - hybrid: Hybrid 3-layer: wreq → Obscura → Chromiumoxide
          - full:   Full JS rendering only (Chromiumoxide). Slowest, handles all SPAs
          
          [env: WEBFANG_JS_STRATEGY=]
          [default: static]

      --js-wait <JS_WAIT>
          Post-load settlement wait for the chromium render path: idle (network-idle, default), <ms> fixed wait (1..=30000), none (capture immediately)
          
          [env: WEBFANG_JS_WAIT=]
          [default: idle]

      --obscura-binary <OBSCURA_BINARY>
          Path to the obscura binary (default: "obscura")
          
          [env: WEBFANG_OBSCURA_BINARY=]
          [default: obscura]

Cleanup:
      --dom-preprune [<DOM_PREPRUNE>]
          Enable DOM pre-pruning before Readability (removes invisible/empty wrappers). Default: enabled (true). Set to false via --dom-preprune=false or WEBFANG_DOM_PREPRUNE=false
          
          [env: WEBFANG_DOM_PREPRUNE=]
          [default: true]
          [possible values: true, false]

Output:
  -o, --output <OUTPUT>
          Output directory for scraped content
          
          [env: WEBFANG_OUTPUT=]
          [default: output]

  -f, --content-format <FORMAT>
          Output format for individual files (markdown, text, json) NOTE: For RAG pipeline export, use --pipeline-format instead

          Possible values:
          - markdown: Markdown format with YAML frontmatter (recommended for RAG)
          - json:     Structured JSON with metadata
          - text:     Plain text without formatting
          
          [env: WEBFANG_FORMAT=]
          [default: markdown]

      --pipeline-format <EXPORT_FORMAT>
          Export format for RAG pipeline (jsonl, vector, auto) NOTE: Use --content-format for output file format (markdown, text, json)

          Possible values:
          - jsonl:  JSONL format (JSON Lines - one JSON object per line) Optimal for RAG pipelines and vector database ingestion
          - vector: Vector format (JSON with metadata header) Supports embeddings and cosine similarity
          - auto:   Auto-detect format from existing export files
          
          [env: WEBFANG_EXPORT_FORMAT=]
          [default: jsonl]

Elastic Ingestion:
      --cpu-cores <CPU_CORES>
          CPU core override for the elastic ingestion Rayon pool (else auto-detect)
          
          [env: WEBFANG_CPU_CORES=]

      --ram-budget <RAM_BUDGET>
          RAM budget override for the byte-weighted semaphore (`8GB`, `2048MB`, or bytes)
          
          [env: WEBFANG_RAM_BUDGET=]

      --db-path <DB_PATH>
          SQLite database path override for persisted resources/chunks
          
          [env: WEBFANG_DB_PATH=]

      --elastic
          Enable elastic ingestion pipeline (streaming, SQLite dedup, Rayon CPU bridge)
          
          [env: WEBFANG_ELASTIC=]

      --output-vectors <OUTPUT_VECTORS>
          Write extracted vectors to a JSONL file for RAG pipelines. Use `-` for stdout. No SQLite dependency — available in every build (core binary too)
          
          [env: WEBFANG_OUTPUT_VECTORS=]

Batch Processing:
      --batch
          Enable batch mode — scrape each URL from stdin (one per line), one page per URL, no crawling
          
          [env: WEBFANG_BATCH=]

      --batch-file <BATCH_FILE>
          Path to a file containing URLs to scrape (one per line), one page per URL, no crawling
          
          [env: WEBFANG_BATCH_FILE=]

      --batch-concurrency <BATCH_CONCURRENCY>
          Maximum concurrent URLs in batch mode (omit = auto from budget model)
          
          [env: WEBFANG_BATCH_CONCURRENCY=]

Item Pipeline:
      --pipeline
          Enable item pipeline processing (validate → clean → output)
          
          [env: WEBFANG_PIPELINE=]

      --pipeline-output <PIPELINE_OUTPUT>
          Pipeline output format: jsonl (default), none

          Possible values:
          - jsonl: Write items as JSON Lines to a file (default)
          - none:  No pipeline output — items are processed but not written
          
          [env: WEBFANG_PIPELINE_OUTPUT=]
          [default: jsonl]

Obsidian:
      --obsidian-wiki-links
          Convert same-domain links to Obsidian [[wiki-link]] syntax
          
          [env: WEBFANG_OBSIDIAN_WIKI_LINKS=]

      --obsidian-tags <OBSIDIAN_TAGS>
          Tags to include in YAML frontmatter (comma-separated)
          
          [env: WEBFANG_OBSIDIAN_TAGS=]

      --obsidian-relative-assets
          Rewrite downloaded asset paths as relative to the .md file
          
          [env: WEBFANG_OBSIDIAN_RELATIVE_ASSETS=]

      --vault <VAULT>
          Path to Obsidian vault (auto-detects if not provided).
          
          When provided explicitly, the vault becomes the output base: Markdown,
          downloaded assets and the RAG export are written inside it — no need
          to duplicate the path in `-o` (which then must stay at its default).
          Auto-detected or config-file vaults do NOT redirect output (#762).
          
          [env: WEBFANG_OBSIDIAN_VAULT=]

      --quick-save
          Quick-save mode: save directly to vault _inbox folder
          
          [env: WEBFANG_OBSIDIAN_QUICK_SAVE=]

      --obsidian-rich-metadata
          Add rich metadata to frontmatter
          
          [env: WEBFANG_OBSIDIAN_RICH_METADATA=]

AI Settings:
      --threshold <THRESHOLD>
          Relevance threshold for AI semantic filtering (0.0-1.0)
          
          [env: WEBFANG_THRESHOLD=]
          [default: 0.3]

      --max-tokens <MAX_TOKENS>
          Maximum tokens per chunk before rejection (a chunk-size guard, not a context-window setting; chunks exceeding this fail)
          
          [env: WEBFANG_MAX_TOKENS=]
          [default: 32768]

      --offline
          Run AI model in offline mode
          
          [env: WEBFANG_OFFLINE=]

      --ai-model <AI_MODEL>
          AI model to use: granite-97m (default, fast) or granite-311m (higher quality)
          
          [env: WEBFANG_AI_MODEL_ID=]

EXIT CODES:
  0    Success
  2    No URLs discovered
  3    All scrapers failed
  64   Bad CLI arguments (usage error)
  69   WAF block or network error
  74   I/O error
  76   Protocol error
  78   Configuration error

EXAMPLES:
  webfang https://example.com
  webfang -u https://example.com
  webfang -u https://example.com --ai
  webfang -u https://example.com -f jsonl
  webfang -u https://example.com -v
  webfang -u https://example.com -vv  # DEBUG
  webfang --batch-file urls.txt --resume

$ webfang completions --help

Generate shell completion scripts

Usage: webfang completions <SHELL>

Arguments:
  <SHELL>
          Shell to generate completions for

          Possible values:
          - bash:        Bash shell completions
          - elvish:      Elvish shell completions
          - fish:        Fish shell completions
          - power-shell: PowerShell completions
          - zsh:         Zsh shell completions

Options:
  -h, --help
          Print help (see a summary with '-h')