CLI Reference
Complete flag reference for the webfang binary. The block in the
"Complete flag reference" section below is auto-generated from the
binary itself by scripts/gen-cli-reference.sh (built with the exact release
feature set, ai mcp), so flags, defaults, and WEBFANG_* environment
variables always match reality. CI drift-checks the chapter against the binary
on every doc run; regenerate locally with just cli-doc-regen.
Quick start
Scrape a single page by passing the URL positionally:
webfang https://example.com
Crawl a whole site following its sitemap.xml:
webfang https://example.com --use-sitemap --max-pages 50
Extract only the elements matching a CSS selector and export for a RAG pipeline:
webfang https://example.com -s "article.main-content" --export-format jsonl
Resume an interrupted crawl from its last checkpoint:
webfang --batch-file urls.txt --resume
Zero values
Numeric flags follow the Zero Silent Loss policy: a zero is never silently turned into another value.
- Zero disables the feature where an "off" state exists:
--delay-ms 0disables request pacing (no token bucket is allocated),--max-depth 0scrapes only the seed URL. - Zero is rejected with a usage error where zero is meaningless or
destructive:
--concurrency,--download-concurrency,--download-timeout,--max-pages,--rate-limit-burst,--timeout-secs. - Legacy exception: a non-numeric
--rate-limit-burstvalue warns and falls back to the hardware-derived default (numeric zero and out-of-range values are still rejected).
Complete flag reference
Everything below is the verbatim output of webfang --help and
webfang completions --help — including every default value and the
matching WEBFANG_* environment variable for each flag.
$ webfang --help
CLI Arguments for the webfang binary.
Parsed using `clap` with derive macros.
# Examples
```no_run use webfang_core::Args; use clap::Parser;
let args = Args::parse_from([ "webfang", "--url", "https://example.com", "--output", "./output", "--export-format", "jsonl", "--resume", ]);
assert_eq!( args.crawler.url.as_ref().map(webfang_core::domain::ValidUrl::as_str), Some("https://example.com") ); ```
Usage: webfang [OPTIONS] [URL]
webfang [OPTIONS] [URL] <COMMAND>
Commands:
completions Generate shell completion scripts
help Print this message or the help of the given subcommand(s)
Arguments:
[URL]
URL to scrape (positional shorthand — equivalent to --url). Hardened through the SAME argv-boundary parser as `--url` (#1239): the shorthand cannot bypass the credential strip
Options:
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
Target:
-u, --url <URL>
URL to scrape (required unless using a subcommand)
[env: WEBFANG_URL=]
-s, --selector <SELECTOR>
CSS selector for content extraction
[env: WEBFANG_SELECTOR=]
[default: body]
Discovery:
--delay-ms <DELAY_MS>
Delay between requests in milliseconds
[env: WEBFANG_DELAY_MS=]
[default: 1000]
--max-pages <MAX_PAGES>
Maximum pages to scrape
[env: WEBFANG_MAX_PAGES=]
[default: 10]
--concurrency <CONCURRENCY>
Concurrency level (auto or number, minimum 1)
[env: WEBFANG_CONCURRENCY=]
[default: auto]
--rate-limit-burst <RATE_LIMIT_BURST>
Explicit rate-limiter burst permits (token-bucket capacity).
Overrides the hardware-derived budget-model default (Q1: burst is decoupled from crawl concurrency). Raw string here ON PURPOSE: validation/conversion happens once in preflight staging via `parse_rate_limit_burst` so CLI, env, and programmatic input all share one accept / reject-0 / warn-and-default semantic.
[env: WEBFANG_RATE_LIMIT_BURST=]
--use-sitemap
Use sitemap for URL discovery NOTE: HTTP redirects (301/302) are resolved at scrape-time, not parse-time. This avoids redundant HEAD requests during sitemap parsing for better performance
[env: WEBFANG_USE_SITEMAP=]
--sitemap-url <SITEMAP_URL>
Explicit sitemap URL
[env: WEBFANG_SITEMAP_URL=]
Behavior:
--single-page
Scrape only the seed URL without discovery or crawling (batch mode always scrapes one page per URL)
[env: WEBFANG_SINGLE_PAGE=]
--resume
Resume mode - skip URLs already processed
[env: WEBFANG_RESUME=]
--state-dir <STATE_DIR>
Custom state directory for resume mode
[env: WEBFANG_STATE_DIR=]
--download-images
Download images from the page
[env: WEBFANG_DOWNLOAD_IMAGES=]
--download-documents
Download documents from the page
[env: WEBFANG_DOWNLOAD_DOCUMENTS=]
--download-assets
Download all assets (images + documents) from the page
[env: WEBFANG_DOWNLOAD_ASSETS=]
--extraction-fingerprint
Record extraction failure fingerprints in SQLite and attach them to low-quality extraction hints (#792). Repeated low-score extractions on the same site/selector pair accumulate a failure count surfaced in the hint, instead of degrading silently
[env: WEBFANG_EXTRACTION_FINGERPRINT=]
--clean-ai
Use AI-powered semantic cleaning for better RAG output
[env: WEBFANG_CLEAN_AI=]
[alias: --ai]
Display:
-v, --verbose...
Verbosity level: -v (INFO), -vv (DEBUG), -vvv (TRACE)
[env: WEBFANG_VERBOSE=]
-q, --quiet
Quiet mode — suppress info/debug output
[env: WEBFANG_QUIET=]
-n, --dry-run
Dry-run mode — discover URLs and print without scraping
[env: WEBFANG_DRY_RUN=]
--trace-file <TRACE_FILE>
Path to write OTel spans as JSONL for offline debugging
[env: WEBFANG_TRACE_FILE=]
Crawler Settings:
--max-depth <MAX_DEPTH>
Maximum depth to crawl (0 = only seed URL)
[env: WEBFANG_MAX_DEPTH=]
[default: 2]
--timeout-secs <TIMEOUT_SECS>
Request timeout in seconds
[env: WEBFANG_TIMEOUT_SECS=]
[default: 30]
--include-pattern <INCLUDE_PATTERNS>
URL patterns to include (glob-style). Three modes:
* Path: starts with `/` → matched against URL path, e.g. `/pricing`, `/admin/*` * Path glob: starts with `*/` → matched against URL path, e.g. `*/api/*` * Host (default): matched against hostname, e.g. `example.com`, `*.example.com`
Example: to exclude a path, use `--exclude-pattern "/admin/*"`, not `*admin*`
[env: WEBFANG_INCLUDE=]
--exclude-pattern <EXCLUDE_PATTERNS>
URL patterns to exclude (glob-style, same three modes as --include-pattern). Deny takes precedence over allow
[env: WEBFANG_EXCLUDE=]
Download Settings:
--asset-naming <ASSET_NAMING>
Estrategia de nombre de archivo para assets descargados: hash (default), slug, content-disposition
[default: hash]
[possible values: hash, slug, content-disposition]
--download-concurrency <DOWNLOAD_CONCURRENCY>
Máximo de descargas de assets concurrentes por página (mínimo 1)
[env: WEBFANG_DOWNLOAD_CONCURRENCY=]
--max-file-size <MAX_FILE_SIZE>
Maximum file size to download in bytes (default: 50MB)
[env: WEBFANG_MAX_FILE_SIZE=]
[default: 52428800]
--download-timeout <DOWNLOAD_TIMEOUT>
Timeout for individual asset downloads in seconds (minimum 1)
[env: WEBFANG_DOWNLOAD_TIMEOUT=]
[default: 30]
HTTP Client Settings:
--max-retries <MAX_RETRIES>
Maximum number of retry attempts
[env: WEBFANG_MAX_RETRIES=]
[default: 3]
--backoff-base-ms <BACKOFF_BASE_MS>
Base delay for exponential backoff (ms)
[env: WEBFANG_BACKOFF_BASE_MS=]
[default: 1000]
--backoff-max-ms <BACKOFF_MAX_MS>
Maximum delay for exponential backoff (ms)
[env: WEBFANG_BACKOFF_MAX_MS=]
[default: 10000]
--accept-language <ACCEPT_LANGUAGE>
Accept-Language header value
[env: WEBFANG_ACCEPT_LANGUAGE=]
[default: en-US,en;q=0.9]
--user-agent <USER_AGENT>
Custom User-Agent header value (overrides Chrome 145 default)
[env: WEBFANG_USER_AGENT=]
-H, --header <NAME: VALUE>
Inject a custom HTTP header as `Name: Value` (repeatable).
Overrides any default header with the same (case-insensitive) name. Example: `-H "Authorization: Bearer TOKEN"`.
[env: WEBFANG_HEADER=]
--cookie <NAME=VALUE>
Inject a custom cookie as `name=value` (repeatable).
Seeded into the cookie jar before the first request so authenticated crawls work without a prior login round-trip. Example: `--cookie "session=abc123"`.
[env: WEBFANG_COOKIE=]
Sitemap Settings:
--sitemap-depth <SITEMAP_DEPTH>
Maximum recursion depth for sitemap indexes
[env: WEBFANG_SITEMAP_DEPTH=]
[default: 3]
Competitive Features:
--checkpoint-interval <CHECKPOINT_INTERVAL>
Pages between automatic checkpoint saves (0 = disabled) — unified via PersistenceMode with --resume (Checkpoint/Full when enabled)
[env: WEBFANG_CHECKPOINT_INTERVAL=]
[default: 100]
--no-checkpoint
Disable checkpoint persistence entirely — PersistenceMode disables checkpoint (Resume only when combined with --resume)
[env: WEBFANG_NO_CHECKPOINT=]
--ignore-robots
Skip robots.txt enforcement
[env: WEBFANG_IGNORE_ROBOTS=]
--ignore-waf
Skip WAF/CAPTCHA classification (REQ-WAF-07): challenged responses are reported as plain HTTP errors instead of WAF blocks. It does not rescue the fetch — a challenge page is never scraped as content (F-11).
[env: WEBFANG_IGNORE_WAF=]
--autoscale
Enable autoscaled concurrency — dynamically adjusts task concurrency based on RAM usage
[env: WEBFANG_AUTOSCALE=]
--no-session-health
Disable session pool health checks
[env: WEBFANG_NO_SESSION_HEALTH=]
--h2-profile <H2_PROFILE>
TLS/HTTP2 profile name (default: Chrome145)
[env: WEBFANG_H2_PROFILE=]
[default: Chrome145]
JS Rendering:
--js-strategy <JS_STRATEGY>
JavaScript rendering strategy: static (wreq only), hybrid (3-layer), full (Chromiumoxide only)
Possible values:
- static: Static HTTP only (wreq). Fastest, no JS rendering
- hybrid: Hybrid 3-layer: wreq → Obscura → Chromiumoxide
- full: Full JS rendering only (Chromiumoxide). Slowest, handles all SPAs
[env: WEBFANG_JS_STRATEGY=]
[default: static]
--js-wait <JS_WAIT>
Post-load settlement wait for the chromium render path: idle (network-idle, default), <ms> fixed wait (1..=30000), none (capture immediately)
[env: WEBFANG_JS_WAIT=]
[default: idle]
--obscura-binary <OBSCURA_BINARY>
Path to the obscura binary (default: "obscura")
[env: WEBFANG_OBSCURA_BINARY=]
[default: obscura]
Cleanup:
--dom-preprune [<DOM_PREPRUNE>]
Enable DOM pre-pruning before Readability (removes invisible/empty wrappers). Default: enabled (true). Set to false via --dom-preprune=false or WEBFANG_DOM_PREPRUNE=false
[env: WEBFANG_DOM_PREPRUNE=]
[default: true]
[possible values: true, false]
Output:
-o, --output <OUTPUT>
Output directory for scraped content
[env: WEBFANG_OUTPUT=]
[default: output]
-f, --content-format <FORMAT>
Output format for individual files (markdown, text, json) NOTE: For RAG pipeline export, use --pipeline-format instead
Possible values:
- markdown: Markdown format with YAML frontmatter (recommended for RAG)
- json: Structured JSON with metadata
- text: Plain text without formatting
[env: WEBFANG_FORMAT=]
[default: markdown]
--pipeline-format <EXPORT_FORMAT>
Export format for RAG pipeline (jsonl, vector, auto) NOTE: Use --content-format for output file format (markdown, text, json)
Possible values:
- jsonl: JSONL format (JSON Lines - one JSON object per line) Optimal for RAG pipelines and vector database ingestion
- vector: Vector format (JSON with metadata header) Supports embeddings and cosine similarity
- auto: Auto-detect format from existing export files
[env: WEBFANG_EXPORT_FORMAT=]
[default: jsonl]
Elastic Ingestion:
--cpu-cores <CPU_CORES>
CPU core override for the elastic ingestion Rayon pool (else auto-detect)
[env: WEBFANG_CPU_CORES=]
--ram-budget <RAM_BUDGET>
RAM budget override for the byte-weighted semaphore (`8GB`, `2048MB`, or bytes)
[env: WEBFANG_RAM_BUDGET=]
--db-path <DB_PATH>
SQLite database path override for persisted resources/chunks
[env: WEBFANG_DB_PATH=]
--elastic
Enable elastic ingestion pipeline (streaming, SQLite dedup, Rayon CPU bridge)
[env: WEBFANG_ELASTIC=]
--output-vectors <OUTPUT_VECTORS>
Write extracted vectors to a JSONL file for RAG pipelines. Use `-` for stdout. No SQLite dependency — available in every build (core binary too)
[env: WEBFANG_OUTPUT_VECTORS=]
Batch Processing:
--batch
Enable batch mode — scrape each URL from stdin (one per line), one page per URL, no crawling
[env: WEBFANG_BATCH=]
--batch-file <BATCH_FILE>
Path to a file containing URLs to scrape (one per line), one page per URL, no crawling
[env: WEBFANG_BATCH_FILE=]
--batch-concurrency <BATCH_CONCURRENCY>
Maximum concurrent URLs in batch mode (omit = auto from budget model)
[env: WEBFANG_BATCH_CONCURRENCY=]
Item Pipeline:
--pipeline
Enable item pipeline processing (validate → clean → output)
[env: WEBFANG_PIPELINE=]
--pipeline-output <PIPELINE_OUTPUT>
Pipeline output format: jsonl (default), none
Possible values:
- jsonl: Write items as JSON Lines to a file (default)
- none: No pipeline output — items are processed but not written
[env: WEBFANG_PIPELINE_OUTPUT=]
[default: jsonl]
Obsidian:
--obsidian-wiki-links
Convert same-domain links to Obsidian [[wiki-link]] syntax
[env: WEBFANG_OBSIDIAN_WIKI_LINKS=]
--obsidian-tags <OBSIDIAN_TAGS>
Tags to include in YAML frontmatter (comma-separated)
[env: WEBFANG_OBSIDIAN_TAGS=]
--obsidian-relative-assets
Rewrite downloaded asset paths as relative to the .md file
[env: WEBFANG_OBSIDIAN_RELATIVE_ASSETS=]
--vault <VAULT>
Path to Obsidian vault (auto-detects if not provided).
When provided explicitly, the vault becomes the output base: Markdown,
downloaded assets and the RAG export are written inside it — no need
to duplicate the path in `-o` (which then must stay at its default).
Auto-detected or config-file vaults do NOT redirect output (#762).
[env: WEBFANG_OBSIDIAN_VAULT=]
--quick-save
Quick-save mode: save directly to vault _inbox folder
[env: WEBFANG_OBSIDIAN_QUICK_SAVE=]
--obsidian-rich-metadata
Add rich metadata to frontmatter
[env: WEBFANG_OBSIDIAN_RICH_METADATA=]
AI Settings:
--threshold <THRESHOLD>
Relevance threshold for AI semantic filtering (0.0-1.0)
[env: WEBFANG_THRESHOLD=]
[default: 0.3]
--max-tokens <MAX_TOKENS>
Maximum tokens per chunk before rejection (a chunk-size guard, not a context-window setting; chunks exceeding this fail)
[env: WEBFANG_MAX_TOKENS=]
[default: 32768]
--offline
Run AI model in offline mode
[env: WEBFANG_OFFLINE=]
--ai-model <AI_MODEL>
AI model to use: granite-97m (default, fast) or granite-311m (higher quality)
[env: WEBFANG_AI_MODEL_ID=]
EXIT CODES:
0 Success
2 No URLs discovered
3 All scrapers failed
64 Bad CLI arguments (usage error)
69 WAF block or network error
74 I/O error
76 Protocol error
78 Configuration error
EXAMPLES:
webfang https://example.com
webfang -u https://example.com
webfang -u https://example.com --ai
webfang -u https://example.com -f jsonl
webfang -u https://example.com -v
webfang -u https://example.com -vv # DEBUG
webfang --batch-file urls.txt --resume
$ webfang completions --help
Generate shell completion scripts
Usage: webfang completions <SHELL>
Arguments:
<SHELL>
Shell to generate completions for
Possible values:
- bash: Bash shell completions
- elvish: Elvish shell completions
- fish: Fish shell completions
- power-shell: PowerShell completions
- zsh: Zsh shell completions
Options:
-h, --help
Print help (see a summary with '-h')