---
canonical_url: "https://docs.composio.dev/toolkits/webcrawlerapi"
markdown_url: "https://docs.composio.dev/toolkits/webcrawlerapi.md"
product: "platform"
version: null
last_updated: null
---

# WebCrawlerAPI

WebCrawlerAPI provides crawling, page scraping, AI web agents, and monitored website feeds through a unified API.

- **Category:** ai web scraping
- **Auth:** API_KEY
- **Composio-managed OAuth available?** N/A
- **Tools:** 10
- **Triggers:** 0
- **Slug:** `WEBCRAWLERAPI`
- **Version:** 00000000_00

## Tools

### Create Crawl

**Slug:** `WEBCRAWLERAPI_CREATE_CRAWL`

Start a metered asynchronous crawl over a site and return its job ID. The crawl may visit and charge for up to items_limit pages; use GET_CRAWL_JOB to monitor it.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `url` | string | Yes | Seed URL where crawling starts. |
| `max_age` | integer | No | Maximum cached-result age in seconds. Defaults to allowing cached content up to seven days old; use 0 to disable cache reuse. |
| `max_depth` | integer | No | Maximum link depth; 0 crawls only the seed URL, while omission leaves the depth unrestricted. |
| `items_limit` | integer | No | Maximum number of pages to crawl and potentially charge for. |
| `webhook_url` | string | No | Public HTTPS URL to notify when the crawl finishes. |
| `output_formats` | array | No | Representations to generate for each crawled page. |
| `blacklist_regexp` | string | No | Exclude discovered URLs matching this regular expression. |
| `whitelist_regexp` | string | No | Only crawl discovered URLs matching this regular expression. |
| `keep_query_params` | boolean | No | Keep query parameters when deduplicating discovered URLs. |
| `main_content_only` | boolean | No | Exclude navigation and peripheral content from page output. |
| `respect_robots_txt` | boolean | No | Honor the target site's robots.txt rules. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Create Feed

**Slug:** `WEBCRAWLERAPI_CREATE_FEED`

Create a recurring website-change feed. Creation immediately schedules an initial crawl that may consume metered credits, and future scheduled runs may also consume credits; delete the feed when it is no longer needed.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `url` | string | Yes | Website URL to monitor. |
| `name` | string | No | Human-readable feed name. |
| `max_depth` | integer | No | Maximum link depth for each run; 0 checks only the seed URL. |
| `items_limit` | integer | No | Maximum number of pages checked per feed run. |
| `webhook_url` | string | No | Public HTTPS URL to receive feed-run notifications. |
| `output_format` | string ("markdown" | "cleaned" | "html") | No | Content representation stored for changed pages. |
| `include_errors` | boolean | No | Whether unavailable or error pages should be included in feed output. |
| `blacklist_regexp` | string | No | Exclude URLs matching this regular expression. |
| `whitelist_regexp` | string | No | Only monitor URLs matching this regular expression. |
| `main_content_only` | boolean | No | Whether to exclude navigation and peripheral content when detecting changes. |
| `respect_robots_txt` | boolean | No | Whether to honor the target site's robots.txt rules. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Delete Feed

**Slug:** `WEBCRAWLERAPI_DELETE_FEED`

Permanently cancel a feed so it cannot be resumed. The feed is removed from feed listings, but retained history may remain directly readable for 30 days.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `feed_id` | string | Yes | Feed ID to permanently cancel. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Get Crawl Job

**Slug:** `WEBCRAWLERAPI_GET_CRAWL_JOB`

Get a crawl job's status, configuration, per-page results, content URLs, errors, and observed costs. Respect recommended_pull_delay_ms when polling.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `crawl_id` | string | Yes | Crawl job ID returned by CREATE_CRAWL. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Get Feed

**Slug:** `WEBCRAWLERAPI_GET_FEED`

Get one feed's configuration, lifecycle status, and recent run history, including per-run crawl counts and cost. Canceled feeds remain directly readable while retained by the provider.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `feed_id` | string | Yes | Feed ID returned by CREATE_FEED or LIST_FEEDS. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Get Organization Costs

**Slug:** `WEBCRAWLERAPI_GET_ORGANIZATION_COSTS`

Return current spendable balance plus request count and USD usage for a date range. The connected API key is used for both organization routes.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `to_date` | string | No | Inclusive usage end date in YYYY-MM-DD format; defaults to the current UTC date. |
| `from_date` | string | No | Inclusive usage start date in YYYY-MM-DD format; defaults to 30 days before to_date. |
| `include_daily` | boolean | No | Include one usage row per day, including dates with no usage. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### List Feed Changes

**Slug:** `WEBCRAWLERAPI_LIST_FEED_CHANGES`

Return one page of detected feed changes as structured JSON, with an opaque continuation cursor for older pages.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `feed_id` | string | Yes | Feed ID whose detected changes should be returned. |
| `page_size` | integer | No | Maximum changes to return on this page (1-1000). |
| `next_cursor` | string | No | Opaque cursor returned by the previous call; omit for the newest page. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### List Feeds

**Slug:** `WEBCRAWLERAPI_LIST_FEEDS`

List active and paused feeds for the connected organization, newest first. Logically canceled feeds are omitted.

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Scrape Page

**Slug:** `WEBCRAWLERAPI_SCRAPE_PAGE`

Scrape one web page synchronously and return requested content or structured extraction. This consumes metered credits; supplying a prompt adds $0.002 per request.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `url` | string | Yes | Absolute HTTP or HTTPS URL of the page to scrape. |
| `prompt` | string | No | AI extraction instruction. Supplying this adds $0.002 to the request. |
| `max_age` | integer | No | Maximum cached-result age in seconds. Defaults to allowing cached content up to seven days old; use 0 to force a fresh scrape. |
| `output_formats` | array | No | Content representations to return. |
| `clean_selectors` | string | No | Comma-separated CSS selectors to remove before content extraction. |
| `response_schema` | object | No | JSON Schema for structured_data when prompt is supplied. |
| `keep_query_params` | boolean | No | Preserve query parameters when identifying the target URL. |
| `main_content_only` | boolean | No | Exclude navigation and other peripheral page content. |
| `respect_robots_txt` | boolean | No | Honor the target site's robots.txt rules. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Set Feed State

**Slug:** `WEBCRAWLERAPI_SET_FEED_STATE`

Pause an active feed's future scheduled runs or resume a paused feed. Pausing does not stop a run that has already started, and resuming schedules future crawls that may consume metered credits.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `feed_id` | string | Yes | Feed ID to pause or resume. |
| `desired_state` | string ("paused" | "active") | Yes | Set paused to stop future schedules or active to resume scheduling. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |