---
canonical_url: "https://docs.composio.dev/toolkits/crawlbase"
markdown_url: "https://docs.composio.dev/toolkits/crawlbase.md"
product: "platform"
version: null
last_updated: null
---

# Crawlbase

Crawlbase provides APIs for crawling web pages, managing crawler queues, storing crawl results, and monitoring account usage.

- **Category:** web scraping & browser
- **Auth:** API_KEY
- **Composio-managed OAuth available?** N/A
- **Tools:** 8
- **Triggers:** 0
- **Slug:** `CRAWLBASE`
- **Version:** 20261014_00

## Tools

### Crawl URL

**Slug:** `CRAWLBASE_CRAWL_URL`

Fetch a public HTTP or HTTPS URL through Crawlbase using the connected Normal token and return its content with separate target and Crawlbase statuses.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `url` | string | Yes | Public HTTP or HTTPS URL to crawl. |
| `store` | boolean | No | Store the crawl result in Crawlbase Cloud Storage when true. |
| `device` | string ("desktop" | "tablet" | "mobile") | No | Device profile used for the request. |
| `cookies` | string | No | Optional cookie string to send to the target. |
| `country` | string | No | Optional country code for geolocated crawling. |
| `scraper` | string | No | Optional Crawlbase scraper name for structured extraction. |
| `autoparse` | boolean | No | Ask Crawlbase to automatically parse supported pages. |
| `user_agent` | string | No | Optional User-Agent value to send to the target. |
| `tor_network` | boolean | No | Route through the Tor network when true. |
| `output_format` | string ("html" | "json" | "markdown") | No | Content representation to request. Markdown uses Crawlbase's md format. |
| `cookie_session` | string | No | Optional Crawlbase cookie-session identifier, at most 32 characters. |
| `request_headers` | object | No | Optional headers to send to the target URL. |
| `custom_success_codes` | array | No | Additional target HTTP status codes Crawlbase should treat as successful. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Delete Stored Pages

**Slug:** `CRAWLBASE_DELETE_STORED_PAGES`

Irreversibly delete explicitly selected pages from Crawlbase Cloud Storage by request ID and return each page's deletion outcome.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `rids` | array | Yes | One to 100 unique request IDs of stored pages to delete irreversibly. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Generate User Agents

**Slug:** `CRAWLBASE_GENERATE_USER_AGENTS`

Generate one to ten realistic randomized User-Agent strings for a selected device class.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `size` | integer | No | Number of User-Agent strings to return, from 1 to 10. |
| `device` | string ("desktop" | "mobile") | No | Device class for generated User-Agent strings. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Get Crawling Usage

**Slug:** `CRAWLBASE_GET_CRAWLING_USAGE`

Return current Crawling API usage and optionally previous-month statistics for the connected Crawlbase account.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `include_previous_month` | boolean | No | Include previous-month crawling usage statistics when available. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Get Storage Count

**Slug:** `CRAWLBASE_GET_STORAGE_COUNT`

Return the number of pages currently held in Crawlbase Cloud Storage.

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Get Stored Page

**Slug:** `CRAWLBASE_GET_STORED_PAGE`

Retrieve the latest Crawlbase Cloud Storage page by exactly one request ID or original URL.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `rid` | string | No | Stored request ID. Provide exactly one of rid or url. |
| `url` | string | No | Original HTTP or HTTPS URL. Provide exactly one of url or rid. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### Get Stored Pages

**Slug:** `CRAWLBASE_GET_STORED_PAGES`

Retrieve up to 100 Crawlbase Cloud Storage pages by request ID without deleting them. Returns decoded page content and identifies requested IDs that were not found.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `rids` | array | Yes | One to 100 unique request IDs for stored pages to retrieve. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |

### List Stored Page IDs

**Slug:** `CRAWLBASE_LIST_STORED_PAGE_IDS`

Return one page of Crawlbase Cloud Storage request IDs and a short-lived continuation cursor.

#### Input Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `limit` | integer | No | Maximum number of stored request IDs to return, from 1 to 1000. |
| `order` | string ("asc" | "desc") | No | Order stored request IDs from oldest to newest (asc) or newest to oldest (desc). |
| `next_cursor` | string | No | Short-lived cursor from the previous response. Omit for the first page and use a returned cursor within about 15 seconds. |

#### Output

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `data` | string | Yes | Data from the action execution |
| `error` | string | No | Error if any occurred during the execution of the action |
| `successful` | boolean | Yes | Whether or not the action execution was successful or not |