Scrapingbee

Learn how to use Scrapingbee with Composio

Overview

SLUG: SCRAPINGBEE

Description

ScrapingBee is a web scraping API that handles headless browsers and proxy rotation, allowing developers to extract HTML from any website in a single API call.

Authentication Details

generic_api_key
stringRequired

Connecting to Scrapingbee

Create an auth config

Use the dashboard to create an auth config for the Scrapingbee toolkit. This allows you to connect multiple Scrapingbee accounts to Composio for agents to use.

1

Select App

Navigate to Scrapingbee.

2

Configure Auth Config Settings

Select among the supported auth schemes of and configure them here.

3

Create and Get auth config ID

Click “Create Scrapingbee Auth Config”. After creation, copy the displayed ID starting with ac_. This is your auth config ID. This is not a sensitive ID — you can save it in environment variables or a database. This ID will be used to create connections to the toolkit for a given user.

Connect Your Account

Using API Key

1from composio import Composio
2
3# Replace these with your actual values
4scrapingbee_auth_config_id = "ac_YOUR_SCRAPINGBEE_CONFIG_ID" # Auth config ID created above
5user_id = "0000-0000-0000" # UUID from database/app
6
7composio = Composio()
8
9def authenticate_toolkit(user_id: str, auth_config_id: str):
10 # Replace this with a method to retrieve an API key from the user.
11 # Or supply your own.
12 user_api_key = input("[!] Enter API key")
13
14 connection_request = composio.connected_accounts.initiate(
15 user_id=user_id,
16 auth_config_id=auth_config_id,
17 config={"auth_scheme": "API_KEY", "val": {"generic_api_key": user_api_key}}
18 )
19
20 # API Key authentication is immediate - no redirect needed
21 print(f"Successfully connected Scrapingbee for user {user_id}")
22 print(f"Connection status: {connection_request.status}")
23
24 return connection_request.id
25
26
27connection_id = authenticate_toolkit(user_id, scrapingbee_auth_config_id)
28
29# You can verify the connection using:
30connected_account = composio.connected_accounts.get(connection_id)
31print(f"Connected account: {connected_account}")

Tools

Executing tools

To prototype you can execute some tools to see the responses and working on the Scrapingbee toolkit’s playground

Python
1from composio import Composio
2from openai import OpenAI
3import json
4
5openai = OpenAI()
6composio = Composio()
7
8# User ID must be a valid UUID format
9user_id = "0000-0000-0000" # Replace with actual user UUID from your database
10
11tools = composio.tools.get(user_id=user_id, toolkits=["SCRAPINGBEE"])
12
13print("[!] Tools:")
14print(json.dumps(tools))
15
16def invoke_llm(task = "What can you do?"):
17 completion = openai.chat.completions.create(
18 model="gpt-4o",
19 messages=[
20 {
21 "role": "user",
22 "content": task, # Your task here!
23 },
24 ],
25 tools=tools,
26 )
27
28 # Handle Result from tool call
29 result = composio.provider.handle_tool_calls(user_id=user_id, response=completion)
30 print(f"[!] Completion: {completion}")
31 print(f"[!] Tool call result: {result}")
32
33invoke_llm()

Tool List

Tool Name: ScrapingBee Data Extraction

Description

Tool to extract structured data from a webpage using css or xpath selectors. use scrapingbee's extract rules feature.

Action Parameters

api_key
stringRequired
block_resources
boolean
country_code
string
custom_headers
object
device
stringDefaults to desktop
extractor
objectRequired
javascript
boolean
premium_proxy
boolean
url
stringRequired
wait
integer

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: ScrapingBee HTML Fetch

Description

Tool to fetch html or screenshot via scrapingbee html api. use when you need page markup or image after optional js rendering and resource controls.

Action Parameters

block_ads
boolean
block_resources
string
cookies
string
country_code
string
device
string
extract_rules
string
js_scenario
string
js_snippet
string
premium_proxy
boolean
render_js
boolean
retry
integer
screenshot
boolean
screenshot_full_page
boolean
screenshot_selector
string
stealth_proxy
boolean
url
stringRequired
wait
integer
wait_for
string

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: ScrapingBee Proxy Mode

Description

Tool to fetch web content via scrapingbee's proxy mode. use when you need to route requests through scrapingbee proxies with optional js rendering and resource blocking.

Action Parameters

block_ads
boolean
block_resources
array
cookies
object
country_code
string
forward_headers
boolean
headers
object
js_scenario
string
premium_proxy
boolean
render_js
boolean
session_id
string
stealth_proxy
boolean
timeout
integerDefaults to 60000
url
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: ScrapingBee Stealth Proxy

Description

Tool to perform stealth scraping via scrapingbee's stealth proxy mode. use when you encounter anti-bot measures requiring undetectable requests.

Action Parameters

block_resources
boolean
cookies
string
country_code
string
device
string
extract_rules
string
forward_headers
boolean
headers
string
js_render
boolean
premium_proxy
boolean
return_page_source
boolean
stealth_proxy
boolean
url
stringRequired
wait
integer

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: ScrapingBee Usage Stats

Description

Tool to retrieve usage statistics for your scrapingbee account. use when you need to monitor remaining credits and request count.

Action Parameters

Action Response

data
objectRequired
error
string
successful
booleanRequired