Scrapegraph ai

Learn how to use Scrapegraph ai with Composio

Overview

SLUG: SCRAPEGRAPH_AI

Description

ScrapeGraphAI is an AI-powered web scraping API that enables developers to extract structured data from any website using natural language prompts. Website https://scrapegraphai.com

Authentication Details

generic_api_key
stringRequired

Connecting to Scrapegraph ai

Create an auth config

Use the dashboard to create an auth config for the Scrapegraph ai toolkit. This allows you to connect multiple Scrapegraph ai accounts to Composio for agents to use.

1

Select App

Navigate to [Scrapegraph ai](https://platform.composio.dev/marketplace/Scrapegraph ai).

2

Configure Auth Config Settings

Select among the supported auth schemes of and configure them here.

3

Create and Get auth config ID

Click “Create Scrapegraph ai Auth Config”. After creation, copy the displayed ID starting with ac_. This is your auth config ID. This is not a sensitive ID — you can save it in environment variables or a database. This ID will be used to create connections to the toolkit for a given user.

Connect Your Account

Using API Key

1from composio import Composio
2
3# Replace these with your actual values
4scrapegraph_ai_auth_config_id = "ac_YOUR_SCRAPEGRAPH_AI_CONFIG_ID" # Auth config ID created above
5user_id = "0000-0000-0000" # UUID from database/app
6
7composio = Composio()
8
9def authenticate_toolkit(user_id: str, auth_config_id: str):
10 # Replace this with a method to retrieve an API key from the user.
11 # Or supply your own.
12 user_api_key = input("[!] Enter API key")
13
14 connection_request = composio.connected_accounts.initiate(
15 user_id=user_id,
16 auth_config_id=auth_config_id,
17 config={"auth_scheme": "API_KEY", "val": user_api_key}
18 )
19
20 # API Key authentication is immediate - no redirect needed
21 print(f"Successfully connected Scrapegraph ai for user {user_id}")
22 print(f"Connection status: {connection_request.status}")
23
24 return connection_request.id
25
26
27connection_id = authenticate_toolkit(user_id, scrapegraph_ai_auth_config_id)
28
29# You can verify the connection using:
30connected_account = composio.connected_accounts.get(connection_id)
31print(f"Connected account: {connected_account}")

Tools

Executing tools

To prototype you can execute some tools to see the responses and working on the [Scrapegraph ai toolkit’s playground](https://app.composio.dev/app/Scrapegraph ai)

Python
1from composio import Composio
2from openai import OpenAI
3import json
4
5openai = OpenAI()
6composio = Composio()
7
8# User ID must be a valid UUID format
9user_id = "0000-0000-0000" # Replace with actual user UUID from your database
10
11tools = composio.tools.get(user_id=user_id, toolkits=["SCRAPEGRAPH_AI"])
12
13print("[!] Tools:")
14print(json.dumps(tools))
15
16def invoke_llm(task = "What can you do?"):
17 completion = openai.chat.completions.create(
18 model="gpt-4o",
19 messages=[
20 {
21 "role": "user",
22 "content": task, # Your task here!
23 },
24 ],
25 tools=tools,
26 )
27
28 # Handle Result from tool call
29 result = composio.provider.handle_tool_calls(user_id=user_id, response=completion)
30 print(f"[!] Completion: {completion}")
31 print(f"[!] Tool call result: {result}")
32
33invoke_llm()

Tool List

Tool Name: Get Credits

Description

Retrieve remaining and used credits for your scrapegraphai account.

Action Parameters

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Markdownify Status

Description

Check status and results of webpage-to-markdown conversion jobs.

Action Parameters

request_id
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Search Scraper

Description

Perform ai-powered web searches with structured, parsed results.

Action Parameters

language
string
num_results
integer
parse
boolean
query
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Start Smart Crawler

Description

Start ai-powered web crawling with structured data extraction across multiple pages.

Action Parameters

depth
integer
headers
object
max_pages
integer
number_of_scrolls
integer
output_schema
object
prompt
stringRequired
rules
object
sitemap
boolean
url
stringRequired
wait
boolean
website_html
string

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: SmartCrawler Status

Description

Check status and results of smartcrawler web crawling jobs.

Action Parameters

task_id
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Start Smart Scraper

Description

Start ai-powered web scraping with natural language extraction prompts.

Action Parameters

headers
object
output_schema
object
user_prompt
stringRequired
wait
boolean
website_html
string
website_url
string

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: SmartScraper Status

Description

Check status and results of smartscraper web scraping jobs.

Action Parameters

request_id
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Convert Webpage to Markdown

Description

Tool to convert a specified webpage into clean, formatted markdown.

Action Parameters

wait
boolean
website_url
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Check SearchScraper Status

Description

Check the status and results of an asynchronous searchscraper job.

Action Parameters

request_id
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Submit Feedback

Description

Submit feedback and ratings for completed scrapegraphai requests.

Action Parameters

feedback_text
string
rating
integerRequired
request_id
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired