Scrapegraph ai

Learn how to use Scrapegraph ai with Composio

Overview

SLUG: SCRAPEGRAPH_AI

Description

ScrapeGraphAI is an AI-powered web scraping API that enables developers to extract structured data from any website using natural language prompts.

Authentication Details

generic_api_key
stringRequired

Connecting to Scrapegraph ai

Create an auth config

Use the dashboard to create an auth config for the Scrapegraph ai toolkit. This allows you to connect multiple Scrapegraph ai accounts to Composio for agents to use.

1

Select App

Navigate to the [Scrapegraph ai toolkit page](https://app.composio.dev/apps/Scrapegraph ai) and click “Setup Integration”.

2

Configure Auth Config Settings

Select among the supported auth schemes of and configure them here.

3

Create and Get auth config ID

Click “Create Integration”. After creation, copy the displayed ID starting with ac_. This is your auth config ID. This is not a sensitive ID — you can save it in environment variables or a database. This ID will be used to create connections to the toolkit for a given user.

Connect Your Account

Using API Key

1from composio import Composio
2from composio.types import auth_scheme
3
4# Replace these with your actual values
5scrapegraph_ai_auth_config_id = "ac_YOUR_SCRAPEGRAPH_AI_CONFIG_ID" # Auth config ID created above
6user_id = "0000-0000-0000" # UUID from database/app
7
8composio = Composio()
9
10def authenticate_toolkit(user_id: str, auth_config_id: str):
11 # Replace this with a method to retrieve an API key from the user.
12 # Or supply your own.
13 user_api_key = input("[!] Enter API key")
14
15 connection_request = composio.connected_accounts.initiate(
16 user_id=user_id,
17 auth_config_id=auth_config_id,
18 config={"auth_scheme": "API_KEY", "val": user_api_key}
19 )
20
21 # API Key authentication is immediate - no redirect needed
22 print(f"Successfully connected Scrapegraph ai for user {user_id}")
23 print(f"Connection status: {connection_request.status}")
24
25 return connection_request.id
26
27
28connection_id = authenticate_toolkit(user_id, scrapegraph_ai_auth_config_id)
29
30# You can verify the connection using:
31connected_account = composio.connected_accounts.get(connection_id)
32print(f"Connected account: {connected_account}")

Tools

Executing tools

To prototype you can execute some tools to see the responses and working on the [Scrapegraph ai toolkit’s playground](https://app.composio.dev/app/Scrapegraph ai)

Python
1from composio import Composio
2from openai import OpenAI
3import json
4
5openai = OpenAI()
6composio = Composio()
7
8# User ID must be a valid UUID format
9user_id = "0000-0000-0000" # Replace with actual user UUID from your database
10
11tools = composio.tools.get(user_id=user_id, toolkits=["SCRAPEGRAPH_AI"])
12
13print("[!] Tools:")
14print(json.dumps(tools))
15
16def invoke_llm(task = "What can you do?"):
17 completion = openai.chat.completions.create(
18 model="gpt-4o",
19 messages=[
20 {
21 "role": "user",
22 "content": task, # Your task here!
23 },
24 ],
25 tools=tools,
26 )
27
28 # Handle Result from tool call
29 result = composio.provider.handle_tool_calls(user_id=user_id, response=completion)
30 print(f"[!] Completion: {completion}")
31 print(f"[!] Tool call result: {result}")
32
33invoke_llm()

Tool List

Tool Name: Start LocalScraper

Description

Tool to initiate the localscraper service. use when you already have raw html content and need ai-based extraction according to your prompt. limited to 2 mb input.

Action Parameters

output_schema
object
user_prompt
stringRequired
website_html
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Search Scraper

Description

Tool to perform ai-powered web searches with structured results. use when you need to gather and optionally parse top search results based on a user query.

Action Parameters

language
string
num_results
integer
parse
boolean
query
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Start SmartScraper

Description

Tool to start the smartscraper ai service. use when you need to extract specific information from a webpage using a natural language prompt. example: "extract the main heading and description from the homepage."

Action Parameters

headers
object
output_schema
object
user_prompt
stringRequired
website_html
string
website_url
string

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: SmartScraper Status

Description

Tool to retrieve the status and results of a smartscraper request. use when you need to check the current state and fetch results of a previously initiated scraping job.

Action Parameters

request_id
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Convert Webpage to Markdown

Description

Tool to convert a specified webpage into clean, formatted markdown. use when you need a markdown representation of a webpage, e.g., for documentation or note-taking workflows.

Action Parameters

website_url
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired