Competitor Researcher
This guide will walk you through setting up a system that scrapes competitor websites and automatically adds detailed research notes to your Notion workspace. Let’s get started!
This project uses Composio to automate the creation and management of competitor pages in Notion. It scrapes data from competitor websites and generates Notion pages under a specified parent page. If a page with the same name already exists, a unique identifier is added. This ensures that your competitor information in Notion is always up-to-date and well-organized.
Competitor Researcher Agent GitHub Repository
Explore the complete source code for the Competitor Researcher Agent project. This repository contains all the necessary files and scripts to set up and run the Competitor Researcher system using CrewAI and Composio.
Run the `setup.sh` file
Fork and Clone this repository, Navigate to the Project Directory
cd python/examples/competitor_researcher
Make the setup.sh Script Executable (if necessary): On Linux or macOS, you might need to make the setup.sh script executable:
Import base packages
In your Python script, import the required libraries:
Initialise Language Model and Define tools
We’ll initialize our language model and set up the necessary tools for our agents.
Scrape Webpage
Define a function to scrape the competitor’s website and extract the information you need.
- Helper function
remove_tags(html)uses BeautifulSoup to remove HTML tags from a string, returning clean text. scrape_website(url)function fetches content from the specified URL. This function uses requests to get the webpage, processes the HTML content to remove tags, and returns the cleaned text. Handles potential request errors gracefully.
Create and Execute Agent
Set up the agent that will interact with Notion and create the required pages with the scraped data.
Putting it All Together
Was this page helpful?