This project uses Composio to automate the creation and management of competitor pages in Notion. It scrapes data from competitor websites and generates Notion pages under a specified parent page. If a page with the same name already exists, a unique identifier is added. This ensures that your competitor information in Notion is always up-to-date and well-organized.

Competitor Researcher Agent GitHub Repository

Explore the complete source code for the Competitor Researcher Agent project. This repository contains all the necessary files and scripts to set up and run the Competitor Researcher system using CrewAI and Composio.

1

Run the `setup.sh` file

Fork and Clone this repository, Navigate to the Project Directory

cd python/examples/competitor_researcher

Make the setup.sh Script Executable (if necessary): On Linux or macOS, you might need to make the setup.sh script executable:

2

Import base packages

In your Python script, import the required libraries:

3

Initialise Language Model and Define tools

We’ll initialize our language model and set up the necessary tools for our agents.

4

Scrape Webpage

Define a function to scrape the competitor’s website and extract the information you need.

  • Helper function remove_tags(html) uses BeautifulSoup to remove HTML tags from a string, returning clean text.
  • scrape_website(url) function fetches content from the specified URL. This function uses requests to get the webpage, processes the HTML content to remove tags, and returns the cleaned text. Handles potential request errors gracefully.
5

Create and Execute Agent

Set up the agent that will interact with Notion and create the required pages with the scraped data.

Putting it All Together