Ai Observability

Honeyhive

Learn how to use Honeyhive with Composio

Overview

SLUG: HONEYHIVE

Description

HoneyHive is a modern AI observability and evaluation platform that enables developers and domain experts to collaboratively build reliable AI applications faster.

Authentication Details

generic_api_key
stringRequired

Connecting to Honeyhive

Create an auth config

Use the dashboard to create an auth config for the Honeyhive toolkit. This allows you to connect multiple Honeyhive accounts to Composio for agents to use.

1

Select App

Navigate to Honeyhive.

2

Configure Auth Config Settings

Select among the supported auth schemes of and configure them here.

3

Create and Get auth config ID

Click “Create Honeyhive Auth Config”. After creation, copy the displayed ID starting with ac_. This is your auth config ID. This is not a sensitive ID — you can save it in environment variables or a database. This ID will be used to create connections to the toolkit for a given user.

Connect Your Account

Using API Key

1from composio import Composio
2
3# Replace these with your actual values
4honeyhive_auth_config_id = "ac_YOUR_HONEYHIVE_CONFIG_ID" # Auth config ID created above
5user_id = "0000-0000-0000"  # UUID from database/app
6
7composio = Composio()
8
9def authenticate_toolkit(user_id: str, auth_config_id: str):
10    # Replace this with a method to retrieve an API key from the user.
11    # Or supply your own.
12    user_api_key = input("[!] Enter API key")
13
14    connection_request = composio.connected_accounts.initiate(
15        user_id=user_id,
16        auth_config_id=auth_config_id,
17        config={"auth_scheme": "API_KEY", "val": {"generic_api_key": user_api_key}}
18    )
19
20    # API Key authentication is immediate - no redirect needed
21    print(f"Successfully connected Honeyhive for user {user_id}")
22    print(f"Connection status: {connection_request.status}")
23    
24    return connection_request.id
25
26
27connection_id = authenticate_toolkit(user_id, honeyhive_auth_config_id)
28
29# You can verify the connection using:
30connected_account = composio.connected_accounts.get(connection_id)
31print(f"Connected account: {connected_account}")

Tools

Executing tools

To prototype you can execute some tools to see the responses and working on the Honeyhive toolkit’s playground

Python
1from composio import Composio
2from openai import OpenAI
3import json
4
5openai = OpenAI()
6composio = Composio()
7
8# User ID must be a valid UUID format
9user_id = "0000-0000-0000"  # Replace with actual user UUID from your database
10
11tools = composio.tools.get(user_id=user_id, toolkits=["HONEYHIVE"])
12
13print("[!] Tools:")
14print(json.dumps(tools))
15
16def invoke_llm(task = "What can you do?"):
17    completion = openai.chat.completions.create(
18        model="gpt-4o",
19        messages=[
20            {
21                "role": "user",
22                "content": task,  # Your task here!
23            },
24        ],
25        tools=tools,
26    )
27
28    # Handle Result from tool call
29    result = composio.provider.handle_tool_calls(user_id=user_id, response=completion)
30    print(f"[!] Completion: {completion}")
31    print(f"[!] Tool call result: {result}")
32
33invoke_llm()

Tool List

Tool Name: Add datapoints to dataset

Description

Tool to add datapoints to a dataset. Use when you need to append multiple entries with specified input, ground truth, and history mappings.

Action Parameters

data
arrayRequired
dataset_id
stringRequired
mapping
objectRequired
project
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Create Batch Model Events

Description

Tool to create multiple model events in a single request. Use when you need to log a batch of event interactions to HoneyHive.

Action Parameters

model_events
arrayRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Create Batch Tool Events

Description

Tool to log a batch of external API calls as tool events. Use when you need to record multiple tool events in one request—use after gathering all event data.

Action Parameters

events
arrayRequired
is_single_session
boolean

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Create Dataset

Description

Tool to create a dataset. Use when you need to initialize a new dataset within a project.

Action Parameters

datapoints
array
description
string
linked_evals
array
metadata
object
name
stringRequired
pipeline_type
stringDefaults to event
project
stringRequired
saved
boolean
type
stringDefaults to evaluation

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Create Tool

Description

Tool to create a new tool. Use when you need to register a new function or plugin for invocation.

Action Parameters

description
string
name
stringRequired
parameters
objectRequired
task
stringRequired
type
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Delete Datapoint

Description

Tool to delete a specific datapoint by its ID. Use when you need to remove a datapoint from HoneyHive after confirming its identifier.

Action Parameters

id
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Delete Dataset

Description

Tool to delete a dataset by ID. Use when you need to remove a dataset after confirming its ID.

Action Parameters

dataset_id
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: End Evaluation Run

Description

Tool to mark an evaluation run as completed. Use after finishing manual evaluations to update the run status to completed.

Action Parameters

configuration
object
datapoint_ids
array
dataset_id
string
event_ids
array
metadata
object
name
string
run_id
stringRequired
status
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Get Configurations

Description

Tool to retrieve a list of configurations. Use when you need to fetch all configurations for a specific project before making changes.

Action Parameters

env
string
name
string
project
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Get Datasets

Description

Tool to retrieve a list of datasets. Use when you need to fetch datasets for a specific project with optional filters.

Action Parameters

dataset_id
string
project
stringRequired
type
string

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Get Metrics

Description

Tool to retrieve all metrics. Use when you need to list metrics for a specific project, after obtaining project context.

Action Parameters

project_name
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Get Projects

Description

Tool to retrieve projects. Use when you need to list all available projects.

Action Parameters

name
string

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: List Tools

Description

Tool to list all available Honeyhive tools. Use when you need to discover which functions or plugins are registered for use.

Action Parameters

limit
integer
page
integer

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Retrieve Datapoint

Description

Tool to retrieve a specific datapoint by its ID. Use when you have a datapoint ID and need its full details.

Action Parameters

id
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Retrieve Datapoints

Description

Tool to retrieve a list of datapoints. Use when you need to fetch datapoints for a project with optional filters.

Action Parameters

datapoint_ids
array
dataset_name
string
project
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Retrieve Events

Description

Tool to retrieve events by filters. Use when you need to export events based on filter criteria, date range, and pagination.

Action Parameters

dateRange
object
filters
arrayRequired
limit
integer
page
integer
project
stringRequired
projections
array

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Retrieve Experiment Result

Description

Tool to retrieve the result of a specific experiment run. Use when you need the status, metrics, and datapoint-level details of a completed experiment.

Action Parameters

aggregate_function
string
project_id
stringRequired
run_id
stringRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Start Evaluation Run

Description

Tool to initiate an evaluation run using external datasets. Use after selecting a project and events; optionally link a dataset.

Action Parameters

configuration
object
datapoint_ids
array
dataset_id
string
event_ids
arrayRequired
metadata
object
name
stringRequired
project
stringRequired
status
string

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Start Session

Description

Tool to start a new session. Use when you need to initiate a new tracking session and retrieve its session_id.

Action Parameters

session
objectRequired

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Update Datapoint

Description

Tool to update a specific datapoint. Use when you need to modify fields of an existing datapoint.

Action Parameters

ground_truth
object
history
array
id
stringRequired
inputs
object
linked_datasets
array
linked_evals
array
metadata
object

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Update Dataset

Description

Tool to update an existing dataset. Use when you need to modify a dataset's details (name, description, datapoints, linked evaluations, or metadata) after confirming its ID.

Action Parameters

datapoints
array
dataset_id
stringRequired
description
string
linked_evals
array
metadata
object
name
string

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Update Event

Description

Tool to update an event. Use when updating event details by ID.

Action Parameters

config
object
duration
number
event_id
stringRequired
feedback
object
metadata
object
metrics
object
outputs
object
user_properties
object

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Update Metric

Description

Tool to update an existing metric. Use when you need to modify a metric’s properties after creation. Ensure you retrieve the metric first to verify its current state.

Action Parameters

code_snippet
string
criteria
string
description
string
enabled_in_prod
boolean
event_name
string
event_type
string
metric_id
stringRequired
name
string
needs_ground_truth
boolean
pass_when
boolean
prompt
string
return_type
string
threshold
object
type
string

Action Response

data
objectRequired
error
string
successful
booleanRequired

Tool Name: Update Project

Description

Tool to update a project's name or description. Use when you need to modify an existing project by its ID after creation.

Action Parameters

description
string
id
stringRequired
name
string

Action Response

data
objectRequired
error
string
successful
booleanRequired