Ai Observability

Honeyhive

Learn how to use Honeyhive with Composio

Overview

SLUG: HONEYHIVE

Description

HoneyHive is a modern AI observability and evaluation platform that enables developers and domain experts to collaboratively build reliable AI applications faster.

Authentication Details

generic_api_key
stringRequired

Connecting to Honeyhive

Create an auth config

Use the dashboard to create an auth config for the Honeyhive toolkit. This allows you to connect multiple Honeyhive accounts to Composio for agents to use.

1

Select App

Navigate to Honeyhive.

2

Configure Auth Config Settings

Select among the supported auth schemes of and configure them here.

3

Create and Get auth config ID

Click “Create Honeyhive Auth Config”. After creation, copy the displayed ID starting with ac_. This is your auth config ID. This is not a sensitive ID — you can save it in environment variables or a database. This ID will be used to create connections to the toolkit for a given user.

Connect Your Account

Using API Key

1from composio import Composio
2
3# Replace these with your actual values
4honeyhive_auth_config_id = "ac_YOUR_HONEYHIVE_CONFIG_ID" # Auth config ID created above
5user_id = "0000-0000-0000"  # UUID from database/app
6
7composio = Composio()
8
9def authenticate_toolkit(user_id: str, auth_config_id: str):
10    # Replace this with a method to retrieve an API key from the user.
11    # Or supply your own.
12    user_api_key = input("[!] Enter API key")
13
14    connection_request = composio.connected_accounts.initiate(
15        user_id=user_id,
16        auth_config_id=auth_config_id,
17        config={"auth_scheme": "API_KEY", "val": {"generic_api_key": user_api_key}}
18    )
19
20    # API Key authentication is immediate - no redirect needed
21    print(f"Successfully connected Honeyhive for user {user_id}")
22    print(f"Connection status: {connection_request.status}")
23    
24    return connection_request.id
25
26
27connection_id = authenticate_toolkit(user_id, honeyhive_auth_config_id)
28
29# You can verify the connection using:
30connected_account = composio.connected_accounts.get(connection_id)
31print(f"Connected account: {connected_account}")

Tools

Executing tools

To prototype you can execute some tools to see the responses and working on the Honeyhive toolkit’s playground

For code examples, see the Tool calling guide and Provider examples.

Tool List

Tool Name: Add datapoints to dataset

Description

Tool to add datapoints to a dataset. Use when you need to append multiple entries with specified input, ground truth, and history mappings.

Action Parameters

data
arrayRequired
dataset_id
stringRequired
mapping
objectRequired
project
stringRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Create Batch Model Events

Description

Tool to create multiple model events in a single request. Use when you need to log a batch of event interactions to HoneyHive.

Action Parameters

model_events
arrayRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Create Batch Tool Events

Description

Tool to log a batch of external API calls as tool events. Use when you need to record multiple tool events in one request—use after gathering all event data.

Action Parameters

events
arrayRequired
is_single_session
boolean

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Create Dataset

Description

Tool to create a dataset. Use when you need to initialize a new dataset within a project.

Action Parameters

datapoints
description
linked_evals
metadata
name
stringRequired
pipeline_type
stringDefaults to event
project
stringRequired
saved
type
stringDefaults to evaluation

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Create Tool

Description

Tool to create a new tool. Use when you need to register a new function or plugin for invocation.

Action Parameters

description
name
stringRequired
parameters
objectRequired
task
stringRequired
type
stringRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Delete Datapoint

Description

Tool to delete a specific datapoint by its ID. Use when you need to remove a datapoint from HoneyHive after confirming its identifier.

Action Parameters

id
stringRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Delete Dataset

Description

Tool to delete a dataset by ID. Use when you need to remove a dataset after confirming its ID.

Action Parameters

dataset_id
stringRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: End Evaluation Run

Description

Tool to mark an evaluation run as completed. Use after finishing manual evaluations to update the run status to completed.

Action Parameters

configuration
datapoint_ids
dataset_id
event_ids
metadata
name
run_id
stringRequired
status
stringRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Get Configurations

Description

Tool to retrieve a list of configurations. Use when you need to fetch all configurations for a specific project before making changes.

Action Parameters

env
name
project
stringRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Get Datasets

Description

Tool to retrieve a list of datasets. Use when you need to fetch datasets for a specific project with optional filters.

Action Parameters

dataset_id
project
stringRequired
type

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Get Metrics

Description

Tool to retrieve all metrics. Use when you need to list metrics for a specific project, after obtaining project context.

Action Parameters

project_name
stringRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Get Projects

Description

Tool to retrieve projects. Use when you need to list all available projects.

Action Parameters

name

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: List Tools

Description

Tool to list all available Honeyhive tools. Use when you need to discover which functions or plugins are registered for use.

Action Parameters

limit
page

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Retrieve Datapoint

Description

Tool to retrieve a specific datapoint by its ID. Use when you have a datapoint ID and need its full details.

Action Parameters

id
stringRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Retrieve Datapoints

Description

Tool to retrieve a list of datapoints. Use when you need to fetch datapoints for a project with optional filters.

Action Parameters

datapoint_ids
dataset_name
project
stringRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Retrieve Events

Description

Tool to retrieve events by filters. Use when you need to export events based on filter criteria, date range, and pagination.

Action Parameters

dateRange
filters
arrayRequired
limit
page
project
stringRequired
projections

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Retrieve Experiment Result

Description

Tool to retrieve the result of a specific experiment run. Use when you need the status, metrics, and datapoint-level details of a completed experiment.

Action Parameters

aggregate_function
project_id
stringRequired
run_id
stringRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Start Evaluation Run

Description

Tool to initiate an evaluation run using external datasets. Use after selecting a project and events; optionally link a dataset.

Action Parameters

configuration
datapoint_ids
dataset_id
event_ids
arrayRequired
metadata
name
stringRequired
project
stringRequired
status

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Start Session

Description

Tool to start a new session. Use when you need to initiate a new tracking session and retrieve its session_id.

Action Parameters

session
objectRequired

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Update Datapoint

Description

Tool to update a specific datapoint. Use when you need to modify fields of an existing datapoint.

Action Parameters

ground_truth
history
id
stringRequired
inputs
linked_datasets
linked_evals
metadata

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Update Dataset

Description

Tool to update an existing dataset. Use when you need to modify a dataset's details (name, description, datapoints, linked evaluations, or metadata) after confirming its ID.

Action Parameters

datapoints
dataset_id
stringRequired
description
linked_evals
metadata
name

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Update Event

Description

Tool to update an event. Use when updating event details by ID.

Action Parameters

config
duration
event_id
stringRequired
feedback
metadata
metrics
outputs
user_properties

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Update Metric

Description

Tool to update an existing metric. Use when you need to modify a metric’s properties after creation. Ensure you retrieve the metric first to verify its current state.

Action Parameters

code_snippet
criteria
description
enabled_in_prod
event_name
event_type
metric_id
stringRequired
name
needs_ground_truth
pass_when
prompt
return_type
threshold
type

Action Response

data
objectRequired
error
successful
booleanRequired

Tool Name: Update Project

Description

Tool to update a project's name or description. Use when you need to modify an existing project by its ID after creation.

Action Parameters

description
id
stringRequired
name

Action Response

data
objectRequired
error
successful
booleanRequired