Fetches all trace logs for an experiment.

Args:
    experiment_uuid (str): UUID of the experiment.

Returns:
    list[TraceLogTreeSchema]: List of trace logs for the experiment.

Get Experiment Trace Logs

Answer Matches Target

Will be automatically populated if this score was from a deployed evaluation metric.

Evaluation Metric Id

Name

Reason

Score

EvaluationScoreSchema

FilterOperator

Inputs for LLM inference such as messages, functions and additional parameters.

Controls how the model responds to function calls. â€œautoâ€ means the model can pick between an end-user or calling a function. To specifying a particular function name use a dictionary with key=â€œnameâ€ and value=â€œmy_functionâ€. This will force the model to call that function.

Function Call

A list of functions the model may generate JSON inputs for. Assumes every item in list has key 'name', 'description' and 'parameters'

Functions

Messages

Model

Provider

InputsSchema

Content

MessageSchema

Additional parameters to run inference of LLMs, such as temperature, top_p, etc.
Following the OpenAI API format.

Frequency Penalty

Max Length

Presence Penalty

Response format. See OpenAI docs for definition

Response Format

Safe Prompt

Temp

Top P

ParamsSchema

Role

Annotation Name

Created At

Defined Annotation Id

Trace Id

User Email Address

User Id

Value

TraceLogAnnotationSchema

Comment

TraceLogCommentSchema

Caption

TraceLogImage

Any annotations on log which were collected on Parea frontend. It maps annoation criterion ID to a dictionary mapping user_id (Parea user ID) to annotation.

Annotations

If specified, evals given with `evaluation_metric_names` will be applied to this log with this fraction.

Apply Eval Frac

Cache Hit

Children

IDs of any children. Will be automatically populated.

Children Ids

Children Logs

Any comments on log which were collected on Parea frontend.

Comments

If this log was a LLM call, this will contain the configuration used for the call.

If this was a LLM call, this will contain the cost of the call.

Cost

Optionally, provide the ID of the used deployed prompt in this log.

Deployment Id

Depth/level of nestedness of span in overall trace. Root-level trace is 0 and it always increments by 1.

Depth

End Timestamp

End User Identifier

If `status=error`, this should contain any additional information such as the stacktrace

Error

Evaluation Metric Ids

Names of evaluation metrics deployed on Parea which should be applied to this log.

Evaluation Metric Names

The execution number of span in trace. It starts with 0 and increments by 1 with every span.

Execution Order

If given, will be used to associate this log with an experiment.

Experiment Uuid

Feedback Score

Fill Children

Images

If this was a LLM call, this will contain the number of tokens in the input.

Input Tokens

Key-value pair inputs of this trace. Note, there is a special field to capture messages in LLM calls. You can still use it in the case of LLM calls to track the key-value pairs for prompt templates.

Inputs

Latency

If specified, this log and its entire associated trace will logged with this probability. Must be between 0 and 1 (incl.). Defaults to 1.0 (i.e., keeping all logs)

Log Sample Rate

Organization ID associated with Parea API key. Will be automatically determined from API key

Organization Id

Response of this step/log/function. If response isnâ€™t a string, it needs to be serialized to a string.

Output

If provided, will be used as output for any specified evaluation metric.

Output For Eval Metrics

If this was a LLM call, this will contain the number of tokens in the output.

Output Tokens

If given, current trace will be a child of this trace. If current child is not a child, `parent_trace_id` should be equal to `trace_id`

Parent Trace Id

Name of the project with which the trace/log should be associated with. Must be provided if `project_uuid` is not provided

Project Name

UUID of project with which this log is associated. Will be automatically filled-in by SDKs

Project Uuid

This is the UUID of the root trace/span of this trace. If current trace is the root trace, `root_trace_id` must be equal to `trace_id`

Root Trace Id

Any scores/eval results associated with this log.

Scores

Unique identifier for a session. Can you be used to associated multiple logs in e.g. chat applications.

Session Id

Start Timestamp

Status

The target or â€œgold standardâ€ response for the inputs of this log.

Target

If this was a LLM call, this will contain the time taken to generate the first token.

Time To First Token

If this was a LLM call, this will contain the total number of tokens in the input and output.

Total Tokens

UUID of the trace log. Ex: e3267953-a16f-47f5-b37e-622dbb29d730

Trace Name

TraceLogTreeSchema

Location

Message

Error Type

ValidationError

FilterBaseSchema

APIKeyHeader

Response Get Experiment Trace Logs Api Api Parea V1 Experiment  Experiment Uuid  Trace Logs Post

HTTPValidationError

Get Experiment Trace Logs Api

x-api-key

Parea AI

What is Parea AI?

Evaluation Quickstart

Monitor your LLM requests and application functions.

Monitoring Quickstart

Evaluation

Datasets

Deploy versioned prompts and use them via the SDK.

Deployed Prompts

Tutorial on improving a Langchain RAG application using Parea's Evals, Tracing, and Playground.

Optimize a LangChain RAG App

How many samples are necessary to achieve good performance with DSPy?

Optimize a RAG DSPy Application

REST API Walkthrough

Leverage user feedback to run A/B tests of prompts, models & other approaches

A/B Testing of LLM Apps

You will need an API Key and Organization to use the API/SDK.

Authentication

Create a project or get the project if it already exists. Is used to find out UUID of project.

Create or get Project

Health Check

Log a (LLM) span to visualize inference results, or chains.

Record Trace Log

Record any (user) feedback & ground truth/correction of output for a log.

Record Feedback

Update Trace Log

Fetches trace logs and returns them as a paginated response.

Get Trace Logs

Get Trace Log

Create an experiment and get the associated experiment uuid.

Create Experiment

Finishes an experiment, calculates stats and returns all stats for root trace logs

Finish Experiment

Fetches aggregated stats for every root-level trace log in an experiment.

Get Experiment Stats

Lists experiments given a set of filters incl. their high-level statistics.

List Experiments

Get a completion response using either one of your organizationâ€™s deployed prompts, or by providing completion details including prompt and inputs in the request.
This endpoint acts as a LLM gateway/proxy endpoint to generate completions from different LLMs.

Completion

Get a completion response using either one of your organizationâ€™s deployed prompts, or by providing completion details including prompt and inputs in the request.

This endpoint acts as a LLM gateway/proxy endpoint to generate completions from different LLMs.

Stream Completion

Given a deployment_id, fetches the deployed prompt and its details. Can be optionally used to fill-in the templated prompt with provided inputs.

Fetch Deployed Prompt

Create Collection

Adds items/test cases to a dataset. If the dataset does not exist, it will be created.
Returns the IDs of the created items/test cases.

Returns:
    List[int]: List of IDs of the created items/test cases.

Create Test Cases For Dataset

Fetches dataset/test case collection by its ID or name.

Args:
    test_collection_identifier (Union[str, int]): ID or name of the test case collection/dataset.

Get Test Case Collection By Identifier

Update the item/test case of the dataset with any given fields.

Args:
    dataset_id (int): ID of the dataset.
    test_case_id (int): ID of the item/test case.
    update_test_case_request (UpdateTestCasePublicRequestSchema): Request body containing the fields to update.

Update Test Case

Python

TypeScript

Automatic creation of LLM evals based on human review of responses

Self-improving, domain-specific LLM Evals

When ground truth values are expensive to obtain & launching fast is important

Hill climbing generative AI problems

LLM logs are useful but it's hard to prioritize which production log to review. High entropy responses are a good starting point.

How to detect unreliable behavior of LLM apps

Practices to improve LLM apps component-wise

Tactics for multi-step AI app experimentation

Practical workflow focusing on continuous improvement and data-driven decision-making.

A Systematic Workflow to build Production-Ready LLM Applications

We will use Instructor in TypeScript to generate synthetic data for a question-answering task.

Synthetic Data Generation for Q&A Tasks

Haiku > GPT-4 Turbo > Opus >>> GPT-3.5 Turbo if not using parallel function calling.

Anthropic's Haiku Beats GPT-4 Turbo in Tool Use - Sometimes

How to measure the performance of retrieval applications without ground truth data.

General

Tracing

Experiments

LLM Proxy

Datasets

SDK

Get Experiment Trace Logs

Authorizations

Path Parameters

Body

Response