Evaluation Quickstart

Installation

First, you’ll need a Parea API key. See Authentication to get started.After you’ve followed those steps, you are ready to install the Parea SDK client.

pip install parea-ai

npm install parea-ai

Create an evaluation script

Start with creating a simple evaluation script.

Python
TypeScript

import os

from dotenv import load_dotenv

from parea import Parea, trace
from parea.evals.general import levenshtein

load_dotenv()

p = Parea(api_key=os.getenv("PAREA_API_KEY"))


# annotate function with the trace decorator and pass the evaluation function(s)
@trace(eval_funcs=[levenshtein])
def greeting(name: str) -> str:
    return f"Hello {name}"

p.experiment(
    "Greetings",  # experiment name
    data=[
        { "name": "Foo", "target": "Hi Foo" },
        { "name": "Bar", "target": "Hello Bar" },
    ],  # test data to run the experiment on (list of dicts)
    func=greeting,
).run()

import { Parea, trace, Completion, CompletionResponse, Log, Message, levenshtein } from 'parea-ai';
import * as dotenv from 'dotenv';

dotenv.config();

const p = new Parea(process.env.PAREA_API_KEY);

// annotate function with the trace decorator and pass the evaluation function(s)
const greet = trace(
  'greetings',
  (name: string): string => {
    return `Hello ${name}`;
  },
  {
    evalFuncs: [levenshtein],
  },
);

export async function main() {
  const e = p.experiment(
    'Greetings',  // experiment name
    [
      { name: 'Foo', target: 'Hi Foo' },
      { name: 'Bar', target: 'Hello Bar' },
    ], // test data to run the experiment on (list of dicts)
    greet, // function to run (callable)
  );
  return await e.run();
}

main().then(() => {
  console.log('Experiment complete!');
});

Run experiment

After you’ve followed the above steps, you are ready run your experiment.

python3 path/to/experiment_file.py

ts-node path/to/experiment_file.ts

View results

The executed script will create a link to the experiment overview & its traces. You will see a high-level overview of your experiment, including average values for metrics such as latencies and cost, and any evaluation functions you’ve defined. You will see a table of your logs, and any chains will be expandable. The log table supports search, filtering, and sorting.

You can create additional statistics by clicking the “Pin stat” button. If you click a log, it will open the detailed trace view. Here, you can step through each span and view inputs, outputs, messages, metadata, and other key metrics associated with a given trace.

Can you improve to 100%?

For our first experiment, we only achieved a 77.8% score. Can you improve the score to 100%? If you run another experiment, you can compare the results like the screenshot below.

What’s Next?

Dive deeper into Experiments or get started with monitoring your application.

What is Parea AI?Monitoring Quickstart

Welcome

Guides

Tutorials

Evaluation Quickstart

What’s Next?

​What’s Next?

What’s Next?