TypeScript
Setup
Set PAREA_API_KEY
as an environment variable, or in a .env file
export PAREA_API_KEY=<your API key>
const p = new Parea(process.env.PAREA_API_KEY);
Install the Parea Package
npm i parea-ai
trace
export type TraceOptions = {
metadata?: any;
endUserIdentifier?: string;
tags?: string[];
evalFuncNames?: string[];
evalFuncs?: any[];
accessOutputOfFunc?: (arg0: any) => string;
applyEvalFrac?: number;
deploymentId?: string;
target?: string;
};
type AsyncFunctionOrNot<TReturn, TArgs extends unknown[]> = (
...args: TArgs
) => Promise<TReturn> | TReturn;
function trace<TReturn,TArgs>(
funcName: string,
func: AsyncFunctionOrNot<TReturn, TArgs>,
options?: TraceOptions
): (
...args: TArgs
) => Promise<awaited Promise<TReturn> | TReturn>
The trace
decorator is used to trace a function and apply evaluation functions to its output.
It automatically attaches the current trace to the parent trace, if one exists, or sets it as the current trace.
This creates a nested trace structure, which can be viewed in the logs.
Parameters
funcName
: This will be used as the name of the trace and is often the name of the function.func
- The function being traced.options
: An optional object that may contain the following properties:metadata
: Any extra data that needs to be attached with the trace, which might help with debugging.endUserIdentifier
: The identifier for the end user that is using your application.tags
: Any tags that need to be attached with the trace.target
: An optional ground truth/expected output for the inputs and can be used by evaluation functions.evalFuncNames
: A list of names of evaluation functions, created in the Datasets tab, to evaluate on the output of the traced function. They will be applied non-blocking and asynchronously in the backend.accessOutputOfFunc
: An optional function that takes the output of the traced function and returns the value which should be used asoutput
of the function for evaluation functions.applyEvalFrac
: An optional parameter to specify the fraction of the time the evaluation functions should be applied. For example, ifapplyEvalFrac
is 0.5, the evaluation functions will be applied 50% of the time.deploymentId
: An optional parameter to specify the deployment ID of a prompt.target
: An optional parameter to specify the target of the trace.
Experiment
export type ExperimentOptions = {
nTrials?: number;
metadata?: { [key: string]: string };
datasetLevelEvalFuncs?: any[];
nWorkers?: number;
};
export type EvaluationResult = {
name: string;
score: number;
reason?: string;
};
export type EvaluatedLog = Log & {
scores?: EvaluationResult[];
};
export type EvaluationScoreSchema = EvaluationResult & {
id?: number;
};
export type TraceStatsSchema = {
trace_id: string;
latency?: number;
input_tokens?: number;
output_tokens?: number;
total_tokens?: number;
cost?: number;
scores?: EvaluationScoreSchema[];
};
export class ExperimentStatsSchema {
parent_trace_stats: TraceStatsSchema[];
// ...
}
class Experiment {
name: string;
runName: string;
data: string | Iterable<DataItem>;
func: (...dataItem: any[]) => Promise<any>;
p: Parea;
experimentStats?: ExperimentStatsSchema;
nTrials: number = 1;
metadata?: { [key: string]: string };
datasetLevelEvalFuncs?: ((
logs: EvaluatedLog[],
) => Promise<number | null | undefined | EvaluationResult | EvaluationResult[]>)[];
nWorkers: number = 10;
// ...
}
The Experiment
class is used to define an experiment of your LLM application. It is initialized the data to run the
experiment on (data
), and the entry point/function (func
). You can read more about running experiments
here.
run
const e = p.experiment(
"Experiment Name",
...);
await e.run();
This method runs the experiment and saves the stats to the experiment_stats
attribute. You can optionally specify the
name
of the experiment as an argument.
Was this page helpful?