​
Setup

Set PAREA_API_KEY as an environment variable, or in a .env file

export PAREA_API_KEY=<your API key>
const p = new Parea(process.env.PAREA_API_KEY);

Install the Parea Package

npm i parea-ai

​
trace

export type TraceOptions = {
  metadata?: any;
  endUserIdentifier?: string;
  tags?: string[];
  evalFuncNames?: string[];
  evalFuncs?: any[];
  accessOutputOfFunc?: (arg0: any) => string;
  applyEvalFrac?: number;
  deploymentId?: string;
  target?: string;
};

type AsyncFunctionOrNot<TReturn, TArgs extends unknown[]> = (
    ...args: TArgs
) => Promise<TReturn> | TReturn;

function trace<TReturn,TArgs>(
    funcName: string,
    func: AsyncFunctionOrNot<TReturn, TArgs>,
    options?: TraceOptions
    ): (
        ...args: TArgs
    ) => Promise<awaited Promise<TReturn> | TReturn>

The trace decorator is used to trace a function and apply evaluation functions to its output. It automatically attaches the current trace to the parent trace, if one exists, or sets it as the current trace. This creates a nested trace structure, which can be viewed in the logs.

​
Parameters

  • funcName: This will be used as the name of the trace and is often the name of the function.
  • func - The function being traced.
  • options: An optional object that may contain the following properties:
    • metadata: Any extra data that needs to be attached with the trace, which might help with debugging.
    • endUserIdentifier: The identifier for the end user that is using your application.
    • tags: Any tags that need to be attached with the trace.
    • target: An optional ground truth/expected output for the inputs and can be used by evaluation functions.
    • evalFuncNames: A list of names of evaluation functions, created in the Datasets tab, to evaluate on the output of the traced function. They will be applied non-blocking and asynchronously in the backend.
    • accessOutputOfFunc: An optional function that takes the output of the traced function and returns the value which should be used as output of the function for evaluation functions.
    • applyEvalFrac: An optional parameter to specify the fraction of the time the evaluation functions should be applied. For example, if applyEvalFrac is 0.5, the evaluation functions will be applied 50% of the time.
    • deploymentId: An optional parameter to specify the deployment ID of a prompt.
    • target: An optional parameter to specify the target of the trace.

​
Experiment

    export type ExperimentOptions = {
      nTrials?: number;
      metadata?: { [key: string]: string };
      datasetLevelEvalFuncs?: any[];
      nWorkers?: number;
    };

    export type EvaluationResult = {
      name: string;
      score: number;
      reason?: string;
    };

    export type EvaluatedLog = Log & {
      scores?: EvaluationResult[];
    };

    export type EvaluationScoreSchema = EvaluationResult & {
      id?: number;
    };

    export type TraceStatsSchema = {
      trace_id: string;
      latency?: number;
      input_tokens?: number;
      output_tokens?: number;
      total_tokens?: number;
      cost?: number;
      scores?: EvaluationScoreSchema[];
    };

    export class ExperimentStatsSchema {
      parent_trace_stats: TraceStatsSchema[];

      // ...
    }

    class Experiment {
      name: string;
      runName: string;
      data: string | Iterable<DataItem>;
      func: (...dataItem: any[]) => Promise<any>;
      p: Parea;
      experimentStats?: ExperimentStatsSchema;
      nTrials: number = 1;
      metadata?: { [key: string]: string };
      datasetLevelEvalFuncs?: ((
        logs: EvaluatedLog[],
      ) => Promise<number | null | undefined | EvaluationResult | EvaluationResult[]>)[];
      nWorkers: number = 10;

        // ...
    }

The Experiment class is used to define an experiment of your LLM application. It is initialized the data to run the experiment on (data), and the entry point/function (func). You can read more about running experiments here.

​
run

  const e = p.experiment(
      "Experiment Name",
      ...);
  await e.run();

This method runs the experiment and saves the stats to the experiment_stats attribute. You can optionally specify the name of the experiment as an argument.