Record discrete events or related events in your application from LLM requests/chains to functions.
In order to build production ready LLM applications, developers need to understand the state of their systems.
You want to track key LLM metrics, such as request inputs and outputs, as well as model specific metadata, model parameters, tokens, cost, etc.
But you also want to track your functions which may manipulate data from one LLM and chain it into another. Parea makes it easy to get this deep visibility into any LLM stack.
If you want to use OpenAI directly, you can still get automatic logging using Parea’s wrap_openai_client helper.
openai.py
Copy
from openai import OpenAIfrom parea import Pareaclient = OpenAI(api_key="OPENAI_API_KEY")# All you need to do is add these two linesp = Parea(api_key="PAREA_API_KEY") # replace with your API keyp.wrap_openai_client(client)response = client.chat.completions.create( model="gpt-3.5-turbo", temperature=0.5, messages=[ { "role": "user", "content": "Write a Hello World program in Python using FastAPI.", } ],)print(response.choices[0].message.content)# Also works with the assistants APIassistant = client.beta.assistants.create( name="Math Tutor", instructions=instructions, tools=[{"type": "code_interpreter"}], model="gpt-4-turbo-preview",)print(assistant)
If you want to use Anthropic’s Claude directly, you can still get automatic logging using Parea’s wrap_anthropic_client helper.
anthropic.py
Copy
import anthropicfrom parea import Pareap = Parea(api_key="PAREA_API_KEY") # replace with your API keyclient = anthropic.Anthropic()p.wrap_anthropic_client(client)message = client.messages.create( model="claude-3-opus-20240229", max_tokens=1024, messages=[ { "role": "user", "content": "Write a Hello World program in Python using FastAPI.", } ],)print(message.content[0].text)
The completion method allows you to call any LLM model you have access to on Parea with the same API interface.
You have granular control over what is logged via the parameters on Parea’s completion method.
log_omit_inputs: bool = field(default=False) # omit the inputs to the LLM call
log_omit_outputs: bool = field(default=False) # omit the outputs from the LLM call
log_omit: bool = field(default=False) # do not log anything
parea_completion.py
Copy
from parea import Pareafrom parea.schemas import LLMInputs, Message, ModelParams, Role, Completionp = Parea(api_key="PAREA_API_KEY") # replace with your API keyresponse = p.completion( Completion(llm_configuration=LLMInputs( model="gpt-3.5-turbo", # this can be any model enabled on Parea model_params=ModelParams(temp=0.5), messages=[Message( role=Role.user, content="Write a Hello World program in Python using FastAPI.", )], )))print(response.content)
Parea also supports frameworks such as Langchain. You can use PareaAILangchainTracer as a callback to automatically log all requests and responses.
langchain.py
Copy
from langchain_core.output_parsers import StrOutputParserfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_openai import ChatOpenAIfrom parea import Pareafrom parea.utils.trace_integrations.langchain import PareaAILangchainTracer# All you need to do is add these two linesp = Parea(api_key="PAREA_API_KEY") # replace with your API keyhandler = PareaAILangchainTracer()llm = ChatOpenAI(openai_api_key="OPENAI_API_KEY") # replace with your API keyprompt = ChatPromptTemplate.from_messages([("user", "{input}")])chain = prompt | llm | StrOutputParser()response = chain.invoke( {"input": "Write a Hello World program in Python using FastAPI."}, config={"callbacks": [handler]}, # <- use the callback handler here)print(response)
Parea supports automatic logging for OpenAI, Anthropic, Langchain, or any model if using Parea’s completion method (schema definition).
If you want to use OpenAI directly, you can still get automatic logging using Parea’s wrap_openai_client helper.
openai.py
Copy
from openai import OpenAIfrom parea import Pareaclient = OpenAI(api_key="OPENAI_API_KEY")# All you need to do is add these two linesp = Parea(api_key="PAREA_API_KEY") # replace with your API keyp.wrap_openai_client(client)response = client.chat.completions.create( model="gpt-3.5-turbo", temperature=0.5, messages=[ { "role": "user", "content": "Write a Hello World program in Python using FastAPI.", } ],)print(response.choices[0].message.content)# Also works with the assistants APIassistant = client.beta.assistants.create( name="Math Tutor", instructions=instructions, tools=[{"type": "code_interpreter"}], model="gpt-4-turbo-preview",)print(assistant)
If you want to use Anthropic’s Claude directly, you can still get automatic logging using Parea’s wrap_anthropic_client helper.
anthropic.py
Copy
import anthropicfrom parea import Pareap = Parea(api_key="PAREA_API_KEY") # replace with your API keyclient = anthropic.Anthropic()p.wrap_anthropic_client(client)message = client.messages.create( model="claude-3-opus-20240229", max_tokens=1024, messages=[ { "role": "user", "content": "Write a Hello World program in Python using FastAPI.", } ],)print(message.content[0].text)
The completion method allows you to call any LLM model you have access to on Parea with the same API interface.
You have granular control over what is logged via the parameters on Parea’s completion method.
log_omit_inputs: bool = field(default=False) # omit the inputs to the LLM call
log_omit_outputs: bool = field(default=False) # omit the outputs from the LLM call
log_omit: bool = field(default=False) # do not log anything
parea_completion.py
Copy
from parea import Pareafrom parea.schemas import LLMInputs, Message, ModelParams, Role, Completionp = Parea(api_key="PAREA_API_KEY") # replace with your API keyresponse = p.completion( Completion(llm_configuration=LLMInputs( model="gpt-3.5-turbo", # this can be any model enabled on Parea model_params=ModelParams(temp=0.5), messages=[Message( role=Role.user, content="Write a Hello World program in Python using FastAPI.", )], )))print(response.content)
Parea also supports frameworks such as Langchain. You can use PareaAILangchainTracer as a callback to automatically log all requests and responses.
langchain.py
Copy
from langchain_core.output_parsers import StrOutputParserfrom langchain_core.prompts import ChatPromptTemplatefrom langchain_openai import ChatOpenAIfrom parea import Pareafrom parea.utils.trace_integrations.langchain import PareaAILangchainTracer# All you need to do is add these two linesp = Parea(api_key="PAREA_API_KEY") # replace with your API keyhandler = PareaAILangchainTracer()llm = ChatOpenAI(openai_api_key="OPENAI_API_KEY") # replace with your API keyprompt = ChatPromptTemplate.from_messages([("user", "{input}")])chain = prompt | llm | StrOutputParser()response = chain.invoke( {"input": "Write a Hello World program in Python using FastAPI."}, config={"callbacks": [handler]}, # <- use the callback handler here)print(response)
Parea supports automatic logging for OpenAI or any model using Parea’s completion method (schema definition).
Support for Anthropic’s Claude is coming soon (reach out for early access).
If you want to use OpenAI directly, you can still get automatic logging using Parea’s patchOpenAI helper.
openai.ts
Copy
import OpenAI from 'openai';import {Parea, patchOpenAI} from "parea-ai";const openai = new OpenAI();// All you need to do is add these two linesnew Parea("PAREA_API_KEY"); // replace with your API keypatchOpenAI(openai);async function main(): Promise<string | null> { const response = await openai.chat.completions.create({ messages: [{role: 'user', content: "Write a hello world program in Python using FastAPI."}], model: 'gpt-3.5-turbo', }) return response.choices[0].message.content;}
The completion method allows you to call any LLM model you have access to on Parea with the same API interface.
You have granular control over what is logged via the parameters on Parea’s completion method.
log_omit_inputs: bool = field(default=False) # omit the inputs to the LLM call
log_omit_outputs: bool = field(default=False) # omit the outputs from the LLM call
log_omit: bool = field(default=False) # do not log anything
parea_completion.ts
Copy
import {Parea} from "parea-ai";const p = new Parea("PAREA_API_KEY"); // replace with your API keyasync function main(): Promise<string | null> { const response = await p.completion({ llm_configuration: { model: 'gpt-3.5-turbo', model_params: {temp: 0.5}, messages: [{role: 'user', content: "Write a hello world program in Python using FastAPI."}], } }); return response.content}
If you want to use Parea to complete and log an LLM request, you can use the completion endpoint as shown below.
To only create a log (doesn’t need to be a LLM request), you can use the log endpoint.
Note, in below example we use the same UUID for trace_id, root_trace_id, and parent_trace_id.
This will create a simple log without any hierarchical trace.
To find out how to associate multiple logs to create a trace, see the Tracing section.
Copy
curl 'https://parea-ai-backend-us-9ac16cdbc7a7b006.onporter.run/api/parea/v1/completion' \-H 'Content-Type: application/json' \-H 'x-api-key: $PAREA_API_KEY' \-d '{ "llm_configuration": { "model": "gpt-3.5-turbo", "model_params": {"temp": 0.5}, "messages": [ { "role": "user", "content": "Write a hello world program in Python using FastAPI." } ] }}'
If your LLM application has complex abstractions such as chains, agents, retrieval, tool usage, or external functions that
modify or connect prompts, then you will want a trace to associate all your related logs.
A Trace captures the entire lifecycle of a request and consists of one or more spans, representing different sub-steps.
The @trace decorator allows you to associate multiple processes into a single parent trace.
You only need to add the decorator to the top level function or any non-llm call function that you want to also track.
If you want to use OpenAI directly, you can still get automatic logging using Parea’s wrap_openai_client helper.
openai_trace_decorator.py
Copy
from openai import OpenAIfrom parea import Parea, traceclient = OpenAI(api_key="OPENAI_API_KEY") # replace with your API key# All you need to do is add these two linesp = Parea(api_key="PAREA_API_KEY") # replace with your API keyp.wrap_openai_client(client)# We generally recommend creating a helper function to make LLM API calls.def llm(messages: list[dict[str, str]]): response = client.chat.completions.create(model="gpt-3.5-turbo", temperature=0.5, messages=messages) return response.choices[0].message.content# (Optional) You can add a trace decorator to each prompt.# This will give the Span the name of the function.# Without the decorator the default name for all LLM call logs is `llm-openai`@tracedef hello_world(lang: str, framework: str): return llm([{"role": "user", "content": f"Write a Hello World program in {lang} using {framework}."}])@tracedef critique_code(code: str): return llm([{"role": "user", "content": f"How can we improve this code: \n {code}"}])# Our top level function is called chain. By adding the trace decorator here,# all sub-functions will automatically be logged and associated with this trace.# Notice, you can also add metadata to the trace, we'll revisit this functionality later.@trace(metadata={"purpose": "example"}, end_user_identifier="John Doe")def chain(lang: str, framework: str) -> str: return critique_code(hello_world(lang, framework))
from parea import Parea, tracefrom parea.schemas import LLMInputs, Message, ModelParams, Completionp = Parea(api_key="PAREA_API_KEY") # replace with your API key# We generally recommend creating a helper function to make LLM API calls.def llm(messages: list[dict[str, str]]): return p.completion( Completion( llm_configuration=LLMInputs( model="gpt-3.5-turbo", model_params=ModelParams(temp=0.5), messages=[Message(**m) for m in messages], ) ) ).content# (Optional) You can add a trace decorator to each prompt.# This will give the Span the name of the function.# Without the decorator the default name for all LLM call logs is `LLM`@tracedef hello_world(lang: str, framework: str): return llm([{"role": "user", "content": f"Write a Hello World program in {lang} using {framework}."}])@tracedef critique_code(code: str): return llm([{"role": "user", "content": f"How can we improve this code: \n {code}"}])# Our top level function is called chain. By adding the trace decorator here,# all sub-functions will automatically be logged and associated with this trace.# Notice, you can also add metadata to the trace, we'll revisit this functionality later.@trace(metadata={"purpose": "example"}, end_user_identifier="John Doe")def chain(lang: str, framework: str) -> str: return critique_code(hello_world(lang, framework))
The @trace decorator allows you to associate multiple processes into a single parent trace.
You only need to add the decorator to the top level function or any non-llm call function that you want to also track.
If you want to use OpenAI directly, you can still get automatic logging using Parea’s wrap_openai_client helper.
openai_trace_decorator.py
Copy
from openai import OpenAIfrom parea import Parea, traceclient = OpenAI(api_key="OPENAI_API_KEY") # replace with your API key# All you need to do is add these two linesp = Parea(api_key="PAREA_API_KEY") # replace with your API keyp.wrap_openai_client(client)# We generally recommend creating a helper function to make LLM API calls.def llm(messages: list[dict[str, str]]): response = client.chat.completions.create(model="gpt-3.5-turbo", temperature=0.5, messages=messages) return response.choices[0].message.content# (Optional) You can add a trace decorator to each prompt.# This will give the Span the name of the function.# Without the decorator the default name for all LLM call logs is `llm-openai`@tracedef hello_world(lang: str, framework: str): return llm([{"role": "user", "content": f"Write a Hello World program in {lang} using {framework}."}])@tracedef critique_code(code: str): return llm([{"role": "user", "content": f"How can we improve this code: \n {code}"}])# Our top level function is called chain. By adding the trace decorator here,# all sub-functions will automatically be logged and associated with this trace.# Notice, you can also add metadata to the trace, we'll revisit this functionality later.@trace(metadata={"purpose": "example"}, end_user_identifier="John Doe")def chain(lang: str, framework: str) -> str: return critique_code(hello_world(lang, framework))
from parea import Parea, tracefrom parea.schemas import LLMInputs, Message, ModelParams, Completionp = Parea(api_key="PAREA_API_KEY") # replace with your API key# We generally recommend creating a helper function to make LLM API calls.def llm(messages: list[dict[str, str]]): return p.completion( Completion( llm_configuration=LLMInputs( model="gpt-3.5-turbo", model_params=ModelParams(temp=0.5), messages=[Message(**m) for m in messages], ) ) ).content# (Optional) You can add a trace decorator to each prompt.# This will give the Span the name of the function.# Without the decorator the default name for all LLM call logs is `LLM`@tracedef hello_world(lang: str, framework: str): return llm([{"role": "user", "content": f"Write a Hello World program in {lang} using {framework}."}])@tracedef critique_code(code: str): return llm([{"role": "user", "content": f"How can we improve this code: \n {code}"}])# Our top level function is called chain. By adding the trace decorator here,# all sub-functions will automatically be logged and associated with this trace.# Notice, you can also add metadata to the trace, we'll revisit this functionality later.@trace(metadata={"purpose": "example"}, end_user_identifier="John Doe")def chain(lang: str, framework: str) -> str: return critique_code(hello_world(lang, framework))
The trace() wrapper allows you to associate multiple processes into a single parent trace.
You only need to add the wrapper to the top level function or any non-llm call function that you want to also track.
import { Parea, trace } from 'parea-ai';import OpenAI from 'openai';import { patchOpenAI } from '../utils/wrap_openai';import { ChatCompletionMessageParam } from 'openai/src/resources/chat/completions';const openai = new OpenAI();const p = new Parea('PAREA_API_KEY'); // replace with your API key// Patch OpenAI to add trace logspatchOpenAI(openai);// We generally recommend creating a helper function to make LLM API calls.async function llm(messages: ChatCompletionMessageParam[]): Promise<string | null> { const response = await openai.chat.completions.create({ model: 'gpt-3.5-turbo', messages, temperature: 0.5 }); return response.choices[0].message.content ?? '';}async function helloWorld(lang: string, framework: string): Promise<string | null> { return llm([{ role: 'user', content: `Write a hello world ${lang} in python using ${framework}.` }]);}async function critiqueCode(code: string | null): Promise<string | null> { return llm([{ role: 'user', content: `How can we improve this code: \n ${code}` }]);}// Our top level function is called Chain. By using the trace wrapper here,// all sub-functions will automatically be logged and associated with this trace.const chain = trace('Chain', async (lang: string, framework: string): Promise<string | null> => { return await critiqueCode(await helloWorld(lang, framework));});
import {CompletionResponse, Message, Parea, trace} from "parea-ai";const p = new Parea("PAREA_API_KEY") // replace with your API key// We generally recommend creating a helper function to make LLM API calls.async function llm(messages: Message[]): Promise<string | null> { const response: CompletionResponse = await p.completion({ llm_configuration: { model: 'gpt-3.5-turbo', model_params: {temp: 0.5}, messages: messages, }, }); return response.content}async function helloWorld(lang: string, framework: string): Promise<string | null> { return llm([{role: 'user', content: `Write a hello world ${lang} in python using ${framework}.`}])}async function critiqueCode(code: string | null): Promise<string | null> { return llm([{role: 'user', content: `How can we improve this code: \n ${code}`}])}// Our top level function is called Chain. By using the trace wrapper here,// all sub-functions will automatically be logged and associated with this trace.const chain = trace( 'Chain', async (lang: string, framework: string): Promise<string | null> => { return await critiqueCode(await helloWorld(lang, framework)); },);
If you use the API directly, you will need to manually associate the logs to create a trace.
To do that, we rely on the following fields:
trace_id: The UUID of the current trace log.
parent_trace_id: The UUID of the parent of the current trace log. If the current trace log is the root, this field will be the same as trace_id.
root_trace_id: The UUID of the root trace log. If the current trace log is the root, this field will be the same as trace_id.
Please, see the API walkthrough for more information & examples on how to implement tracing.
The trace decorator relies on Python’s contextvars to create traces.
However, when spawning threads from inside a trace the decorator will not work correctly as the contextvars are not correctly copied to the new threads or processes.
There is an existing issue in Python’s standard library and a great explanation in the FastAPI repo that discusses this limitation.
For example when a @trace-decorated function uses a ThreadPoolExecutor to make concurrent LLM requests the context that holds important info on the nesting hierarchy (“we are inside another trace”) is not copied over correctly to the child threads.
So, the created generations will not be linked to the trace and be ‘orphaned’.
In the UI, you will see a trace missing those generations.
A workaround is to manually copy over the context to the new threads or processes via contextvars.copy_context.
This is the recommended approach when using threading or multi-processing in Python.
Copy
from concurrent.futures import ThreadPoolExecutorimport contextvarsfrom parea import Parea, tracep = Parea(api_key="PAREA_API_KEY") # replace with your Parea API key@tracedef llm_call(question): return f"I can't answer that question: {question}"@tracedef multiple_llm_calls(question, n_calls: int = 2): answers = [] with ThreadPoolExecutor(max_workers=2) as executor: for _ in range(n_calls): context = contextvars.copy_context() future = executor.submit(context.run, llm_call, question) answers.append(future.result()) return answersresponse = multiple_llm_calls("Who are you?")print(response)
You can either disable logging or only store a percentage of all logs in Parea.
In Python, you can disable logging by setting the environment variable TURN_OFF_PAREA_LOGGING to True.
Alternatively, you can also deactivate logging by using the parea.helpers.TurnOffPareaLogging context manager.
In order to reduce the amount of logs stored in Parea, you can specify the log_sample_rate in the trace decorator or completion function
In Python, you can disable logging by setting the environment variable TURN_OFF_PAREA_LOGGING to True.
Alternatively, you can also deactivate logging by using the parea.helpers.TurnOffPareaLogging context manager.
In order to reduce the amount of logs stored in Parea, you can specify the log_sample_rate in the trace decorator or completion function
In TypeScript, you can disable logging by setting the environment variable PAREA_TRACE_ENABLED to false.
In order to reduce the amount of logs stored in Parea, you can specify the logSampleRate in the trace decorator or completion function
When using the REST API directly, you can specify the log_sample_rate in the record trace log and completion endpoints.
from openai import AsyncOpenAI, OpenAIfrom parea import Parea, tracep = Parea(api_key="PAREA_API_KEY") # replace with your API key# Syncclient = OpenAI(api_key="OPENAI_API_KEY") # replace with your API keyp.wrap_openai_client(client)# or Asyncaclient = AsyncOpenAI(api_key="OPENAI_API_KEY") # replace with your API keyp.wrap_openai_client(aclient)simple_example = { "model": "gpt-3.5-turbo-0125", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello!"} ], "stream": True}@tracedef call_openai_stream(data): stream = client.chat.completions.create(**data) for chunk in stream: print(chunk.choices[0].delta or "")@traceasync def acall_openai_stream(data): stream = await aclient.chat.completions.create(**data) async for chunk in stream: print(chunk.choices[0].delta or "")