Instrumenting your Langchain applications with Parea AI
Parea AI works seamlessly with LangChain. LangChain provides a callbacks system to hook into the various stages of your LLM application.
With Parea’s PareaAILangchainTracer you can automatically subscribe to all exposed LangChain events.
Add the PareaAILangchainTracer as a callback when running your Langchain model/chain/agent:
pip install parea-ai langchain
First initialize Parea and the PareaAILangchainTracer. Follow these steps to get an API key.
from parea import Pareafrom parea.utils.trace_integrations.langchain import PareaAILangchainTracerp = Parea(api_key="PAREA_API_KEY") # Replace with your Parea API keyhandler = PareaAILangchainTracer()
Then initialize the tracer and add it as a callback to your LangChain application.
from langchain.chat_models import ChatOpenAIfrom langchain.prompts import ChatPromptTemplatechat_model = ChatOpenAI()prompt = ChatPromptTemplate.from_messages([ ("system", "You are a helpful AI."), ("human", "{input}")])chain = prompt | chat_modelchain.invoke( {"input": "What is the meaning of life?"}, config={"callbacks": [handler]})
There are two ways callbacks can be passed - as constructor callbacks and as request callbacks.
Constructor callbacks are defined in the constructor of an object and apply to all calls made on that specific object. The scope is limited to the object itself.
Request callbacks are defined in the run() or apply() methods used for sending a request. They are applied to a specific request and all sub-requests it contains.
For example, passing a handler to the chain.run() method will be used for that particular request and any subsequent sub-requests triggered. Additional details here.
Below are some examples of how to set callbacks in different scenarios.
from langchain.llms import OpenAIfrom langchain.utilities import SQLDatabasefrom langchain_experimental.sql import SQLDatabaseChaindb = SQLDatabase.from_uri("sqlite:///Chinook.db")llm = OpenAI(temperature=0, verbose=True)db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True)db_chain.run( "How many employees are there?", callbacks=[PareaAILangchainTracer()])
Every time you run a LangChain component with tracing, the call hierarchy for the run is saved and can be visualized
in the app on the Logs tab. You can drill down into the components inputs and outputs, parameters, response time, token usage, and other
important metadata.
Below is an example of how to use the PareaAILangchainTracer and Langchain to evaluate a RAG chain. The code can be found in the cookbook here.
Python
import osfrom datetime import datetimefrom operator import itemgetterfrom dotenv import load_dotenvfrom langchain.chat_models import ChatOpenAIfrom langchain.document_loaders import RecursiveUrlLoaderfrom langchain.document_transformers import Html2TextTransformerfrom langchain.embeddings import OpenAIEmbeddingsfrom langchain.prompts import ChatPromptTemplatefrom langchain.schema.output_parser import StrOutputParserfrom langchain.text_splitter import TokenTextSplitterfrom langchain.vectorstores import Chromafrom parea import Pareafrom parea.utils.trace_integrations.langchain import PareaAILangchainTracerfrom parea.evals.general import answer_matches_target_llm_grader_factoryfrom parea.evals.rag import context_query_relevancy_factory, percent_target_supported_by_context_factoryfrom parea.evals import EvalFuncTuple, run_evals_in_thread_and_logfrom parea.schemas import Logload_dotenv()# Need to instantiate Parea for tracing and evalsp = Parea(api_key=os.getenv("PAREA_API_KEY"))CONTEXT = Nonedef format_docs(docs) -> str: global CONTEXT # Declare CONTEXT as a global variable context = "\n\n".join(doc.page_content for doc in docs) CONTEXT = context return contextraw_documents = RecursiveUrlLoader("https://en.wikipedia.org/wiki/New_York_City").load()transformed = Html2TextTransformer().transform_documents(raw_documents)documents = TokenTextSplitter(model_name="gpt-3.5-turbo", chunk_size=2000, chunk_overlap=200).split_documents(transformed)retriever = Chroma.from_documents(documents, OpenAIEmbeddings()).as_retriever()prompt = ChatPromptTemplate.from_messages( [ ( "system", "You are a helpful documentation Q&A assistant, trained to answer questions from the provided context." "\nThe current time is {time}.\n\nRelevant documents will be retrieved in the following messages.", ), ("system", "{context}"), ("human", "{question}"), ]).partial(time=str(datetime.now()))model = ChatOpenAI(model="gpt-3.5-turbo-16k", temperature=0)response_generator = prompt | model | StrOutputParser()chain = ( # The runnable map here routes the original inputs to a context and a question dictionary to pass to the response generator {"context": itemgetter("question") | retriever | format_docs, "question": itemgetter("question")} | response_generator)# Get started with Parea's auto evaluation metrics for RAGEVALS = [ EvalFuncTuple(name="matches_target", func=answer_matches_target_llm_grader_factory(question_field="question")), EvalFuncTuple(name="relevancy", func=context_query_relevancy_factory(question_field="question", context_fields=["context"])), EvalFuncTuple(name="supported_by_context", func=percent_target_supported_by_context_factory(question_field="question", context_fields=["context"])),]def main(): handler = PareaAILangchainTracer() question = "What is the population of New York City as of 2020?" output = chain.invoke({"question": question}, config={"callbacks": [handler]}) # get parent trace id from the tracer parent_trace_id = handler.get_parent_trace_id() # build log component needed for evaluation metric functions log = Log(inputs={"question": question, "context": CONTEXT}, output=output, target="8,804,190") # parea provided helper function to run evaluation metrics in a thread to avoid blocking run_evals_in_thread_and_log(trace_id=str(parent_trace_id), log=log, eval_funcs=EVALS, verbose=True)# ###Eval Results#### NamedEvaluationScore(name='matches_target', score=1.0)# NamedEvaluationScore(name='relevancy', score=0.0053)# NamedEvaluationScore(name='supported_by_context', score=1.0)# View trace at: https://app.parea.ai/logs/detailed/{parent_trace_id}
If you want to observe or test LangChain code and other code together, the trace decorator plays well with that and will associate any traces, LLM calls, and LangChain logs.
You can see a schematic example below:
langchain_inside_trace.py
oai_client = OpenAI()p = Parea(api_key=os.getenv("PAREA_API_KEY"))handler = PareaAILangchainTracer()p.wrap_openai_client(oai_client)chain = ...@tracedef main(): programming_language = ( oai_client.chat.completions.create(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Suggest one programming languages"}]).choices[0].message.content ) return chain.invoke( {"input": f"Write a Hello World program in {programming_language}."}, config={"callbacks": [handler]}, )