Instrumenting your DSPy application with Parea AI
DSPy is a framework for automatically prompting and fine-tuning language models. It provides:
nn.Module
),To observe your DSPy application, you can use trace_dspy
to trace the execution of your DSPy modules.
This will create traces like the one below:
Note, that optimization/compilation with DSPy can create a lot of logs which aren’t necessarily actionable.
You can suppress these logs by using the TurnOffPareaLogging
context manager.
After optimization, you will likely want to assess the performance of your module as outlined below.
The DSPy integration automatically creates nested traces by relying on Python’s contextvars
.
That means, if you are using threading or multi-processing in your DSPy application, the traces of the DSPy modules in that get orphaned from the main trace.
There is an existing issue in Python’s standard library and a great explanation in the FastAPI repo that discusses this limitation.
To avoid this, you need to manually copy over the context to the new thread/process via contextvars.copy_context()
.
See the below example:
Threading & Multi-processing in Python
You can evaluate & track the performance of your DSPy modules by running experiments.
To evaluate DSPy modules, you need to attach the evaluation metrics to the module (attach_evals_to_module
) and convert the DSPy examples to dictionaries (convert_dspy_examples_to_parea_dicts
).
If you have evaluation functions which don’t require reference/target answers, you can evaluate your DSPy modules by attaching those evals to the module via attach_evals_to_module
.
This will automatically apply your list of evals to the module whenever you call it. See the example below for more details.
Example: Attach Evaluation to Module
Checkout our tutorial on how to improve a DSPy RAG application with Parea AI, here. Or read more about how experiments work to assess variance of LLMs by using multiple trials, make experiments reproducible, and more.
Instrumenting your DSPy application with Parea AI
DSPy is a framework for automatically prompting and fine-tuning language models. It provides:
nn.Module
),To observe your DSPy application, you can use trace_dspy
to trace the execution of your DSPy modules.
This will create traces like the one below:
Note, that optimization/compilation with DSPy can create a lot of logs which aren’t necessarily actionable.
You can suppress these logs by using the TurnOffPareaLogging
context manager.
After optimization, you will likely want to assess the performance of your module as outlined below.
The DSPy integration automatically creates nested traces by relying on Python’s contextvars
.
That means, if you are using threading or multi-processing in your DSPy application, the traces of the DSPy modules in that get orphaned from the main trace.
There is an existing issue in Python’s standard library and a great explanation in the FastAPI repo that discusses this limitation.
To avoid this, you need to manually copy over the context to the new thread/process via contextvars.copy_context()
.
See the below example:
Threading & Multi-processing in Python
You can evaluate & track the performance of your DSPy modules by running experiments.
To evaluate DSPy modules, you need to attach the evaluation metrics to the module (attach_evals_to_module
) and convert the DSPy examples to dictionaries (convert_dspy_examples_to_parea_dicts
).
If you have evaluation functions which don’t require reference/target answers, you can evaluate your DSPy modules by attaching those evals to the module via attach_evals_to_module
.
This will automatically apply your list of evals to the module whenever you call it. See the example below for more details.
Example: Attach Evaluation to Module
Checkout our tutorial on how to improve a DSPy RAG application with Parea AI, here. Or read more about how experiments work to assess variance of LLMs by using multiple trials, make experiments reproducible, and more.