You can integrate Parea with experiment tracking in DVC to iterate on your LLM app without polluting your git history with a commit for every experiment and still have the ability to revert your code to the state of any experiment. Once integrated, every time you run an experiment, Parea will automatically capture the state of the workspace as a DVC experiment together with the associated metrics. This will enable you to compare experiments via the DVC CLI (dvc exp show) and to revert to the state of them (dvc exp apply).

This is currently only supported for Python. Please, reach out if you want that functionality for TypeScript.

Pre-requisites

We assume that you have:

  1. Installed DVC and initialized it via dvc init.
  2. Defined an experiment

Setting-up the integration with DVC

You can integrate Parea with DVC by running the following command:

parea dvc-init

This command will check if DVC is installed as well as create a .parea directory with a dvc.yaml and a metrics.json file if they don’t exist. If the files haven’t been committed to git before, it will ask you to commit them. The dvc.yaml file will point to the metrics.json file and the metrics.json file will contain the metrics of the experiments. Both files are necessary for DVC. You can always re-run the command to check if the integration is set up properly.

Running an experiment

After updating your code, you can run an experiment as usual:

parea experiment <path/to/experiment_file.py>

This will capture the state of the workspace as a DVC experiment together with the associated metrics. To compare all ran experiments since the last commit you can run the following:

dvc exp show

dvc exp show

To revert the code to the state of an experiment run the following:

dvc exp apply <experiment-name>

Learn more about DVC

You can learn more about