Custom evaluation functions

You can also define your own arbitrary evaluation functions by clicking Create new custom metric.

Evaluation function

The editor will be pre-populated with a template for you to get started. You can delete all the code and retain the eval_fun signature def eval_fun(log: Log) -> float:. If you have this function and return a float or boolean, your new metric will be valid.

A simple example could be:

def eval_fun(log: Log) -> float:
    return float(log.output ==