Skip to main content

Active Learning

Active Learning in the context of AutoEval

Active learning is the process of using an algorithm (an LLM in this case) to interactively label the training data. You can continuously improve the active learning by providing additional samples of ground truth data.

active-learning#diagram

Active Learning for Datasets:​

In AutoEval, we refer to active learning as LLM Judge++. This flow uses the LLM to generate an initial set of labels, and refines it with human input:

Labeling Flow

We are actively (no pun intended) adding more capabilities to better support active learning, because having a high quality labeled dataset that accurately reflects your evaluation criteria is critical to getting a high quality evaluator model that learns from those labels.