Pipeline checkpointing

[julian-risch]

With the new Pipeline run logic, we discussed that introducing checkpoints to pipelines becomes possible. Users could then set breakpoints, execute a pipeline from a breakpoint, execute up until a breakpoint. Save a checkpoint/snapshot. Set a callback on the checkpoint.
Breakpoints will be beneficial for pipeline debugging. They can also be useful for pipeline evaluation, in particular for isolated component evaluation to find bottlenecks as described in Haystack 1.x docs. https://docs.haystack.deepset.ai/v1.24/docs/evaluation#integrated-and-isolated-node-evaluation

In integrated evaluation, a node receives the predictions from the preceding node as input. It shows the performance users can expect when running the pipeline and it's the default mode when calling pipeline.eval().
In isolated evaluation, a node is isolated from the predictions of the preceding node. Instead, it receives ground-truth labels as input. Isolated evaluation shows the maximum performance of a node if it receives the perfect input from the preceding node. You can activate it by running pipeline.eval(add_isolated_node_eval=True).
For example, in an ExtractiveQAPipeline comprised of a Retriever and a Reader, isolated evaluation would measure the upper bound of the Reader's performance, that is the performance of the Reader assuming that the Retriever passes on all relevant documents.
If the isolated evaluation result of a Node differs significantly from its integrated evaluation result, you may need to improve the preceding node to get the best results from your pipeline. If the difference is small, it means that you should improve the Node that you evaluated to improve the pipeline's overall result quality.