Handle ES Down Cases More Gracefully in Bulk Indexing

[sunil-saini]

Hey ES Team,

I am writing a logs tailer program, which tails logs continuously from a file and bulk indexes to elasticsearch

My requirement is to stop indexing If there is any problem in ES, and start again when previously submitted all batches are successfully flushed

Currently if ES is down bulkIndexer.Add() keeps on adding items to indexer and doesn't return any ES error, leading to uncontrolled memory spike of program and if it gets killed in between, all added items are lost

So was trying to understand how to design a flow where can stop adding items to bulk indexer as soon as it detects N no of flush errors and resumes back when these errors falls below N again

I see that flush worker have OnError callback support, so that can be used to detect if there is error in flushing(though if retries are added, it will become complicated to know exact no of errors), but how to resume it back ? like how to detect how many flush errors are remaining now, as there is no OnSuccess function callback for flush workers

One solution i am able to think is frequently checking bulkIndexer.Stats().NumFailed somehow to detect if failures are fixed

Is my understanding correct or there is some native way in library itself to support such requirement ?