feature: add river inspector that returns various info about the system

[krhubert]

Motivation

I want to periodically upload metrics about jobs count, state, and kind.

The API that allows that would be great, but at the same time I have a workaround and I think this is a low prio issue.

API

I think about something similar as asynq inspector, but only with few methods like:

• JobsCountByState
• JobsCountByKind
• etc...

This API might be utilized by riverui, because right now riverui uses its own sql queries

API dosen't need to guarantee backward compatibility as it will probably change in upcoming versions.

Possible solutions

1. River middleware that sends telemetry data about each job.

This would work, but many projects enable sampling to reduce the cost of storing large amount of data.

2. subscriptions

This approach requires a leader election to make sure only one instance sends data.
Btw - I think exposing a leader election as an API would be a great value added by river.

3. Collector

This is the most common setup and collector could aggregate metrics. But if you want to keep things simple then instances send the telemetry data straight to the external service.

4. Perioding job + sql

Right now I use periodic job and custom sql queries to fetch the data and send it once a while.

[brandur]

Yeah, something like this seems like a good idea.

One problem that was raised recently is that some of those job count queries can get quite slow when job tables become too large. It makes me wonder if we should have some kind of system that tries to count things incrementally and keep track of stats, which would work fine, except then you either have only per-client stats, or need to aggregate them somehow.

We could also leverage the river_client table that already exists (I added it some time ago for future use, but haven't put it to work yet) so have clients write stats periodically which could be aggregated by any client easily/cheaply.