Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logs enrich #2910

Open
v-zhuravlev opened this issue Mar 5, 2025 · 0 comments
Open

Add logs enrich #2910

v-zhuravlev opened this issue Mar 5, 2025 · 0 comments
Labels
proposal A proposal for new functionality.

Comments

@v-zhuravlev
Copy link
Contributor

v-zhuravlev commented Mar 5, 2025

Background

Logs and metrics correlation is very powerful concept that easy to achieve in Grafana with alloy, if logs/metrics are collected with the same agent.

However, it is not always the case:

  • For network devices logs can be received via syslog protocol or other push protocol, while metrics could be collected with snmp_exporter.

Proposal

To make sure that logs/metrics correlation is possible even with logs pushed without any additional metadata/labels, I would like to propose a component similar to https://www.rsyslog.com/doc/configuration/lookup_tables.html but instead of files, powerful prometheus discovery components can be used, since good probability that those could also be used for snmp metrics collection as well.

Examples:

1)Enrich Network devices syslog messages with device metadata(as labels) from IP address management (IPAM)/Data center infrastructure management(DCIM) like Netbox (https://github.com/FlxPeters/netbox-plugin-prometheus-sd). It could be same labels as used by prometheus.exporter.snmp or other metric scrapers.
2) Ensure metrics and logs use same labelset (gathered from same discovery.* component) even if metrics are scraped from prometheus endpoints and logs are received via loki.source.api.

Going forward, i created a component loki.enrich that could just do exactly that:
PR,

However there is also function array.combine_maps that does something similar, but, unfotrunately not working for logs scenarios describred right now.

@v-zhuravlev v-zhuravlev added the proposal A proposal for new functionality. label Mar 5, 2025
@github-project-automation github-project-automation bot moved this to Incoming in Alloy proposals Mar 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
proposal A proposal for new functionality.
Projects
Status: Incoming
Development

No branches or pull requests

1 participant