Zerolog in Context #507
Zerolog in Context #507
Conversation
|
I was unable to restore #501, so this PR will replace it |
| return nil, nil | ||
| } | ||
|
|
||
| estimate := 500 * len(events.logs) | ||
| estimate := averageLogSizeEstimate * len(events.logs) |
| // internal variable names and constants | ||
| const ( | ||
| // number of bytes expected to be needed for the average log message | ||
| averageLogSizeEstimate = 400 |
Adds the ability to ingest and harvest log events that have been structured and formatted to the proper json spec. This is meant to be used by wrapper libraries for popular logging frameworks, and should not be called by users.
Modifies the agent to support passing log data from the new zerolog plugin into the harvest pool and sending that data to our servers.
Creates a suite of tests to measure and benchmark the performance of features of the Go agent at scale.
| | ------- | --------- | | ||
| | Forwarding | :heavy_check_mark: | | ||
| | Metrics | :heavy_check_mark: | | ||
| | Enrichment | :x: | |
There was a problem hiding this comment.
I see span.id and trace.id being put on the log lines -- is there more enrichment that needs to be supported? Also, there are the three common attributes of entity GUID, entity name, and hostname.
There was a problem hiding this comment.
By enrichment, I mean that we do not support enriching the local logs that get printed to console in this release. We add all of the enrichments to the logs, including entity GUID, entity name, and hostname at harvest
|
|
||
| // MarshalJSON is used for testing. | ||
| func (e *logEvent) MarshalJSON() ([]byte, error) { | ||
| buf := bytes.NewBuffer(make([]byte, 0, averageLogSizeEstimate)) |
There was a problem hiding this comment.
Does hinting the capacity with that 400 averageLogSizeEstimate really make that much of a difference? Cool that we're thinking this hard about performance, though. I'd love to hear about your grouping of ints that I read earlier, too.
There was a problem hiding this comment.
Yes! The bytes buffer gets allocated on the heap, and when it grows, it has to estimate/request more memory. Each time it does that, we incur a cpu cost. By estimating a little more closely to what we think we need, we can cut down the number of buffer resizes we need, and save us cpu time, and run time!
There was a problem hiding this comment.
The struct field one blew my mind when I first read it: https://wagslane.dev/posts/go-struct-ordering/
TLDR, go allocates contiguous memory for structs in the order you specify the fields in. If the fields are mismatched sizes, it will just allocate the same amount of memory for everything. If you have big structs, this will add up very fast.
There was a problem hiding this comment.
Yes, it's similar to C in that respect.
| buf.WriteString(`"entity.guid":`) | ||
| jsonx.AppendString(buf, events.entityGUID) | ||
| buf.WriteByte(',') | ||
| buf.WriteString(`"entity.name":`) | ||
| jsonx.AppendString(buf, events.entityName) | ||
| buf.WriteByte(',') | ||
| buf.WriteString(`"hostname":`) | ||
| jsonx.AppendString(buf, events.hostname) |
There was a problem hiding this comment.
It's great that we can take advantage of the "common" section of the logging payload format in the context of Go apps.
| // events. | ||
| func (events *logEvents) split() (*logEvents, *logEvents) { | ||
| // numSeen is conserved: e1.numSeen + e2.numSeen == events.numSeen. | ||
| sc1, sc2 := splitSeverityCount(events.severityCount) |
Logging for the Go agent is here. This feature adds: