Skip to content

Commit

Permalink
Fixed issue #213 to lower logging level of spurious errors from 8T lo…
Browse files Browse the repository at this point in the history
…ad balancing
  • Loading branch information
nr-swilloughby committed Mar 12, 2021
1 parent df4d9c2 commit e3a6dd6
Showing 1 changed file with 16 additions and 3 deletions.
19 changes: 16 additions & 3 deletions v3/newrelic/trace_observer.go
Original file line number Diff line number Diff line change
Expand Up @@ -272,9 +272,22 @@ func (to *gRPCtraceObserver) rcvResponses(spanClient v1.IngestService_RecordSpan
for {
s, err := spanClient.Recv()
if nil != err {
to.log.Error("trace observer response error", map[string]interface{}{
"err": err.Error(),
})
// (issue 213) These two specific errors were reported as nuisance
// but are really harmless so we'll report them as DEBUG level events
// instead of ERROR.
// This error comes from our Infinite Tracing load balancers.
// We believe the EOF error comes from the gRPC getting reset every 30 seconds
// from the same cause (rebalancing 8T)
if err.Error() == "rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: NO_ERROR" || err.Error() == "EOF" {
to.log.Debug("trace observer response error", map[string]interface{}{
"err": err.Error(),
})
} else {
to.log.Error("trace observer response error", map[string]interface{}{
"err": err.Error(),
})
}

// NOTE: even when the trace observer is shutting down
// properly, an EOF error will be received here and a
// supportability metric created.
Expand Down

0 comments on commit e3a6dd6

Please sign in to comment.