GODRIVER-3156 Detect and discard closed idle connections.

[matthewdale]

GODRIVER-3156

Summary

• When checking if a connection is closed (either during check out or during the background maintenance routine), try to read from the connection to check if it's still alive.
  • Only check for the liveness of the connection if it's been idle for more than 10 seconds to prevent adding latency to most check-out operations.
• Change connection.idleDeadline to connection.idleStart, allowing us to calculate different idle deadlines for different purposes.

Background & Motivation

Currently the Go driver never attempts to detect connections that were closed by the other side when checking out connections. As a result, it's possible to return a connection that's been closed by the other side where any read or write operations will immediately fail. Instead, we should check if the connection has been closed when checking out a connection.

[prestonvasquez]

What is the significance of 10 seconds?

[matthewdale]

My reference was the PyMongo driver, which checks connection liveness when it's idle for >1 second. I wanted to mitigate the risk that checking liveness too often would cause performance problems (since a liveness check takes at least 1ms), so I increased that threshold to 10s.

[blink1073]

This looks relevant: x/mongo/driver/topology/pool_test.go:1230:4: undefined: noerr

[mongodb-drivers-pr-bot]

API Change Report

No changes found!

[prestonvasquez]

LGTM, I recommend adding a unit test for isAlive:

func TestConnection_IsAlive(t *testing.T) {
	t.Run("idle time lt 10 seconds", func(t *testing.T) {
		mockConn := &drivertest.ChannelNetConn{}
		conn := &connection{
			nc: mockConn,
		}

		conn.idleStart.Store(time.Now())
		assert.True(t, conn.isAlive())
	})

	t.Run("idle time gt 10 seconds", func(t *testing.T) {
		mockConn := &drivertest.ChannelNetConn{
			ReadResp: make(chan []byte, 1),
		}

		mockConn.ReadResp <- []byte{5, 0, 0, 0, 0}

		conn := &connection{
			nc: mockConn,
		}

		conn.idleStart.Store(time.Now().Add(-11 * time.Second))
		assert.False(t, conn.isAlive())
	})

	t.Run("read error", func(t *testing.T) {
		mockConn := &drivertest.ChannelNetConn{
			ReadErr: make(chan error, 1),
		}

		mockConn.ReadErr <- errors.New("")

		conn := &connection{
			nc: mockConn,
		}

		conn.idleStart.Store(time.Now().Add(-11 * time.Second))
		assert.False(t, conn.isAlive())
	})

	// TODO:
	// case: deadline exceeded
}

[blink1073]

drivers-pr-bot please backport to release/1.17