Can not assign requested address error

[seanbrant]

BitmapError: Post http://x.x.x.155:10101/index/i295/query: dial tcp x.x.x.155:10101: connect: cannot assign requested address

SetBit(rowID=24078501, frame='d', columnID=4001656)
SetRowAttrs(rowID=24078501, frame='d', category=0)
SetBit(rowID=70592988, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70592988, frame='tag_n', category=0)
SetBit(rowID=70592989, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70592989, frame='tag_n', category=2408311)
SetBit(rowID=70592992, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70592992, frame='tag_n', category=0)
SetBit(rowID=70592993, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70592993, frame='tag_n', category=2408320)
SetBit(rowID=70592994, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70592994, frame='tag_n', category=0)
SetBit(rowID=70592995, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70592995, frame='tag_n', category=2408326)
SetBit(rowID=70592996, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70592996, frame='tag_n', category=2408329)
SetBit(rowID=70593001, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70593001, frame='tag_n', category=0)
SetBit(rowID=70593003, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70593003, frame='tag_n', category=0)
SetBit(rowID=70593004, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70593004, frame='tag_n', category=2408358)
SetBit(rowID=70593009, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70593009, frame='tag_n', category=2408350)
SetBit(rowID=70593345, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70593345, frame='tag_n', category=0)
SetBit(rowID=70593346, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70593346, frame='tag_n', category=2408574)
SetBit(rowID=70593347, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70593347, frame='tag_n', category=2408576)
SetBit(rowID=70593348, frame='tag_n', columnID=4001656)
SetRowAttrs(rowID=70593348, frame='tag_n', category=2408578)

[seanbrant]

Errors like this happen frequently (we've had 11k so far) it seems to happen when the cluster is under load. We have not observed errors like this in older builds.

[jaffee]

I believe we're using the default http Client for node->node communication. There may be some settings we can tweak to make it more resilient. This post has some pointers, but there are many like it.

The idea is to make sure we're dealing with network connections in a sanitary way. Always setting timeouts, cleaning up old connections, closing response bodies, etc.

[raskle]

Could it be we are running out of free TCP connections that are not in TIME_WAIT state
golang/go#16012
Maybe this could help if we have a way to test and repo the issue
http.DefaultTransport.(*http.Transport).MaxIdleConnsPerHost = 100

[jaffee]

that's definitely worth a try @raskle. @seanbrant - if you're up for a cluster reboot, I'm sure we could cut you a build with that change pretty easily.

[seanbrant]

Yeah lets do it, just let me know when it's ready!