[BUG] gRPC load balancer does not work

[mostafafarzaneh]

Summary

Service Connect failed to distribute traffic between the two services correctly when using gRPC

Description

We utilize gRPC for communication between two services: Service A (client) and Service B (server), with Service B running multiple instances for scalability.
Under AppMesh, load balancing across these instances was effective. However, since migrating to Service Connect, all traffic from Service A is directed to a single instance of Service B, bypassing the expected load distribution.

Expected Behavior

Service Connect should distribute traffic across all instances of Service B similarly to how AppMesh did.

Observed Behavior

All traffic is routed to only one instance of Service B, leading to potential performance bottlenecks.

Environment Details

We are using AWS ECS with fargate.

[adventureisyou]

Can confirm, we're seeing the same behaviour.

gRPC prefers to hold a single connection and run many streams (requests) over it. ALB and AppMesh both correctly round robin individual streams to backend services, but ECS connect only round robins the connections themselves, meaning that all the cross-service calls from one task go to the same task of the other service. This effect is worsened if all tasks of one service happen pick a single task from the other service to connect to by chance.

We've had to abandon ECS connect for now because of this.