Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

secret create, config create: refactor, use limit reader, and touch up errors #5912

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

thaJeztah
Copy link
Member

secret create: refactor, use limit reader, and touch up errors

Swarm has size constraints on the size of secrets, but the client-side would
read content into memory, regardless its size. This could lead to either the
client reading too much into memory, or it sending data that's larger than
the size limit of gRPC, which resulted in the error not being handled by
SwarmKit and a generic gRPC error returned.

Reading a secret from a file was added in moby@c6f0b7f, which used a
system.OpenSequential for reading (FILE_FLAG_SEQUENTIAL_SCAN). While
there could be a very marginal benefit to prevent polluting the system's
cache (Windows won’t aggressively keep it in the cache, freeing up system
memory for other tasks). These details were not documented in code, and
possibly may be too marginal, but adding a comment to outline won't hurt
so this patch also adds a comment.

This patch:

  • Rewrites readSecretData to not return a nil-error if no file was
    set, in stead only calling it when not using a driver.
  • Implements reading the data with a limit-reader to prevent reading
    large files into memory.
  • The limit is based on SwarmKits limits (MaxSecretSize), but made
    twice that size, just in case larger sizes are supported in future;
    the main goal is to have some constraints, and to prevent hitting
    the gRPC limit.
  • Updates some error messages to include STDIN (when used), or the
    filename (when used).

Before this patch:

ls -lh largefile
-rw-------  1 thajeztah  staff   8.1M Mar  9 00:19 largefile

docker secret create nosuchfile ./nosuchfile
Error reading content from "./nosuchfile": open ./nosuchfile: no such file or directory

docker secret create toolarge ./largefile
Error response from daemon: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (8462870 vs. 4194304)

docker secret create empty ./emptyfile
Error response from daemon: rpc error: code = InvalidArgument desc = secret data must be larger than 0 and less than 512000 bytes

cat ./largefile | docker secret create toolarge -
Error response from daemon: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (8462870 vs. 4194304)

cat ./emptyfile | docker secret create empty -
Error response from daemon: rpc error: code = InvalidArgument desc = secret data must be larger than 0 and less than 512000 bytes

With this patch:

docker secret create nosuchfile ./nosuchfile
error reading from ./nosuchfile: open ./nosuchfile: no such file or directory

docker secret create empty ./emptyfile
error reading from ./emptyfile: data is empty

docker secret create toolarge ./largefile
Error response from daemon: rpc error: code = InvalidArgument desc = secret data must be larger than 0 and less than 512000 bytes

cat ./largefile | docker secret create toolarge -
Error response from daemon: rpc error: code = InvalidArgument desc = secret data must be larger than 0 and less than 512000 bytes

cat ./emptyfile | docker secret create empty -
error reading from STDIN: data is empty

config create: refactor, use limit reader, and touch up errors

Swarm has size constraints on the size of configs, but the client-side would
read content into memory, regardless its size. This could lead to either the
client reading too much into memory, or it sending data that's larger than
the size limit of gRPC, which resulted in the error not being handled by
SwarmKit and a generic gRPC error returned.

Reading a config from a file used a system.OpenSequential for reading
(FILE_FLAG_SEQUENTIAL_SCAN). While there could be a very marginal benefit
to prevent polluting the system's cache (Windows won’t aggressively keep it
in the cache, freeing up system memory for other tasks). These details were
not documented in code, and possibly may be too marginal, but adding a comment
to outline won't hurt so this patch also adds a comment.

This patch:

  • Factors out the reading code to a readConfigData, analogous to the
    equivalent in secret create.
  • Implements reading the data with a limit-reader to prevent reading
    large files into memory.
  • The limit is based on SwarmKits limits (MaxConfigSize), but made
    twice that size, just in case larger sizes are supported in future;
    the main goal is to have some constraints, and to prevent hitting
    the gRPC limit.
  • Updates some error messages to include STDIN (when used), or the
    filename (when used).

Before this patch:

ls -lh largefile
-rw-------  1 thajeztah  staff   8.1M Mar  9 00:19 largefile

docker config create nosuchfile ./nosuchfile
Error reading content from "./nosuchfile": open ./nosuchfile: no such file or directory

docker config create toolarge ./largefile
Error response from daemon: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (8462870 vs. 4194304)

docker config create empty ./emptyfile
Error response from daemon: rpc error: code = InvalidArgument desc = config data must be larger than 0 and less than 1024000 bytes

cat ./largefile | docker config create toolarge -
Error response from daemon: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (8462870 vs. 4194304)

cat ./emptyfile | docker config create empty -
Error response from daemon: rpc error: code = InvalidArgument desc = config data must be larger than 0 and less than 1024000 bytes

With this patch:

docker config create nosuchfile ./nosuchfile
error reading from ./nosuchfile: open ./nosuchfile: no such file or directory

docker config create empty ./emptyfile
error reading from ./emptyfile: data is empty

docker config create toolarge ./largefile
Error response from daemon: rpc error: code = InvalidArgument desc = config data must be larger than 0 and less than 1024000 bytes

cat ./largefile | docker config create toolarge -
Error response from daemon: rpc error: code = InvalidArgument desc = secret data must be larger than 0 and less than 1024000 bytes

cat ./emptyfile | docker config create empty -
error reading from STDIN: data is empty

- Human readable description for the release notes

- A picture of a cute animal (not mandatory but encouraged)

Swarm has size constraints on the size of secrets, but the client-side would
read content into memory, regardless its size. This could lead to either the
client reading too much into memory, or it sending data that's larger than
the size limit of gRPC, which resulted in the error not being handled by
SwarmKit and a generic gRPC error returned.

Reading a secret from a file was added in [moby@c6f0b7f], which used a
system.OpenSequential for reading ([FILE_FLAG_SEQUENTIAL_SCAN]). While
there could be a very marginal benefit to prevent polluting the system's
cache (Windows won’t aggressively keep it in the cache, freeing up system
memory for other tasks). These details were not documented in code, and
possibly may be too marginal, but adding a comment to outline won't hurt
so this patch also adds a comment.

This patch:

- Rewrites readSecretData to not return a nil-error if no file was
  set, in stead only calling it when not using a driver.
- Implements reading the data with a limit-reader to prevent reading
  large files into memory.
- The limit is based on SwarmKits limits ([MaxSecretSize]), but made
  twice that size, just in case larger sizes are supported in future;
  the main goal is to have some constraints, and to prevent hitting
  the gRPC limit.
- Updates some error messages to include STDIN (when used), or the
  filename (when used).

Before this patch:

    ls -lh largefile
    -rw-------  1 thajeztah  staff   8.1M Mar  9 00:19 largefile

    docker secret create nosuchfile ./nosuchfile
    Error reading content from "./nosuchfile": open ./nosuchfile: no such file or directory

    docker secret create toolarge ./largefile
    Error response from daemon: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (8462870 vs. 4194304)

    docker secret create empty ./emptyfile
    Error response from daemon: rpc error: code = InvalidArgument desc = secret data must be larger than 0 and less than 512000 bytes

    cat ./largefile | docker secret create toolarge -
    Error response from daemon: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (8462870 vs. 4194304)

    cat ./emptyfile | docker secret create empty -
    Error response from daemon: rpc error: code = InvalidArgument desc = secret data must be larger than 0 and less than 512000 bytes

With this patch:

    docker secret create nosuchfile ./nosuchfile
    error reading from ./nosuchfile: open ./nosuchfile: no such file or directory

    docker secret create empty ./emptyfile
    error reading from ./emptyfile: data is empty

    docker secret create toolarge ./largefile
    Error response from daemon: rpc error: code = InvalidArgument desc = secret data must be larger than 0 and less than 512000 bytes

    cat ./largefile | docker secret create toolarge -
    Error response from daemon: rpc error: code = InvalidArgument desc = secret data must be larger than 0 and less than 512000 bytes

    cat ./emptyfile | docker secret create empty -
    error reading from STDIN: data is empty

[moby@c6f0b7f]: moby/moby@c6f0b7f
[FILE_FLAG_SEQUENTIAL_SCAN]: https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea#FILE_FLAG_SEQUENTIAL_SCAN
[MaxSecretSize]: https://pkg.go.dev/github.com/moby/swarmkit/[email protected]/api/validation#MaxSecretSize

Signed-off-by: Sebastiaan van Stijn <[email protected]>
Swarm has size constraints on the size of configs, but the client-side would
read content into memory, regardless its size. This could lead to either the
client reading too much into memory, or it sending data that's larger than
the size limit of gRPC, which resulted in the error not being handled by
SwarmKit and a generic gRPC error returned.

Reading a config from a file used a system.OpenSequential for reading
([FILE_FLAG_SEQUENTIAL_SCAN]). While there could be a very marginal benefit
to prevent polluting the system's cache (Windows won’t aggressively keep it
in the cache, freeing up system memory for other tasks). These details were
not documented in code, and possibly may be too marginal, but adding a comment
to outline won't hurt so this patch also adds a comment.

This patch:

- Factors out the reading code to a readConfigData, analogous to the
  equivalent in secret create.
- Implements reading the data with a limit-reader to prevent reading
  large files into memory.
- The limit is based on SwarmKits limits ([MaxConfigSize]), but made
  twice that size, just in case larger sizes are supported in future;
  the main goal is to have some constraints, and to prevent hitting
  the gRPC limit.
- Updates some error messages to include STDIN (when used), or the
  filename (when used).

Before this patch:

    ls -lh largefile
    -rw-------  1 thajeztah  staff   8.1M Mar  9 00:19 largefile

    docker config create nosuchfile ./nosuchfile
    Error reading content from "./nosuchfile": open ./nosuchfile: no such file or directory

    docker config create toolarge ./largefile
    Error response from daemon: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (8462870 vs. 4194304)

    docker config create empty ./emptyfile
    Error response from daemon: rpc error: code = InvalidArgument desc = config data must be larger than 0 and less than 1024000 bytes

    cat ./largefile | docker config create toolarge -
    Error response from daemon: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (8462870 vs. 4194304)

    cat ./emptyfile | docker config create empty -
    Error response from daemon: rpc error: code = InvalidArgument desc = config data must be larger than 0 and less than 1024000 bytes

With this patch:

    docker config create nosuchfile ./nosuchfile
    error reading from ./nosuchfile: open ./nosuchfile: no such file or directory

    docker config create empty ./emptyfile
    error reading from ./emptyfile: data is empty

    docker config create toolarge ./largefile
    Error response from daemon: rpc error: code = InvalidArgument desc = config data must be larger than 0 and less than 1024000 bytes

    cat ./largefile | docker config create toolarge -
    Error response from daemon: rpc error: code = InvalidArgument desc = secret data must be larger than 0 and less than 1024000 bytes

    cat ./emptyfile | docker config create empty -
    error reading from STDIN: data is empty

[FILE_FLAG_SEQUENTIAL_SCAN]: https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea#FILE_FLAG_SEQUENTIAL_SCAN
[MaxConfigSize]: https://pkg.go.dev/github.com/moby/swarmkit/[email protected]/manager/controlapi#MaxConfigSize

Signed-off-by: Sebastiaan van Stijn <[email protected]>
@thaJeztah thaJeztah added status/2-code-review area/swarm kind/refactor PR's that refactor, or clean-up code labels Mar 9, 2025
@thaJeztah thaJeztah changed the title config create: refactor, use limit reader, and touch up errors secret create, config create: refactor, use limit reader, and touch up errors Mar 9, 2025
@codecov-commenter
Copy link

codecov-commenter commented Mar 9, 2025

Codecov Report

Attention: Patch coverage is 61.19403% with 26 lines in your changes missing coverage. Please review.

Project coverage is 59.28%. Comparing base (2eec746) to head (5b9d046).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5912      +/-   ##
==========================================
+ Coverage   59.26%   59.28%   +0.01%     
==========================================
  Files         357      357              
  Lines       29771    29806      +35     
==========================================
+ Hits        17645    17671      +26     
- Misses      11153    11158       +5     
- Partials      973      977       +4     
🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/swarm kind/refactor PR's that refactor, or clean-up code status/2-code-review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants