Can't get large file

[ZergsLaw]

Hello guys!

I have a problem, if I try to download large file from ObjectStore by nats.go lib, it's not working, because you have added a hardcode timeout
r.SetReadDeadline(time.Now().Add(2 * time.Second))

1. nats-server version - 2.8.2
2. nats.go version - v1.16.0

Platform:
linux/macos/std-docker

Steps or code to reproduce the issue:

fVideo, err := os.Open(movFilePath)
assert.NoError(err)
t.Cleanup(func() { assert.NoError(fVideo.Close()) })

id := uuid.Must(uuid.NewV4())
meta := &nats.ObjectMeta{
	Name: id.String(),
}

store, err := conn.JetStream.ObjectStore("post_attachments")
assert.NoError(err)

_, err = store.Put(meta, fVideo, nats.Context(ctx))
assert.NoError(err)

file, err := store.Get(id.String(), nats.Context(ctx))
assert.NoError(err)

buf, err := io.ReadAll(file)
assert.NoError(err)

Expected result:

getting file

Actual result:

getting error

[derekcollison]

Is this reproduce-able with 2.8.4 server? Does increasing the timeout help solve the problem and allow the test to pass above?

Also how large is the asset and how far away (RTT) is the client application from the NATS system and the store asset?

Thanks..

[ZergsLaw]

Hi, yeah, on 2.8.4 same situation

I made fix, and it's working, but I'm not sure, mb my fix isn't correct

[ZergsLaw]

#1005 - I'm not sure, mb it is not correct, but with fix it's working

[ZergsLaw]

But for file with size 3gb+ not working anyway :(

[derekcollison]

So for 3Gb+ your fix is not working regardless yes?

What is RTT between your app and the asset? nats rtt?

[derekcollison]

What in the example above is the context? Can you share the code that creates that?

[ZergsLaw]

Yeah, for 3Gb+ is not working anyway, I'm getting lock

What do you mean? (about RTT), I'm using single node nats with js (I can share test cfg)

about ctx

If I set ctx with my fix it's working (only for not big file size less than 3Gb)

ctx, cancel := context.WithTimeout(context.Background, time.Second * 10)

[derekcollison]

Thanks If you set context timeout to say 60s does the 3Gb+ test complete?

The reads are for individual reads from the pipe but the context spans the whole get operation I believe.

[ZergsLaw]

No, 3Gb is not working anyway :(

[derekcollison]

ok will take a look, thanks for the info and the report.

[derekcollison]

Hmm ok. I added a large asset to the demo.nats.io. With a updated version of NATS cli I could download the asset which is over 3GB.

/tmp> ls -l BT
-rw-r--r--  1 derek  wheel  3317569928 Jun 24 13:25 BT
/tmp> nats obj ls BIG_TEST
╭────────────────────────────────────────────╮
│              Bucket Contents               │
├──────┬─────────┬───────────────────────────┤
│ Name │ Size    │ Time                      │
├──────┼─────────┼───────────────────────────┤
│ M1   │ 1.5 GiB │ 2022-06-24T12:53:25-07:00 │
│ BT   │ 3.1 GiB │ 2022-06-24T13:22:47-07:00 │
╰──────┴─────────┴───────────────────────────╯

/tmp> nats obj get BIG_TEST BT
Wrote: 3.1 GiB to /tmp/BT in 58.43s average 54 MiB/s
/tmp> ls -l BT
-rw-r--r--  1 derek  wheel  3317569928 Jun 24 13:25 BT

[derekcollison]

I can check to see if local things go too fast and you get a slow consumer possibly. Did you notice and logs from the server or disconnects in the app during the GET?

[derekcollison]

Any updates from your side? I still have not looked at doing a test with server and client co-located.

[derekcollison]

Wanted to ping here again and see if this issue is still present?

[nitzan-entro]

@derekcollison im having the same problem
is there a resolution? i saw the pr @ZergsLaw had, looks like it might help. why wast it merged?

this is the error i get read pipe: i/o timeout

[piotrpio]

Hey @nitzan-entro - we recently merged a PR which changes the read timeout to whatever you have configured in JetStreamContext (default is 5 seconds). If your problem is caused by hitting the read deadline set by client, this should help. It's not released yet, but will be a part of the next release.

[goingaround]

Thanks @piotrpio , do you know when this release will be out?

[nitzan-entro]

Thanks @piotrpio, do you know when this release will be out? we've been waiting for some time now and this is currently a blocking problem for us. additionally the pr link u added is leading to the issue page not to a pr...

[piotrpio]

The release will be out this week. Sorry about the link - it's this PR: #1531