Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PUT appends empty bytes to the end of files #478

Closed
tampajohn opened this issue Sep 16, 2021 · 4 comments
Closed

PUT appends empty bytes to the end of files #478

tampajohn opened this issue Sep 16, 2021 · 4 comments

Comments

@tampajohn
Copy link

tampajohn commented Sep 16, 2021

Issue description

When executing a PUT on a parquet file (83620KB) 32KB of 0x00 bytes are written to the end of the file. This causes issues with parquet as it's expecting parquet's magic bytes footer.

Tell us what should happen and what happens instead

Ignoring compression, the file uploaded should contain the same content.

Example code

	_, err = s.sfdb.Exec(fmt.Sprintf("PUT 'file://%s' @%s auto_compress=true overwrite=true", pathname, tableFile.TableName))

Error log

No errors on PUT

Configuration

Driver version (or git SHA): v1.6.1

Go version: go1.17.1 linux/amd64

Server version: 5.34.1

@tampajohn
Copy link
Author

This doesn't seem to happen always, but it does occur with various file sizes.

@jgraettinger
Copy link

jgraettinger commented Sep 20, 2021

Seeing this as well with PUT JSON files, every attempt thus far. Example query:

PUT file:///tmp/82e2839d-3ffd-40da-ab0f-23abedec2c5b @flow_v1 AUTO_COMPRESS=FALSE SOURCE_COMPRESSION=NONE OVERWRITE=TRUE ;

Reported error on trying to use it:

Error parsing JSON: invalid character outside of a string: '\0' File 'stages/5f17cf82-7b1d-40a3-9ded-75f753d1e4aa/82e2839d-3ffd-40da-ab0f-23abedec2c5b3abedec2c5b', line 239187, character 1 Row 239186 starts at line 239186, column $1 If you would like to continue loading when an error is encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option. For more information on loading options, please run 'info loading_data' in a SQL client.

@sfc-gh-jbahk
Copy link
Contributor

Investigating this. Will update soon.

@sfc-gh-jbahk
Copy link
Contributor

This has been addressed and merged. It is available in the newest release. @tampajohn @jgraettinger

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants