SNOW-521578 PUT Memory Enhancements #527
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sfc-gh-jbahk
force-pushed
the
SNOW-521578-fix-golang-perf
branch
from
2890498 to
57c10bb
Compare
sfc-gh-jbahk
force-pushed
the
SNOW-521578-fix-golang-perf
branch
from
57c10bb to
47a5940
Compare
sfc-gh-jbahk
changed the title
6 tasks
sfc-gh-cshi
reviewed
Feb 7, 2022
file_util.go
Outdated
| for { | ||
| chunk := make([]byte, chunkSize) | ||
| chunk := make([]byte, fileChunkSize) |
There was a problem hiding this comment.
dow e need to always 'make' a new chunk each time?
There was a problem hiding this comment.
You are correct, we can pull this out of the loop.
sfc-gh-jbahk
force-pushed
the
SNOW-521578-fix-golang-perf
branch
from
a106068 to
53665ec
Compare
sfc-gh-cshi
approved these changes
Feb 9, 2022
agam
pushed a commit
to sigmacomputing/gosnowflake
that referenced
this pull request
Feb 15, 2022
* modify block size for azure * fix incorrect source * open file v read * move chunk out of loop
agam
added a commit
to sigmacomputing/gosnowflake
that referenced
this pull request
Feb 22, 2022
* Refactor & Lint (snowflakedb#525) * move function calls * fix lint * SNOW-530866: Bumped up GoLang connector PATCH version from 1.6.5 to 1.6.6 (snowflakedb#529) * SNOW-534077 Multi-Statement Missing Result IDs (snowflakedb#534) * SNOW-440362 Implement UUIDv4 (snowflakedb#541) * implement uuid * unexport rfc * SNOW-535399 Fix GetQueryStatus Bug (snowflakedb#539) * change errorcode to string * fix emtpy query status * fix null pointer dereference * SNOW-521578 PUT Memory Enhancements (snowflakedb#527) * modify block size for azure * fix incorrect source * open file v read * move chunk out of loop * Arrow Record Result Batches (snowflakedb#544) * arrow record channel in context impl + unit test * iteration on comments 1 * changed mutex map struct to sync.map * final iteration on comments * added converter for snowflake records * added converter tests * releasing record as soon as possible * fix value has non-zero nanoseconds error * iteration on comments * withDistributedBatches refactor * reverting withArrowRecordChan changes * adjusting go.mod for forking * first iteration on comments * final iteration on comments * reverting module change * revert exposure of SnowflakeRows, QueryContext, and change signature of Fetch * improved chunk_test to query more than 1 chunk * improved chunk_test to show download workers, added metadata, and made Rec private * nomenclature tweaks * Formatting & Lint (snowflakedb#545) * Fix Select 1 Bug (snowflakedb#511) Make port optional Co-authored-by: sfc-gh-jbahk <[email protected]> * SNOW-526255 Fix Time Scale for Arrow (snowflakedb#547) * fix time scale issue * add timezone in ltz * remove high precision context * Allow Client to specify ClientTimeout in the DSN (snowflakedb#484) * add failing test * add fix * SNOW-544029: Bumped up GoLang connector PATCH version from 1.6.6 to 1.6.7 (snowflakedb#548) (snowflakedb#549) * Fixes after syncing with upstream. * Modify instructions to reflect the process I followed this time. Co-authored-by: sfc-gh-jbahk <[email protected]> Co-authored-by: Kiran Dama <[email protected]> Co-authored-by: ddl-giuliocapolino <[email protected]> Co-authored-by: yiksanchan <[email protected]> Co-authored-by: Greg Owen <[email protected]> Co-authored-by: Agam Brahma <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Modified Azure PUT block size & changed file reading mechanism from reading it all into memory to simply opening the file.
Checklist
make fmtto fix inconsistent formatsmake lintto get lint errors and fix all of them