-
Notifications
You must be signed in to change notification settings - Fork 245
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DRIVERS-716 Improved Bulk Write API #1534
Merged
Merged
Changes from 75 commits
Commits
Show all changes
92 commits
Select commit
Hold shift + click to select a range
fb27ce1
initial work
isabelatkinson 0ede8ae
retryability
isabelatkinson afb1ab6
changelog
isabelatkinson e2cb197
transactions
isabelatkinson a24c7bd
unified test runner changes
isabelatkinson 5a98994
add schema
isabelatkinson 98fbfda
add batching prose tests
isabelatkinson 7f59210
restructure spec
isabelatkinson dbb2083
various cleanup
isabelatkinson bc888f8
schema version updates
isabelatkinson b425c23
move run on requirements to top level
isabelatkinson d7294cd
always use verbose results
isabelatkinson 82799a7
clean up yaml anchors
isabelatkinson 905e2ee
clean up command started events
isabelatkinson 981e6c3
move tests into crud directory
isabelatkinson 03c3bde
fix typo
isabelatkinson 01c9ae7
update some language
isabelatkinson fab1deb
self review changes
isabelatkinson 7e1e897
language updates
isabelatkinson d907502
minor updates
isabelatkinson a4cf43f
review changes
isabelatkinson 13f88a7
fix ordered
isabelatkinson ee27603
add future work
isabelatkinson 94ed555
add errinfo
isabelatkinson b7a9d86
remove note escapes
isabelatkinson d557b6e
must -> should
isabelatkinson 89c99f6
remove q
isabelatkinson 82aa4f2
more review changes
isabelatkinson fb6007e
remove outdated test, add bypassDocumentValidation language
isabelatkinson c34e432
clarify
isabelatkinson 0491cec
insert -> write
isabelatkinson c4ddf5b
clarify naming
isabelatkinson c308b1b
add network errors tests
isabelatkinson 48b7cbf
add comment option
isabelatkinson d5d98f0
use exists for upsertedId
isabelatkinson 43a4ffb
rename write concern error message
isabelatkinson d1e1a33
update upserted
isabelatkinson ec6863b
language
isabelatkinson 7c8eb73
test lsid and txnNumber
isabelatkinson bd384b5
improve size batching length check
isabelatkinson 1633501
add bypassdocumentvalidation false test
isabelatkinson 7f31275
writeconcernerror batch prose test
isabelatkinson 3ab6d00
writeerror batch prose test
isabelatkinson 06cf1e3
command monitoring, cursor tests
isabelatkinson 1677d57
killcursors
isabelatkinson c74634c
add pipeline tests
isabelatkinson 7460b42
update vs. replace validation
isabelatkinson a83fe6b
add validation tests
isabelatkinson 1077086
partial result note
isabelatkinson 947004b
fix failed iteration test
isabelatkinson e6d8bc5
empty models test
isabelatkinson e2effb8
basic write concern test
isabelatkinson fa5395b
more write concern tests
isabelatkinson c231068
add retryWrites:false test
isabelatkinson bef5231
add unacknowledged test
isabelatkinson 9b82f6f
reduce _id size
isabelatkinson 6b4f623
iteration test updates
isabelatkinson 1540e57
errorResponse
isabelatkinson aaf6a21
add files
isabelatkinson c5ba181
add unacknowledged monitoring test
isabelatkinson 6995d24
add handshake error test
isabelatkinson e6f8100
operation ID server selection test
isabelatkinson c09c1d4
add pinning test
isabelatkinson fa4b434
add write concern transaction test
isabelatkinson 919c506
another transaction test
isabelatkinson af0a16d
add csot test
isabelatkinson 72eda1b
cleanup
isabelatkinson f389db6
add q&a
isabelatkinson 8f54d52
nsInfo clarifications and test
isabelatkinson ec71c96
retry bulkWrite when getMore fails
isabelatkinson 6cde925
Revert "retry bulkWrite when getMore fails"
isabelatkinson 5993389
add future work section, rewrite size batch splitting rules
isabelatkinson ccb03ed
add ns batch split test
isabelatkinson 834cca8
too large test
isabelatkinson 80b7499
language
isabelatkinson 2c9702d
language
isabelatkinson 3fdd543
improve CSOT test
isabelatkinson 7ef7c8c
jeremy language suggestions
isabelatkinson e133ed3
add oid to calculation info
isabelatkinson b9f39a6
update batching tests
isabelatkinson fb659af
remove auto-encryption support
isabelatkinson c12cf6c
server versions
isabelatkinson 35b2250
Merge branch 'bulk-write' of github.com:isabelatkinson/specifications…
isabelatkinson a3d1652
update csot test formatting
isabelatkinson d23399c
1.20 -> 1.21
isabelatkinson 91bec49
save files
isabelatkinson ebe592e
Merge branch 'master' into bulk-write
isabelatkinson cd64072
fix lint errors
isabelatkinson ae0d729
fix schema versions
isabelatkinson 05852ad
json
isabelatkinson ad57a1e
bump schema version in makefile
isabelatkinson 5a6532f
merge 1.20 and 1.21
isabelatkinson File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
186 changes: 186 additions & 0 deletions
186
source/command-logging-and-monitoring/tests/monitoring/unacknowledged-client-bulkWrite.json
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
93 changes: 93 additions & 0 deletions
93
source/command-logging-and-monitoring/tests/monitoring/unacknowledged-client-bulkWrite.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
description: "unacknowledged-client-bulkWrite" | ||
|
||
schemaVersion: "1.0" | ||
|
||
createEntities: | ||
- client: | ||
id: &client client | ||
observeEvents: | ||
- commandStartedEvent | ||
- commandSucceededEvent | ||
- commandFailedEvent | ||
uriOptions: | ||
w: 0 | ||
- database: | ||
id: &database database | ||
client: *client | ||
databaseName: &databaseName command-monitoring-tests | ||
- collection: | ||
id: &collection collection | ||
database: *database | ||
collectionName: &collectionName test | ||
|
||
initialData: | ||
- collectionName: *collectionName | ||
databaseName: *databaseName | ||
documents: | ||
- { _id: 1, x: 11 } | ||
- { _id: 2, x: 22 } | ||
- { _id: 3, x: 33 } | ||
|
||
_yamlAnchors: | ||
namespace: &namespace "command-monitoring-tests.test" | ||
|
||
tests: | ||
- description: 'A successful mixed client bulkWrite' | ||
operations: | ||
- object: *client | ||
name: clientBulkWrite | ||
arguments: | ||
models: | ||
- insertOne: | ||
namespace: *namespace | ||
document: { _id: 4, x: 44 } | ||
- updateOne: | ||
namespace: *namespace | ||
filter: { _id: 3 } | ||
update: { $set: { x: 333 } } | ||
expectResult: | ||
insertedCount: | ||
$$unsetOrMatches: 0 | ||
upsertedCount: | ||
$$unsetOrMatches: 0 | ||
matchedCount: | ||
$$unsetOrMatches: 0 | ||
modifiedCount: | ||
$$unsetOrMatches: 0 | ||
deletedCount: | ||
$$unsetOrMatches: 0 | ||
insertResults: | ||
$$unsetOrMatches: {} | ||
updateResults: | ||
$$unsetOrMatches: {} | ||
deleteResults: | ||
$$unsetOrMatches: {} | ||
expectEvents: | ||
- | ||
client: *client | ||
events: | ||
- commandStartedEvent: | ||
commandName: bulkWrite | ||
databaseName: admin | ||
command: | ||
bulkWrite: 1 | ||
errorsOnly: true | ||
ordered: true | ||
ops: | ||
- insert: 0 | ||
document: { _id: 4, x: 44 } | ||
- update: 0 | ||
filter: { _id: 3 } | ||
updateMods: { $set: { x: 333 } } | ||
multi: false | ||
nsInfo: | ||
- ns: *namespace | ||
- commandSucceededEvent: | ||
commandName: bulkWrite | ||
reply: | ||
ok: 1 | ||
nInserted: { $$exists: false } | ||
nMatched: { $$exists: false } | ||
nModified: { $$exists: false } | ||
nUpserted: { $$exists: false } | ||
nDeleted: { $$exists: false } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would an insert batch of 50 1MB documents take an "exceedingly long time"? Also, if a fail point is being used on each command, couldn't you also get by using fewer, larger documents that split into 2+ commands and trigger the timeout?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, updated to insert a few large documents. I lifted most of this test from the existing
insertMany
test for consistency, but the actual writes happening shouldn't matter much as long as the sameblockConnection
is being used. Some basic local benchmarking of inserting a few large documents didn't show any time differences based on topology so I also removed that language.Caveat: Kevin and I can't verify that this works as Rust and C both don't implement CSOT, so we'll need to wait for a driver that does have CSOT to implement this.