You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We want to add shuffling to a non-spark dataset but we don't need to use Flyte/DynamoDB/etc to do this. We can simply sequentially shuffle locally and not worry about the runtime being O(N) for right now. We can come back later in time to update this to user multi-proc approach.
Acceptance Criteria:
Sequential shuffling of local basic persistor can be turned on
Tests exist to test that the sequential shuffling shuffles but persists all of the data correctly
The text was updated successfully, but these errors were encountered:
Overview:
We want to add shuffling to a non-spark dataset but we don't need to use Flyte/DynamoDB/etc to do this. We can simply sequentially shuffle locally and not worry about the runtime being O(N) for right now. We can come back later in time to update this to user multi-proc approach.
Acceptance Criteria:
The text was updated successfully, but these errors were encountered: