-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[save] Stream Worksheet XML to Disk #1255
Conversation
Cherry pick 15daf92 |
Using this, is a game changer memory wise. Writing a 500,000 x 100 rnorm data frame previously required - with the optimizations from this branch - ~28GB. Writing the output file with pugixml using a custom xml writer, reduces the required memory amount by 12-14GB. options("openxlsx2.export_with_pugi" = FALSE) This branch skips building the XML file in memory before it is written to the disk. The custom XML writer simply flushes what is available into a text file. Without any checks for correct XML etc. |
485e774
to
6e430fd
Compare
6e430fd
to
306a58c
Compare
|
b097ae0
to
0b8633e
Compare
This is an experimental branch, using a custom C++ function to stream the XML file to disk. It is working, but not extensively tested and might still lack features. It has shown that it can reduce the memory consumption significantly.