Ideas Test

5 VOTE

Loading and saving files in a compressed format

Files can get big, very big. and they compress quite well. a 4gig file goes down to 200meg when zipped. Of course saving and loading those files is slower when working on an SSD, but might be faster when working with a spinning disk or over the network. and save lots of space when storing files for longer periods of time

  • Avatar32.5fb70cce7410889e661286fd7f1897de Guest
  • Aug 2 2018
  • Attach files
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    02 Aug 17:41

    I did a quick benchmark using tar using xz compression at level 1 compression a 1.99GB logic data file down to to 75MB in 40 seconds on my 2.6 GHz Intel Core i7, 16GB MacBook Pro.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    02 Aug 17:41

    Typically loading and saving compressed files is faster, as io bandwidth is a limiting factor. Even with fast SSD drives minimal compression should help. Network drives and other interfaces would benefit as well.
    See:
    - zlib https://en.wikipedia.org/wiki/Zlib
    - xz https://en.wikipedia.org/wiki/Xz

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    02 Aug 17:41

    It would be great if .logicdata files were simply compressed always (or some new extension that becomes the new default), just like how .xlsx files are zip'ed XML files. As noted they do tend to compress quite well so they would take much less space and be easier/faster to transfer.