Ideas Test

16 VOTE

Using hard drive space for longer/unlimited capture

Why not use both RAM and hard drive space to store even more data?

I understand that hard drives are slower, but since you are implementing RLE (or something like it) anyways, why not at least attempt it? Dump the RAM data into a file as fast as possible. If the hard drive can't keep up, then the RAM will get full like usual.

I have a SSD so it should help quite a bit if the data toggle is sparse.

  • Avatar32.5fb70cce7410889e661286fd7f1897de Guest
  • Jun 26 2018
  • Planned
  • Attach files
  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    26 Jun 19:40

    The current software doesn't internally use a buffer that can be paged directly to disk, unfortunately. We are working on the back end of the software now, and long term, we want to make sure that our new back end architecture supports this. We are also looking into more short term solutions to make better use of the installed RAM. If you are recording analog data, please disable the 'upsampled pipeline' in the options to improve performance and reduce memory usage. If you are using protocol decoders, you can also try removing them during the capture to reduce memory usage.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    26 Jun 19:40

    The green motto of "Reduce, Reuse, Recycle" seems useful. Dropping unwanted data Reduces the storage requirements...

    Reuse would be a set of ring buffers in RAM for tracking each channel in a circular buffer and keeping XXXX samples or XXX.XXX seconds in memory per channel. That way when a triggered even occurs you already have the previous data and then either XXX.XXX seconds of trailing capture or XXXX numbers of sample later... This would keep the RAM requirement to something more predictable and allow captures to run for hours or days, but not consume massive resources...

    Recycle would be to allocate a set of ring buffers for the current capture, one set for the next capture and once the current set is full, the addresses of the buffers to write to would be swapped and a secondary thread would take the old ring buffers and write them to disk files in the background. Once the background threads complete the write, the buffers are either zeroed and passed back to be used on the next round, freed and released back to the OS, or some such... Maybe there are more than two sets of buffers. Maybe there are 10 smaller ones, and they are rotated...

    Lots of options, but I would think something like this might help alleviate other design limitations... This would likely have issues at the upper end of USB 3.0 bandwidths. With so many CPU cores available today, though and even laptops having M.2 storage with ~1GB/sec storage, this sort of design seems workable... Just my $0.02 worth...

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    26 Jun 19:40

    I have a few items that sort of fit under this topic.

    One could be that the 7Zip library or something similar could help:
    http://www.7-zip.org/sdk.html
    It can do multithreaded large memory compression that could possible be done asynchronously with/alongside/behind the actual capture. Unused/idle CPU cores would be able to reduce either memory allocation/usage or disk size consumed. It could also be that file sizes are limited to say 1GB and rotated in filename/sequence.

    In any case the direct to disk capture helps a great deal for some circumstances.

    However the biggest problem with this is the lack of capture edit capabilities in the software. With such huge captured disk files, it should be possible to easily edit/throw away unneeded (boring) parts of any capture. Like editing in Audacity. This is a similar type of sequential data set that has tremendous length problems as well (but on a different scale in kHz vs MHz).

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    26 Jun 19:40

    This is actually possible to do in real time. You currently limit the usb 3.0 data stream to 2Gbps and there's SSDs that MAY be able to handle that continuously.
    But if do a RAID 0 or RAID 5 configuration, it'll definitely be able to handle the full stream.

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    26 Jun 19:40

    Another thing you'll get for free with this (at least on Linux) is the ability to stream your results to a Linux fifo (mkfifo) and then write standard Linux processes to find interesting features in the data streams. (You could then stop the capture when you find a particular pattern and only write out to disk the small amount of data near this point.)

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    26 Jun 19:40

    Definite requirement and unsure why you can't do that simply, especially if the sampling rate is not excessive (it's a trade-off some of us would be willing to make). The current max time window is much too small for most analysis applications.

    Thanks,
    Pierre Costa, LMTS
    ATT Labs
    Austin, TX

  • Avatar40.8f183f721a2c86cd98fddbbe6dc46ec9
    Guest commented
    26 Jun 19:40

    Maybe using a better compression on the data? Like zip?