Data Compression - offloading?

The technical support forum for Firestreamer (the virtual tape library).
Locked
johnmantle
Posts: 6
Joined: 07 Feb 2012, 09:45

Post by johnmantle »

jsf
Cristalink Support
Posts: 300
Joined: 29 Aug 2010, 09:03

Post by jsf »

Firestreamer uses the same compression as NTFS. Each tape drive runs in its own thread and uses a separate instance of the "compressor". It's up to Windows to schedule the threads on different processors. In theory, we can add settings to associate specific tape drives with specific processors, but we don't think it will be really useful. The simpler, the better.

Note that the compression consumes a lot of CPU time if the data is uncompressible (for example, .zip, movies, photos, etc). You may want to isolate the protection groups that cause the high CPU usage and disable the compression for them.

Our customers doesn't complain about the compression often, so we didn't feel the need for improvements in this area. However, your suggestion to use a hardware compression card looks interesting, and we will look into it.

>>we have registered very heavy usage on CPU0/Core0

Does the usage of CPU0/Core0 improve if you disable the compression?
Best regards,
John Smith
Cristalink Support
johnmantle
Posts: 6
Joined: 07 Feb 2012, 09:45

Post by johnmantle »

The usage of CPU0/core0 is significantly reduced when we disable compression (from 70%-99% (high base load, frequent spikes) back to 15%-60% (with infrequent spikes), and the frequency of spiking is also not visible on the other cores - so this does seem to be compression related.
It seems that the O/S is managing the threads to some degree so your explanation fits with our observations and this is NOT an operational problem for us!
Thus, this post is not a complaint - we just wanted to bring the observed behaviour to your attention - we feel that FS is a very good product and we want to help keep it that way! :)
Compression offloading to custom ASIC or a GPU software driver would be the "icing on the cake".
Thanks for the suggestion regarding Protection Group content - we'll keep an eye on it and see if we can do any tuning.
John Mantle.
jsf
Cristalink Support
Posts: 300
Joined: 29 Aug 2010, 09:03

Post by jsf »

Thank you for your feedback.
Best regards,
John Smith
Cristalink Support
jsf
Cristalink Support
Posts: 300
Joined: 29 Aug 2010, 09:03

Post by jsf »

I checked BridgeSTOR's website, and it seems to me that their card provides "transparent" compression and therefore can be used with any software, including Firestreamer. The only problem is that the card is designed for local drives. It won't compress, for example, a network share, unless it's installed in the server that provides the share. It doesn't seem to support removable drives either. We will see if there are other cards available that provide pure compression without being tied to specific storage.
Best regards,
John Smith
Cristalink Support
Locked