Slow backup speed to network SOFS shares

The technical support forum for Firestreamer (the virtual tape library).
abeaudoin
Posts: 6
Joined: 02 Jun 2016, 12:17

Slow backup speed to network SOFS shares

Post by abeaudoin » 03 Jun 2016, 12:53

We have DPM 2012 R2 Update Rollup 4 with Firestreamer 4.0 virtual tape libraries, writing to tapes on a SOFS share. We access the shares over multiple 10g links. There is one share (named 3) that has some of the hard disks on it that are SSDs that are used as write caches, and one share (named 4) that has only normal spinning disks.

We are experiencing very slow sending speeds to pools on the SOFS shares, and more specifically, on the pools with the spinning disks. We are writing at about 1.5 MB/s to 3 MB/s to the tapes on those shares. On the SSD pool, we are seeing speeds up to 20 MB/s, but that is still a far cry from what we can see when copying files manually over or using diskspd to test writing and reading.

I have tried creating a vhdx file on the share 4, adding it to the DPM VM(this vhdx is seen as a local disk in the DPM VM), and then created a few tapes on it. I then started a tape backup onto a tape in this VHDX, and the speeds would be at around 40 MB/s, a lot closer to what they should be. So Firestreamer can write at fast speeds, except when it is writing to a network share.

Have you seen this before? Are there any settings we can tweak to make Firestreamer write quicker to the disks when the tapes are directly on the SOFS share?

jsf
Cristalink Support
Posts: 300
Joined: 29 Aug 2010, 09:03

Re: Slow backup speed to network SOFS shares

Post by jsf » 03 Jun 2016, 23:37

Firestreamer makes no distinction between local disks and network shares. All virtual tape files are opened with the same API call:

Code: Select all

OBJECT_ATTRIBUTES tObjAttr;
InitializeObjectAttributes( &tObjAttr, <file name>, OBJ_CASE_INSENSITIVE | OBJ_KERNEL_HANDLE | OBJ_FORCE_ACCESS_CHECK, NULL, NULL );

DWORD access = FILE_SEQUENTIAL_ONLY; // By default
ZwCreateFile( &hFile, GENERIC_READ | GENERIC_WRITE | SYNCHRONIZE, &tObjAttr, &tIosb, NULL, FILE_ATTRIBUTE_NORMAL, 0, FILE_CREATE, FILE_NON_DIRECTORY_FILE | FILE_SYNCHRONOUS_IO_NONALERT | access, NULL, 0 );
You have tested (A) a file on a network share, and (B) a file on a VHD located on the same share and mounted as a local disk. One would expect (B) to perform a bit slower than (A) because of the intermediate VHD layer. To the contrary, you observed that (B) is much faster than (A). It therefore seems like a problem with the storage stack that handles the network share, or with the share itself. The best way to resolve the problem would be to contact Microsoft Support. There is not much we can do.

You can also try changing the following Firestreamer parameters in the registry, under

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\clfs3chr\Parameters

Value name: FileMediaAccessMode
Value type: DWORD
Value data: 0 - unspecified access mode; 1 - FILE_SEQUENTIAL_ONLY (default); 2 - FILE_RANDOM_ACCESS.
No reboot is required.

Value name: ExtendFileMediaFile
Value type: DWORD
Value data: 0 - virtual tape files are extended by the actual amount of data appended to them (default); 1 - virtual tape files are extended in large blocks.
No reboot is required.

Try the following, one by one:
  1. FileMediaAccessMode=0
  2. FileMediaAccessMode=2
  3. Keep the best setting for FileMediaAccessMode (or delete the value altogether if it didn't help), and add ExtendFileMediaFile=1
Please let us know the outcome. Thanks!
Best regards,
John Smith
Cristalink Support

abeaudoin
Posts: 6
Joined: 02 Jun 2016, 12:17

Re: Slow backup speed to network SOFS shares

Post by abeaudoin » 06 Jun 2016, 20:30

Good afternoon.

We will try to change the registry keys. I understand we do not have to reboot the server, but we will need to restart the Firestreamer service, correct?

Thank you

jsf
Cristalink Support
Posts: 300
Joined: 29 Aug 2010, 09:03

Re: Slow backup speed to network SOFS shares

Post by jsf » 08 Jun 2016, 10:15

No, you don't need to do anything apart from changing the registry. The settings are applied each time a tape is loaded.
Best regards,
John Smith
Cristalink Support

abeaudoin
Posts: 6
Joined: 02 Jun 2016, 12:17

Re: Slow backup speed to network SOFS shares

Post by abeaudoin » 09 Jun 2016, 12:46

I tried all the settings for the registry keys, and none of them changed the speed at which the backups were taken.

Is it normal that I had to create the keys, or were they supposed to be there in the first place?

jsf
Cristalink Support
Posts: 300
Joined: 29 Aug 2010, 09:03

Re: Slow backup speed to network SOFS shares

Post by jsf » 09 Jun 2016, 21:53

The key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\clfs3chr\Parameters should be already present.

The other settings I mentioned are values, not keys. You need to create them.

Have you tried copying a big file (several GB in size) from the DPM VM to the network share using Windows Explorer?
Best regards,
John Smith
Cristalink Support

abeaudoin
Posts: 6
Joined: 02 Jun 2016, 12:17

Re: Slow backup speed to network SOFS shares

Post by abeaudoin » 14 Jun 2016, 18:38

I have tried to copy a 7 GB ISO file to the network locations.

Copying to the SSD pool: Max speed seen is 130 MB/s
Copying to the normal disk pool: Max Speed seen is about 110 MB/s

Much faster than any operation to backup to tape.

So I know I can send data quickly to the SOFS pools. I don't know why the backups over tape cannot.

jsf
Cristalink Support
Posts: 300
Joined: 29 Aug 2010, 09:03

Re: Slow backup speed to network SOFS shares

Post by jsf » 14 Jun 2016, 22:41

Backing up to a virtual tape on a local disk (without SOFS involved) would give you an indication of the maximum transfer rate you should expect. If the local transfer rate significantly differs from the SOFS transfer rate, then it's a problem with SOFS because Firestreamer writes virtual tapes in the same way regardless of the underlying storage.

How do you access the SOFS share, with \\server\share or c:\share?
Best regards,
John Smith
Cristalink Support

abeaudoin
Posts: 6
Joined: 02 Jun 2016, 12:17

Re: Slow backup speed to network SOFS shares

Post by abeaudoin » 15 Jun 2016, 12:21

We access using \\[clustername]\share. All our tapes are setup this way. Copying to [clustername] is fast.

jsf
Cristalink Support
Posts: 300
Joined: 29 Aug 2010, 09:03

Re: Slow backup speed to network SOFS shares

Post by jsf » 15 Jun 2016, 12:40

Can you try to back up to a non-SOFS share on a fast network, and compare the performance of SOFS and non-SOFS shares?
Best regards,
John Smith
Cristalink Support

Locked