DPM 2010 Firestreamer backup hangs - clfs3chr / STOP errors
Posted: 30 Nov 2010, 16:53
Hi,
We were running Firestreamer 3.95.9 (4.0 RC) with our Dell RD1000 removable hard drive media and Microsoft DPM 2007 to back up our Hyper-V servers for a few months now, and the system has been working without issues.
However we recently upgraded to DPM 2010 and Firestreamer 4.0 (drivers 4.0.1), and we have been having some serious problems. Specifically, the overnight short-term backup to tape hangs after a few hours, and the following errors start to appear in the System Event Log every 30 seconds from the time of the hang:
Log Name: System
Source: clfs3chr
Date: 30/11/2010 16:05:59
Event ID: 129
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: hyper1server.crgs.local
Description:
Reset to device, \Device\RaidPort4, was issued.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="clfs3chr" />
<EventID Qualifiers="32772">129</EventID>
<Level>3</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-11-30T16:05:59.027294300Z" />
<EventRecordID>11666</EventRecordID>
<Channel>System</Channel>
<Computer>hyper1server.crgs.local</Computer>
<Security />
</System>
<EventData>
<Data>\Device\RaidPort4</Data>
<Binary>0F001800010000000000000081000480040000000000000000000000000000000000000000000000000000000000000000010000810004800000000000000000</Binary>
</EventData>
</Event>
When I try to restart the server to resolve the problem, the server hangs for about 20 minutes at the 'Shutting down...' phase of the reboot process, and then generates the following STOP error:
DRIVER_POWER_STATE_FAILURE
STOP : 0x0000009F (0x0000000000000003, 0xFFFFFA800CDE1700, 0xFFFFF800015DA518, 0xFFFFFA801A5A7760)
At first when I was experiencing these issues I thought it was some sort of hardware or DPM issue, but since there are no other errors in the event log to indicate any problem with DPM or the RD1000 drive, I was at a loss to explain it. In the end I investigated 'clfs3chr' further and found that this is a Firestreamer driver, which seemed to indicate this might be the source of the problem.
The first time this error occurred I tried uninstalling Firestreamer 4.0 and reinstalling it again, as we had originally done an in-place upgrade from 3.95.9. This seemed to resolve the issue for a while, which seems to indicate that it was indeed a problem with Firestreamer. However recently I had to reformat the server again, and after installing the same software as before, including a 'clean' install of Firestreamer 4.0, after a few days of apparently working fine, the problem has reoccurred.
Please could you tell me what I can do to troubleshoot this problem, as it seems to be an issue with the latest version of Firestreamer, and I cannot use it to protect my servers until this is resolved.
We were running Firestreamer 3.95.9 (4.0 RC) with our Dell RD1000 removable hard drive media and Microsoft DPM 2007 to back up our Hyper-V servers for a few months now, and the system has been working without issues.
However we recently upgraded to DPM 2010 and Firestreamer 4.0 (drivers 4.0.1), and we have been having some serious problems. Specifically, the overnight short-term backup to tape hangs after a few hours, and the following errors start to appear in the System Event Log every 30 seconds from the time of the hang:
Log Name: System
Source: clfs3chr
Date: 30/11/2010 16:05:59
Event ID: 129
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: hyper1server.crgs.local
Description:
Reset to device, \Device\RaidPort4, was issued.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="clfs3chr" />
<EventID Qualifiers="32772">129</EventID>
<Level>3</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2010-11-30T16:05:59.027294300Z" />
<EventRecordID>11666</EventRecordID>
<Channel>System</Channel>
<Computer>hyper1server.crgs.local</Computer>
<Security />
</System>
<EventData>
<Data>\Device\RaidPort4</Data>
<Binary>0F001800010000000000000081000480040000000000000000000000000000000000000000000000000000000000000000010000810004800000000000000000</Binary>
</EventData>
</Event>
When I try to restart the server to resolve the problem, the server hangs for about 20 minutes at the 'Shutting down...' phase of the reboot process, and then generates the following STOP error:
DRIVER_POWER_STATE_FAILURE
STOP : 0x0000009F (0x0000000000000003, 0xFFFFFA800CDE1700, 0xFFFFF800015DA518, 0xFFFFFA801A5A7760)
At first when I was experiencing these issues I thought it was some sort of hardware or DPM issue, but since there are no other errors in the event log to indicate any problem with DPM or the RD1000 drive, I was at a loss to explain it. In the end I investigated 'clfs3chr' further and found that this is a Firestreamer driver, which seemed to indicate this might be the source of the problem.
The first time this error occurred I tried uninstalling Firestreamer 4.0 and reinstalling it again, as we had originally done an in-place upgrade from 3.95.9. This seemed to resolve the issue for a while, which seems to indicate that it was indeed a problem with Firestreamer. However recently I had to reformat the server again, and after installing the same software as before, including a 'clean' install of Firestreamer 4.0, after a few days of apparently working fine, the problem has reoccurred.
Please could you tell me what I can do to troubleshoot this problem, as it seems to be an issue with the latest version of Firestreamer, and I cannot use it to protect my servers until this is resolved.