SAS drives, Firmware and SCSI mode pages

[Airbozo]

I have been dealing with some SAS drive issues and thought I would document them here. Some of these issues apply to SATA and PATA drives and CD/DVD optical drives as well (since they use the SCSI commands). Some of this will bore people to death and some will beg for more detail.

WARNING: Following some of my examples will not only brick your HDD, it will void the warranty! Proceed with caution!

Most of these issues are related to the fact that the drives I am dealing with are end of life and my customer has not gone through the approval process for new drives.

We first encountered a problem when we bought a batch of drives that were new, but were OEM drives (for Dell I later found out). Since there were no custom labels or any indication of these being OEM drives we were blind sided when all of them had problems when configured in a raid array. Two problems were encountered; the drive activity LED was on constantly and flickered during use and the drive would randomly drop out of the array and then pop back in, causing a rebuild and all the associated alarms and alerts. Turns out that even though the drive shows no OEM information and a valid Firmware version (an older version), it is indeed an OEM drive. Dell has changed several values in the SCSI mode pages for these drives. One of them was the RLM bit which controls the function of the drive LED. Default value is off and Dell has it turned on. This caused inconsistencies and broke our process (very picky medical device customer).

I did some research and others are having issues with this in a wide variety of OEM drives. Turns out my solution was a utility that is fairly familiar to me, "sdparm". Since I have been using sdparm (and its cousin "hdparm") for many years dealing with real parallel scsi drives and tape drives I did some research on the LED bit and was able to turn the LED off. I first updated the firmware to version 0004 which I knew worked and then modified the RLM bit on scsi mode page 0x19.

OK, now some of you have already glazed over while others will be waiting for more info. That's OK. This is really kind of boring, but it shows the different options available to people to tune drives to operate a specific way or with a specific controller. We used to tune certain parameters at SGI to limit head travel, sectors, block sizes and other items to squeeze every ounce of performance out of our drives.

Next up, some examples of using sdparm in windows to verify settings between drives.

Stay tuned...

[NightrainSrt4]

Interesting. Can't wait to read more!

[Airbozo]

OK, next installment.

I wrote about what sdparm is and some simple examples of what it can do. Now I will walk you through how I am using it to compare the same model HDD with different scsi mode page entries (and hopefully why some are different).

Back to the issues that got me here in the first place:

• Older firmware on OEM drives (0002 vs 0004).
• Green drive led always on when normal behavior is off unless activity (then it flashes).

The first one was a pain to resolve and didn't involve sdparm (this time), but I will explain how I got around my issues just in case someone needs to know.

Firmware issue:
Since these drives (Seagate ST3300655SS) have been end of life for a while now, we have resorted to the "gray" market (brokers and lot resellers) to acquire them. In the beginning, we had no issues since the drives on the market were original Seagate drives, some were used, but we weeded those out (Using Seatools to verify the time the drive has been powered up). Recently the only drives available have been OEM drives. These are drives made by Seagate for DELL, IBM and HP (etc...) that have original Seagate firmware with tweaks to the scsi mode pages to work in the OEM hardware. There are other OEM drives with "custom" firmware and those are pretty easy to tell apart due to the labels on the drives. I won't go too much into that mess except to say most of those drives are locked and NOT flash-able by Seatools (or other flash utilities). Or at least that is what they want you to believe. Again the solution is the scsi mode pages. (I may touch on that later)

Seagate will not help you with these drives and instead refer you to the OEM company, but will not tell you who the OEM company is unless you talk with a supervisor. To me it didn't matter who the OEM was I just needed the firmware updated (among other things). So I broke out seatools, plugged in the drive, booted from the Seatools image and the program did not recognize any drives in the system. !! Long story short: Turns out most raid cards will not let anything talk directly with the drive and instead intercepts any commands and inquiries and tries to act on them. Not good.

After digging through my SAS controllers I found one that I knew had potential. An LSI 3442 raid controller can be flashed into IT mode so that the controller no longer intercepts drive messages and commands and presents the drive to the system as is. This also disables any raid functionality of the controller which can then be re-flashed back in IR mode to become a raid controller again.

OK, fire up Seatools and still, no drives recognized... Turns out the regular version of Seatools has a hard time with SAS drives and I had to download the Enterprise version and install it on a system (will not operate in standalone or DOS mode). Plus I found out that it will not even install on a 64 bit OS. Really? An enterprise tool that does not run on an enterprise version OS? Since all of the systems I have here with SAS controllers and compatible back planes are running 64 bit os's, I had to reload one with windows XP.

OK, that done I was finally able to see and flash the drives to version 0004. But the LED is still on. Still not usable. Now comes "sdparm". I remembered suing it to do other things with drives so I did an investigation and found out many people are having this same issue and an easy fix was already on the web. Plus I found out that sdparm had been ported to windows! Even better, now I can flash the drives and fix the LED issue on the same system.

Here are a few commands and expected output. sdparm has lots of options and I will not go through all of them, but I will add to this thread as I learn more. Just typing sdparm with no option will spit out a usage page.

The first thing I need to do is figure out what device I need to send commands to. This is accomplished using the --wscan (or -w) option to sdparm which spits out something like this(your system may show something different):

PD1 [E] SEAGATE ST3300655SS 0004 3lmxxxxxxxxxxxx

In some cases you will not get the drive letter. OK the device I need is PD1 (PD0 is my boot drive), so the next thing I need to do is find out what I need to change. A little research (thanks google) tells me that the bit I need to modify is the RLM bit on page 0x19h. When I query the contents of page 0x19h with the command "sdparm --page=0x19 pd1", I only see:

PPID 6 [cha: n, def: 6, sav: 6]

Hmm....(more on what that output means in a minute)

OK, since the RLM bit is not listed I need to query the value of that bit to see if it does exist. I do that with this command: "sdparm --get RLM=1 -H pd1" and my output is:

0x00

Which indicates the LED is on (I learned this by trial and error). Now I need to turn it of with this command: "sdparm --set=RLM pd1". If I wanted to turn it on, I would use this command: "sdparm --clear=RLM pd1" which seems counter intuitive, but whatever.

OK, now for some explanation before I get into more sdparm stuff.

Lets take the output from the earlier page 0x19 output (the full output this time):

pd1: SEAGATE ST330065SS 0004
Protocol specific port mode page:
PPID 6 [cha: n, def: 6, sav: 6]

If you add "--long" to the command, you get this:

pd1: SEAGATE ST330065SS 0004
Direct access device specific parameters: WP=0 DPOFUA=1
Protocol specific port [pp] mode page:
PPID 6 [cha: n, def: 6, sav: 6] Port's <transport> protocol identifier

which gives us some important information. One thing that is shown is the definition of the PPID variable. Most important to me is the "<transport>" tag. This allows me to query the mode pages for even more info like this:
"sdparm --transport=sas --long pd1" Which provides me the information I need:

pd1: SEAGATE ST3300655SS 0004
Direct access device specific parameters: WP=0 DPOFUA=1
Disconnect-reconnect (SAS) [dr] mode page:
BITL 0 [cha: n, def: 0, sav: 0] Bus inactivity time limit (100us)
MCTL 0 [cha: n, def: 0, sav: 0] Maximum connect time limit (100us)
MBS 314 [cha: y, def:314, sav:314] Maximum burst size (512 bytes)
Protocol specific logical unit (SAS) [pl] mode page:
LUPID 6 [cha: n, def: 6, sav: 6] Logical unit's (transport) protocol
identifier
Protocol specific port (SAS) [pp] mode page:
PPID 6 [cha: n, def: 6, sav: 6] Port's (transport) protocol identif
ier
RLM 0 [cha: y, def: 0, sav: 0] Ready LED meaning
ITNLT 2000 [cha: y, def:2000, sav:2000] I_T nexus loss time (ms)
IRT 0 [cha: y, def: 0, sav: 0] Initiator response timeout (ms)
Phy control and discover (SAS) [pcd] mode page:
NOP 2 [cha: n, def: 2, sav: 2] Number of phys
SASA 0x5000c50007d2a2e9 [cha: n, def:0x5000c50007d2a2e9, sav:0x5000c5000
7d2a2e9] SAS address
ASASA 0x500605b002a6bd52 [cha: n, def:0x500605b002a6bd52, sav:0x500605b00
2a6bd52] Attached SAS address

Notice the RLM bit now shows up?

Now for a little explanation of what we are seeing here.

Let's take the original (truncated) output:

PPID 6 [cha: n, def: 6, sav: 6]

The first field,

PPID 6 [cha: n, def: 6, sav: 6]

Indicates the tag name "PPID" and current value "6".

The second filed,

PPID 6 [cha: n, def: 6, sav: 6]

tells us if we can change that value, which in this case is no.

The third field,

PPID 6 [cha: n, def: 6, sav: 6]

indicates the default value.

The fourth field,

PPID 6 [cha: n, def: 6, sav: 6]

indicates the saved value.

So in the case of my RLM bit,

RLM 0 [cha: y, def: 0, sav: 0]

if I issue the command:
"sdparm --set=RLM pd1", the output of the RLM bit will show:

RLM 1 [cha: y, def: 0, sav: 0]

Which indicates that the RLM bit is set to "1" but not yet saved sop a reboot of the system will turn the bit back to the sav value or "0" which means the led will be on.

In order to save it, the same command can be issued with and extra option: "sdparm --set=RLM --save pd1" and the output will now look like this:

RLM 1 [cha: y, def: 0, sav: 1]

Confused yet?

Next up I have to verify that the mode pages from the OEM drives match the original Seagate drives...

[enjoy]

hi Airbozo,
well done ! the leds working good now on my poweredge 2950 with ST3300655SS.
did you find a way to make it certificated ? my MD3000i doesn't accept it.
Or have you find the way to edit the firmware from 0004 to S527 ?
thanks

[Airbozo]

hi Airbozo,
well done ! the leds working good now on my poweredge 2950 with ST3300655SS.
did you find a way to make it certificated ? my MD3000i doesn't accept it.
Or have you find the way to edit the firmware from 0004 to S527 ?
thanks

You have to have the firmware to flash onto the drive to change it. That value is dependent on the version of firmware loaded on the drive. You can copy the firmware from one drive to another or just save it into a file. I don't have the time right now, but I will look into the procedure for doing this.

[enjoy]

I have done a lot of research and it doesn't look so easy !

[Airbozo]

I have done a lot of research and it doesn't look so easy !

I've done it a couple of times, but I never documented it and now I would have to actually get into it to recall what I did. Most of the drive manufacturers don't want to publish the procedure, but it isn't too hard if I remember right. I'll see if I can dig up some of my notes to get you started.

[enjoy]

it will be amazing if you could do it ! thanks