This article shares my experience in implementing a iSCSI based SAN storage in RHEL 3 environment. This article doesn't warrant that your SAN solution will work perfectly if you follow the instructions in this article. These are merely my personal experiences, my own way of implementing the storage and may not be acknowledged by DELL or EMC. You must have their appointed SAN Experts to design and deploy your storage.

I recently had an opportunity to play around with a DELL|EMC AX150i on a HP Compaq Server running RHEL 3 Update 8. I've done the following in sequence to get the storage up and running in the Linux Environment.  

The first thing to do in any SAN environment is to look for the compatibility matrix and for DELL|EMC if you have the PowerLink access you can check the AX-Series ESM (Enterprise Support Matrix) prior to your deployment. I knew that my RHEL box running Taroon (2.4.21-47.0.1.ELsmp) is in the support matrix. I'm not going to discuss in detail about the Kernel update and the editing the Grub boot loader to boot to the newly installed kernel as I presume that the reader who's going to deploy the SAN would be well versed in Kernel updates and all.

Once the Kernel update is completed, the next logical step is to detect the new SAN devices in the Operating System. We may either re-compile the kernel with the needed drivers or best leave it modularized. We can edit the modules.conf to add our new parameters

1. Edit the modules.conf file as below

vi /etc/modules.conf

2. In the last line of the modules.conf file add the below options

options scsi_mod max_scsi_luns=256 scsi_allow_ghost_devices=1

3. In order to get these changes into effect, we have to rebuild the ramdisk. Go to the /boot directory and type the following commands

cd /boot
mkinitrd -v -f initrd-2.4.21-47.0.1.ELsmp.img 2.4.21-47.0.1.ELsmp

This command will rebuild the ramdisk and on next boot the changes to the modules.conf will be effective

The next step is to get the iSCSI initiator working. For that we need to check the ethernet config files. In my case, I wanted to isolate my iSCSI traffic from my production LAN. I had my eth0 configured already for my production LAN. I had a second NIC which is identified as eth1 dedicated for iSCSI.

1. Goto the ethernet configuration directory

cd /etc/sysconfig/network-scripts

2. Edit the configuration file

vi ifcfg-eth1

3. Set the IP address and other parameters for the NIC

DEVICE=eth1
BOOTPROTO=none
TYPE=Ethernet
IPADDR=10.10.10.10
NETMASK=255.255.255.0
ONBOOT=yes

You can by all means set these parameters through the "ifconfig" command without the need to edit the config files using vi

4. Start the NIC

ifup eth1

or

service network restart

[Note: This will restart your other production NIC eth0 as well]

Since our network is ready for iSCSI communication, we have to install the iSCSI initiator tools to complete the task

The latest iscsi-initiator-utils package available from RHN for RHEL3 would be iscsi-initiator-utils-3.6.3-4.i386.rpm

1. Install the latest iSCSI Initiator

rpm -ivh iscsi-initiator-utils-3.6.3-4.i386.rpm

2. Start the iSCSI service

service iscsi start

3. Now we have to ensure that the iSCSI service is started on every reboot automatically

chkconfig --level 345 iscsi on

We still have not connected the iSCSI cable to the target yet and before connecting it, we have to ensure that the path management software is in place and is running. It can either be the EMC proprietary PowerPath or the native DM-MPIO available on Linux.

In this case, we're going to use the EMC provided software PowerPath.

1. Mount the EMC PowerPath CD

mount cdrom
cd /mnt/cdrom

2. Install PowerPath

rpm -ivh EMCpower.LINUX-4.5.3-004.rpm

3. Start PowerPath

service PowerPath start

The next step is to configure the iSCSI initiator parameters.

Note: You can configure your iSCSI target with the following options and your iSCSI initiator configuration will vary according to your target settings
- No CHAP Authentication
- CHAP Authentication
- Mutual CHAP Authentication

In my case, I didn't have any CHAP authentication configured on the AX150i storage and hence the below configuration on iSCSI initiator applies only for "No CHAP Authentication". Note that the iSCSI initiator on RHEL 3 is provided by sfNET and the parameters look mostly the same to the open iSCSI initiators.

1. Edit the iscsi.conf (Most of the settings are already there and you just have to uncomment the needed setting)

vi /etc/iscsi.conf

#Discovery Address Category
#IP address of storage system’s iSCSI port (SPA & SPB)
DiscoveryAddress=10.10.10.1:3260
DiscoveryAddress=10.10.10.2:3260

#DIGEST SETTINGS
HeaderDigest=never
DataDigest=never

#PORTAL FAILOVER SETTINGS
PortalFailover=no

#MULTIPATH SETTINGS
#when EMC PowerPath or DM-MPIO is installed
Multipath=yes

#LUN SETTINGS
LUNs=0-255

#SESSION TIMEOUT SETTINGS  
DiskCommandTimeout=10

#CONTINUOUS DISCOVERY SETTINGS
Continuous=no

#OPERATIONAL PARAMETER SETTINGS
InitialR2T=yes
ImmediateData=yes.

Now all our settings on the host are completed. We have to initialize the storage array to get connected to the SAN LUNS.

There are two ways to initialize the storage

- Install the Navisphere Storage System Initialization utility on your notebook and connect through the console cable to the storage.
- Install the Navisphere Storage System Initialization utility on atleast one of the server connected to the storage.

This wizard has numerous help functions and can be invoked any time during the initialization. You can set Set network parameters for the AX-Series storage system, such as the IP addresses for the storage system SPs, the subnet mask, and the default gateway address. You can also assign a username and password to create a user account for accessing the storage system.

NOTE: If you forget your username and or password, the only way to reset is to uninitialize the storage array. The uninitialization steps can be found in the storage documentation that you've received with your storage or call the relevant technical support for procedure.

Installing Navi Initialization tool

1. Get the AX-Series Setup CD in the CDROM drive

mount cdrom
cd /mnt/cdrom

2. Install the Initialization tool

rpm - ivh naviinittool-6.20.**********.i386.rpm

3. Change to the install directory and start the initialization tool

cd /opt/Navisphere/bin/
./naviinittoolcli

4. Accept the Licence agreement when prompted

5. The next step you will see the initialization tool has automatically discovered the available storage arrays.
Please choose a storage system by "Item Number" or "0" to discover again.
Please type "e" to exit the application.

Type 1 in this case

6. Follow the on screen instructions to change the
- Array Name
- SPA IP Address
- SPB IP Address
- Subnet Mask
- Gateway
- User Name
- Password

7. Finally accept the changes by typing "a"

Note: It takes a while to initialize the storage and please be patient during this time.

There's a Navisphere Server Utility tool provided in the CD which can be installed to the server or run from the CD itself.
The utility allows you to
- Change the host name, IP address of the server
- Update file system information
- Add / Remove / Replace a HBA on the host
- Manage SnapShot

While the AX-Series CDROM is still mounted

1. Install the Navi Server Util as well

rpm -ivh axnaviserverutil.6.20.*********.i386.rpm

2. Change to the installed directory and start the server utility

cd /opt/Navisphere/bin/
./axnaviserverutilcli

3. You will be given a list of options to choose

type "1" to choose Update Server Information

4. Follow the on screen instructions and select "u" to update the server information.

Now we have completed all the configurations and are yet to create the LUNS. Login to the Storage and create the Virtual disks and assign it to the preferred host.

While we create the LUNS in the storage

1. In the host stop the following

service PowerPath Stop
service iscsi stop

2. Then start the following

service iscsi start
service PowerPath start

3. Check for available devices

powermt display dev=all

You should obviously see the pseudo name assigned as emcpowera, the logical device information, state, owner etc

4. Enable this configuration

powermt config

5. We have to partition and format this LUN using fdisk

fdisk /dev/emcpowera

6. Create a new partition by typing "n"

7. Make it a primary partition by typing "p"

8. Save the partition table by typing "w"

9. Format the partition with ext3 file system

mke2fs /dev/emcpowera1

or

/sbin/mkfs -t ext3 /dev/emcpowera1

10. Create a mount point

mkdir /mnt/SANLUN01

11. Mount the partition

mount -t ext3 -o _netdev /dev/emcpowera1 /mnt/SANLUN01

12. Edit the /etc/fstab to make the mount point persistent

/dev/emcpowera1 /mnt/SANLUN01 ext3 _netdev 0 0
 
13. Save this configuration in PowerPath

powermt save

14. Change to the Navi Installation directory

cd /opt/Navisphere/bin directory
./axnaviserverutilcli

15. You will be given a list of options to choose

type "1" to choose Update Server Information

16. Follow the on screen instructions and select "u" to update the server information.


Issues / Challenges

1.    var/log/messages started getting filled with Messages like "Connection to Discovery Address ***** Failed" and then "Connected to Discovery Address *****" every second.
2.    /etc/fstab entries are not loaded during boot up and the SAN LUN’s are not mounted automatically.



For issue 1

1. Edit the iscsi.conf (Most of the settings are already there and you just have to uncomment the needed setting)

vi /etc/iscsi.conf

#Discovery Address Category
#IP address of storage system’s iSCSI port (SPA & SPB)
DiscoveryAddress=10.10.10.1:3260
DiscoveryAddress=10.10.10.2:3260

#DIGEST SETTINGS
HeaderDigest=never
DataDigest=never

#PORTAL FAILOVER SETTINGS
PortalFailover=no

#MULTIPATH SETTINGS
#when EMC PowerPath or DM-MPIO is installed
Multipath=yes

#LUN SETTINGS
LUNs=0-255

#SESSION TIMEOUT SETTINGS  
DiskCommandTimeout=10

#CONTINUOUS DISCOVERY SETTINGS
Continuous=no

#OPERATIONAL PARAMETER SETTINGS
InitialR2T=yes
ImmediateData=yes.

Do note that this variable Continuous=no in the last few lines of the iscsi.conf file should be commented out and instead should be moved above the DiscoveryAddress variable as below

Continuous=no
DiscoveryAddress=10.10.10.1:3260
DiscoveryAddress=10.10.10.2:3260

For issue 2

1. The /etc/fstab issue can be resolved by two ways. In most of the systems, the mount option _netdev should do the trick, as the OS notices the SAN volume as the net device and delays the fstab until the iSCSI drivers are loaded

/dev/emcpowera1    /mnt/sanLUN    ext3    _netdev    0 0

2. For some reason, some Red Hat servers load the drivers / modules in a wrong sequence during the kernel boot like -->1. fstab -->2. PowerPath -->3. iSCSI kernel driver whereas it should be loaded in reverse like this -->1. iSCSI Kernel driver -->2. PowerPath -->3. fstab. The iSCSI sessions are established after the PowerPath startup script is run. As a result, PowerPath is not able to see iSCSI devices until the "powermt config" is run manually after the iSCSI session has been established. Refer to EMC Primus article EMC110819 for more information on "emcp_delayconfig".

Look at /etc/rc.d/rc.local, it should contain the following lines(usually at the end of the file):

#EMCpower begin. DO NOT REMOVE THIS LINE
if [-f /sbin/powermt]; then
echo "PowerPath is sleeping for 120 second(s)...please wait."
sleep 120
/sbin/powermt config
fi
#EMCpower end. DO NOT REMOVE THIS LINE

3. If the above lines do not exist at the /etc/rc.d/rc.local file append the above script to the file.
4. You can now manually restart the services or do a clean reboot to see the SAN LUN is mounted successfully during the boot and there will be no more error messages on the /var/log/messages.

Useful Tips:

1. Always ensure that you strictly follow the DELL|EMC ESM (Enterprise Support Matrix)

2. Check your iSCSI initiator status

service iscsi status

or

iscsi-ls

3. Check your iSCSI initiator is always started while boot

chkconfig --list iscsi

and make the service to start automatically while booting on runlevels 3, 4 and 5

chkconfig --level 345 iscsi on

4. Ensure that the authentication setting matches between the iSCSI Target and the Initiator.

Please login or register to add comments