Provisioning Aggregates

In the previous posts we installed our ONTAP simulator and applied some basic configuration settings. We also connected to the cluster via System Manager. From within System Manager we can login to the cluster, expand "storage" in the left pane and take a look at our disks.

It looks like the vsim comes with 2 "shelves" of 14 disks each. You'll also notice the first 3 disks are already assigned to an aggregate, aggr0, the node miamicl-01's root aggregate. This leaves 11 spares to create another aggregate with. Below the 11 spares you'll notice another "shelf" of disks which have a state of "present". These disks have not been assigned to the node yet. So let's assign them to miamicl-01. This cannot be done from System Manager and must be done via command line. We're just going to assign all of the unowned disks to miamicl-01. But you can specify individual disks, all unowned disks, or all disks on a specific adapter. See the below example:
miamicl::> storage disk assign -disk miamicl-01:v4.* -owner miamicl-01

miamicl::> storage disk show
                     Usable           Container
Disk                   Size Shelf Bay Type        Position   Aggregate Owner
---------------- ---------- ----- --- ----------- ---------- --------- --------
miamicl-01:v4.16     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.17     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.18     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.19     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.20     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.21     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.22     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.24     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.25     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.26     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.27     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.28     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.29     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v4.32     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v5.16     1020MB     -   - aggregate   dparity    aggr0     miamicl-01
miamicl-01:v5.17     1020MB     -   - aggregate   parity     aggr0     miamicl-01
miamicl-01:v5.18     1020MB     -   - aggregate   data       aggr0     miamicl-01
miamicl-01:v5.19     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v5.20     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v5.21     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v5.22     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v5.24     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v5.25     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v5.26     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v5.27     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v5.28     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v5.29     1020MB     -   - spare       present    -         miamicl-01
miamicl-01:v5.32     1020MB     -   - spare       present    -         miamicl-01
28 entries were displayed.


Now we have assigned all the previously unowned disks to node miamicl-01. And if we refresh the disk page in System Manager we see all disks are assigned and have been made spares.

 We are ready to carve up an aggregate. Now how you want to do this really depends on what you're planning to do. Do you want one large aggregate or multiple smaller ones? For my lab it doesn't really matter, it gets tore down and rebuilt every few months anyway. But for this post I will make two, one via System Manager and one via the command line. 

From System Manager expand "Storage" and click on "Aggregates". Then click on "create" 

This will bring up the "Create Aggregate Wizard". 

Click "Next".

Name your aggregate (Its nice to use meaningful names instead of the auto-generated name)

Specify RAID Type, RAID-DP is default and what we'll use in this post.

Click "next"

Click the button "Select Disks"

The wizard automatically determines the minimum number of hot spares (in this case 1 disk) and leaves the remaining spares in a group (in this case 24). We select the group. If we had other spares of another type there would be a second group in this list:

In this post I'm creating two 12 disk aggregates, so I will drop the number of capacity disks to use from 24 down to twelve, leaving us another 12 spares to make a second aggregate with via CLI. 

Then I hit "save and close" 

And now I can change the size of my RAID groups. 

In this example I have a 12 disk aggregate , I could put them all in one RAID group, but I'm going to split this one into two RAID groups. I'll try to remember to illustrate why for you later. So I change my RAID group size to 6 and hit save and close, then hit create.

Now let's do it from the command line.

miamicl::*> storage aggregate create -aggregate aggr2 -diskcount 12

[Job 16] Job succeeded: DONE. Warning: Creation of aggregate "aggr2" has been initiated.  12 disks need to be zeroed before they can be added to the aggregate.  The process has been initiated.  Once zeroing completes on these disks, all disks will be added at once.  Note that if the system reboots before the disk zeroing is complete, the aggregate will not exist.

miamicl::*> storage aggregate show

Aggregate     Size Available Used% State   #Vols  Nodes            RAID Status

--------- -------- --------- ----- ------- ------ ---------------- ------------

aggr0        900MB   43.43MB   95% online       1 miamicl-01       raid_dp,

                                                                   normal

aggr1_2rg   7.03GB    7.03GB    0% online       0 miamicl-01       raid_dp,

                                                                   normal

aggr2           0B        0B    0% creating     0 miamicl-01       raid_dp,

                                                                   initializing

3 entries were displayed.

miamicl::*>

And storage aggregate show now displays aggr0, aggr1_2rg (created from SysMgr), and aggr2 which we just made in the CLI and is still initializing. Time to grab a beer while disks zero and aggregate creation completes. 

OK, aggrs are created, and I wanted to show you why I made an aggr with 2 RAID groups. DATA ONTAP uses RAID-DP so there is really 2 parity disks in each RAID group. This allows the RAID group to suffer double disk failures and remain online. Not like we're too concerned with data protection or losing disks to parity in this post, but I think it's worth demonstrating.

So above when we created aggr1_2rg using System Manager we made it out of two, six-disk RAID groups. I unowned my one spare and disabled disk autoassign to help demonstrate this. Check out how many disks have to be failed to bring the aggregate offline:

Aggregate aggr1_2rg (online, raid_dp, degraded) (block checksums)

  Plex /aggr1_2rg/plex0 (online, normal, active)

    RAID group /aggr1_2rg/plex0/rg0 (double degraded, block checksums)

      RAID Disk Device  HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------  ------------- ---- ---- ---- ----- --------------    --------------

      dparity   v4.22   v4    ?   ?   FC:B   -  FCAL 15000 1020/2089984      1027/2104448

      parity    FAILED          N/A                        1020/ -

      data      v4.24   v4    ?   ?   FC:B   -  FCAL 15000 1020/2089984      1027/2104448

      data      v4.25   v4    ?   ?   FC:B   -  FCAL 15000 1020/2089984      1027/2104448

      data      v4.29   v4    ?   ?   FC:B   -  FCAL 15000 1020/2089984      1027/2104448

      data      FAILED          N/A                        1020/ -

    RAID group /aggr1_2rg/plex0/rg1 (double degraded, block checksums)

      RAID Disk Device  HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------  ------------- ---- ---- ---- ----- --------------    --------------

      dparity   FAILED          N/A                        1020/ -

      parity    v5.29   v5    ?   ?   FC:B   -  FCAL 15000 1020/2089984      1027/2104448

      data      FAILED          N/A                        1020/ -

      data      v5.32   v5    ?   ?   FC:B   -  FCAL 15000 1020/2089984      1027/2104448

      data      v5.27   v5    ?   ?   FC:B   -  FCAL 15000 1020/2089984      1027/2104448

      data      v4.32   v4    ?   ?   FC:B   -  FCAL 15000 1020/2089984      1027/2104448

So after failing out 4 disks in aggr1_2rg, the aggregate is still online. Aggr2 which only has one RAID group would only be able to survive half that many disk failures. Out of my 12 1GB disks aggr1_2rg, with it's two RAID groups, has 7GB of usable space. While aggr2 with its 12 1GB disks lumped into one RAID group provides an additional 1.7GB of usable space (8.7GB total). There is a pretty good document on the NetApp Support Site that does a way better job of explaining RAID-DP and RAID groups than I ever could. Check it out here.

Upcoming articles will go over provisioning and testing a vserver, or storage virtual machine, for CIFS.

Add Cluster to System Manager

In my last post we walked through installing the ONTAP simulator on ESX. If you're following along, you should have a single node running 8.2.1 Cluster Mode. This post we will go through adding our cluster to System Manager so that we can perform some management tasks through the GUI. While I prefer the command line, especially in 7-mode, there are some tasks in Cluster Mode I find System Manager almost indispensable for.

This post assumes the following:
1. You have a running cluster that you can access via SSH to the cluster management LIF.
2. You have installed OnCommand System Manager 3.1 on the system you're managing the storage cluster from.

Step 1. We will need to enable SNMP on the cluster to allow System Manager to discover the cluster. Beginning in Ontap 8.2 SNMP versions 1 and 2c are disabled by default.
You need to SSH to the cluster management LIF (could also use console) and use the command line to accomplish this. Once logged into the cluster we will use the "system snmp community" command to create a new SNMP community named "public" as illustrated below.

miamicl::> system snmp community add -community-name public -type ro
miamicl::> system snmp community show

miamicl

        ro  public

Step 2. Now that we have created the "public" SNMP community in the cluster we can discover and add it to System Manager. Open System Manager and near the top left corner there is an icon labeled "Discover". Click "Discover".

In the "Discover Storage Systems" dialog box, enter your cluster management IP address, and click discover. It will take a few seconds to discover the cluster, but it should show up. Highlight it and click "Add Selected Systems"

Step 3. Let's login and see what we have to play around with. Your System Manager window should now have your cluster showing. Highlight it and click "Login"

Login as admin and use the admin password you used during cluster setup, and click "Sign In" And you should see something like this:

Now as you can see there are no aggregates, volumes, vservers (SVMs) or any way for the cluster to serve any data. In the coming posts we will work on that and begin to explore just what we can do with some of the features of Data ONTAP.

Installing Data ONTAP Simulator on ESXi 5.1.0

One of the best ways to get your feet wet with Data ONTAP is to get your hands on the Simulate ONTAP 8 software which allows you to install a "vsim", or virtual machine that runs Data ONTAP 8. It provides a great way to experiment with and learn about many of the features of Data ONTAP. The simulator can be downloaded from the NetApp support site NetApp customers and select partners. This guide assumes you already have the following:

1. You have already downloaded the vsim_esx-cm.tgz package (I'll be using Clustered ONTAP) from the support site.
2. Installed and configured your ESX host and can access it via SSH as well as through the vSphere client.

The following steps will have your simulator up and running Clustered ONTAP pretty quick. Let me know if you have questions, if I left something out, or if this guide helped you.

Step 1. The first thing we need to do is copy the vsim package to the datastore we'll be using. I used the vSphere client to accomplish this, though there are other ways. Click on your host in the left pane of vSphere Client> Select the configuration tab> and select storage in the Hardware pane on the left.Your screeen should look like this:

Right click the datastore you want to house the simulator and click "browse data store". Then click the icon with a green up arrow on top of a hard drive.

Select "upload file" and browse to the location you downloaded the simulator to and select it to begin the upload.

Step 2. Once the file has uploaded we need to untar it so we can use it. 

SSH to your ESX host and cd to the datastore we uploaded the vsim.tgz file to. In my case it looked like this:

~ # cd vmfs/volumes/storage

/vmfs/volumes/53a81f39-7f7d23f5-1240-001a64ca2d40 #

From here you will run the following tar command:

# tar xvf vsim_esx-cm.tgz

It should look something like this at first:

/vmfs/volumes/53a81f39-7f7d23f5-1240-001a64ca2d40 # tar xvf vsim_esx-cm.tgz

vsim_esx-cm/

vsim_esx-cm/cfcard/

vsim_esx-cm/cfcard/env/

vsim_esx-cm/cfcard/env/env

vsim_esx-cm/nvram

vsim_esx-cm/DataONTAP.vmdk

vsim_esx-cm/DataONTAP-flat.vmdk

It will take sometime for the vmdk files to unpack. This is a good time to go grab a beer ;)

Step 3. Once complete you can refresh the datastore browser and see the unpacked folder. I usually rename it to something more meaningful, then drill down and rename the vmx file to something meaningful as well, but this is optional.

Now its showtime! Let's fire up our simulator and lay down some basic configuration of the cluster. 

Step 4. Within the "browse datastore" window drill down to the vsim's vmx file, right click it, and select "add to inventory"

Name the vsim and click "next"

Select a resource pool. and click "next" For me this is simple as I'm only using one host in my lab and therefore only have one resource pool. 

On the "ready to complete" screen click "finish"

Close the datastore browser window.

You should now see a new, powered off ONTAP simulator vm in your inventory in the left pane of vSphere.

Right-click the new vm, mouse-over "power" then click "power-on".

At this point if all goes well, your ONTAP simulator vm should power on. In which case you can skip this next section. My vm however, would not power on. I received the following error:

Lucky for me I have run into this before the last time I rebuilt my lab. ESXi 5.1 no longer loads the vmkernel multiextent module by default. To load it I ran the following command on the ESX host via SSH:

vmkload_mod multiextent

You should see something like this:

~ # vmkload_mod multiextent

Module multiextent loaded successfully

And now you can can power on the vm.

Step 5. To apply some basic configuration, right-click the vm and select "open console"

We want to interrupt the boot process by hitting "ctrl+c" when prompted for the boot menu.

At the boot menu, type "4" and enter.

You will likely see some console spam and within it the system will ask "Zero disks, reset config and install a new file system?"

Hit "y" and enter

It will ask you again, "Zero disks, reset config and install a new file system?:"

Hit "y" and enter

The simulator will reboot to wipe config/disks.

When it reboots youll see some more console spam and the first time I did this (long ago) I thought the simulator was panicking and dumping core. It's not, its just wiping the disks.

Let it do it's thing. When it boots, it will automatically fire off the cluster setup script. You'll want to type "create" and hit enter.

We're standing up a single node cluster in this post. I may do another one with multiple nodes later, but there's some limitations with the simulator when it comes to HA, so for now, we'll just use a single node.

Type "yes" and hit enter

There will be a delay after you hit enter. But just let it work.

You will then be asked to name your cluster. Type a name and hit enter

You'll then be asked to enter the base license key. Type the key in and hit enter.

You'll then have an opportunity to add additional license keys. Lets skip this for now and get SSH working so we don't have to use the console and can cut/paste the other license keys.

You'll be prompted to set and confirm the admin password. 

Now configure the management interface. This interface would be used to manage the entire cluster in a multi-node cluster. As soon as the management interface is configured, you can use SSH to connect to the cluster and complete configuration.

SSH to your management interface that you configured above. Log in as admin and the password you set earlier. If you managed to lose/forget the password, go back to step 5.

Once you're logged in via SSH, just re-run the cluster setup script and hitting enter to keep the same settings you already applied, until you get to adding licenses, DNS setup, etc. After completing the setup script, your cluster is ready to serve some data. I'll have another post up soon about actually doing some evaluation of the many features of Data ONTAP.