Root Volume Dangerously Low on Space

So I wanted to walk through a customer's data migration plan in my home lab to brush up on some SnapMirror and FlexClone commands in clustered ONTAP. To do this I would need a functional cluster. Which to my knowledge, I had a several ONTAP simulators running clustered ONTAP, and a couple functional clusters. It's been a week or more since I logged into anything in the home lab and have been neglecting/neglected the usual administrative duties that go along with managing storage and virtual machines. So needless to say, before I could do what I really wanted to do, I needed  to clean up my mess, so to speak.

My single-node cluster was unavailable via System Manager, so I opened the node's console from VSphere and was welcomed with the following:

Notice the console message "The root volume (/mroot) on miamicl-01 is dangerously low on space; less than 10 MB remaining. To make space available, delete old snapshot copies, delete unneeded files, or expand the root volume capacity. After space is made available, reboot the controller. If needed contact technical support for assistance." And also the message where the controller calls home with the following: "Call home for ROOT VOLUME NOT WORKING PROPERLY: RECOVERY REQUIRED."

The first error message seems pretty self-explanatory. The second message sounds scary, but in this case its a quick fix. Free up some space in the root volume on miamicl-01 and we will be back in business in the "Miami office" which is critical. Luckily I have run into this before and even had people ask how to manage space in a node's root volume as it is not accessible from System Manager. As far as I know the only way to manage a nodes root volume is via the nodeshell CLI interface.

So let's login and free up some space, shall we? As soon as I login I get the same message as above about low space on root volume. From the prompt (which doesn't look like the normal cluster shell) I enter node shell via the following to drop to the nodeshell:

miamicl-01::> system node run local

Type 'exit' or 'Ctrl+D' to return to the CLI

Now I need to determine the name of the root volume so I use "vol status". In the output in the "options" column I find vol0 has the root attribute. 

I then use "snap list vol0" to identify all snapshots in volume vol0. And I use "snap delete vol0 "  to delete all snapshots, replacing "" with "nightly.x" or "hourly.x" as necessary until I have deleted all snapshots. Then I reboot the node, and it comes up clean, I login, and am returned to the normal and familiar clustershell.

Great! My single node cluster is back online and now I can begin preparations of creating SnapMirror destinations and other tasks I need to familiarize myself with. But, let's take it a step further so next week when I login to test/verify some action plans or commands we don't have to recover the root volume. Let's configure auto-deletion of snapshots and a target free space to prevent this moving forward. I can now log the output using Putty/SSH instead of the VSphere console. And it looks like this:

login as: admin

Using keyboard-interactive authentication.

Password:

miamicl::>

miamicl::> node run local

Type 'exit' or 'Ctrl-D' to return to the CLI

miamicl-01> snap autodelete vol0 on

snap autodelete: snap autodelete enabled

miamicl-01> snap autodelete vol0 target_free_space 35

snap autodelete: snap autodelete configuration options set

miamicl-01> snap autodelete vol0

snapshot autodelete settings for vol0:

state                           : on

commitment                      : try

trigger                         : volume

target_free_space               : 35%

delete_order                    : oldest_first

defer_delete                    : user_created

prefix                          : (not specified)

destroy_list                    : none

miamicl-01>

This should prevent the root volume from having less than 35% free space. Now I can create SnapMirror relationships from my 7-mode vsim to my cluster. That should be fun so grab a beer and stick around, though I'm running short of time and probably won't get around to documenting it until next week.