hwork.org

The Tapped Out Network Share

About a year ago we had an issue with svn access to our RoboCup project. We weren’t able to commit any files. Poking around, I saw that our nfs share, microwave, was completely out of space. 0.0 GB flashed in the Finder when I brought it up.

So I thought that was odd, as I remembered seeing plenty of space available on the share. So I chimed my boss, who contacted IT. I got an email pretty quickly — as we have decent weight on campus for IT responsiveness, probably for being fellow nerds — saying that they had installed 200 extra gigabytes of space. Free space popped up on the share and we were able to resume our normal coding habits, so I continued working on robots.

The next day, svn stopped working again. I checked, and space had run out. That’s pretty odd, I thought. So we told IT, and a few minutes later, they called to say that they had partitioned 350 more gigabytes to our share. Surely, they said, that would be enough for our apparently growing robocup needs. At this point, we knew something funny was going on.

Investigation

Talking to some of the professors, we decided to investigate. Prof. Toma and I reported to stations in one of the labs. She took the command-line route, being more of a hacker than I at that time, and I tried to find an OSX application to help us out. Neither of us had initial luck.

Eventually, by going into the Finder I was somehow able to scan the network drive and sort the folders by size. The process was very suspenseful. As Finder was slowly sorting user folders by size, Prof. Toma and I mused at what we’d find.

The overwhelming conclusion we made — or maybe hoped for — was that we’d find some massive, illicit den of warez from some random student (or better, professor!) and it would be totally scandalous and awesome.

There was a point where my own user folder topped the list, much to my embarrassment. But my account was only ~5 gigs of space, and all that it was filled with was robocup movies, so I was in the clear.

My user folder continued unabated atop the list while the remaining folders dwindled to just a few left. But then the whopper hit.

Discovery

One folder topped out at 732 GBs. And worse: it was growing. And even worse: it belonged to a Professor.

Soon, we found the culprit and, sadly, it wasn’t star wars dvd images. It was an out-of-control, growing Matlab file. That was 732 Gigabytes. Presumably, it was caused by some crash or freeze in the program while saving that just continued to save and append data over and over again. We’ll never know.

So after contacting the professor, we rm -rf’ed the file (for we had the sudo power). We watched as space slowly refilled the drive. Much to our happiness, no replacement file appeared. We felt l33t.

Lessons Learned

When IT solves a problem by throwing more hard drive space your way, accept it gladly. We now have well over a terabyte of space that we’ll never use but it’s cool to have. They’ve never repartitioned it.

Writing software that can continually write a 732 gigabyte file to disk without crashing is pretty impressive.

Nothing scandalous ever happens at a liberal arts computer science department.