Unix Administration Horror Stories!!! - Technology, Humor, and Computer (pdf document) free file download at fliiby.com

"From: ani0349@cs.rit.edu (Anatoly N Ivasyuk) Date: 1 Mar 93 04:54:06 GMT Newsgroups: comp.unix.admin Subject: Unix Administration Horror Stories!!! (part 1 of 4) ============================================================================ The Unofficial Unix Administration Horror Story Summary, version 1.1 ---------------------------------------------------------------------------compiled by: Anatoly Ivasyuk (anatoly@nick.csh.rit.edu) ============================================================================ Introduction -----------This is version 1.1 of "The Unofficial Unix Administration Horror Story Summary". I put this together for two reasons: 1) Some of these stories are damn amusing. 2) Many people can learn many things about what *not* to do when they're in charge of a system. As rickf@pmafire.inel.gov (Rick Furniss) puts it: "More systems have been wiped out by admins than any hacker could do in a lifetime." This is not an FAQ, but more like the questions that *should* have been asked (and answered). There are success stories, and... well... other stories. I'm certain that everyone can learn something from reading these stories. The organization of the Summary has been changed quite a bit (maybe I should bump the version number up to 2.0?). Instead of leaving the stories in more or less chronological order of the postings, they have been separated into sections. There are currently sections for all different types of stories, and a brief table of contents to go along with it. Any new stories that I have received since version 1.0 of the Summary have been integrated with the rest of the stories, but usually appear at the end of their respective sections. The new stories are marked by '*NEW*'. The miscellaneous section is a little large, but I had no idea where to stick those stories. If anyone cares to suggest a place, or comes up with a new section, let me know. Submitting stories -----------------If there are additional stories that anyone wants to submit, I'll be glad to add them to the Summary . Send them to me at: anatoly@nick.csh.rit.edu. About posting the stories ------------------------This is probably the last time that the stories will be posted to USENET in their entirety. However, I do plan to make updates more frequent as more stories roll in. Further announcements of updates to the horror stories will be posted to the comp.unix.admin newsgroup, but the horror stories will themselves be available through an ftp site or ftpmail. Initially, the stories will be available for ftp at sunsite.unc.edu. Thanks to jem@sunsite.unc.edu for letting me put them there. They will probably appear within the next few days. They will probably be in /pub/docs/humor or /pub/archives/comp.unix.admin. More ftp sites may follow. How to get the stories through ftpmail: --------------------------------------There are a few sites that provide ftp services by mail for those people who don't have ftp access. To find out more, mail one of the following locations with a subject header 'help': ftpmail@decwrl.dec.com ftpmail@sunsite.unc.edu As always, send more stories! -Anatoly Ivasyuk ----------------------------------------------------------------------------The posting that started it all: -------------------------------aras@multix.no (Arne Asplem) wrote: > > > > I'm the program chair for a one day conference on Unix system administration in Oslo in 3 weeks, including topics like network management, system admininistration tools, integration, print/file-servers, securitym, etc. > I'm looking for actual horror stories of what have gone wrong because > of bad system administration, as an early morning wakeup. > I'll summarise to the net if there is any interest. > -- Arne ----------------------------------------------------------------------------Table of Contents: -----------------Section 1) 2) 3) 4) 5) 6) 7) 8) 9) 10) 11) 12) -| Anatoly Ivasyuk @ Rochester Institute of Technology | |-----------------------------------------------------| | anatoly@nick.csh.rit.edu | ani0349@cs.rit.edu | | Computer Science House | Computer Science Dept. | From: ani0349@cs.rit.edu (Anatoly N Ivasyuk) Date: 1 Mar 93 04:55:21 GMT Newsgroups: comp.unix.admin Subject: Unix Administration Horror Stories!!! (part 2 of 4) ============================================================================= Section 1: Creative uses of rm(1)... ============================================================================= ~From: dbrillha@dave.mis.semi.harris.com (Dave Brillhart) Organization: Harris Semiconductor We can laugh (almost) about it now, but... Our operations group, a VMS group but trying to learn UNIX, was assigned account administration. They were cleaning up a few non-used accounts like they do on VMS - backup and purge. When they came across the account "sccs", which had never been accessed, away it went. The "deleteuser" utility fom DEC asks if you would like to delete all the files in the account. Seems reasonable, huh? ----------------------------------------------------------------------------~From: broadley@neurocog.lrdc.pitt.edu (Bill Broadley) Organization: University of Pittsburgh On a old decstation 3100 I was deleting last semesters users to try to dig up some disk space, I also deleted some test users at the same time. One user took longer then usual, so I hit control-c and tried ls. Creative uses of rm(1) How not to free up space on your drive Dealing with /dev files Making backups Blaming it on the hardware Partitioning the drives Configuring the system Upgrading the system All about file permissions Machine dependencies Miscellaneous stories (a.k.a. 'oops') What we have learned "ls: command not found" Turns out that the test user had / as the home directory and the remove user script in ultrix just happily blew away the whole disk. ftp, telnet, rcp, rsh, etc were all gone. one LONG rebuild of X11R5. Had to go to tapes, and had Fortunately it wasn't our primary system, and I'm only a student.... ----------------------------------------------------------------------------~From: cjc@ulysses.att.com (Chris Calabrese) Organization: AT&T Bell Labs, Murray Hill, NJ, USA We have a home-grown admin system that controls accounts on all of our machines. It has a remove user operation that removes the user from all machines at the same time in the middle of the night. Well, one night, the thing goes off and tries to remove a user with the home directory '/'. All the machines went down, with varying ammounts of stuff missing (depending on how soon the script, rm, find, and other importing things were clobbered). Nobody knew what what was going on! The systems were restored from backup, and things seemed to be going OK, until the next night when the remove-user script was fired off by cron again. This time, Corporate Security was called in, and the admin group's supervisor was called back from his vacation (I think there's something in there about a helicopter picking the guy up from a rafting trip in the Grand Canyon). By chance, somebody checked the cron scripts, and all was well for the next night... ----------------------------------------------------------------------------~From: tzs@stein.u.washington.edu (Tim Smith) Organization: University of Washington, Seattle I was working on a line printer spooler, which lived in /etc. I wanted to remove it, and so issued the command "rm /etc/lpspl." There was only one problem. Out of habit, I typed "passwd" after "/etc/" and removed the password file. Oops. I called up the person who handled backups, and he restored the password file. A couple of days later, I did it again! he made a link, /etc/safe_from_tim. This time, after he restored it, About a week later, I overwrote /etc/passwd, rather than removing it. After he restored it again, he installed a daemon that kept a copy of /etc/passwd, on another file system, and automatically restored it if it appeared to have been damaged. Fortunately, I finished my work on /etc/lpspl around this time, so we didn't have to see if I could find a way to wipe out a couple of filesystems... ----------------------------------------------------------------------------~From: bill@chaos.cs.umn.edu ( bill pociengel ) Organization: University of Minnesota After a real bad crash (tm) and having been an admin (on an RS/6000) for less than a month (honest it wasn't my fault, yea right stupid) we got to test our backup by doing: # cd / # rm -rf * ohhhhhhhh sh*t i hope those tapes are good. Ya know it's kinda funny (in a perverse way) to watch the system just slowly go away. ----------------------------------------------------------------------------~From: barrie@calvin.demon.co.uk (Barrie Spence) Organization: DataCAD Ltd, Hamilton, Scotland My mistake on SunOS (with OpenWindows) was to try and clean up all the '.*' directories in /tmp. Obviously "rm -rf /tmp/*" missed these, so I was very careful and made sure I was in /tmp and then executed "rm -rf ./.*". I will never do this again. If I am in any doubt as to how a wildcard will expand I will echo it first. ----------------------------------------------------------------------------~From: robjohn@ocdis01.UUCP (Contractor Bob Johnson) Organization: Tinker Air Force Base, Oklahoma Cleaning out an old directory, I did 'rm *', then noticed several files that began with dot (.profile, etc) still there. So, in a fit of obtuse brilliance, I typed... rm -rf .* & By the time I got it stopped, it had chewed through 3 filesystems which all had to be restored from tape (.* expands to ../*, and the -r makes it keep walking up the directory tree). Live and learn... ----------------------------------------------------------------------------~From: JRowe@cen.ex.ac.uk (John Rowe) Organization: Computer Unit. - University of Exeter. UK rik@nella15.cc.monash.edu.au (Rik Harris) writes: [snippet about "using 'find' in an auto-cleanup script which blew away half of the source" deleted. -ed.] If you're doing this using find always put -xdev in: find /tmp/ -xdev -fstype 4.2 -type f -atime +5 -exec rm {} \; This stops find from working its way down filesystems mounted under /tmp/. If you're using, say, perl you have to stat . and .. and see if they are mounted on the same device. The fstype 4.2 is pure paranoia. Needless to say, I once forgot to do this. All was well for some weeks until Convex's version of NQS decided to temporarily mount /mnt under /tmp... Interestingly, only two people noticed. Yes, the chief op. keeps good backups! Other triumphs: I created a list of a user's files that hadn't been accessed for three months and a perl script for him to delete them. Of course, it had to be tested, I mislaid a quote from a print statement... This did turn into a triumph, he only wanted a small fraction of them back so we saved 20 MB. I once deleted the only line from within an if..then statement in rc.local, the sun refused to come up, and it was surprisingly difficult to come up single user with a writeable file system. AIX is a whole system of nightmares strung together. If you stray outside of the sort of setup IBM implicitly assume you have (all IBM kit, no non IBM hosts on the network, etc.) you're liable to end up in deep doodoo. One thing I would like all vendors to do (I know one or two do) is to give root the option of logging in using another shell. Am I the only one to have mangled a root shell? ----------------------------------------------------------------------------~From: rheiger@renext.open.ch (Richard H. E. Eiger) Organization: Olivetti (Schweiz) AG, Branch Office Berne Just imagine having the sendmail.cf file in /etc. Now, I was working on the sendmail stuff and had come up with lots of sendmail.cf.xxx which I wanted to get rid of so I typed "rm -f sendmail.cf. *". At first I was surprised about how much time it took to remove some 10 files or so. Hitting the interrupt key, when I finally saw what had happened was way to late, though. Fortune has it that I'm a very lazy person. That's why I never bothered to just back up directories with data that changes often. Therefore I managed to restore /etc successfully before rebooting... :-) Happy end, after all. Of course I had lost the only well working version of my sendmail.cf... ----------------------------------------------------------------------------~From: gfowler@javelin.sim.es.com (Gary Fowler) Organization: Evans & Sutherland Computer Corporation Once I was going to make a new file system using mkfs. The device I wanted to make it on was /dev/c0d1s8. The device name that I used, however, was /dev/c0d0s8 which held a very important application. I had always been a little annoyed by the 10 second wait that mkfs has before it actually makes the file system. I'm sure glad it waited that time though. I probably waited 9.9 seconds before I realized my mistake and hit that DEL key just in time. That was a near disaster avoided. [ I wish all systems were like that. Linux mkfs doesn't wait, but at ] [ least I have the source! -ed. ] Another time I wasn't so lucky. I was a very new SA, and I was trying to clean some junk out of a system. I was in /usr/bin when I noticed a sub directory that didn't belong there. A former SA had put it there. I did an ls on it and determined that it could be zapped. Forgetting that I was still in /usr/bin, I did an rm *. No 10 second idiot proofing with rm. Now if some one would only create an OS with a "Do what I mean, not what I say" feature. Gary "Experience is what allows you to recognize a mistake the second time you make it." Fowler ----------------------------------------------------------------------------~From: russells@ccu1.aukuni.ac.nz (Russell Street) Organization: University of Auckland, New Zealand. I once had "gnu-emacs" aliased to 'em' (and 'emacs' etc) One day I wanted to edit the start up file and mistyped # rm /etc/rc.local instead of the obvious. *Fortunately* I had just finished a backup and was now finding out the joys of tar and it's love of path names. [./etc/rc.local and /etc/rc.local and etc/rc.local) are *not* the same for tar and TK-50s take a *long* time search for non-existant files :(] Of course the BREAK (Ctrl-P) key on a VAX and an Ultrix manual and a certain /etc/ttys line are just a horror story waiting to happen! Especially when the VAX and manuals are in a unsupervised place :) ----------------------------------------------------------------------------~From: rik@nella15.cc.monash.edu.au (Rik Harris) Organization: Monash University, Melb., Australia. Most of our disks reside on a single, high-powered server. We decided this probably wasn't too good an idea, and put a new disk on one of the workstations (particularly since the w/s has a faster transfer rate than the server does!). It's still really useful to be able to use all disks from the one machine, so I mounted the w/s disk on the server. I said to myself (being a Friday afternoon...see previous post) "it's only temporary.../mnt is already being used...I'll mount it in /tmp". So, I mounted on /tmp/a (or something). This was fine for a few hours, but then the auto-cleanup script kicked in, and blew away half of my source (the stuff over 2 weeks old). I didn't notice this for a few days, though. After I figured out what had happened, and restored the files (we _do_ have a good backup strategy), everything was OK. Until a few months later. We were trying to convince a sysadmin from another site that he shouldn't NFS export his disks rw,root to everyone, so I mounted the disk to put a few suid root programs in his home directory to convince him. Well, it's only a temporary mount, so.... You guessed it, another Friday afternoon. I did a umount /tmp/b, and forgot about it. I noticed this one about halfway through the next day. (NFS over a couple of 64k links is pretty slow). The disk had not unmounted because it was busy...busy with two find scripts, happily checking for suid programs, and deleting anything over a week old. A df on the filesystem later showed about 12% full :-( Sorry Craig. Now, I create /mnt1, /mnt2, /mnt3.... :-) Remember....Friday afternoons are BAD news. ----------------------------------------------------------------------------~From: ranck@joesbar.cc.vt.edu (Wm. L. Ranck) Well, after reading some of the stories in this thread I guess I can tell mine. I got an RS/6000 mod. 220 for my office about 6 months ago. The OS was preloaded so I had little chance to learn that process. Being used to a full-screen editor I was not happy with vi so I read in the manual that INED (IBM's editor for AIX) was full-screen and I logged in as root and installed it. I immediately started to play with the new editor and somehow found a series of keys that told the editor to delete the current directory. To this day I don't know what that sequence of keys was, but I was unfortunately in the /etc directory when I found it, and I got a prompt that said "do you want to remove this?" and I thought i was just removing the file I had been playing with but instead I removed /etc! I got the chance to learn how to install AIX from scratch. I did reinstall INED even though I was a little gun-shy but I made sure that whenever I used it from then on I was *not* root. I have since decided that EMACS may be a better choice. ----------------------------------------------------------------------------~From: root@rulcvx.LeidenUniv.nl (root) Organization: CRI, institute for telecommunication and computerservices. Well, waddya know... Some half hour ago, coming back from root (I was installing m4 on our system) [Shit, all my neato emacs tricks won't work. Damn, damn, damn kill, kill, KILL] to my own userid, I got this little message: "Can't find home directory /mnt0/crissl." and an other: "Can't lstat .". [Grrrrr, ^S and ^Q haven't been remapped...] Guess what happened, not an hour ago... A collegue of mine was emptying some directories of computer-course accounts. As I did a "ps -t" on his tty, what did I see? "rm -rf .*" Well, I'm not alone, he got sixteen other homedirectories as well. And guess what filesystems we don't make incremental backups of... And why not? Beats me... I haven't killed him yet, he first has to restore the lot. And for those "touch \-i" fans out there: you wouldn't have been protected... ----------------------------------------------------------------------------~From: jcm@coombs.anu.edu.au (J. McPherson) Organization: Australian National University A few months ago in comp.sys.hp, someone posted about their repairs to an HP 7x0, after a new sysadmin had just started work. They {the new person} had been looking throught the file system to try to make some space, saw /dev and the mainly 0 length files therein. Next command was "rm -f /dev/*" and they wondered why they couldn't login ;) I think the result was that the new person was sent on a sysamin's course a.s.a.p ----------------------------------------------------------------------------~From: msb@sq.sq.com (Mark Brader) Organization: SoftQuad Inc., Toronto, Canada > ... if you're trying rm -rf / you'll NEVER get a clear disk - at least > /bin/rm (and if it reached /bin/rmdir before scanning some directories > then add a lot of empty directories). I've seen it once... Then it must be version-dependent. On this Sun, "cp /bin/rm foo" followed by "./foo foo" does not leave a foo behind, and strings shows that rm appears not to call rmdir (which makes sense, as it can just use unlink()). In any case, I'm reminded of the following article. This is a classic which, like the story of Mel, has been on the net several times; it was in this newsgroup in January. It was first posted in 1986. ----Have you ever left your terminal logged in, only to find when you came back to it that a (supposed) friend had typed "rm -rf ~/*" and was hovering over the keyboard with threats along the lines of "lend me a fiver 'til Thursday, or I hit return"? Undoubtedly the person in question would not have had the nerve to inflic..."

You need to upgrade your Flash Player , or try to enable javascript in order see this document properly.