myrkat's nightmares come true!

[myrkat]

Well, I've been quiet the last couple of days (and next few) for good reasons. Well, "good" isn't what I'd call them. Reasons, yes; but "good" - no.

The place where I work isn't exactly equipped with cutting edge technology; in fact, it's more like the "hilt" (assuming that's the furthest spot from the bleeding edge of the knife of technology). OK: we have computers. That's about it.

So I'm pokin' along doing my work (Architecture) and lo and behold there's an AVG upgrade available (AVG 7.0 is out). So I install it on the server - here's where it gets good: our server is a 1996 HP LH Pro 200MHz PPro running SBS 4.5 (NT 4 based with Exchange 5.5 rolled in). Just a note: this is not AVG's fault... read on:

Anyway, install goes off without a hitch and I need to reboot (typical). No problem. "NET SEND * SERVER GOING DOWN IN 5 MIN!" message is out... people are ready... reboot!

That was 9:30am Wed. morning.

Server starts rebooting... gets to the VGA screen (just before the logon graphic thing) and BSOD! Automatically reboots... BSOD (same place)... again... again... "OK, this isn't going to cure itself..." I'm thinking:

STOP: c000021a (FATAL SYSTEM ERROR)
The Windows Logon System system failed... etc. etc.

Oh, joy! Just what I always wanted: a troubled NT system! For those of you too young to ever appreciate the joys of recovering an NT workstation, let alone an NT server, it is quite an adventure.

Here's where it gets REALLY good:[LIST=1]
There is no ERD (Emergency Recovery Disk).
There are no original CD's/Floppies (though I tracked them down later)
The drives are formatted NTFS...
...and MIRRORED...
...and COMPRESSED[/LIST=1]
Can things get any worse? Yes. Whole office pages me 6 minutes later and asks, "Is it ok for me to go back to work?" ... "Hello?" ... "You there?" All I can manage is a weak, "Uh. There might be a small problem..." as I'm about to faint (because I have a hunch where this is going)...

So a few hours later, server torn apart, spare workstation torn apart to create a spare server. Cats living with Dogs, everything is going crazy! And I have effectively gone: NOWHERE. Jeeze, I'm so screwed.

Then the first couple of people are sent home... Anyone do any tech support where it's so bad, you have to tell them, "I think now would be a good time to start sending people home..." - and you know the responses you typically get? Yeah, it's like that.

Right about now, the real "tech support" guys are cringing, I imagine.

So I'm trying to repair, recover, and finally reinstall... NOTHING works. . . and I mean NOTHING. No reinstall, nor install of anykind (reformat, etc). It's all spotty. About 12 attempts (no kidding) with varying degrees of success as far as getting beyond the initial install/reboot. Sometimes I don't even make it through the initial install, sometimes I do. No telling why (well, many guesses why, just no one tells me).

My best guess is that it's a funky controller (dual Adaptec 7800's on the mobo). No this server is a real piece of work. Hewlett-Packard LH Pro 200MHz Pentium Pro (dual capable, but only one cpu) with a whopping 192MB (I just *had* to add that 128MB chip a couple years back), and FOUR 9GB SCSI drives (mirrored). 18GB of storage. TWICE what we originally had (9GB). It's a HUGE-ASS BRICK, too. "Tower" is too skinny-sounding. This is like the fat lady at the circus, and she ain't singin' no more.

19 hours later; yeah, buddy, those are 19 straight hours of hell; I finally am ready to admit DEFEAT. Beaten by technology: my worst nightmare. I figure, "OK, no problem, I have all these backup tapes that we do EVERY DAY" - nope, gotta RE-INSTALL Windows, then re-install Veritas (the backup program), THEN restore. Well, I cannot re-install anything, so best I can do is set up an NT 4.0 workstation (all the clients are on 2000, but this version of veritas, 7.0, only lists NT and I'm in no mood to be adventurous). So I set up the workstation and install the backup software... catalog the last few tapes... restore to a recently-purchased LINKSYS EFG80 (we had 200MB +/- left on our server for free space!) and dump EVERYTHING to the 80GB drive that comes in it. No problem (well, there were, but this post is horrible long already).

Needless to say, I got all the DATA back (CAD files, documents, accounting, etc) but NO EXCHANGE. Yes, the other half of my worst nightmare: restoring an EXCHANGE database. I decided to head home for 3 hours sleep and a shower.

I get back in to work this morning (Thursday) at 6am (jee, I just left at 2:30am), start re-mapping drives and printers for everyone as they come in... late last night I killed the MX records at our provider to start pulling our mail via POP3 (they don't hae imap) since we no longer had exchange up... ok, mail barely trickles in... supposed to take 24-48 hours, well, I believe them now.

Another 9 hours of fixing, band-aiding, patching, etc. and still no exchange restored. Ugh. The server is officially dead. $15,000 (back in '96 remember) is now worth $0. So I ordered a DELL 2600 Tower server today. I won't even compare specs. But hey, for $2500, it's a lot of server.

Now the HUGE question of: To Exchange or NOT Exchange, that is the question.

Do I stick with SBS2000 (w/ a $99 2-year assurance for 2003 SBS) or just go with Windows 2003 server? At this point, I don't know. Exchange is nice when it works, but is a REAL BITCH to maintain (I like low-maintenence things). So I'll decide this in the next few days while Dell builds the server and ships it.

My worst nightmare... at least it's now headed for the scrap heap.

-myrkat

PS: If you made it this far, go get a life! ;) :smash:

[Dragon57]

I can really identify with your adventure and pain, been there, done that, ugh.

I really considered changing professions after my adventure. Perhaps becoming a chef is the thing. Pots and pans do not require upgrades very often if you get good stuff in the beginning. Hmm, I may have to rethink things here. :)

[Shadowhwk]

I think your server needs a better send off. Why don't you load it with fireworks and poor gasoline all over it and then light it. I'm sure that would be a very satisfying thing to watch. Film it then post.

[Kaleva]

Yowzers, I can sympathize, that is definately a bad day at the office. I've been there too and know how un-fun it can be. Sounds like you are well on the road to recovery though...

Whether or not to Exchange is certainly a big question. I don't know if it will help any to point out the obvious, but staying with Exchange is certainly going to be your path of least resistance. The newest versions of Exchange are pretty darn easy to setup and will more or less 'just work' once you have them installed. Users will notice little to no difference and life will go on (until something else fails, and then all bets are off). But, when a similar thing happened to me, I used it as an excuse to smash the Exchange disks and go with a Linux based Sendmail/IMAP solution instead. There were some growing pains getting mail moved over and getting users used to the few noticable differences, but it was worth it in the long run.

Given that you have just been through hell, you may want to stick with the better known evil, but this may be a great opportunity to explore the alternatives...

Good luck! :cheers:

[JeffL]

STOP: c000021a (FATAL SYSTEM ERROR)


Oh dear god NO!!!!

I've NEVER been able to fix an error like that, only a format job has ever worked.

Sorry to hear what happened, that is a real horror story alrighty. Sounds like a case of overwork which made the old frankenstein stop in its tracks :(

Personally, I would switch to IMAP because it is a more open standard, and fairly well supported by most email clients. Not much experience with either though, so take that with a large grain of salt.

[MARQUISDARQUIS]

If you go the Fireworks and Gasoline route to retire your ex, you realize its critically important that it be done right. That means a raft on a good sized body of water and some bag pipes wailing (a recording of a good cat fight will do in a pinch). Its only fitting and respectful. After all, it did its job (I assume) for something like 8 years.

Goes to show, though, its important to constantly upgrade your systems and a back-up system really is a necessity. Unfortunately, we usually realize that truth just after the primary failed as in your and most other's cases.

If you are adequately paranoid, (paranoia is just nature's way of telling you something is about to go terribly wrong) you will have a good back up for your back up as well.

I am in architecture as well, used to do work for a company called Tandem Computers, they specialized in satisfying serious technical paranoia by providing multiple backup computer systems. I learned a lot from those guys, especially about handling some of the nasty chemicals used in making circuit boards. The concept applies to anything though.

Well, glad you survived your ordeal and as they say, if it doesn't kill you, it will make you stronger. Congratulations on your new found strength.

[cfukai]

Hehehehe... jessh, so many of the battles myrkat shared bring back not so fond memories for me. I've fought a lot of those too :) I worked in a CAD startup as a programmer but I also had to handle a lot of the IT side too. Learned way too much about NT, Exchange, Tape Backups, etc., than I ever cared too ;)

I only worked with Exchange 5.5. I didn't really like it all that much but it was better than Sendmail (in some ways). I hear Exchange 2000 is a lot easier to manage but since I have never used it I couldn't personally vouch for it.

Do you really need all of the other groupware apps (calendars, meetings, etc) you get with Exchange or are you strictly interested in email? For a project I'm working on now we're looking at setting up an email server for a small group of people. Here are some of the Exchange alternatives I've looked at:

http://www.altn.com
http://www.merakmailserver.com
http://netwinsite.com/surgemail/
http://www.lavasoftware.net
http://www.floosietek.com

Haven't made a decision yet, still browsing. Anyway, good luck with everything and be sure to get some sleep this weekend!!!

:D

[beakmyn]

My suggestion:

Buy two servers.
One for Exchange
One for the PDC/data storage/etc.
Life is much easier that way.

Get a backup solution. IE Tape drive, CDR whatever.


Backup up daily!

[br2]

wow, my sympathies. now i remember why i left IT...
at least you have everything backed up. i lke exchange but i never had to administer it except briefly for dev purposes.
"restore exchange" search in msdn returns several pages. let me know if you don't have msdn and i'll email you the notes.

Small Business Server 2000? i thot that was a version of office...
2003 is nice but i have 2000 advanced server in production. that's not advice, i'm just cautious about prod.

good luck.

[myrkat]

First of all, thanks for the sympathies guys, it helps.

Now...

I walk in FRIDAY morning, ready to add another drive to the Linksys EFG80 that is acting as peer-to-peer server for the 15 or so clients (i.e. not what it's designed to do). But it has an easy add-a-second-drive option that provides backup.

Or so I thought.

Following Linksys's instructions, I powered it down, pulled the second tray, added the drive, put it back in, and powered up. Whew. Yeah, it was about that quick and easy.

Then I couldn't connect to the IP, but I could browse to the files on the first (default) HD. Hrmm...

Power down, wait, reboot. The EFG is a RedHat mini computer with two ATA drives (comes with one 80GB and an empty tray for another).

Beeep-Beeep... wait.... Beeep-Beeep... oh, joy. Can't get to ANYTHING now.

One hour with Linksys tech support (different countries each time?) and it turns out the NIC inside the EFG80 is D.O.A.

Nice. Just perfect timing.

So I peeled myself off the ceiling and sat down for a short break. Oh, this was all before 8am - now the rest of the employees are coming in.

Now here's the kicker. The temp server-setup thing I did, I scheduled a tape backup (we have a Sony SDT-10000 DDS-4 drive) to run the night before...

Well, I came in that morning and see the tape on the top of the computer: I forgot to put it in before I left. Blame it on lack of sleep, but what a bone-headed move (or lack-of move).

NO backup. Not too bad, there is only the one day's worth of work lost (which was minimal because of all the tweaking and running around I was doing on everyone's client-side computer).

So not a HUGE thing, but coming on the heels of the SBS 4.5 fiasco, it sure didn't help any.

Turns out it is cheaper to stick with SBS 2000, as we have the UPGRADE path (about $740) vs. $1400 if we don't include Exchange (like SBS does).

Funny MS pricing. That and the $99 2-year assurance is also... reassuring! :smash:

So tomorrow I'll go back in for a few hours (my son's 4th b-day party is at 11am, so I have to go in early then head home). Hopefully, by Monday, I'll have the DATA up on *something* and when the new server and OS get here, I'll go down the infamous Exchange 5.5 to Exchange 2000 Migration - according to MS, it's simple. Ask the rest of the world, and it's a bit different.

Again, thanks for all the sympathy and encouragements. I would love to switch to a *nix, but I don't want to be IT, I want to be "Architect" (which is what I am, remember, computers are a HOBBY).

I'm affraid I'll need to outsource our tech support, as it's taking away from my projects.

-myrkat

[mattew]

OK, I have to ask the question. I know that computers are a hobby.... but why would you do your own IT? Do you get extra pay? How come you do not have a back up IT guy on call?

I see this all of the time. I do management/software consulting for lawyers and other time critical business (accountant, engineers etc.) and there always is a person or two that likes computers and wants to do IT. In my line of work, I support several software pacakges (time,billing and accounting and practice management --kind of like Microsoft project, but you can manage many projects at once) and there is no way that a part-timer could solve advanced issues. The basics and the day to day.... but once into some heavy trouble... they do not have a chance, no matter how goof their intentions.

You have to start interviewing someone/companies so that you can out source your advanced IT problems. Something blows up..... call in someone. If they can't fix it, they get in trouble and not you. I can say that when thinking of all my clients(100+) I make the most money off the ones that either want to do it cheaply at the beginning, or have a part timer administering and trouble-shooting. It takes me twice as long as I have to undo what they did wrong and then re-do it from scratch.

You have to at least have a company or person that you can call when the sh*t hits the fan. If you interview in advance...you will be more comfortable making that phone call and if they have had any input in set up ... much better. Two heads are better than one.

Glad you survived!!!!


Matt

[myrkat]

Matt,

You of course have excellent points, and I am not surprised you see this often. I can easily handle anything short of the server blowing up (a direct quote to my boss some time ago). Well, the server blew up. (onboard raid controller went bad intermittently).

Anyway, I am making a few calls to establish some outsourcing. Usually, it was difficult to find anyone who could service us, as we were on NT4-based SBS4.5/Exchange 5.5.

And this is the first "disaster" we've had in 15 years of owning computers, so I'm not sure it's an easy sell on the pay upfront bit (when you consider the TCO). However, considering my sanity, it's getting outsourced and then some. I don't mind walking people through how-to's or OJT, but I really do not want to give the time to go through and troubleshoot a server (though I've been doing it for some years).

Software is easy. Our accounting package and CAD software are all under annual licensing/support agreements, and Office apps are office apps (easy), so the software end doesn't need ANY out-sourced tech support. Hardware is another thing, though.

One thing I am going to suggest we impliment is a 36-month refresh with annual reviews. Both HW and SW (OS, mainly). Establish a more sound IT budget to accomodate the upgrades and go from there. It will make whatever figures I get in from outsourced tech support that much smaller; as the older the equipment and software is, the higher the fee (if you can find someone competant enough to service it).

Thanks for the pointers, and you are right-on. And I'm sure those companies you see this happen in will agree with you, but usually someone is tying their hands (the boss, accounting, etc).

-myrkat

[mattew]

It is always about the money. I don't like it when you have a good person trying as hard as they can to keep the computers and the system together. Something goes wrong and they get hung, when they were nights and week ends keeping things going and working beyond the scope of their job.

I am 100% mission critical database applications. When I get those, "it blew up" calls it is my neck on the line... which sucks, but is kind of fun.

I think that you have to stay involved in it all, but you need that safety blanket. After all you have been through... good excuse to have a few extra beers.

Good Luck keeping it all together,

Matt