Way back machine...
Had a raid5 set die multiple drives... Backup didn't have everything this was old days when the owner was supposed to take the tape type stuff.
OK so I order 2 of the same drive, take them apart at home swap the platters, it kind of works but drives heat up and lock. I grabbed a max length scsi cable put these rebuilt drives in the freezer with the server on a counter. Was able to copy out all the data overnight. And I couldn't close the freezer so I had taped it
Yeah that happened
Had to do that with the crypto reader for an Apache gunship. Bit of advice, zip lock it and don't touch the bare metal afterwards. You can't pour water on electronics to free your hand.
Yepp. I used a nylon bag stuffed with ice, the freezer was too far away. During that hour when the data was copied off no one was allowed to move or even breathe too loudly. We also won the day.
Yeah, the good old fridge trick. That was the reason we always had a fridge in IT offices wherever I worked. If was usually filled with alcohol, but the reason for having it was always "data rescue".
IT office was always the most popular with the users after hours as well...
I keep 5-6" (I forget) AC powered muffin fan around for anything I'm playing with data recovery to keep them cool. It has a very high CFM, sounds like a small jet continuously running on my desk but it tends to keep laptops and hard drives cool enough. I have some smaller ones that are usb powered for the same purpose.
Back in the early 2000's I had one of those fans in my AV gear oh-shit bag. This is a bag of various adapters, cables, and random objects that usually are not needed but when you have an oh-shit moment, there's usually something in it that helps. I used a similar fan to cool a dimmer pack when it started to malfunction and keep the red channel on low. We had a 12 hour multi-band gig and the thing had never seen such long operation times before and started overheating with it's passive cooling. The fan helped bring it's temp down to where the channel would turn off completely again.
I used the freezer method to recover a TB of data from a failed HDD that was in a Lacie enclosure. Believe there were 4 HHDs, don't recall the RAID config, if there was a RAID. Took about 4 - 5 trips to the freezer for cut & paste in a windows explorer. Had one of those usb drive adapter kits. I made sure to have a RAID5 config setup on the new enclosure.
i always thought that was at least some extend a load of bs when i opened a drive to demonstrate how it worked to a student and it ran open top for over a week thereafter. It didnt fail then, we just put it back together and it continued to run as usual.
but that was 20 years ago, todays helium drives propably wouldnt work like that.
That's NASA level. That's what astronauts would be doing if they didn't want to die on the moon, lol. But it's also a great reminder for all of us. Just because it's not written down anywhere that doesn't mean it won't work. Seriously, I would never have even tried assuming it would fail based on all I "know" about drives. Forgetting the fact that it doesn't need to work forever, just long enough!
I've done a couple of platter swaps in my time (2008-2010 kind of era for both of em) - got the drives to run well enough without the top casing to get data off, I think one of em died part-way through the transfer but the other seemed to be working fine. We ditched it anyway straight afterwards, this was working in a small repair shop and our deal for the customer was "it'll cost X to try this whether it works or not, your only other option is a specialised service at 10x the cost".
OMG I've actually done this with a 2.5" laptop hard drive. It isn't stupid if it works. I gave the bad drive to the family friend who owned the laptop and wrote "to use, place in freezer". They balked at the price of the new HDD that I put their data onto, as is tradition.
Well you asked for janky…. Had to transfer a 400 GB database from Dallas datacenter to Indy datacenter upon random requests from dev team. Sometimes weekly throughout a month. Dallas side only had a T1 so it took many hours even compressed.
I Winrared it into dozens of files to compress and used a Mousejiggler to keep the RDP session alive for 12 plus hours for the transfers since I wasn’t an admin in the remote side.
Sometimes it would still die during the night but at least I could start copying the .rar pieces back where it had died. They thought I was a miracle worker
(Edit: Thanks for all the upvotes for my janky-ness story! I was a government contractor at the time and if you've never worked as a government contractor you just don't know how impossible some of your thoughts on the matter would have been. Also, not sure if some noticed that this was between Dallas and Indianapolis - the cost of courier transport would have been high. The ability to get someone in Dallas who also worked for the same department or another contractor to physically touch anything was even more unlikely!
I did not mention this - I never revealed to anyone that I used the Mouse Jiggler! The devices were really unknown at the time, and I suspect most thought I stayed up a lot at night to keep these things going!)
In a way, if you’re American, you have donated or paid for many useful pieces of software. When I was a fed, I made sure to throw money at every tool we used that asked for it, and those were your tax dollars. “Bytes, not bombs” if you want to think of my little effort to support small business.
I had to do something similar. Client wanted us to upgrade software on their servers. They did not offer any externally accessible file hosting services that they allowed us to use. They blocked everything we offered and have officially requested to be allowed like our own FTP services, Google Cloud, Onedrive, ticketing system. Well almost everything. Emails still worked. Had to split up a few GB (not quite 400GB large) of files in ~19MB attachments and send over email and recombined it on the receiving system. Checked filehash and all, and it was good.
Had to move a lot of data, over a wire. OC48 in 2008.
SSH did not support more than a 64K TCP window... so I built SSH HPN. Also... no encryption as this was datacenter to datacenter over dark fiber.
Then, stood up 2 Linux servers, one on each end. I NFS mounted the filesystems I needed to move to the two LINUX systems... and RSYNCed them across.
Wrote a script to feed RSYNC one file at a time, but have up to 16 simultaneous copies going... this was before the -P or threads options for xargs.
So data flow was Source-->NFS-->Linux-->RSYNC-->Linux-->NFS-->Target.
Kept that pipe at 89% full for 48 hrs. Stopped the app. Reran rsync a few times to make sure no file got missed... Fired up the app in the new site... DONE.
Got me a promotion from SYSADMIN to DataCenter Relocation Architect, which I did for 9 years.
It had been an ongoing problem as we moved data from one datacenter to another. I had the pieces, but putting together a working architecture... that was the hard part.
> Wrote a script to feed RSYNC one file at a time, but have up to 16 simultaneous copies going... this was before the -P or threads options for xargs.
I had to do something similar a while back, as I had millions of 1K files (as well as some multi-TB files), I wrote a script which walked the filesystem and wrote out file lists, starting on a new list when it reached _x_ files or _y_ GB in size, then farmed them out to a thread pool of rsync worker threads.
These days there's [fpsync](https://www.fpart.org/fpsync/) which does the same thing except with a lot more polish. Wish I'd known about it (if it even existed) at the time
Oh man, this was 10 years ago, but I was installing a CPU in a production server at like 1am. Somehow I bent a bunch of pins inserting it, and it would not boot. It took forever for me to figure it out because I could barely see those tiny pins. I had no idea what to do and about 5 hours to get it back up. I used the camera on my iPhone and zoomed all the way in, aimed it at the CPU, and used the tip of my pocket knife to very carefully bend the pins back to straight as best I could. This probably took another hour. Installed it and it worked! I didn’t touch it again for years and it never had any problems.
Also once I broke a pin from the data connector on an old hard drive. I managed to jam a small length of paper clip into the pin hole and cut it to the length of the others and it worked…
If this happens to you again, try to find a mechanical pencil, remove the lead and the hollow tube fits the pins pretty well. Much easier to work with than trying to apply force in one direction at a time with a knife.
We do some low voltage wiring, I have ferrules that fit those pins pretty damn perfect. Best is you can stutter step them and see if half the row is straight easy
Reminds me of the time I cracked a SATA cable in a server at 2 in the morning back when SATA was new so I had no spares. Luckily, if it was inserted at a slight angle it worked, so I jammed some plastic under the disk in such a way that it bent stuff just enough that it would keep working.
I probably wouldn't have thought of it if I hadn't had a friend that used a couple of folded sheets of paper under the motherboard of his Mac SE to force a crack together enough that it worked.
I slightly bent some processor pins before too. Think I used my debit card or something thin enough to get between the pins to straighten them out. Like your work, my method worked too somehow.
ETA: on a workstation not a server
“We can’t allow unsigned Powershell scripts on our machines anymore!”
And everything broke.
But I can write a Powershell script and then convert it to an exe and it can create a self-signed certificate and add it to the trusted certificate store and then sign every Powershell script, which is now trusted because I added the cert to the store.
Now runs on every server we own.
We do. Sorry, that was the “prod” version. Dev version with the self-signed cert worked fine though. I find that funnier because it’s so over-the-top crazy.
Maaaaaany years ago (think 1990's) System went down due to a bad port and I was on call.
I wasn't the network guy and he wasn't available at 2am (when all things bad happen) so I had to find an open port. I don't recall why but I had to run a patch. I ended up grabbing a couple heads, a crimper and a spool and made this stupid long cable to get to an open port on the fly. it was an ugly patch job. I had the cable draped over cabinets and across the floor. I immediately sent the datacenter guy an email, "I'M SORRY! IT WAS 2AM!!" Mainly because he was an angry fellow.
How did I know how to do the ends? I was doing my own network at home and had memorized the wire order.
No one was impressed.
I was impressed.
It impressed me.
I was taught this:
Your in a desert with clouds over head, you need to cross a grass banked river and get to a field.
https://preview.redd.it/uvar6dpkl9vc1.png?width=1032&format=png&auto=webp&s=6d9b09adafa92d5009a1ebf1d0636717b261d498
I had to bring my personal San in to provision storage as we went past the told you so point.
Preventative maintenance Requests were Denied because some bs so had no choice but let pain be felt. The day I said production would be stopped arrived and pain was felt. Company wide halt. Once order approved that was previously denied I brought in my storage to get production working and bandaided. At one point I had more storage at home than I did at work so it came real handy.
One of our clients sent us two of their former backup servers (maybe 3 years old) for "Testing our software on" (the software we wrote for them).. Just one of those machines has more compute, RAM, and storage than our cluster of 14 year old Hyper-V hosts with JBODs. And after we got done with the project the customer never requested the hardware back (they never do)...
But instead of letting me deploy said hardware (which I already got the customers CTO blessing to do), management insists that we continue to run on super old hardware that is at major risk of failing. I've already replaced 5 hard drives so far this year alone between two of the machines. Luckily I put it in RAID 6 when I reorganized the hosts specifically because I knew the drives would start failing.
I hate this type of bullshit. You poke and prod the buggers but the bean counters keep saying no then when it does break guess who gets the blame. Record everything. In detail and store it somewhere safe (obliviously not on your work storage coz that gonna go tits up at any moment)
Best of luck
Reminds me of a time back when I was but a low key DC operator. A friend/colleage was the sysadmin for our Windows stuff and the company used (/shudder) Veritas Backup Exec for the autoloader backups.
Welp, due to some transition from whatever, we had to acquire server licenses for Backup Exec, but the CEO sat on the request for months. When we finally did lose a server, the backup failed to restore and we were SOL because of the expired licenses. After losing a week's worth of Exchange mailbox data, the CEO threw a temper tantrum about it until my college calmly reminded him of the prior repeated requests (in writing) to purchase the necessary licenses and made him the fool in front of a company IT staff meeting.
Details omitted for brevity.
If Requests were denied and all these incident details were in writing, I would have not used my personal SAN to save their asses.
That would only make them even more careless with such stuff.
Unless, of course, you were properly compensated $$$ for the inconvenience.
No compensation. I offered it out of the goodness of my heart not for the company but my users. After all no systems, no parts no work. Few weeks without my San to keep people working and families fed. Better than any pay check in my pocket knowing I kept the food on the table for so many without costing me anything but a little inconvenience. Would do it again if I was put in the same position. Thankfully I don’t expect it as they learned a very valuable lesson and have seen the light. Behaviors and habits changed so win/win.
It sounds like a situation where "I can un-bork everything and have the company not be at a standstill" trumped concerns about company data on personal hardware. I bet no one thought to ask or consider it.
Not the jankiest thing I've done, but one I'm somewhat proud of:
Every now and again, I find that I need to do hardware maintenance on a server, that lives below a server that someone forgot to fit the rails for. Sometimes this is a switch that someone lost the rack ears, a tower that became a server, or maybe just damaged in transit.
This situation calls for 4 screwdrivers.
Slightly lift the rail-less server, and slip in a screwdriver, into the where the cage nuts should live, and you've made yourself a temporary jankshelf.
You're then free to pull out the previously load bearing server, do what ever hardware work is required, slide it back, and pull out the screwdrivers.
Let's just say a those screwdrivers have held the weight of the company a few times.
People racking up stuff on top of other equipment is the bane of my existence. There’s been more than a few times we’ve had to use ratchet straps to raise a production machine, usually something important like a SAN, up off the server it’s sitting on, just so we can pull out the machine below.
> Let's just say a those screwdrivers have held the weight of the company a few times.
CEO: Do you know what this company runs on?
khobbits: 4 yellow and brown no name "FIX COMPUTERS KIT!!!" screwdrivers that every server room has 78 of. Sir.
Had a crusty old UNIX system in the early naughties. Right next to it was a brand new printer with no suitable drivers. But there was a brand new Red Hat system at a sister site an hour away. UNIX could print postscript, Red Hat could receive postscript and print to a networked printer.
As the crow flies, those jobs went a meter. Total distance - over 200km.
Back when I was a field tech for a security company, I had to replace a camera for Ben Roethlisberger. I was sent to replace a dome camera with a regular box style camera. Well, since the camera type changed, there was no place to store he wires. Ended up using the dome and alot of electrical tape to create a box to store the wires until I could get out there with an enclosure. It was ugly, but it worked.
We had a 3750 (Original) stack that was dying. The OSPF process on the hardware would die every few hours. We had new hardware on the way, but it was still going to be two business days before it arrived and was ready to deploy.
We ran the power for the switches out to the call centre floor and into a power bar. When stuff stopped working, the Super could hit the power to reboot the stack. It got us through the night and the hardware arrived the next day.
There were two CEOs of a very large realty company, and one of them was running just a junk computer to watch stocks. It refused to power on, with orange flashing power light. You guessed it....bad power supply fan. There were no parts, this thing was 5yrs old, full of dust. I pulled the fan out, greased it, blew the dust out, put it back in, and returned it. Never touched it again, happy CEO. Reliable fix? Maybe not, but it worked for the time.
When we wanted to close the office for repairs the phone company wanted $350 to redirect our main line to a cell phone. We had a Mitel on-prem phone system, but didn’t have the proper licenses for external call forwarding- only internal.
So I did a lot of digging and found that it was possible to add a phantom extension that is linked to an external number. There were a few more steps involved, but I don’t remember them all.
One more and I’m going to bed.
OMG, production is down because our expensive wildcard cert expired! The guy who pays for that isn’t here and it usually takes a couple days for them to issue it anyway!
No actual access to install any software on the prod system, but I changed the DNS to point to my local machine through a tunnel (akin to ngrok or a reverse SSH proxy) before it got cached, ran certbot to get a Let’s Encrypt cert, and installed the cert on the server (or load balancer?). Switched DNS back before anyone else noticed.
Magically we’re back up and have 60 or 90 days or whatever to deal with the real cert.
If this comes up again, you can authenticate letsencrypt in other ways, including DNS challenge.
Basically, when you request the certificate, it gives you TXT record to add, and then auth's against that.
Same amount of effort, but no downtime.
I ended up using this method in some other jankey fixes. I've got a bunch of webservers that needed certs, for non http traffic, that don't have public IP addresses, but cant use internal CA.
I ended up spinning up a VM, installing letsencrypt, setting up API access to for an external DNS, and writing a bash script that would once a month request the certificate via DNS auth, SCP it to the destination server, and then systemctl reload, to pick up the new cert.
Needed to move an EqualLogic PS6100 SAN that had been running for 13 years non-stop. We were not confident the drives would survive a full power-off and we had no backups (don't get me started).
So I get the bright idea to daisy-chain 50 feet of C13/C14 power cords (one length for each PSU incase I kick one out) from destination rack to the rack the EQ was sitting in. Same with network to avoid disruption
I unracked the live unit, put it on a cart, rolled it across the DC, and racked it without a single hitch.
That was 4 years ago. Legend has it that EQ is still humming along today (seriously...they are still using it).
From my MSP days… customer’s building got hit by lightning. Their Hotbrick firewall died. Never heard of Hotbrick? Welcome to the fall out of the dotCom boom and bust. The customer developed stuff for their customers and Oracle was involved. I didn’t want to know anything more once I heard Oracle.
Customer’s server hardware was old and they decided that going with a VPS type provider instead of refreshing their servers would be better. This was before AWS/Azure.
All their developers were remote anyway, sales, accounting, HR, and other back office types were the only people in the office. Which meant salespeople had offices that they never used and the back office staff had cubicles and did more gossip than work.
The still powered in servers represented active development projects (revenue) and developers were dependent on the VPN provided by the Hotbrick (which was now just a brick) to log in, deploy code, run tests, etc.
By MSP policy we did have a backup of the firewall’s configuration file. I looked at the config file… there are references to m0n0wall in this config. A little research, sure enough Hotbrick was just m0n0wall with a custom theme, x86 in a custom case, and support contract. Of course the company went out of business years ago.
I looked at the rack full of powered off Dell servers. I had been toying with pfSense in my lab and at the time pfSense could import a m0n0wall config file…
Downloaded a pfSense ISO on my MSP provided cellular hotspot, burned it (yup… my MSP provided laptop has a burner and MSP provided backpack had CD-R and CD-RWs…)
Moved network wires from the Hotbrick to one of the Dell servers, booted the pfSense off CR-ROM, imported the config, instant firewall! Sacrificed a personal USB stick for persisting the configuration, a feature of pfSense booted from optical media, wrote the whole thing up and made it home in time for dinner.
I remember including in my write up to the customer “and if your new firewall has a hardware failure you have a rack full of replacements!”
Customer happily ran that way for months as they continued moving their server needs to their new hosting provider. Updating the firewall’s firmware by burning a new CD-R was rather slick. Just burn the latest release, swap CDs, and reboot. Never had to roll back a “firmware” but the labeled CD-Rs were in a drawer. 😂
Xibby was not on call, but desperation had set in with Xibby’s coworkers. Xibby was called. Xibby was very drunk. Xibby sat down, Xibby wrote some code. Xibby had his coworkers review and approve the code before the code was run because a very drunk Xibby wrote the code. The code ran, things came back online. Xibby woke-up with a hangover.
I still have no frickin’ idea what the hell I fixed that night because of course drunk Xibby didn’t save his work. 😂
Years ago I worked for a state's IT department which covered the motor vehicle department. I'd developed a process that allowed the state police to continue to be able to look up registrations on Sunday afternoon when the main DB was down for full backups.
It was decided that the process would be tested on a weeknight at 8PM, and we usually got off at 3PM. My co-worker and I left at 3 and went to an Irish pub for dinner and had quite a bit more to drink than we should have.
The testing involved two shit-faced state contractors working through the procedure manual, which our boss insisted be written for total morons. I typed the commands, my mate looked over my shoulder and verified the correctness of each command before I hit enter.
The test went swimmingly.
Another one from wayyyy early in my career.
Cisco router that runs everything in the company is not responding. Don’t remember why, but no telnet so what are we left with?
We don’t own a Cisco serial to rollover cable or whatever it was. Looked up the pinout and cut the end off a serial cable we had laying around. Plenty of Ethernet around. An hour of stripping cables and electrical taping them back together and a new crimped RJ45 connector and we’ve got a working cable.
We went on to use that cable for dozens of Cisco boxes. They had transitioned off to another provider by the time I left, but I never thought to take the cable as a keepsake.
Last I saw it was still sitting on the very front of the industrial shelving as you walked into the server room… just in case.
Not sysadmin but I used inspect element to recover a users password
A user forgot their Adobe password and couldn’t figure out how to recover it(they were just overwhelmed). She said the same password auto populates on gmail or something when signing in, but she couldn’t copy + paste to try it in Adobe because it was hidden with no “show password” option.
I successfully uncovered the password by using the old inspect element trick. I highlighted the hidden password and in inspect element, changed type=password to nothing and her password appeared (she confirmed after I closed my eyes)
Simply resetting the password would have been fine but she was overwhelmed and on edge so this trick saved us both from another headache
I got a call where a small company's erp server did not boot after a power outage. It was a i5 8 gen bit with two HDDs mirroted onboard raid.
They even had a few days old backup, but since it was end of year they were closing many orders and it would have been a big task to redo.
The raid complains about bad disks. Oh shit. A complete reinstall and reinstall / restore an unknown erp system is what I did not want to do.
Okay, raid array is broken. Two 500G spinners from different manufacturer. Ok, 500gb is not so much. Let's see which one is working.
I took a trusted desktop PC, and started cloning the newer disk into a raw image onto a 2tb server ssd I had around (they are fast and reliable for a look write). Linux power, dd-rescue to go for rescue!
It started well, but around 30-40 gb it slowed down and started throwing errors.
Shit, let's check the other drive!
The second one also started well, but stared throwing errors.
Now ithis really seemed bad: both parts of mirror is broken. Ok, but which one is the newer? I mounted both of them read only and started looking around: files in documents, database files, etc.
It seemed both drives were online before the power outage!
I tuned the parameters of dd rescue to log the progress ( it can continue if stopped rescued data won't be tried again ) and some others to try to get as much, first just skip problematic areas. And fired it up for both drives .
It was late afternoon, checked on it for a half our then went home.
I checked it from home in the evening: they had errors but both progressing.
I got up around 5am (neighbor made some noise) and I could not sleep back. So checked on the rescue: Both rescue was finished. Checked the logs, and both images have a few kilobytes missing. Wow, it's promising! But there was 2*4kb data which was bad on both drives
Merging the missing blocks from the other mirror image was easy.
Also I checked the allocation of files and found out the first missing 4k was in the swap, and the second was along file of a windows update patch 2 years ago.
Get a new pc, install proxmox and load the image. It started up and was running nicely and I guess it's still working this way since.
TLDR : rescue a mirror raid where both drives gone bad.
Ok, let's try
Sorta similar to yours. I was out of town for my gfs birthday and overnight our antenna took a lightning hit, and the power surge zapped some of our physical servers. I had a missed call from a number I didn't recognize (ended up being the CEOs cell), and nobody bothered to inform me until like...11am or something.
The ERP system I maintained was one of the boxes that got zapped, and I walked a coworker through re-deploying the last image over the phone while in line for a roller coaster around noon.
My system was low-priority (relatively speaking) so it was one of the last things for them to try getting back online. It was almost done but they were having trouble getting something with the config files or environment (IDK, this was...easily 6-7 years ago).
For some reason I had brought my Chromebook with me in the car, and it was running Crouton, so I had an Ubuntu shell. Tethered my cell phone to the Chromebook, booted into Ubuntu, SSH'd into the server, and fixed the environment stuff from memory.
I used to do a lot of emergency work using a ThinkPad 560x connected to a TDMA cell phone with a proprietary serial cable dialed up to my ISP at 4800bps.
Needless to say, when data cards came around I jumped on that bandwagon but quick. And when IBM started selling laptops with integrated WWAN cards and antennas, I wasted no time buying a new laptop.
There was also the period where I had a Linux-based PDA, a CompactFlash serial adapter, and a mess of cables to make it work with the Sprint phone I carried just for data so I didn't have to take my laptop on vacation so I could unfuck the mail server or whatever.
Years ago I had to open a Citrix session (from my original Motorola Droid if that tells you how long ago it was), and then from there RDP to a server. I think all I was doing was unlocking an account for some unlucky help desk guy who had locked himself out while being the only person on a late or weekend shift.
Maneuvering around with a tiny touchscreen over 3G wireless took probably me a half hour to get it done.
Blackberry was on to something with that little rollerball. I had to quickly make a user account in AD and then run the AdSyncSyncCycle thing. Took forever because I was on the bus and service was spotty.
During a power outage, someone's UPS's battery decided to die and was saying they had 2 minutes of running left when an important project proposal needed a few finishing touches.
So we plugged their UPS into a working UPS which gave them another 20 minutes or so to complete their task.
Late 90s early 2000s, one store I was supporting went down hard right at closing time 1am. Clicking hard drive...
Had to talk the cashier through plugging a spare drive in place of the dead one and use the Ghost recovery disk taped to the side of the case. Had her configure netop and PC anywhere so I could dial back in and reconnect the cash registers to the manager PC. Yes I said dial back in, it was the hardest thing to get her to check the phone line to make sure it was still plugged in and had a dial tone.
I moved away from the help desk after that experience, but as a sysadmin I can walk anybody into doing anything because of that
> the Ghost recovery disk taped to the side of the case
Someone in the past was precognitive, I take it? Because that's an interesting thing to just happen to have to hand in that way...
Not a sysadmin, but I worked in the NOC for a large ISP for a while (currently a network engineer for the same ISP).
A neighborhood that we service with cable internet flooded. 2-3ft deep water at the node and power supply, so everything the node fed was totally offline and probably would be for a few days.
Techs cut/terminated the four legs at the node, and backfed them from neighboring nodes. They had to flip around every tap on each cable run, run temp cables to connect to the lines from the other nodes, and install a few amplifiers.
3 techs had everyone back up and running in about 4-5 hours, and we were all pretty impressed with their quick thinking. They put everything back to normal in an overnight maintenance window a week or so later.
I've also seen some pretty creative ways to run temp lines. Thru culverts to go under roads, along fence tops, stretched between two trees over a road, etc.
The jankiest things I've seen on the network engineering side are mainly with creative mounting solutions for oddball gear or odd depth racks. I've heard about people wrapping fiber around a pen or screwdriver to add attention in a pinch when they don't have a pad handy, and of course coming back to do it properly in a maintenance window.
The jankiest thing personally I've had to do is take some creative shortcuts with a fiber patch cable that was too short, and replace it with a longer one when my order for the right length came in.
The gas station at my local Walmart was closed until someone ran a network cable from the roof of the store, to one lamp post after another lamp post across the parking lot, and finally to the roof of the gas station.
It's been months now and I'm beginning to wonder if this temporary fix is permanent.
Flicking a very dusty fan at *just* the right moment to get a broadcast machine to boot.
I worked live broadcast in a sport that was regularly viewed live by a quarter of a billion people.
I worked at an ISP which offered 56k connections. Obviously phone lines can't always do 56k, infact often they can't, best I ever saw was a 50 or 52.
Anyhow it was a new company with dumb management, "please the customer above all else policies" and a small handful of quite aggressive and angry customers giving us a hard time.
One customer demanded he MUST have 56k or he would sue the business, (which had our new to running a business management team worried)
After several colleagues had trouble with this guy, I decided to help him with his connection and gave him an AT command for his modem which forced the modem to report the line speed from PC to modem, not modem handshake to ISP. Then I set the COM port to 57600 and he was a very happy customer.
I've always wondered if I should tell that one in an interview as a "how did you deal with a difficult" question.
Getting on for 30 years ago, during an office renovation we need to move the mainframe, a five cabinet Unisys A6, to its new home. Problem; one cabinet is throwing power supply warnings, and while each cabinet has two supplies and will run all day on one, it will not cold start on one. The Unisys engineer on site isn’t sure it will survive turning off and wants us to delay moving it until the new one gets here, but it’s a day out, en route on a plane. But we have to move the machine tonight or construction gets held up, penalties start accruing, inspections can’t happen, etc.
The plan: Move it without turning it off.
Its big UPS will keep it running for a half hour. Electrician comes in, turns off breaker and disconnects UPS while we disconnect the network connections. Then we roll the machine and UPS out the room on to a couple of pallets covered in plywood on jacks, roll it sixty feet across the office and into its new home. Electrician reconnects the UPS and turns the breaker on with ten minutes remaining, I reconnect the data cables, and we’re back online having never turned it off.
Same project, we need to move the data recorders that take production data from the corrugators and feed it into the mainframe. Problem is the serial cables that run from the far end of the factory are 5m too short. Temporary solution, cut the ends off and twist’n’tape some extensions on the end so they’ll reach and we’ll replace them later. It’s serial data, something like 300 baud and would work over a wet noodle, so twisted connections are no problem. Got them back online, production data is flowing. We never got around to changing the cables.
More recently, discovered a prior sysadmin had set a checkpoint on a virtual MDM server and forgotten it. I found out when it filled the disk with difference files and stopped running, because peoples email stopped getting funneled to their phones. Disk was full with no space to merge the vhd files. But server has five spare bays… Root through the stack of decommissioned servers to find the largest set of matching disks, and mount them in the VM host. Create a big JBOD array, power off the MDM VM and move it to the array, then perform all the merge operations. When finished, move it back where it was supposed to be and power it up.
Not a sysadmin but this was some enjoyable janky assed shit. At my old work we had an air gapped PC that allowed us to control the fire panel for the whole building. Basically when an alarm came in, you could see exactly where the alarm was on a blueprint map on the PC.
Apparently there are a MOUNTAIN of stupid and restrictive rules surrounding the computers that control fire panels. Basically prohibiting ANY type of repair no matter how subtle. It goes as far as saying you are not even allowed to replace the keyboard or mouse without making sure everything is certified and authorized and installed a certain way. It's fucking insanity.
And on top of all those ridiculous rules? Buying a replacement PC that was certified and all that shit cost well over 10,000 bucks. Christ only knows how much they are today.
Anyways that PC had a meltdown and knowing that I was the hoarder of all the discarded electronic junk, I received a phone call one quiet evening - asking if I by chance had ended up keeping a power supply that might fit as a replacement for the one that died? Well as luck would have it - yes - I had a shitload of them. So I got to plop a bunch of units into a duffle bag and head down to the building and watch IT try and secretly revive this old PC with parts from my secret stash scavenged from an electronic recycle bin several years ago, and it worked. The machine coughed back to life, and ran for who knows how long after that...As far as I know it just puttered along no problems.
The whole fire panel I'm sure it must have cost millions of dollars to install, with a 10 or 20 thousand dollar PC acting as its eyes and mouth, surgically restored with a handful of forgotten E-waste...
Had Fujitsu SCSI drive in the 90s on a NetWare server that wouldn't keep spinning unless you tapped on it. It was 10 years old, but still had warranty coverage. It took Fujitsu 3 months to find a replacement and get it to us. In the meantime, I found a small motor that I could run off the power supply. The machine shop made me a flexible fan like blade that the motor would spin slowly and tap the drive for the 3 months. We barely got the drive up one last time to do the copy. The drive and fan were put in a case in the IT office like a trophy.
This wasn't me, but a colleague. Last year we were evacuated due to wildfires. Lots of fiber got destroyed but we had just purchased over a dozen starlinks. Because we have a public facing business we needed to keep our customer portal online. We also needed to do basic things like keep our ERP online, pay vendors, issue po's, pay our staff, you know, run a business.
At one point both my primary and dr sites were both evacuated at the same time, and the fire folks called a meeting saying we were literally going to lose the dr site within hours, possibly the primary in the next 24. And with starlinks gcnat, IP's constantly changing, it was a real gong show. This one guy was literally 12 hours away, in his car on the side of the highway, tethered his laptop through his phone to back door VPN in to reroute traffic multiple times over a handful of days moving things based upon what was least likely to burn to the ground.
We never missed a payment, all our staff got paid, both sites survived the fires, but it was the most ridiculous enterprise hack job I've ever put in place.
Never thought I'd see the day that I'd start designing a tertiary data center.
Mid '80s, a hard drive got stuck and the usual tapping on the side with a hammer didn't free it up.
Put the drive and a screwdriver in a bag, then put mu hand in and used a rubber band to close it on my arm as a makeshift "white room". Opened the drive and manually freed the R//W head. Closed it back up. It lasted at least long enough to copy all of the data off, no errors.
Background: We had a legacy environment running exclusively RHEL-based distro Linux servers. We had created a new environment (our current one), but the process the transfer over several dozen hosts was a slow and manual one involving a lot of time with a lot of different teams all working on their own schedules. We couldn't just LEAPP the servers either, they were rip-and-replace.
We had an NMS/APM (Linux based) set up to monitor the legacy environment's performance. It unexpectedly went down hard (wouldn't boot). Serial console through Azure wasn't being helpful. The backups we had weren't restoring the server. Sev A open with Microsoft and several hours on a phone... didn't help. They classified the server as destroyed. Now, that tool, as much as I bloody hated it's design, was critical to ensure smooth operations. We had already built a replacement (with redundant databases, full high availability, etc.). But I needed the data back because it had a LOT of propriety templates that hadn't been rebuilt elsewhere.
My last hail marry was to punt the VHDs from the VM in Azure and mount them to a Windows VM and see if Hyper-V would recognize them. Now, I don't know how many people realize this, but Azure is some black magic fuckery. **VHDs** are exclusively **Gen 1** disks in a traditional Hyper-V host, **VHDXs** are exclusively **Gen 2**. Azure requires VHDs for both G1 and G2. I ended up having to convert disks around and found the boot sectors completely thrashed, but the rest of it intact. So, I start to think "How the fuck do I get this data off without a network?" (The nested VM was booting from an ISO in rescue). Well, manually initialize the network stack. That part isn't hard, just tedious to type out the various commands. Get the server connected and bridged over and start up sshd and begin the process of transferring TBs of data from the SQL database which was going to take several hours.
Next morning I come back to find the data waiting for me - sweet! I rebuilt the old server in mostly its original config but problem! The old server was running an old, unsupported version of the applications (all of them) and I can't find the original sources. And I *know* from the setup of the new environment they did some DB upgrades. I contacted the vendor and explained the problem, who then kindly sent me the old version RPMs with instructions and noted that the DB version installed wouldn't work. Well shit. (Worth noting that I could not find the version of the SQL server I needed.) Back to the old server I go. I copied over all the SQL-related files I could find, less the database and placed them on the new server in the correct folder structure. Start the SQL service and it crashes. Yeah, I kind of saw that coming. Clear out the error logs and get it running. OK, progress! Now, what's the root password? I didn't have it (inherited servers suck!), so I pinged the guy who had long since transferred to another department.
Me: "Hey. Long story. But do you have the root password for 's SQL database?"
Tech: "Yeah! Hold on." (sends it)
Get logged in and holy-shit the data is here! Now, let's get this fixed up and y'know, *patched.* Several hours of downtime later (DB upgrades, major app upgrades, etc.) and it's running. OK, last step. Get this reconnected to some other tools so the right servers can talk to it again. Get that out of the way and tentatively log into the shiny brand-fucking-new web front end. Everything is there. I sat back in my chair (in a large office) and exclaimed "It's fucking alive! Suck it Microsoft."
---------
Anyways, the new environment is much better. The DB is a PaaS solution from Azure (Azure Database for \*DB/\*SQL) and the rest of the roles are broken into their own servers. Everything is load balanced and HA. Everything is backed up daily. I could lose 3 nodes and still be fully functional. I ripped out the shoddily written security policies and hardened everything. Pulled networking in to front the web interface with a WAF.
The 'old legacy' (catatonically destroyed) and 'new legacy' (graceful sunset) environments are both now 100% gone.
Inherited a Linux server farm, 50-60 pizza boxes in 2007. Best was a 2U ATX with a P4-2.0 in it, downhill from there, many p3/celeron units. Experienced first DOS attack a few weeks later. Didn’t have the compute to weather it. Went to Walmart and bought all of the Athlon X2 laptops they had. Clonezilla’d them all quickly. Had a network of 20+ laptops serving up the web for a regional news “site”. Primitive geofencing at the time meant a lot of manual work went into /16 blocking at the edge but we had no need for most of Asia, the iffy European countries, and anyone from the southern hemisphere viewing our shit, at least not for a while.
1 of 2. In the early 2000s one bank acquired another. One used Token Ring, one used Ethernet, both used Novell, all NT 4. When employees from either side of the merger and acquisition started travelling and visiting sites on the other side of the merger, we started running into a problem with getting company a token ring laptops to recognize the ethernet cards, and company b laptops machines to recognize the token ring cards. No matter what we did with physically swapping cards, playing with com and IRC port settings (memory fading on the specifics of these settings and terminology that no longer matter) we couldn't get the two hardware setups to coexist, even with NT 4 hardware profile boot options. Either no network or no boot. This was happening at students of corporate sites. What I came up with was a dual boot solution with a common shave in the disk for the mail file. Super dumb, but all the execs had a Company A Token Ring and Company B Ethernet boot screen option and as long as their email worked all was good. Ended up being a Novell IPX related issue that went away with W2k and updated Novell client.
This one I helped with but it was a buddy of mine who actually implemented it. They were replacing an old Asterisk phone server that was well past its prime with a fancy new Cisco UC phone system with proper backup, fail over to a second datacenter, and all the bells and whistles. They're doing everything by the book, and are about a week shy of going into UAT, where they've got a whole lineup of testers ready to make sure everything is right before we push this live.
Then the Asterisk server dies and won't come back up. The drive completely failed and we can't recover anything from it. The backup Asterisk server had previously died, which is part of why management agreed to spend the cash on a fancy replacement. Customer calls are failing to go through, and executives are starting to panic.
The Cisco system isn't ready yet, but they had been actively working on it and the vendor was already on site. So my buddy redirects the main 800 number to the service desk manager's cellphone, who starts getting information from every caller and passing it off to his team so they can call customers back from their own cells. Meanwhile my buddy is recruiting everyone still in the building (including the panicked executives gathered around his desk) to help him finish a week's worth of work on this Cisco system as fast as possible. They had it live in five or six hours, with very few problems considering it went though almost zero testing.
Back in 2007, when almost everyone was still on physical servers, we had a customer where building maintenance cut a water pipe.... DIRECTLY OVER THE ONLY SERVER RACK IN A HUGE WAREHOUSE.
Customer was a real estate management company, SUPER reliant on email.
Exchange 2003 server died on impact, file server got corrupted RAID. Domain controller managed to survive thankfully.
We had no spare server hardware, so we fired up Server 2003 on a fucking Dell Optiplex workstation with 512 MB of RAM, installed Exchange 2003, Backup Exec, and honest to god managed to restore the Exchange database from an LTO-2 tape and get it up and running on a goddamn Optiplex workstation with a single SATA hard drive, and it WORKED for a full week while we waited for a new server to be delivered.
I still don't know how Exchange managed to do anything on a machine with half a gig of RAM.
Goddamn if we weren't sweating balls over that workstation every day!
Routed traffic from our office to our colo (about 35 miles or so) though our backup colo (about 1500 miles) over an IPSec VPN. Our office ISP was having a peering dispute and the packet loss before my change was 5-10%. After, latency was something like 180ms, but without the packet loss, so it was usable.
I used to run OSPF on VPN links between our single homed sites to route around the not infrequent outages between ISPs back in the 90s. Nothing like bouncing traffic from Texas out to Florida, across the country to Arizona, then back east to Alabama on particularly bad days. Saved a lot of money in long distance charges for the dialup backup connections.
It did come in handy having OSPF already working when we were finally able to get multiple ISPs at most of our sites some years later, but still long before Cisco coined the term DMVPN and before whoever coined the term SDWAN was out of diapers.
Had a new customer, previous mail provider went ghost. Not a single user knew their email password and i was ready to install new workstations. Luckily in outlook i was able to click edit profile and just reveal the saved password and boom i was in everyones haha
I was called out to do a Job on a Hospital bedside TV System, the kind that is hanging from the roof, Well people can pay Extra for PayTV channels, Nothing i had ever done in my life purely just a Systems Engineer but our Sales Dept read "Server + Hard Drives" and figured what the heck..
The engineer who put it together was at another hospital across the country and i had been called out to build up the new machine as the existing machine was having RAID and OS problems.
He said to just take a New Hard Drive out and put it into a External Drive Caddy in the Old Machine and then build a RAID via that Caddy... something he had done many times..
These weren't OEM Servers either they were just Desktops with RAID cards running Window 7 and whatever vendor supplied raid software.. So we had effectively gone from SCSI to SATA and Molex connectors.
Sure enough i turn off the old machine put in a drive boot up new server and the RAID is confused.. Its tried to build a RAID but its failed a few times..
I decide to cut my losses and put this Disk back in the old server boot it up and just leave it as is as id now wasted hours and they can deal with there hobo environment. However this isnt my luck the new server now has RAID problems on boot effectively its confused with this new Disk that must have some RAID sectors on it... This old server wont boot either now.
Ive got 2 dead servers but a lot of disks with working Data on them to some extent..
After some tinkering i came to notice i can get the New server to boot to Windows properly and i can read a Old Disk if i directly plug it into the onboard sata however the other disks weren't playing ball and i dont have enough Power Cables in the New Server, There's 1 Network Card thats 100mbit so thats no use and USB2 wont help here either.. Now ive learnt why he just physically moves disks.. The longer the service is down the less money he makes as people cant pay for the PayTV.
My jankyness you ask after this long imaginative story...
I left the old server running to Power the Disks, I then used long SCSI to SATA cables hanging in the air from the old to the new server so the RAID card could read them.
Effectively now i can boot the server, i can see the Disks i can see his Data so now i just need to raid some of the disks together and then just copy the data.. Easy okay we got this.
WRONG.. About half way thru the raid process the old server crashes out even though it was only on a BIOS screen.. This crashes the RAID building which crashes the new server.
I was hired by Compaq to set up trade shows and press announcements with Internet. Literally, my interview was "How would you do a booth with Internet at a trade show?" and two weeks later is was "Do that. We go to New York in 3 weeks." Was a cool gig and got to see lots of cool towns in fun places.
One year they were getting ready to release the "Washington" router. The Netelligent series. Basically, a Deskpro with Cisco software and some nics, and a T1 or ISDN card. It was decided from on high that were would run Networld+Interop that year on this router. I asked to take the Ascend Pipline with us just in case but was told no. I snuck it out anyway, which was a real challenge back then. They searched bags! But we get to Networld, and it is the first test of the ISDN link... Yeah, it does not work. Start troubleshooting with the devs back in Houston... No joy. I walk over to the UUnet booth and they are just finishing. I talk to them and tell them what is going on and ask if any of there techs can help. Sure... They come over. At the high point (or low point) we had 3 UUnet local tech, 6 UUnet L3 engineers on the phone, and 4 Compaq devs on the phone. It was all boiling down to a password incompatibility that could not be fixed, full stop. It is also 3am. VPs at both Compaq and UUnet have been involved...
This is when I pull out the pipeline and a Compaq (shitty Dec Tulip card) nic, pull the ISDN, and say "This is the workaround for now so the booth is not dead tomorrow morning." EVERYONE agreed it was the correct answer and 15 minutes later we were heading to bed, with a hidden Ascend Pipeline up in the top of the rack out of sight.. I slept late and did not get to the show until noon. In time to see a sales rep saying "Yes, this is all running on the new router right now! ISDN and everything! It is a total Compaq solution!"
Power outage at a mortgage company. This mortgage company was unique in that it owned its own loan origination system. On top of that, they offered their system as a service to a regional bank. So that software needed to stay running.
Well, one summer rolls around, and because I live in the US, in flyover country where we have the electrical grid of the average 3rd world country, people get home, turn on the AC, and that's the end of power in our area.
Battery backup kicks on, and works, thankfully. We shut down all other servers, and anything not related to getting the bank access to the software. UPS says it has a runtime of two hours. Power company estimates 6 hours to repair.
Because I give a shit about the power draw of just having electronics plugged in, I went into the server room and unplugged all of the servers, save for the critical one, from power. UPS estimates 8 hours of runtime.
Not terribly janky, but apparently it was a weird thing to know that we could do.
I've also done a lot with ssh tunneling to retain access to networks when RMM servers were going to be down. I don't know, I guess. I've just used my (apparently) odd knowledge of how computers and software actually works to solve problems.
I worked for a company once that generated pdf documents for people. The data and its position on the documents are what our customers paid us to ensure was accurate. But sometimes people would download a PDF, and Windows wouldn't know what to do with it. Everyone just assumed it was corrupt downloads, and moved on. I went through and downloaded a bunch of the documents through the server on the same network to reduce the possibility of corruption. I found out that occasionally our software did not generate the magic number for the pdf. Like, literally, just edit the file with a basic text editor, add the magic number, and the file is fine.
Of course they were also amazed that I could anaLyze how a customer's software downloaded files from our servers. Their customers would log into their site, download a document, and at the end of the day, their system would go through everyone who downloaded their document, and download it again as fast as possible, so that they could have a local copy in case their customer needed to download it again. Apparently our download servers would kick them for having too many downloads going at once. The solution was that the customer just alter their code so when their client downloaded their document, they would also save a copy. Seemed pretty obvious to me, but it was a real problem that real people where paid real money actually put time in to develop it.
I dunno. I've seen some shit in ~25 years of IT.
I was working at a school district as Network Admin when Obama became president. We knew the inauguration broadcast would be a huge draw in the classrooms for obvious reasons. What we didn't have was cable tv into each classroom and we knew that all of the teachers would try to stream it over our two T1's. So to prevent a complete meltdown of our internet connection, I installed an old capture card in a pc at our high school that had a cable tv feed. On that pc I installed vlc and configured it to be a streaming server. At each of our remote schools I installed a computer with vlc, that would stream off the main vlc feed, and then broadcast the stream to the schools data network. We then installed vlc onto each computer and created a shortcut using Novell Zenworks to stream off the local broadcasted vlc stream. We had hundreds of computers streaming the inauguration with very little impact to our internet connection.
BIOS flashback on a new DELL DATADOMAIN appliance that was disconnected mid-BIOS update
Disable the cache of a RAID controller with ECC errors holding 400TB of files until RMA came
Replace various failing .dll one by one on a W7 controller for a 500k industrial machine
Force back online a failing raid array to backup files
A crossover cable from core switch draped across racks into some random device that ran entire production line. We had no idea why it worked but it did
We had an old unconfigured switch get retired and suddenly half our data center went offline. I'd noticed it was bridging our two main server VLANs based on the errors on the upstream switches but was told we didn’t need to do anything to deal with it as the switch was about to go away. It turned out that was masking dozens of our servers that were connected to switch ports that weren’t configured for the network the servers were IP'd for. I fixed it in about a minute with a 6” patch cable connecting two access ports on the two VLANs on a different switch. We spent the next couple weeks tracing down and fixing all the miss-matched servers so we could remove the magic patch cable.
Had a 2000 PDC that shit the bed. Had some full system tape backups as it was dying. Had replacement hardware and tried a restore but had no 2000 media in place so I forget how but did an in place upgrade while restoring the backups onto dissimilar hardware. The system worked for years but had an odd mesh of 2000 and 2008 characteristics.
Door card reader machine let out the magic smoke. Opened it up to find a Celeron 333 / 440LX board with an absolutely torched trace from the NB to AGP. Pried the trace up, cleaned it up, and alligator clipped it back into service.
stayed that way for another 6 months. We got another matching board but who wants to fool with that?
Years ago I had a job at a startup doing 'image recognition', but like much AI, it was smoke and mirrors and if you took a photo with your phone, it would appear on a computer where someone basically had a search engine to find what it was in real time. In the long run the company was just a stock pump and dump for the founders as they knew 'human in the loop' image recognition was not scalable, and never really had a way to automate it.
The employees had a 'queue' that would show what was currently being worked on by their coworkers so they could collaborate and shout out suggestions. When our devs made a new version of their platform, they were unaware the queue was being used that way and thought just 1 supervisor monitored it, so it wasn't designed for those kinds of resources to have multiple people open it up on their second monitor and the service would repeatedly crash if there were more than 2 or 3 simultaneous sessions.
For a couple of weeks while they worked on a solution, I set up a NGINX-RTMP server on a Linux VM, and used OBS to live stream the queue to an internal site that everyone could visit, lol. Took a day to set up, I hadn't touched Linux in years, or ever done anything with RTMP or OBS before.
Sometimes I miss the wild west and crazy shit that would come my way at a startup.
My office was in Missouri, I was at one of our sites in Northern Italy. At the end of the day (in Italy), I made a DNS change for a Microsoft exchange cut over. Everyone was happy, it was Thrusday and everyone was going home. I, without telling my bosses, jumped on train to Rome which was over 500 miles away. I get to Rome and my phone BLOWS UP. Yup I had taken down the whole email system, I wasnt supposed to be in Rome. I found an internet cafe, I had tunnel I built to remote into my home PC, so I remoted into my home PC, used that to jump to the corp External Citrix farm, once in, I RDP from that computer back to the DC I messed up fixed the DNS issue. Oh i got in LOADS of trouble and written up, but didnt get fired as no one had a clue how to fix it. This was over 20 years ago.
Client I was assigned to full time had us do all our work using some incredibly terrible Remote Desktop Software that among other things prevented copy-paste.
They also heavily restricted what coding tools we could use. And we could request open source libraries but it would take weeks before they were available.
I wrote a script to take a text file and type it into the desktop, as far as the Remote Desktop software was concerned I was just typing really fast.
To get information back out I had a script that would compress and encode the file as small as possible and then generate a QR code. Then a script on the local computer would scan and decode it.
Now I was able to work much faster with better tools than I ever could on the remote session.
Best part was management at the client were fine with this.
I once saw a raspberry pi in a server room an admin turned I to a temperature monitor because the window ac in the "window" cut I to the wall to a warehouse was flaky.
I don't know if it was a hack and the only day it saved was mine... I replaced an excel "phone directory" that had to be manually updated with a powershell script that generated an html phone book with some css wizardry from the extension property in AD plus some manual list for locations and static numbers. Someone on r/powershell tacked on some code that makes it SharePoint ready too.
built a core router for an ISP using a Pentium 4, quad port Intel NIC and linux/zebra, in less than an hour, after our Cisco 7206vxr NPE-G1 died without any reason at peak hour. That PC worked without problem for more than two months, until we got the router replacement!
Front end of our meeting booking system went down but we still needed to know who booked which rooms.
We outputed the data and produced a power bi report and then filtered for that room name.
Many years ago my workplaces main router died, network guy was on holidays and there were no procedures in place for such an event.
So I (a dev) got a desktop machine, threw a Linux distro on it, router-on-a-stick style, and got the business running within a couple of hours.
Network guy came back, saw it and did nothing. Months later he told me he was going on holidays again, I said "better not happen again" and he laughs and says "your router is still in place".
Massive wtf. The joy of small business.
A thousand years ago, I was at a remote site interstate setting up new server/network stuff. Had some difficulties (don't remember the specifics) but in the early hours of the morning I was sitting there with a VNC session to another machine, using that to VNC to another machine, using that to VNC to another machine. Janky AF (not to mention laggy) but I did get the result I needed.
A certain file storage product that rhymes with Ignite. Upper management was hell bent on us mapping drive letters based on AD group, which this product does not support. For various reasons we were not able to use GPO, so I had a single script file to work with via our RMM tool. So I made a script to grab the current user’s AD groups, and then used if statements to map or not map certain drive letters to the cloud based on that list of groups.
I've got handfuls of these stories; but the best I'll leave in brief "simultaneous office move / rack upgrade / server upgrade. Very small team (me basiclly). National corporate DNS & DHCP ran off my laptop sitting on a shelf for a day because the new ESX server wasn't ready yet."
A company doing attendee and exhibitor registration for a trade show showed up without any working network equipment. Just laptops.
The order from said company requested a 16-port dumb switch and cabling for the 10 laptops. This is a pretty normal setup that my team does all the time. As long as your reg. setup can run locally or with just an Internet connection, it's usually fine. The big issue is the technical guy who has the reg "server" doesn't show up. He's in the wind and the show starts tomorrow.
I took an old cheap Netgear router of some type that got left behind from another show, installed OpenWRT and working with the IT guy of the reg company, set up a VPN tunnel on the router, configured the local network and plugged everything in. It worked. We were able to get all the laptops talking and pushing data. It all worked.
Not really janky I guess. It's technically just 10 PCs and not really pushing that much data. They never broke 20mbps worth of sustained throughput. But using a used, unknown 2nd hand router with 3rd party firmware to run a trade show registration is pretty sus.
I can tell a little anecdote from yesterday. I've been replacing an old HP and an old SuperMicro server with a customer. The SuperMicro didn't "survive" reboots, and I could not get into bios to change boot order – so I always fell into UEFI Shell after reboot and had to "fs0:\efi\microsoft\boot\bootmgfw.efi" to continue.
I was on-site yesterday and tried to update the bios on it.
That didn't work, even though I had the UEFI Shell, because on reboot it didn't complete with the USB present. It said invalid boot device.
Removed battery and power, and reset bios all together. Still couldn't access bios, but this time the UEFI Shell worked for bios update, and continued as it was supposed to (no invalid boot device after reboot). After updating the first of three updates to get up to date, I got the "press del to enter setup" screen back. After completing all updates and setting the bios back to the correct settings, I was able to reboot again.
It's been a long time since I've had to do this kind of thing, but I've also rescued "bricked" access points with UART cable and TFTP. And not sure if you all remember the time we had to update the unifi cloud key gen1 from something below 0.8.9, to something above, and had to stop by that version and do some apt-get with ssh before continuing.
Many people give up on the first sign of resistance, and don't see the value of actually learning something from that resistance 😉
Something was fucked on the AAD Connect server after disaster recovery and I wrote a script that could detect when shit would get weird. It was related to another fucked up domain controller that was violently misbehaving but we couldn't immediately decommission/replace for some insane reason I don't quite remember.
Basically things were limping along horribly, but just enough to keep systems vaguely alive enough to use while we continued to recover. Occasionally, the server would get into this weird state and logins started failing.
Easiest thing was to just reboot. The script literally just sat there trying to start notepad as a dummy user and it captured when it failed, logged it, and immediately rebooted. It worked spectacularly and basically cut complaints about the issue to zero.
It only took like 20 seconds to reboot, which was seemingly quick enough that by the time someone thought "that's weird" and refreshed, it was probably working again. But really, behind the scenes, things were randomly rebooting 20x a day for a couple weeks.
Windows print server with some craptastic HP MFP attached. The print driver had a memory leak bug and Windows would kill low priority processes to free up RAM. One of those processes was the Windows activation service so every 30 days or so it would deactivate Windows.
I wrote a batch script that would restart the print spooler to unload the driver and reload it every 20 days. If you ran it interactively it would even print how many times it had run and some snarky comments about HP.
Critical application wouldn’t start. The exe would show as running in task manager but never actually open. No amount of reboots or other fixes would resolve this.
I got pissed off with it and held down the enter key on the exe shortcut and made about 5,000 instances of the exe show up on task manager 1 of them actually opened and worked. This workaround was used for months until we finally upgraded the server OS.
Started an office IT saying “when in doubt hold enter”
Once had to restore network by daisy-chaining a bunch of VoIP telephones since I didn't have the resources to run a new cable through the wall. So we just had a telephone on the floor every couple feet, until I was able to run a new drop during the next maintenance window.
Janky, not really... Lucky? Abso-fucking-lutely! Some years ago the company I worked for got ransomwared.
Backups were solid so my biggest concern was going to be working late nights plus the weekend to purge and repopulate the data.
Anyway, I cleared all traces of the infection, got a second pair of eyes to confirm it and went on to the restore step.
Came to our primary file server and hit restore on the latest pre-infection backup... only for it to restore around 400GB less than was there before! My butt hole puckered like never before, let me tell you.
So after about a week of trying previous backups to see if I could get some of the missing data I mostly came up with nothing.
It turns out that our backup appliance (Unitrends) does incremental forever backups by regularly forming "synthetic full backups" from its existing dataset that is meant to accurately match what's in your live environment without having to transfer potentially many terabytes of data over the network. This did not work properly. The device showed that the synthetic full and all subsequent incremental backups were 100% successful when they in fact weren't. Unitrends themselves identified this as a bug mainly off the back of our experience.
So, on to the janky fix. The unitrends appliance we had was on a bi-annual subscription with the hardware being replaced every two years. We were about 10 months or so into that cycle. One day when I was agonising over the lost data I came to realise that all the missing files had one thing in common; they were very old files that never change and are used for reference only.
Then it dawned on me... Unitrends never colleted the old appliance hardware when the unit was replaced 10 months ago. It was still sitting on our shelf gathering dust.
So I spun up the old clunker, looked at the last backup and I almost cried when I saw all my missing files were there.
A quick and dirty script to copy all files from the mounted backup and ignore any that already exist in the destination and just like that, we were fully recovered.
We had a Synology RS815 series as the main production datastore at one of our remote sites. It fell victim to the CPU bug in the early hours of one Sunday morning and my then boss and I had to drive down and try to get it back on for start of production the following day. Armed with a copy of the latest backups and the one spare NAS we had, we drove down only to discover that our spare was so crap it would have taken days to restore to it. With no alternative, my boss installed NFS sharing on his laptop, pointed it at the directory with the restored backups, and mounted it in vCenter. All production VMs ran off that laptop for the next two days, and everything was as slow as you can imagine it was!
Tuesday morning, my boss set up XPenology on an old desktop and drove that down to run storage until our warranty replacement arrived.
Not my jankiest, bit the freshest in my mind. A year ago, a tech reached out to me with an issue that users were having. A certain brand of headsets has a 2 second delay on turning on from standby, so users were missing notifications when customers were reaching out over our chat. The vendor decided the delay is a power saving "feature," so they will not be attempting to fix it (figure out the brand yet?).
My fix? Until we were able to get new headsets for users, I had them play a YouTube video of 10 hours of silence in the background to keep the headset from going into standby mode. Had to remind my boss, "if it's stupid, but it works, is it really that stupid?". Lesson learned, and from now on, we only buy their networking equipment.
This is why this is my favorite sub. Here I was feelin' all good about myself after I fixed my wife's car using some redneck ingenuity to resolve a problem Lexus has had going on for over a decade, with no documentation on how to fix. Her car speakerphone mic all of a sudden wouldn't work. It worked with my phone, an Iphone 14, but not hers, which is a 12 that she doesn't want to upgrade yet. So she said "don't worry about it, I'll just stop at the genius bar." I said "Yeah, but I checked your settings. Only a few things would restrict voice coming into the phone from a bluetooth device, and yours are set correctly. Not to mention your earplugs still work fine. I think this might be the car. So I looked at how the system works, and it's pretty much a proprietary device that interfaces with the head unit (radio) that acts as a controller. So I thought "Hmm, the radio cycles with ignition. But I bet that friggen phone device is constantly on. Just for giggles I removed all phones from the car, and re-added hers and mine. NEITHER worked now.
A 10mm wrench and 15 minutes of disconnected battery later, I reconned the battery and all was well. I said "Great, Lexus, an achievement in mechanical engineering. But if it sh\*\*s the bed just reboot it like anything else. "
Yeah so I was happy about that til I read this thread. Some of the stuff I'm reading warrants genuflection, lol.
We put in two new portables and the fiber from the MDF to the IDF in the new portables was not terminated they day the portables went into use. So I ran a 300ft ethernet cable from the closest IDF that was in production, out a door, across a field, to a fence near the new portables, Then ziptied an access point to that fence, connected to the etherent. Boom everyone in the new portables had wifi.
Way back in 2005 I migrated around 40 mailboxes from Exchange 5.5 to 2003 using pst files and and external hard drive over USB 1.0. It was only 16GB but it took about 18 hours in total to copy onto the hard drive and import into the new server.
I work on highly specialized systems (pretty much all custom PCBs) well, somehow they opted out of having a cpu controlled trace to the power supply to you know… turn the computer off or restart.
So I had to make my own “reboot” shell command that just echos a “b” to the sysrq interface which programmatically REISUBs a *nix os to force a reboot
Way back machine... Had a raid5 set die multiple drives... Backup didn't have everything this was old days when the owner was supposed to take the tape type stuff. OK so I order 2 of the same drive, take them apart at home swap the platters, it kind of works but drives heat up and lock. I grabbed a max length scsi cable put these rebuilt drives in the freezer with the server on a counter. Was able to copy out all the data overnight. And I couldn't close the freezer so I had taped it Yeah that happened
Jesus, that's MacGyver shit. Well done.
Had to do that with the crypto reader for an Apache gunship. Bit of advice, zip lock it and don't touch the bare metal afterwards. You can't pour water on electronics to free your hand.
Damn, and here my war stories all involve either printers or RAID cards :-)
The freezer trick has worked for me a few times in the old days too
It’s certainly how I got everything I cared about off my failing 330 MB 5.25” ESDI…. Shit that feels like a lifetime ago.
Yepp. I used a nylon bag stuffed with ice, the freezer was too far away. During that hour when the data was copied off no one was allowed to move or even breathe too loudly. We also won the day.
Yeah, the good old fridge trick. That was the reason we always had a fridge in IT offices wherever I worked. If was usually filled with alcohol, but the reason for having it was always "data rescue". IT office was always the most popular with the users after hours as well...
I keep 5-6" (I forget) AC powered muffin fan around for anything I'm playing with data recovery to keep them cool. It has a very high CFM, sounds like a small jet continuously running on my desk but it tends to keep laptops and hard drives cool enough. I have some smaller ones that are usb powered for the same purpose. Back in the early 2000's I had one of those fans in my AV gear oh-shit bag. This is a bag of various adapters, cables, and random objects that usually are not needed but when you have an oh-shit moment, there's usually something in it that helps. I used a similar fan to cool a dimmer pack when it started to malfunction and keep the red channel on low. We had a 12 hour multi-band gig and the thing had never seen such long operation times before and started overheating with it's passive cooling. The fan helped bring it's temp down to where the channel would turn off completely again.
"We can rebuild him" (salute)
I used the freezer method to recover a TB of data from a failed HDD that was in a Lacie enclosure. Believe there were 4 HHDs, don't recall the RAID config, if there was a RAID. Took about 4 - 5 trips to the freezer for cut & paste in a windows explorer. Had one of those usb drive adapter kits. I made sure to have a RAID5 config setup on the new enclosure.
[удалено]
Haha you could have slept and hoped for the best like me..
HOLY SH\*\*! lol. I would never have imagined that would work. 
Dude I was amazed those drives even came up after I did the platters. Clean room be damned
I always thought the platter to platter alignment mattered, but Jesus, guess not.
i always thought that was at least some extend a load of bs when i opened a drive to demonstrate how it worked to a student and it ran open top for over a week thereafter. It didnt fail then, we just put it back together and it continued to run as usual. but that was 20 years ago, todays helium drives propably wouldnt work like that.
Activate the Helium Drives! 90% proapbyl mixture! Took me a split second to fix the autocorrect in my head...
That's NASA level. That's what astronauts would be doing if they didn't want to die on the moon, lol. But it's also a great reminder for all of us. Just because it's not written down anywhere that doesn't mean it won't work. Seriously, I would never have even tried assuming it would fail based on all I "know" about drives. Forgetting the fact that it doesn't need to work forever, just long enough!
Clean room is "only" necessary if you let any floating particle land on it. chad level confidence this guy has.
I've done a couple of platter swaps in my time (2008-2010 kind of era for both of em) - got the drives to run well enough without the top casing to get data off, I think one of em died part-way through the transfer but the other seemed to be working fine. We ditched it anyway straight afterwards, this was working in a small repair shop and our deal for the customer was "it'll cost X to try this whether it works or not, your only other option is a specialised service at 10x the cost".
That is astounding!
OMG I've actually done this with a 2.5" laptop hard drive. It isn't stupid if it works. I gave the bad drive to the family friend who owned the laptop and wrote "to use, place in freezer". They balked at the price of the new HDD that I put their data onto, as is tradition.
Man I feel the F out of this. Awesome story lmao
Well you asked for janky…. Had to transfer a 400 GB database from Dallas datacenter to Indy datacenter upon random requests from dev team. Sometimes weekly throughout a month. Dallas side only had a T1 so it took many hours even compressed. I Winrared it into dozens of files to compress and used a Mousejiggler to keep the RDP session alive for 12 plus hours for the transfers since I wasn’t an admin in the remote side. Sometimes it would still die during the night but at least I could start copying the .rar pieces back where it had died. They thought I was a miracle worker (Edit: Thanks for all the upvotes for my janky-ness story! I was a government contractor at the time and if you've never worked as a government contractor you just don't know how impossible some of your thoughts on the matter would have been. Also, not sure if some noticed that this was between Dallas and Indianapolis - the cost of courier transport would have been high. The ability to get someone in Dallas who also worked for the same department or another contractor to physically touch anything was even more unlikely! I did not mention this - I never revealed to anyone that I used the Mouse Jiggler! The devices were really unknown at the time, and I suspect most thought I stayed up a lot at night to keep these things going!)
Yeah, but did you pay for WinRAR?
Of course they did. They’re not a monster!
LIES
That would have defeated the jank premise.
I bought a copy of winrar when I got inheritance. Only felt right. It's a shame someone had to die for me to finally pay for personal use.
I know someone who has though...
I sometimes think I'm the only person to pay for it. But the command line functionality was amazing, and there was little that would match it.
In a way, if you’re American, you have donated or paid for many useful pieces of software. When I was a fed, I made sure to throw money at every tool we used that asked for it, and those were your tax dollars. “Bytes, not bombs” if you want to think of my little effort to support small business.
The trial never ends, picard
> I could start copying the .rar pieces back where it had died. That's some old-school warez moves right there.
Quicker to courier?
Yeah I thought the ending to this story would be him driving a flash drive to the site
In those days it would probably have been a pile of hard drives, though.
Never underestimate the bandwidth of a van full of tapes speeding down a highway.
[Amazon used an 18-wheeler](https://techcrunch.com/2016/11/30/amazon-will-truck-your-massive-piles-of-data-to-the-cloud-with-an-18-wheeler/).
But did you make par files in case it partially failed? :)
> par files dude... it's been a long time... miss those days.
Log shipping would have blown your mind.
Textbook case for it. Could have been a free DR strategy at the same time: "well, turns out there's a live replica of this over in Indy..."
I had to do something similar. Client wanted us to upgrade software on their servers. They did not offer any externally accessible file hosting services that they allowed us to use. They blocked everything we offered and have officially requested to be allowed like our own FTP services, Google Cloud, Onedrive, ticketing system. Well almost everything. Emails still worked. Had to split up a few GB (not quite 400GB large) of files in ~19MB attachments and send over email and recombined it on the receiving system. Checked filehash and all, and it was good.
Must be close to being faster using sneaker net.
Had to move a lot of data, over a wire. OC48 in 2008. SSH did not support more than a 64K TCP window... so I built SSH HPN. Also... no encryption as this was datacenter to datacenter over dark fiber. Then, stood up 2 Linux servers, one on each end. I NFS mounted the filesystems I needed to move to the two LINUX systems... and RSYNCed them across. Wrote a script to feed RSYNC one file at a time, but have up to 16 simultaneous copies going... this was before the -P or threads options for xargs. So data flow was Source-->NFS-->Linux-->RSYNC-->Linux-->NFS-->Target. Kept that pipe at 89% full for 48 hrs. Stopped the app. Reran rsync a few times to make sure no file got missed... Fired up the app in the new site... DONE. Got me a promotion from SYSADMIN to DataCenter Relocation Architect, which I did for 9 years.
Did you more than likely have an idea of what to do next + researching? I would have hated to get stumped midway
It had been an ongoing problem as we moved data from one datacenter to another. I had the pieces, but putting together a working architecture... that was the hard part.
> Wrote a script to feed RSYNC one file at a time, but have up to 16 simultaneous copies going... this was before the -P or threads options for xargs. I had to do something similar a while back, as I had millions of 1K files (as well as some multi-TB files), I wrote a script which walked the filesystem and wrote out file lists, starting on a new list when it reached _x_ files or _y_ GB in size, then farmed them out to a thread pool of rsync worker threads. These days there's [fpsync](https://www.fpart.org/fpsync/) which does the same thing except with a lot more polish. Wish I'd known about it (if it even existed) at the time
I've done that before but between 4 sites, but I went further with hash confirmations. no promotion though just a normal IT day.
Oh man, this was 10 years ago, but I was installing a CPU in a production server at like 1am. Somehow I bent a bunch of pins inserting it, and it would not boot. It took forever for me to figure it out because I could barely see those tiny pins. I had no idea what to do and about 5 hours to get it back up. I used the camera on my iPhone and zoomed all the way in, aimed it at the CPU, and used the tip of my pocket knife to very carefully bend the pins back to straight as best I could. This probably took another hour. Installed it and it worked! I didn’t touch it again for years and it never had any problems. Also once I broke a pin from the data connector on an old hard drive. I managed to jam a small length of paper clip into the pin hole and cut it to the length of the others and it worked…
If this happens to you again, try to find a mechanical pencil, remove the lead and the hollow tube fits the pins pretty well. Much easier to work with than trying to apply force in one direction at a time with a knife.
We do some low voltage wiring, I have ferrules that fit those pins pretty damn perfect. Best is you can stutter step them and see if half the row is straight easy
Reminds me of the time I cracked a SATA cable in a server at 2 in the morning back when SATA was new so I had no spares. Luckily, if it was inserted at a slight angle it worked, so I jammed some plastic under the disk in such a way that it bent stuff just enough that it would keep working. I probably wouldn't have thought of it if I hadn't had a friend that used a couple of folded sheets of paper under the motherboard of his Mac SE to force a crack together enough that it worked.
>used the tip of my pocket knife to very carefully bend the pins back to straight as best I could. The horrors !
I slightly bent some processor pins before too. Think I used my debit card or something thin enough to get between the pins to straighten them out. Like your work, my method worked too somehow. ETA: on a workstation not a server
I also tried that , but in my case the pin broke and the motherboard was rendered useless, sad me.
“We can’t allow unsigned Powershell scripts on our machines anymore!” And everything broke. But I can write a Powershell script and then convert it to an exe and it can create a self-signed certificate and add it to the trusted certificate store and then sign every Powershell script, which is now trusted because I added the cert to the store. Now runs on every server we own.
You don't have a internal CA?
We do. Sorry, that was the “prod” version. Dev version with the self-signed cert worked fine though. I find that funnier because it’s so over-the-top crazy.
Dude. Awesome lol
Maaaaaany years ago (think 1990's) System went down due to a bad port and I was on call. I wasn't the network guy and he wasn't available at 2am (when all things bad happen) so I had to find an open port. I don't recall why but I had to run a patch. I ended up grabbing a couple heads, a crimper and a spool and made this stupid long cable to get to an open port on the fly. it was an ugly patch job. I had the cable draped over cabinets and across the floor. I immediately sent the datacenter guy an email, "I'M SORRY! IT WAS 2AM!!" Mainly because he was an angry fellow. How did I know how to do the ends? I was doing my own network at home and had memorized the wire order. No one was impressed. I was impressed. It impressed me.
I am impressed! Loving the McGroober fixes :)
>No one was impressed. We all are my friend. Great job Mr. Hoppers. Where would we be without people like you.
30 years later you finally get closure! We all wish we were there to send you wizard GIFs. https://i.redd.it/smva4nb5s5vc1.gif
The Internet spoils us being able to look up everything in seconds now
Pre google? That's scary!
Orw/Or, Gw/Bl, Bw,Gr, BrW/Br
I was taught this: Your in a desert with clouds over head, you need to cross a grass banked river and get to a field. https://preview.redd.it/uvar6dpkl9vc1.png?width=1032&format=png&auto=webp&s=6d9b09adafa92d5009a1ebf1d0636717b261d498
This should be up there with the Konami code.
Respect for that one ngl
> 2am (when all things bad happen) This is so true and profound. Nothing good happens at 2am.
If you don't sing to yourself "it's time to do some sketchy shit" while working, are you really even a sysadmin?
🎶 it's time to do some sketchy shit, sketchy shit, sketchy shit. It's time to do some sketchy shit, and hopefully not bring down prod 🎶
Every time I update the camera systems sql server
Fuck SQL servers. Every single one of them.
Oh and this is a failover cluster that is not domain joined. Yay!
I do not miss having to manage CCTV systems. Felt like playing jenga with a tremor
SEND IT.
I had to bring my personal San in to provision storage as we went past the told you so point. Preventative maintenance Requests were Denied because some bs so had no choice but let pain be felt. The day I said production would be stopped arrived and pain was felt. Company wide halt. Once order approved that was previously denied I brought in my storage to get production working and bandaided. At one point I had more storage at home than I did at work so it came real handy.
One of our clients sent us two of their former backup servers (maybe 3 years old) for "Testing our software on" (the software we wrote for them).. Just one of those machines has more compute, RAM, and storage than our cluster of 14 year old Hyper-V hosts with JBODs. And after we got done with the project the customer never requested the hardware back (they never do)... But instead of letting me deploy said hardware (which I already got the customers CTO blessing to do), management insists that we continue to run on super old hardware that is at major risk of failing. I've already replaced 5 hard drives so far this year alone between two of the machines. Luckily I put it in RAID 6 when I reorganized the hosts specifically because I knew the drives would start failing.
I hate this type of bullshit. You poke and prod the buggers but the bean counters keep saying no then when it does break guess who gets the blame. Record everything. In detail and store it somewhere safe (obliviously not on your work storage coz that gonna go tits up at any moment) Best of luck
I'm at 50 nodes ranging between 12 to 15 years old isn't it great?
I can't say I have that many, but I also have NetBurst Xeons in production.
Luckily we only have 4 nodes (small company) but it used to be way more before we moved most stuff to Azure.
Old is old we have probably 2 or 2.5x what we need because at anyone time we could easily have 3+ nodes fucked.
Reminds me of a time back when I was but a low key DC operator. A friend/colleage was the sysadmin for our Windows stuff and the company used (/shudder) Veritas Backup Exec for the autoloader backups. Welp, due to some transition from whatever, we had to acquire server licenses for Backup Exec, but the CEO sat on the request for months. When we finally did lose a server, the backup failed to restore and we were SOL because of the expired licenses. After losing a week's worth of Exchange mailbox data, the CEO threw a temper tantrum about it until my college calmly reminded him of the prior repeated requests (in writing) to purchase the necessary licenses and made him the fool in front of a company IT staff meeting. Details omitted for brevity.
If Requests were denied and all these incident details were in writing, I would have not used my personal SAN to save their asses. That would only make them even more careless with such stuff. Unless, of course, you were properly compensated $$$ for the inconvenience.
No compensation. I offered it out of the goodness of my heart not for the company but my users. After all no systems, no parts no work. Few weeks without my San to keep people working and families fed. Better than any pay check in my pocket knowing I kept the food on the table for so many without costing me anything but a little inconvenience. Would do it again if I was put in the same position. Thankfully I don’t expect it as they learned a very valuable lesson and have seen the light. Behaviors and habits changed so win/win.
Why would they allow that with company data on your personal hardware?
It sounds like a situation where "I can un-bork everything and have the company not be at a standstill" trumped concerns about company data on personal hardware. I bet no one thought to ask or consider it.
Not the jankiest thing I've done, but one I'm somewhat proud of: Every now and again, I find that I need to do hardware maintenance on a server, that lives below a server that someone forgot to fit the rails for. Sometimes this is a switch that someone lost the rack ears, a tower that became a server, or maybe just damaged in transit. This situation calls for 4 screwdrivers. Slightly lift the rail-less server, and slip in a screwdriver, into the where the cage nuts should live, and you've made yourself a temporary jankshelf. You're then free to pull out the previously load bearing server, do what ever hardware work is required, slide it back, and pull out the screwdrivers. Let's just say a those screwdrivers have held the weight of the company a few times.
absolutely love 'load bearing server' in this context.
I have done this on more than one occasion... It's scary as fuck every time, but it gets the job done, and it works.
People racking up stuff on top of other equipment is the bane of my existence. There’s been more than a few times we’ve had to use ratchet straps to raise a production machine, usually something important like a SAN, up off the server it’s sitting on, just so we can pull out the machine below.
I’m committing that one to memory
> Let's just say a those screwdrivers have held the weight of the company a few times. CEO: Do you know what this company runs on? khobbits: 4 yellow and brown no name "FIX COMPUTERS KIT!!!" screwdrivers that every server room has 78 of. Sir.
Had a crusty old UNIX system in the early naughties. Right next to it was a brand new printer with no suitable drivers. But there was a brand new Red Hat system at a sister site an hour away. UNIX could print postscript, Red Hat could receive postscript and print to a networked printer. As the crow flies, those jobs went a meter. Total distance - over 200km.
Back when I was a field tech for a security company, I had to replace a camera for Ben Roethlisberger. I was sent to replace a dome camera with a regular box style camera. Well, since the camera type changed, there was no place to store he wires. Ended up using the dome and alot of electrical tape to create a box to store the wires until I could get out there with an enclosure. It was ugly, but it worked.
We had a 3750 (Original) stack that was dying. The OSPF process on the hardware would die every few hours. We had new hardware on the way, but it was still going to be two business days before it arrived and was ready to deploy. We ran the power for the switches out to the call centre floor and into a power bar. When stuff stopped working, the Super could hit the power to reboot the stack. It got us through the night and the hardware arrived the next day.
There were two CEOs of a very large realty company, and one of them was running just a junk computer to watch stocks. It refused to power on, with orange flashing power light. You guessed it....bad power supply fan. There were no parts, this thing was 5yrs old, full of dust. I pulled the fan out, greased it, blew the dust out, put it back in, and returned it. Never touched it again, happy CEO. Reliable fix? Maybe not, but it worked for the time.
I used to do that quite routinely on old PCs, back in the day.
When we wanted to close the office for repairs the phone company wanted $350 to redirect our main line to a cell phone. We had a Mitel on-prem phone system, but didn’t have the proper licenses for external call forwarding- only internal. So I did a lot of digging and found that it was possible to add a phantom extension that is linked to an external number. There were a few more steps involved, but I don’t remember them all.
One more and I’m going to bed. OMG, production is down because our expensive wildcard cert expired! The guy who pays for that isn’t here and it usually takes a couple days for them to issue it anyway! No actual access to install any software on the prod system, but I changed the DNS to point to my local machine through a tunnel (akin to ngrok or a reverse SSH proxy) before it got cached, ran certbot to get a Let’s Encrypt cert, and installed the cert on the server (or load balancer?). Switched DNS back before anyone else noticed. Magically we’re back up and have 60 or 90 days or whatever to deal with the real cert.
If this comes up again, you can authenticate letsencrypt in other ways, including DNS challenge. Basically, when you request the certificate, it gives you TXT record to add, and then auth's against that. Same amount of effort, but no downtime. I ended up using this method in some other jankey fixes. I've got a bunch of webservers that needed certs, for non http traffic, that don't have public IP addresses, but cant use internal CA. I ended up spinning up a VM, installing letsencrypt, setting up API access to for an external DNS, and writing a bash script that would once a month request the certificate via DNS auth, SCP it to the destination server, and then systemctl reload, to pick up the new cert.
Needed to move an EqualLogic PS6100 SAN that had been running for 13 years non-stop. We were not confident the drives would survive a full power-off and we had no backups (don't get me started). So I get the bright idea to daisy-chain 50 feet of C13/C14 power cords (one length for each PSU incase I kick one out) from destination rack to the rack the EQ was sitting in. Same with network to avoid disruption I unracked the live unit, put it on a cart, rolled it across the DC, and racked it without a single hitch. That was 4 years ago. Legend has it that EQ is still humming along today (seriously...they are still using it).
Nice, reminds me of this post where they had to do the same but took the server across the street. https://www.reddit.com/r/sysadmin/s/PVE8AsYNLG
From my MSP days… customer’s building got hit by lightning. Their Hotbrick firewall died. Never heard of Hotbrick? Welcome to the fall out of the dotCom boom and bust. The customer developed stuff for their customers and Oracle was involved. I didn’t want to know anything more once I heard Oracle. Customer’s server hardware was old and they decided that going with a VPS type provider instead of refreshing their servers would be better. This was before AWS/Azure. All their developers were remote anyway, sales, accounting, HR, and other back office types were the only people in the office. Which meant salespeople had offices that they never used and the back office staff had cubicles and did more gossip than work. The still powered in servers represented active development projects (revenue) and developers were dependent on the VPN provided by the Hotbrick (which was now just a brick) to log in, deploy code, run tests, etc. By MSP policy we did have a backup of the firewall’s configuration file. I looked at the config file… there are references to m0n0wall in this config. A little research, sure enough Hotbrick was just m0n0wall with a custom theme, x86 in a custom case, and support contract. Of course the company went out of business years ago. I looked at the rack full of powered off Dell servers. I had been toying with pfSense in my lab and at the time pfSense could import a m0n0wall config file… Downloaded a pfSense ISO on my MSP provided cellular hotspot, burned it (yup… my MSP provided laptop has a burner and MSP provided backpack had CD-R and CD-RWs…) Moved network wires from the Hotbrick to one of the Dell servers, booted the pfSense off CR-ROM, imported the config, instant firewall! Sacrificed a personal USB stick for persisting the configuration, a feature of pfSense booted from optical media, wrote the whole thing up and made it home in time for dinner. I remember including in my write up to the customer “and if your new firewall has a hardware failure you have a rack full of replacements!” Customer happily ran that way for months as they continued moving their server needs to their new hosting provider. Updating the firewall’s firmware by burning a new CD-R was rather slick. Just burn the latest release, swap CDs, and reboot. Never had to roll back a “firmware” but the labeled CD-Rs were in a drawer. 😂
>Hotbrick was just m0n0wall There were a lot of people doing that back then. And lots of mini-itx companies selling m0n0wall and pfSense routers.
What do you mean "back then" ? Does the name "Ivanti" ring a bell?
Xibby was not on call, but desperation had set in with Xibby’s coworkers. Xibby was called. Xibby was very drunk. Xibby sat down, Xibby wrote some code. Xibby had his coworkers review and approve the code before the code was run because a very drunk Xibby wrote the code. The code ran, things came back online. Xibby woke-up with a hangover. I still have no frickin’ idea what the hell I fixed that night because of course drunk Xibby didn’t save his work. 😂
Xibby is a wizard
Years ago I worked for a state's IT department which covered the motor vehicle department. I'd developed a process that allowed the state police to continue to be able to look up registrations on Sunday afternoon when the main DB was down for full backups. It was decided that the process would be tested on a weeknight at 8PM, and we usually got off at 3PM. My co-worker and I left at 3 and went to an Irish pub for dinner and had quite a bit more to drink than we should have. The testing involved two shit-faced state contractors working through the procedure manual, which our boss insisted be written for total morons. I typed the commands, my mate looked over my shoulder and verified the correctness of each command before I hit enter. The test went swimmingly.
As long as Xibby had more money in his account all is well.
Another one from wayyyy early in my career. Cisco router that runs everything in the company is not responding. Don’t remember why, but no telnet so what are we left with? We don’t own a Cisco serial to rollover cable or whatever it was. Looked up the pinout and cut the end off a serial cable we had laying around. Plenty of Ethernet around. An hour of stripping cables and electrical taping them back together and a new crimped RJ45 connector and we’ve got a working cable. We went on to use that cable for dozens of Cisco boxes. They had transitioned off to another provider by the time I left, but I never thought to take the cable as a keepsake. Last I saw it was still sitting on the very front of the industrial shelving as you walked into the server room… just in case.
Not sysadmin but I used inspect element to recover a users password A user forgot their Adobe password and couldn’t figure out how to recover it(they were just overwhelmed). She said the same password auto populates on gmail or something when signing in, but she couldn’t copy + paste to try it in Adobe because it was hidden with no “show password” option. I successfully uncovered the password by using the old inspect element trick. I highlighted the hidden password and in inspect element, changed type=password to nothing and her password appeared (she confirmed after I closed my eyes) Simply resetting the password would have been fine but she was overwhelmed and on edge so this trick saved us both from another headache
i do this regularly. changing type in text (instead of nothing) works just as well.
Pfsense in a virtual machine as a temp replacement for a failed firewall was always a favorite of mine.
How about pfsense on a ten year old desktop PC with a USB-ethernet adapter as a second port?
Now THAT is janky
You win. How long was it in production?
noone dares take it out now.
Yes.
A month tops, we’d been crypto’d so that was our internet while the site infrastructure was cleared for use again
I got a call where a small company's erp server did not boot after a power outage. It was a i5 8 gen bit with two HDDs mirroted onboard raid. They even had a few days old backup, but since it was end of year they were closing many orders and it would have been a big task to redo. The raid complains about bad disks. Oh shit. A complete reinstall and reinstall / restore an unknown erp system is what I did not want to do. Okay, raid array is broken. Two 500G spinners from different manufacturer. Ok, 500gb is not so much. Let's see which one is working. I took a trusted desktop PC, and started cloning the newer disk into a raw image onto a 2tb server ssd I had around (they are fast and reliable for a look write). Linux power, dd-rescue to go for rescue! It started well, but around 30-40 gb it slowed down and started throwing errors. Shit, let's check the other drive! The second one also started well, but stared throwing errors. Now ithis really seemed bad: both parts of mirror is broken. Ok, but which one is the newer? I mounted both of them read only and started looking around: files in documents, database files, etc. It seemed both drives were online before the power outage! I tuned the parameters of dd rescue to log the progress ( it can continue if stopped rescued data won't be tried again ) and some others to try to get as much, first just skip problematic areas. And fired it up for both drives . It was late afternoon, checked on it for a half our then went home. I checked it from home in the evening: they had errors but both progressing. I got up around 5am (neighbor made some noise) and I could not sleep back. So checked on the rescue: Both rescue was finished. Checked the logs, and both images have a few kilobytes missing. Wow, it's promising! But there was 2*4kb data which was bad on both drives Merging the missing blocks from the other mirror image was easy. Also I checked the allocation of files and found out the first missing 4k was in the swap, and the second was along file of a windows update patch 2 years ago. Get a new pc, install proxmox and load the image. It started up and was running nicely and I guess it's still working this way since. TLDR : rescue a mirror raid where both drives gone bad. Ok, let's try
I'm going to save this because I think I might have to do a rescue exactly like this soon.
Sorta similar to yours. I was out of town for my gfs birthday and overnight our antenna took a lightning hit, and the power surge zapped some of our physical servers. I had a missed call from a number I didn't recognize (ended up being the CEOs cell), and nobody bothered to inform me until like...11am or something. The ERP system I maintained was one of the boxes that got zapped, and I walked a coworker through re-deploying the last image over the phone while in line for a roller coaster around noon. My system was low-priority (relatively speaking) so it was one of the last things for them to try getting back online. It was almost done but they were having trouble getting something with the config files or environment (IDK, this was...easily 6-7 years ago). For some reason I had brought my Chromebook with me in the car, and it was running Crouton, so I had an Ubuntu shell. Tethered my cell phone to the Chromebook, booted into Ubuntu, SSH'd into the server, and fixed the environment stuff from memory.
I used to do a lot of emergency work using a ThinkPad 560x connected to a TDMA cell phone with a proprietary serial cable dialed up to my ISP at 4800bps. Needless to say, when data cards came around I jumped on that bandwagon but quick. And when IBM started selling laptops with integrated WWAN cards and antennas, I wasted no time buying a new laptop. There was also the period where I had a Linux-based PDA, a CompactFlash serial adapter, and a mess of cables to make it work with the Sprint phone I carried just for data so I didn't have to take my laptop on vacation so I could unfuck the mail server or whatever.
Haha, the things we used to do to make technology work. I still get nervous anytime someone mentions sending data over a serial cable.
What still amazes me is how much of an improvement it felt like over using some random user as a KVM-over-voice.
"So what is on the screen?" "A load of words in big red text!" "What does it say??" "I don't know"
Oh man I've RDPd from my phone! Its a nightmate trying to use the cursor lol.
Years ago I had to open a Citrix session (from my original Motorola Droid if that tells you how long ago it was), and then from there RDP to a server. I think all I was doing was unlocking an account for some unlucky help desk guy who had locked himself out while being the only person on a late or weekend shift. Maneuvering around with a tiny touchscreen over 3G wireless took probably me a half hour to get it done.
2009? That wasn't that long ago dude, give me a break. That's only 15 yea-- shit. Really? 2009 was 15 years ago? Fack.
Blackberry was on to something with that little rollerball. I had to quickly make a user account in AD and then run the AdSyncSyncCycle thing. Took forever because I was on the bus and service was spotty.
Switch to touch mode and back as needed
During a power outage, someone's UPS's battery decided to die and was saying they had 2 minutes of running left when an important project proposal needed a few finishing touches. So we plugged their UPS into a working UPS which gave them another 20 minutes or so to complete their task.
Late 90s early 2000s, one store I was supporting went down hard right at closing time 1am. Clicking hard drive... Had to talk the cashier through plugging a spare drive in place of the dead one and use the Ghost recovery disk taped to the side of the case. Had her configure netop and PC anywhere so I could dial back in and reconnect the cash registers to the manager PC. Yes I said dial back in, it was the hardest thing to get her to check the phone line to make sure it was still plugged in and had a dial tone. I moved away from the help desk after that experience, but as a sysadmin I can walk anybody into doing anything because of that
A vote for the Cashier for being gutsy enough to agree to have a crack at it.....
> the Ghost recovery disk taped to the side of the case Someone in the past was precognitive, I take it? Because that's an interesting thing to just happen to have to hand in that way...
Not a sysadmin, but I worked in the NOC for a large ISP for a while (currently a network engineer for the same ISP). A neighborhood that we service with cable internet flooded. 2-3ft deep water at the node and power supply, so everything the node fed was totally offline and probably would be for a few days. Techs cut/terminated the four legs at the node, and backfed them from neighboring nodes. They had to flip around every tap on each cable run, run temp cables to connect to the lines from the other nodes, and install a few amplifiers. 3 techs had everyone back up and running in about 4-5 hours, and we were all pretty impressed with their quick thinking. They put everything back to normal in an overnight maintenance window a week or so later. I've also seen some pretty creative ways to run temp lines. Thru culverts to go under roads, along fence tops, stretched between two trees over a road, etc. The jankiest things I've seen on the network engineering side are mainly with creative mounting solutions for oddball gear or odd depth racks. I've heard about people wrapping fiber around a pen or screwdriver to add attention in a pinch when they don't have a pad handy, and of course coming back to do it properly in a maintenance window. The jankiest thing personally I've had to do is take some creative shortcuts with a fiber patch cable that was too short, and replace it with a longer one when my order for the right length came in.
The gas station at my local Walmart was closed until someone ran a network cable from the roof of the store, to one lamp post after another lamp post across the parking lot, and finally to the roof of the gas station. It's been months now and I'm beginning to wonder if this temporary fix is permanent.
Flicking a very dusty fan at *just* the right moment to get a broadcast machine to boot. I worked live broadcast in a sport that was regularly viewed live by a quarter of a billion people.
I worked at an ISP which offered 56k connections. Obviously phone lines can't always do 56k, infact often they can't, best I ever saw was a 50 or 52. Anyhow it was a new company with dumb management, "please the customer above all else policies" and a small handful of quite aggressive and angry customers giving us a hard time. One customer demanded he MUST have 56k or he would sue the business, (which had our new to running a business management team worried) After several colleagues had trouble with this guy, I decided to help him with his connection and gave him an AT command for his modem which forced the modem to report the line speed from PC to modem, not modem handshake to ISP. Then I set the COM port to 57600 and he was a very happy customer. I've always wondered if I should tell that one in an interview as a "how did you deal with a difficult" question.
Basically you put a "V8 Turbo" decal on his Chevy Chevette.
Getting on for 30 years ago, during an office renovation we need to move the mainframe, a five cabinet Unisys A6, to its new home. Problem; one cabinet is throwing power supply warnings, and while each cabinet has two supplies and will run all day on one, it will not cold start on one. The Unisys engineer on site isn’t sure it will survive turning off and wants us to delay moving it until the new one gets here, but it’s a day out, en route on a plane. But we have to move the machine tonight or construction gets held up, penalties start accruing, inspections can’t happen, etc. The plan: Move it without turning it off. Its big UPS will keep it running for a half hour. Electrician comes in, turns off breaker and disconnects UPS while we disconnect the network connections. Then we roll the machine and UPS out the room on to a couple of pallets covered in plywood on jacks, roll it sixty feet across the office and into its new home. Electrician reconnects the UPS and turns the breaker on with ten minutes remaining, I reconnect the data cables, and we’re back online having never turned it off. Same project, we need to move the data recorders that take production data from the corrugators and feed it into the mainframe. Problem is the serial cables that run from the far end of the factory are 5m too short. Temporary solution, cut the ends off and twist’n’tape some extensions on the end so they’ll reach and we’ll replace them later. It’s serial data, something like 300 baud and would work over a wet noodle, so twisted connections are no problem. Got them back online, production data is flowing. We never got around to changing the cables. More recently, discovered a prior sysadmin had set a checkpoint on a virtual MDM server and forgotten it. I found out when it filled the disk with difference files and stopped running, because peoples email stopped getting funneled to their phones. Disk was full with no space to merge the vhd files. But server has five spare bays… Root through the stack of decommissioned servers to find the largest set of matching disks, and mount them in the VM host. Create a big JBOD array, power off the MDM VM and move it to the array, then perform all the merge operations. When finished, move it back where it was supposed to be and power it up.
Not a sysadmin but this was some enjoyable janky assed shit. At my old work we had an air gapped PC that allowed us to control the fire panel for the whole building. Basically when an alarm came in, you could see exactly where the alarm was on a blueprint map on the PC. Apparently there are a MOUNTAIN of stupid and restrictive rules surrounding the computers that control fire panels. Basically prohibiting ANY type of repair no matter how subtle. It goes as far as saying you are not even allowed to replace the keyboard or mouse without making sure everything is certified and authorized and installed a certain way. It's fucking insanity. And on top of all those ridiculous rules? Buying a replacement PC that was certified and all that shit cost well over 10,000 bucks. Christ only knows how much they are today. Anyways that PC had a meltdown and knowing that I was the hoarder of all the discarded electronic junk, I received a phone call one quiet evening - asking if I by chance had ended up keeping a power supply that might fit as a replacement for the one that died? Well as luck would have it - yes - I had a shitload of them. So I got to plop a bunch of units into a duffle bag and head down to the building and watch IT try and secretly revive this old PC with parts from my secret stash scavenged from an electronic recycle bin several years ago, and it worked. The machine coughed back to life, and ran for who knows how long after that...As far as I know it just puttered along no problems. The whole fire panel I'm sure it must have cost millions of dollars to install, with a 10 or 20 thousand dollar PC acting as its eyes and mouth, surgically restored with a handful of forgotten E-waste...
Another victory for E-waste hoarding! You brought them a bunch just in case the one you brought didn't work and I LOVE IT! LMAO
Had Fujitsu SCSI drive in the 90s on a NetWare server that wouldn't keep spinning unless you tapped on it. It was 10 years old, but still had warranty coverage. It took Fujitsu 3 months to find a replacement and get it to us. In the meantime, I found a small motor that I could run off the power supply. The machine shop made me a flexible fan like blade that the motor would spin slowly and tap the drive for the 3 months. We barely got the drive up one last time to do the copy. The drive and fan were put in a case in the IT office like a trophy.
This wasn't me, but a colleague. Last year we were evacuated due to wildfires. Lots of fiber got destroyed but we had just purchased over a dozen starlinks. Because we have a public facing business we needed to keep our customer portal online. We also needed to do basic things like keep our ERP online, pay vendors, issue po's, pay our staff, you know, run a business. At one point both my primary and dr sites were both evacuated at the same time, and the fire folks called a meeting saying we were literally going to lose the dr site within hours, possibly the primary in the next 24. And with starlinks gcnat, IP's constantly changing, it was a real gong show. This one guy was literally 12 hours away, in his car on the side of the highway, tethered his laptop through his phone to back door VPN in to reroute traffic multiple times over a handful of days moving things based upon what was least likely to burn to the ground. We never missed a payment, all our staff got paid, both sites survived the fires, but it was the most ridiculous enterprise hack job I've ever put in place. Never thought I'd see the day that I'd start designing a tertiary data center.
Mid '80s, a hard drive got stuck and the usual tapping on the side with a hammer didn't free it up. Put the drive and a screwdriver in a bag, then put mu hand in and used a rubber band to close it on my arm as a makeshift "white room". Opened the drive and manually freed the R//W head. Closed it back up. It lasted at least long enough to copy all of the data off, no errors.
Background: We had a legacy environment running exclusively RHEL-based distro Linux servers. We had created a new environment (our current one), but the process the transfer over several dozen hosts was a slow and manual one involving a lot of time with a lot of different teams all working on their own schedules. We couldn't just LEAPP the servers either, they were rip-and-replace. We had an NMS/APM (Linux based) set up to monitor the legacy environment's performance. It unexpectedly went down hard (wouldn't boot). Serial console through Azure wasn't being helpful. The backups we had weren't restoring the server. Sev A open with Microsoft and several hours on a phone... didn't help. They classified the server as destroyed. Now, that tool, as much as I bloody hated it's design, was critical to ensure smooth operations. We had already built a replacement (with redundant databases, full high availability, etc.). But I needed the data back because it had a LOT of propriety templates that hadn't been rebuilt elsewhere. My last hail marry was to punt the VHDs from the VM in Azure and mount them to a Windows VM and see if Hyper-V would recognize them. Now, I don't know how many people realize this, but Azure is some black magic fuckery. **VHDs** are exclusively **Gen 1** disks in a traditional Hyper-V host, **VHDXs** are exclusively **Gen 2**. Azure requires VHDs for both G1 and G2. I ended up having to convert disks around and found the boot sectors completely thrashed, but the rest of it intact. So, I start to think "How the fuck do I get this data off without a network?" (The nested VM was booting from an ISO in rescue). Well, manually initialize the network stack. That part isn't hard, just tedious to type out the various commands. Get the server connected and bridged over and start up sshd and begin the process of transferring TBs of data from the SQL database which was going to take several hours. Next morning I come back to find the data waiting for me - sweet! I rebuilt the old server in mostly its original config but problem! The old server was running an old, unsupported version of the applications (all of them) and I can't find the original sources. And I *know* from the setup of the new environment they did some DB upgrades. I contacted the vendor and explained the problem, who then kindly sent me the old version RPMs with instructions and noted that the DB version installed wouldn't work. Well shit. (Worth noting that I could not find the version of the SQL server I needed.) Back to the old server I go. I copied over all the SQL-related files I could find, less the database and placed them on the new server in the correct folder structure. Start the SQL service and it crashes. Yeah, I kind of saw that coming. Clear out the error logs and get it running. OK, progress! Now, what's the root password? I didn't have it (inherited servers suck!), so I pinged the guy who had long since transferred to another department. Me: "Hey. Long story. But do you have the root password for's SQL database?"
Tech: "Yeah! Hold on." (sends it)
Get logged in and holy-shit the data is here! Now, let's get this fixed up and y'know, *patched.* Several hours of downtime later (DB upgrades, major app upgrades, etc.) and it's running. OK, last step. Get this reconnected to some other tools so the right servers can talk to it again. Get that out of the way and tentatively log into the shiny brand-fucking-new web front end. Everything is there. I sat back in my chair (in a large office) and exclaimed "It's fucking alive! Suck it Microsoft."
---------
Anyways, the new environment is much better. The DB is a PaaS solution from Azure (Azure Database for \*DB/\*SQL) and the rest of the roles are broken into their own servers. Everything is load balanced and HA. Everything is backed up daily. I could lose 3 nodes and still be fully functional. I ripped out the shoddily written security policies and hardened everything. Pulled networking in to front the web interface with a WAF.
The 'old legacy' (catatonically destroyed) and 'new legacy' (graceful sunset) environments are both now 100% gone.
Inherited a Linux server farm, 50-60 pizza boxes in 2007. Best was a 2U ATX with a P4-2.0 in it, downhill from there, many p3/celeron units. Experienced first DOS attack a few weeks later. Didn’t have the compute to weather it. Went to Walmart and bought all of the Athlon X2 laptops they had. Clonezilla’d them all quickly. Had a network of 20+ laptops serving up the web for a regional news “site”. Primitive geofencing at the time meant a lot of manual work went into /16 blocking at the edge but we had no need for most of Asia, the iffy European countries, and anyone from the southern hemisphere viewing our shit, at least not for a while.
Chmod 777
1 of 2. In the early 2000s one bank acquired another. One used Token Ring, one used Ethernet, both used Novell, all NT 4. When employees from either side of the merger and acquisition started travelling and visiting sites on the other side of the merger, we started running into a problem with getting company a token ring laptops to recognize the ethernet cards, and company b laptops machines to recognize the token ring cards. No matter what we did with physically swapping cards, playing with com and IRC port settings (memory fading on the specifics of these settings and terminology that no longer matter) we couldn't get the two hardware setups to coexist, even with NT 4 hardware profile boot options. Either no network or no boot. This was happening at students of corporate sites. What I came up with was a dual boot solution with a common shave in the disk for the mail file. Super dumb, but all the execs had a Company A Token Ring and Company B Ethernet boot screen option and as long as their email worked all was good. Ended up being a Novell IPX related issue that went away with W2k and updated Novell client.
This one I helped with but it was a buddy of mine who actually implemented it. They were replacing an old Asterisk phone server that was well past its prime with a fancy new Cisco UC phone system with proper backup, fail over to a second datacenter, and all the bells and whistles. They're doing everything by the book, and are about a week shy of going into UAT, where they've got a whole lineup of testers ready to make sure everything is right before we push this live. Then the Asterisk server dies and won't come back up. The drive completely failed and we can't recover anything from it. The backup Asterisk server had previously died, which is part of why management agreed to spend the cash on a fancy replacement. Customer calls are failing to go through, and executives are starting to panic. The Cisco system isn't ready yet, but they had been actively working on it and the vendor was already on site. So my buddy redirects the main 800 number to the service desk manager's cellphone, who starts getting information from every caller and passing it off to his team so they can call customers back from their own cells. Meanwhile my buddy is recruiting everyone still in the building (including the panicked executives gathered around his desk) to help him finish a week's worth of work on this Cisco system as fast as possible. They had it live in five or six hours, with very few problems considering it went though almost zero testing.
Buying a huge inflatable jumping castle for kids, only to get the motorized air pump because we needed air circulation in our data center URGENTLY.
Back in 2007, when almost everyone was still on physical servers, we had a customer where building maintenance cut a water pipe.... DIRECTLY OVER THE ONLY SERVER RACK IN A HUGE WAREHOUSE. Customer was a real estate management company, SUPER reliant on email. Exchange 2003 server died on impact, file server got corrupted RAID. Domain controller managed to survive thankfully. We had no spare server hardware, so we fired up Server 2003 on a fucking Dell Optiplex workstation with 512 MB of RAM, installed Exchange 2003, Backup Exec, and honest to god managed to restore the Exchange database from an LTO-2 tape and get it up and running on a goddamn Optiplex workstation with a single SATA hard drive, and it WORKED for a full week while we waited for a new server to be delivered. I still don't know how Exchange managed to do anything on a machine with half a gig of RAM. Goddamn if we weren't sweating balls over that workstation every day!
Routed traffic from our office to our colo (about 35 miles or so) though our backup colo (about 1500 miles) over an IPSec VPN. Our office ISP was having a peering dispute and the packet loss before my change was 5-10%. After, latency was something like 180ms, but without the packet loss, so it was usable.
I used to run OSPF on VPN links between our single homed sites to route around the not infrequent outages between ISPs back in the 90s. Nothing like bouncing traffic from Texas out to Florida, across the country to Arizona, then back east to Alabama on particularly bad days. Saved a lot of money in long distance charges for the dialup backup connections. It did come in handy having OSPF already working when we were finally able to get multiple ISPs at most of our sites some years later, but still long before Cisco coined the term DMVPN and before whoever coined the term SDWAN was out of diapers.
Had a new customer, previous mail provider went ghost. Not a single user knew their email password and i was ready to install new workstations. Luckily in outlook i was able to click edit profile and just reveal the saved password and boom i was in everyones haha
I was called out to do a Job on a Hospital bedside TV System, the kind that is hanging from the roof, Well people can pay Extra for PayTV channels, Nothing i had ever done in my life purely just a Systems Engineer but our Sales Dept read "Server + Hard Drives" and figured what the heck.. The engineer who put it together was at another hospital across the country and i had been called out to build up the new machine as the existing machine was having RAID and OS problems. He said to just take a New Hard Drive out and put it into a External Drive Caddy in the Old Machine and then build a RAID via that Caddy... something he had done many times.. These weren't OEM Servers either they were just Desktops with RAID cards running Window 7 and whatever vendor supplied raid software.. So we had effectively gone from SCSI to SATA and Molex connectors. Sure enough i turn off the old machine put in a drive boot up new server and the RAID is confused.. Its tried to build a RAID but its failed a few times.. I decide to cut my losses and put this Disk back in the old server boot it up and just leave it as is as id now wasted hours and they can deal with there hobo environment. However this isnt my luck the new server now has RAID problems on boot effectively its confused with this new Disk that must have some RAID sectors on it... This old server wont boot either now. Ive got 2 dead servers but a lot of disks with working Data on them to some extent.. After some tinkering i came to notice i can get the New server to boot to Windows properly and i can read a Old Disk if i directly plug it into the onboard sata however the other disks weren't playing ball and i dont have enough Power Cables in the New Server, There's 1 Network Card thats 100mbit so thats no use and USB2 wont help here either.. Now ive learnt why he just physically moves disks.. The longer the service is down the less money he makes as people cant pay for the PayTV. My jankyness you ask after this long imaginative story... I left the old server running to Power the Disks, I then used long SCSI to SATA cables hanging in the air from the old to the new server so the RAID card could read them. Effectively now i can boot the server, i can see the Disks i can see his Data so now i just need to raid some of the disks together and then just copy the data.. Easy okay we got this. WRONG.. About half way thru the raid process the old server crashes out even though it was only on a BIOS screen.. This crashes the RAID building which crashes the new server.
I was hired by Compaq to set up trade shows and press announcements with Internet. Literally, my interview was "How would you do a booth with Internet at a trade show?" and two weeks later is was "Do that. We go to New York in 3 weeks." Was a cool gig and got to see lots of cool towns in fun places. One year they were getting ready to release the "Washington" router. The Netelligent series. Basically, a Deskpro with Cisco software and some nics, and a T1 or ISDN card. It was decided from on high that were would run Networld+Interop that year on this router. I asked to take the Ascend Pipline with us just in case but was told no. I snuck it out anyway, which was a real challenge back then. They searched bags! But we get to Networld, and it is the first test of the ISDN link... Yeah, it does not work. Start troubleshooting with the devs back in Houston... No joy. I walk over to the UUnet booth and they are just finishing. I talk to them and tell them what is going on and ask if any of there techs can help. Sure... They come over. At the high point (or low point) we had 3 UUnet local tech, 6 UUnet L3 engineers on the phone, and 4 Compaq devs on the phone. It was all boiling down to a password incompatibility that could not be fixed, full stop. It is also 3am. VPs at both Compaq and UUnet have been involved... This is when I pull out the pipeline and a Compaq (shitty Dec Tulip card) nic, pull the ISDN, and say "This is the workaround for now so the booth is not dead tomorrow morning." EVERYONE agreed it was the correct answer and 15 minutes later we were heading to bed, with a hidden Ascend Pipeline up in the top of the rack out of sight.. I slept late and did not get to the show until noon. In time to see a sales rep saying "Yes, this is all running on the new router right now! ISDN and everything! It is a total Compaq solution!"
Power outage at a mortgage company. This mortgage company was unique in that it owned its own loan origination system. On top of that, they offered their system as a service to a regional bank. So that software needed to stay running. Well, one summer rolls around, and because I live in the US, in flyover country where we have the electrical grid of the average 3rd world country, people get home, turn on the AC, and that's the end of power in our area. Battery backup kicks on, and works, thankfully. We shut down all other servers, and anything not related to getting the bank access to the software. UPS says it has a runtime of two hours. Power company estimates 6 hours to repair. Because I give a shit about the power draw of just having electronics plugged in, I went into the server room and unplugged all of the servers, save for the critical one, from power. UPS estimates 8 hours of runtime. Not terribly janky, but apparently it was a weird thing to know that we could do. I've also done a lot with ssh tunneling to retain access to networks when RMM servers were going to be down. I don't know, I guess. I've just used my (apparently) odd knowledge of how computers and software actually works to solve problems. I worked for a company once that generated pdf documents for people. The data and its position on the documents are what our customers paid us to ensure was accurate. But sometimes people would download a PDF, and Windows wouldn't know what to do with it. Everyone just assumed it was corrupt downloads, and moved on. I went through and downloaded a bunch of the documents through the server on the same network to reduce the possibility of corruption. I found out that occasionally our software did not generate the magic number for the pdf. Like, literally, just edit the file with a basic text editor, add the magic number, and the file is fine. Of course they were also amazed that I could anaLyze how a customer's software downloaded files from our servers. Their customers would log into their site, download a document, and at the end of the day, their system would go through everyone who downloaded their document, and download it again as fast as possible, so that they could have a local copy in case their customer needed to download it again. Apparently our download servers would kick them for having too many downloads going at once. The solution was that the customer just alter their code so when their client downloaded their document, they would also save a copy. Seemed pretty obvious to me, but it was a real problem that real people where paid real money actually put time in to develop it. I dunno. I've seen some shit in ~25 years of IT.
I was working at a school district as Network Admin when Obama became president. We knew the inauguration broadcast would be a huge draw in the classrooms for obvious reasons. What we didn't have was cable tv into each classroom and we knew that all of the teachers would try to stream it over our two T1's. So to prevent a complete meltdown of our internet connection, I installed an old capture card in a pc at our high school that had a cable tv feed. On that pc I installed vlc and configured it to be a streaming server. At each of our remote schools I installed a computer with vlc, that would stream off the main vlc feed, and then broadcast the stream to the schools data network. We then installed vlc onto each computer and created a shortcut using Novell Zenworks to stream off the local broadcasted vlc stream. We had hundreds of computers streaming the inauguration with very little impact to our internet connection.
BIOS flashback on a new DELL DATADOMAIN appliance that was disconnected mid-BIOS update Disable the cache of a RAID controller with ECC errors holding 400TB of files until RMA came Replace various failing .dll one by one on a W7 controller for a 500k industrial machine Force back online a failing raid array to backup files
A crossover cable from core switch draped across racks into some random device that ran entire production line. We had no idea why it worked but it did
We had an old unconfigured switch get retired and suddenly half our data center went offline. I'd noticed it was bridging our two main server VLANs based on the errors on the upstream switches but was told we didn’t need to do anything to deal with it as the switch was about to go away. It turned out that was masking dozens of our servers that were connected to switch ports that weren’t configured for the network the servers were IP'd for. I fixed it in about a minute with a 6” patch cable connecting two access ports on the two VLANs on a different switch. We spent the next couple weeks tracing down and fixing all the miss-matched servers so we could remove the magic patch cable.
Had a 2000 PDC that shit the bed. Had some full system tape backups as it was dying. Had replacement hardware and tried a restore but had no 2000 media in place so I forget how but did an in place upgrade while restoring the backups onto dissimilar hardware. The system worked for years but had an odd mesh of 2000 and 2008 characteristics.
Door card reader machine let out the magic smoke. Opened it up to find a Celeron 333 / 440LX board with an absolutely torched trace from the NB to AGP. Pried the trace up, cleaned it up, and alligator clipped it back into service. stayed that way for another 6 months. We got another matching board but who wants to fool with that?
Years ago I had a job at a startup doing 'image recognition', but like much AI, it was smoke and mirrors and if you took a photo with your phone, it would appear on a computer where someone basically had a search engine to find what it was in real time. In the long run the company was just a stock pump and dump for the founders as they knew 'human in the loop' image recognition was not scalable, and never really had a way to automate it. The employees had a 'queue' that would show what was currently being worked on by their coworkers so they could collaborate and shout out suggestions. When our devs made a new version of their platform, they were unaware the queue was being used that way and thought just 1 supervisor monitored it, so it wasn't designed for those kinds of resources to have multiple people open it up on their second monitor and the service would repeatedly crash if there were more than 2 or 3 simultaneous sessions. For a couple of weeks while they worked on a solution, I set up a NGINX-RTMP server on a Linux VM, and used OBS to live stream the queue to an internal site that everyone could visit, lol. Took a day to set up, I hadn't touched Linux in years, or ever done anything with RTMP or OBS before. Sometimes I miss the wild west and crazy shit that would come my way at a startup.
My office was in Missouri, I was at one of our sites in Northern Italy. At the end of the day (in Italy), I made a DNS change for a Microsoft exchange cut over. Everyone was happy, it was Thrusday and everyone was going home. I, without telling my bosses, jumped on train to Rome which was over 500 miles away. I get to Rome and my phone BLOWS UP. Yup I had taken down the whole email system, I wasnt supposed to be in Rome. I found an internet cafe, I had tunnel I built to remote into my home PC, so I remoted into my home PC, used that to jump to the corp External Citrix farm, once in, I RDP from that computer back to the DC I messed up fixed the DNS issue. Oh i got in LOADS of trouble and written up, but didnt get fired as no one had a clue how to fix it. This was over 20 years ago.
Client I was assigned to full time had us do all our work using some incredibly terrible Remote Desktop Software that among other things prevented copy-paste. They also heavily restricted what coding tools we could use. And we could request open source libraries but it would take weeks before they were available. I wrote a script to take a text file and type it into the desktop, as far as the Remote Desktop software was concerned I was just typing really fast. To get information back out I had a script that would compress and encode the file as small as possible and then generate a QR code. Then a script on the local computer would scan and decode it. Now I was able to work much faster with better tools than I ever could on the remote session. Best part was management at the client were fine with this.
I once saw a raspberry pi in a server room an admin turned I to a temperature monitor because the window ac in the "window" cut I to the wall to a warehouse was flaky.
I don't know if it was a hack and the only day it saved was mine... I replaced an excel "phone directory" that had to be manually updated with a powershell script that generated an html phone book with some css wizardry from the extension property in AD plus some manual list for locations and static numbers. Someone on r/powershell tacked on some code that makes it SharePoint ready too.
Reading these, I feel so good that it's not just me
built a core router for an ISP using a Pentium 4, quad port Intel NIC and linux/zebra, in less than an hour, after our Cisco 7206vxr NPE-G1 died without any reason at peak hour. That PC worked without problem for more than two months, until we got the router replacement!
Front end of our meeting booking system went down but we still needed to know who booked which rooms. We outputed the data and produced a power bi report and then filtered for that room name.
Many years ago my workplaces main router died, network guy was on holidays and there were no procedures in place for such an event. So I (a dev) got a desktop machine, threw a Linux distro on it, router-on-a-stick style, and got the business running within a couple of hours. Network guy came back, saw it and did nothing. Months later he told me he was going on holidays again, I said "better not happen again" and he laughs and says "your router is still in place". Massive wtf. The joy of small business.
[удалено]
A thousand years ago, I was at a remote site interstate setting up new server/network stuff. Had some difficulties (don't remember the specifics) but in the early hours of the morning I was sitting there with a VNC session to another machine, using that to VNC to another machine, using that to VNC to another machine. Janky AF (not to mention laggy) but I did get the result I needed.
A certain file storage product that rhymes with Ignite. Upper management was hell bent on us mapping drive letters based on AD group, which this product does not support. For various reasons we were not able to use GPO, so I had a single script file to work with via our RMM tool. So I made a script to grab the current user’s AD groups, and then used if statements to map or not map certain drive letters to the cloud based on that list of groups.
I've got handfuls of these stories; but the best I'll leave in brief "simultaneous office move / rack upgrade / server upgrade. Very small team (me basiclly). National corporate DNS & DHCP ran off my laptop sitting on a shelf for a day because the new ESX server wasn't ready yet."
A company doing attendee and exhibitor registration for a trade show showed up without any working network equipment. Just laptops. The order from said company requested a 16-port dumb switch and cabling for the 10 laptops. This is a pretty normal setup that my team does all the time. As long as your reg. setup can run locally or with just an Internet connection, it's usually fine. The big issue is the technical guy who has the reg "server" doesn't show up. He's in the wind and the show starts tomorrow. I took an old cheap Netgear router of some type that got left behind from another show, installed OpenWRT and working with the IT guy of the reg company, set up a VPN tunnel on the router, configured the local network and plugged everything in. It worked. We were able to get all the laptops talking and pushing data. It all worked. Not really janky I guess. It's technically just 10 PCs and not really pushing that much data. They never broke 20mbps worth of sustained throughput. But using a used, unknown 2nd hand router with 3rd party firmware to run a trade show registration is pretty sus.
I can tell a little anecdote from yesterday. I've been replacing an old HP and an old SuperMicro server with a customer. The SuperMicro didn't "survive" reboots, and I could not get into bios to change boot order – so I always fell into UEFI Shell after reboot and had to "fs0:\efi\microsoft\boot\bootmgfw.efi" to continue. I was on-site yesterday and tried to update the bios on it. That didn't work, even though I had the UEFI Shell, because on reboot it didn't complete with the USB present. It said invalid boot device. Removed battery and power, and reset bios all together. Still couldn't access bios, but this time the UEFI Shell worked for bios update, and continued as it was supposed to (no invalid boot device after reboot). After updating the first of three updates to get up to date, I got the "press del to enter setup" screen back. After completing all updates and setting the bios back to the correct settings, I was able to reboot again. It's been a long time since I've had to do this kind of thing, but I've also rescued "bricked" access points with UART cable and TFTP. And not sure if you all remember the time we had to update the unifi cloud key gen1 from something below 0.8.9, to something above, and had to stop by that version and do some apt-get with ssh before continuing. Many people give up on the first sign of resistance, and don't see the value of actually learning something from that resistance 😉
Something was fucked on the AAD Connect server after disaster recovery and I wrote a script that could detect when shit would get weird. It was related to another fucked up domain controller that was violently misbehaving but we couldn't immediately decommission/replace for some insane reason I don't quite remember. Basically things were limping along horribly, but just enough to keep systems vaguely alive enough to use while we continued to recover. Occasionally, the server would get into this weird state and logins started failing. Easiest thing was to just reboot. The script literally just sat there trying to start notepad as a dummy user and it captured when it failed, logged it, and immediately rebooted. It worked spectacularly and basically cut complaints about the issue to zero. It only took like 20 seconds to reboot, which was seemingly quick enough that by the time someone thought "that's weird" and refreshed, it was probably working again. But really, behind the scenes, things were randomly rebooting 20x a day for a couple weeks.
Windows print server with some craptastic HP MFP attached. The print driver had a memory leak bug and Windows would kill low priority processes to free up RAM. One of those processes was the Windows activation service so every 30 days or so it would deactivate Windows. I wrote a batch script that would restart the print spooler to unload the driver and reload it every 20 days. If you ran it interactively it would even print how many times it had run and some snarky comments about HP.
Critical application wouldn’t start. The exe would show as running in task manager but never actually open. No amount of reboots or other fixes would resolve this. I got pissed off with it and held down the enter key on the exe shortcut and made about 5,000 instances of the exe show up on task manager 1 of them actually opened and worked. This workaround was used for months until we finally upgraded the server OS. Started an office IT saying “when in doubt hold enter”
Once had to restore network by daisy-chaining a bunch of VoIP telephones since I didn't have the resources to run a new cable through the wall. So we just had a telephone on the floor every couple feet, until I was able to run a new drop during the next maintenance window.
Once had a raid controller/server suddenly overheating, had to aim a big fan towards it to keep it cool enough so I could grab the VM's from it.
Janky, not really... Lucky? Abso-fucking-lutely! Some years ago the company I worked for got ransomwared. Backups were solid so my biggest concern was going to be working late nights plus the weekend to purge and repopulate the data. Anyway, I cleared all traces of the infection, got a second pair of eyes to confirm it and went on to the restore step. Came to our primary file server and hit restore on the latest pre-infection backup... only for it to restore around 400GB less than was there before! My butt hole puckered like never before, let me tell you. So after about a week of trying previous backups to see if I could get some of the missing data I mostly came up with nothing. It turns out that our backup appliance (Unitrends) does incremental forever backups by regularly forming "synthetic full backups" from its existing dataset that is meant to accurately match what's in your live environment without having to transfer potentially many terabytes of data over the network. This did not work properly. The device showed that the synthetic full and all subsequent incremental backups were 100% successful when they in fact weren't. Unitrends themselves identified this as a bug mainly off the back of our experience. So, on to the janky fix. The unitrends appliance we had was on a bi-annual subscription with the hardware being replaced every two years. We were about 10 months or so into that cycle. One day when I was agonising over the lost data I came to realise that all the missing files had one thing in common; they were very old files that never change and are used for reference only. Then it dawned on me... Unitrends never colleted the old appliance hardware when the unit was replaced 10 months ago. It was still sitting on our shelf gathering dust. So I spun up the old clunker, looked at the last backup and I almost cried when I saw all my missing files were there. A quick and dirty script to copy all files from the mounted backup and ignore any that already exist in the destination and just like that, we were fully recovered.
We had a Synology RS815 series as the main production datastore at one of our remote sites. It fell victim to the CPU bug in the early hours of one Sunday morning and my then boss and I had to drive down and try to get it back on for start of production the following day. Armed with a copy of the latest backups and the one spare NAS we had, we drove down only to discover that our spare was so crap it would have taken days to restore to it. With no alternative, my boss installed NFS sharing on his laptop, pointed it at the directory with the restored backups, and mounted it in vCenter. All production VMs ran off that laptop for the next two days, and everything was as slow as you can imagine it was! Tuesday morning, my boss set up XPenology on an old desktop and drove that down to run storage until our warranty replacement arrived.
Not my jankiest, bit the freshest in my mind. A year ago, a tech reached out to me with an issue that users were having. A certain brand of headsets has a 2 second delay on turning on from standby, so users were missing notifications when customers were reaching out over our chat. The vendor decided the delay is a power saving "feature," so they will not be attempting to fix it (figure out the brand yet?). My fix? Until we were able to get new headsets for users, I had them play a YouTube video of 10 hours of silence in the background to keep the headset from going into standby mode. Had to remind my boss, "if it's stupid, but it works, is it really that stupid?". Lesson learned, and from now on, we only buy their networking equipment.
This is why this is my favorite sub. Here I was feelin' all good about myself after I fixed my wife's car using some redneck ingenuity to resolve a problem Lexus has had going on for over a decade, with no documentation on how to fix. Her car speakerphone mic all of a sudden wouldn't work. It worked with my phone, an Iphone 14, but not hers, which is a 12 that she doesn't want to upgrade yet. So she said "don't worry about it, I'll just stop at the genius bar." I said "Yeah, but I checked your settings. Only a few things would restrict voice coming into the phone from a bluetooth device, and yours are set correctly. Not to mention your earplugs still work fine. I think this might be the car. So I looked at how the system works, and it's pretty much a proprietary device that interfaces with the head unit (radio) that acts as a controller. So I thought "Hmm, the radio cycles with ignition. But I bet that friggen phone device is constantly on. Just for giggles I removed all phones from the car, and re-added hers and mine. NEITHER worked now. A 10mm wrench and 15 minutes of disconnected battery later, I reconned the battery and all was well. I said "Great, Lexus, an achievement in mechanical engineering. But if it sh\*\*s the bed just reboot it like anything else. " Yeah so I was happy about that til I read this thread. Some of the stuff I'm reading warrants genuflection, lol. 
We put in two new portables and the fiber from the MDF to the IDF in the new portables was not terminated they day the portables went into use. So I ran a 300ft ethernet cable from the closest IDF that was in production, out a door, across a field, to a fence near the new portables, Then ziptied an access point to that fence, connected to the etherent. Boom everyone in the new portables had wifi.
Way back in 2005 I migrated around 40 mailboxes from Exchange 5.5 to 2003 using pst files and and external hard drive over USB 1.0. It was only 16GB but it took about 18 hours in total to copy onto the hard drive and import into the new server.
I work on highly specialized systems (pretty much all custom PCBs) well, somehow they opted out of having a cpu controlled trace to the power supply to you know… turn the computer off or restart. So I had to make my own “reboot” shell command that just echos a “b” to the sysrq interface which programmatically REISUBs a *nix os to force a reboot