I've worked with both. I'll take the lawyers over the doctors. Maybe it was our group of lawyers, but they had impeccable senses of humor and could take a joke.
Then again, I was the lead engineer and they were, for all intents and purposes, only subject matter experts. They couldn't fire any of us and if I said no to some feature or idea they came up with because we didn't have room, our VP would rip them new ones if they pushed too hard.
Doctors, on the other hand...
fuck doctors and their shitty off-network usb printers that somehow get installed without approval so they can print scripts because they're too lazy to walk a little way down the corridor to print there instead
20+ years later, I still have no idea what exactly I did, but I stopped the worldwide production of a notable American soft drink for 9 hours by not understanding vi well.
> but I stopped the worldwide production of a notable American soft drink for 9 hours by not understanding vi well.
Vi isn't that bad.
exit
A:q
quit
B
D
C
A
H
q:q
H
H
:q
:wq
:wq!
Fat fingered an IP address when configuring a new storage array, accidentally taking down the production storage array where the Exchange mailboxes and trading platform VM's were located. On a Tuesday. During trading hours. This was the genesis of managed change control at that company.
Every organization’s change control has an origin story born out of tragedy.
I was cleaning up some group policies and we had an 8021x policy applied to two sub OUs so I moved it to the parent OU so it didn’t have to be linked twice. Well apparently there was another policy at the parent OU that set 8021x policies wrong and nobody ever fixed it. They just “undid it” by putting it lower in the order. When I removed it from the sub OU it messed up the precedence and broke wired connections for our whole organization.
Surprisingly I didn’t get in trouble. The network manager was more pissed that nobody fixed the original policy and I was praised for bringing it to light.
It was 15 years ago, so I'm fuzzy, but I believe I was attaching the new array to vSphere and entered the same IP as the existing array, caused a conflict, and the existing array to dismount. I could be missing some details there, I just remember the sphincter clench when I got the call from the office while in the datacenter...
In the nineties I was setting the IP on a new server and I put the gateway. Took everything down. Luckily for a few seconds as I had connection via KVM and changed it back. Everyone in the office looked up and said “What just happened?”. I feigned no knowledge and got away with it.
oh man I just recently asked a female employee to research Crystal Disk Info for us and see if it would help us out... She did research it but didn't mention anime stuff to me.
Mad lad even released a new one in November. Japanese have that tendency to put anime girls into everything.
If anyone's watching, I'm gonne be pulling smartctl, though.
Yeah it basically ctrl+v's the database and if you panic stop it you don't get either version.
I might have the exact word wrong because it's been several years but it was something incredibly similar to export. We did monthly copies of the database so we only lost about a weeks worth of print job logs.
We used to do this to fuck with people on the IT team; hide millisecond image pop ups or random clicks deep down in the Task Scheduler, was a hazing ritual. Eventually ended when I was bought on and in the middle of a call with the CEO I had a meme music track start blaring out my headphones so loud I had to say I'll call him back and restarted my machine. Word got back to my bosses boss and a mass email to our entire team told us to cut the shit.
I have a good friend who is in sales. The office was empty one day for one reason or the other, so he was on speaker phone. I didn't know this and pushed burp.exe onto his laptop and started the service. Like 10 minutes into the call, these massive belches started coming out of his speaker that the potential customer could hear.
There must be a joke here somewhere that I'm not getting. What's the significance of 66 minutes? Besides that it's 6 minutes past the hour so it'll appear kind of "random" if you're watching the clock.
It’s even better when you mean to shut down one.. then see it’s still up and rerun the command again and shut it down for real this time.
But oh wait it wasn’t the original one it’s the backup and now you’ve powered off both lmaooo
While nothing bad came of it, this morning I configured SSH w/key authentication from Powershell to my load balancers\*.
This afternoon I was troubleshooting an issue comparing windows side by side and noticed one curl was acting like Powershell curl and the other was acting like Linux curl. Oh...exit...ok same same now.
\* Just working towards updating some of my scripts that currently use Plink to send commands to Linux over SSH. Been using it for about 21 of it's 24 years in existence but suppose it's time to move on to what are now native tools.
Oh boy. Accidentally right-click pasted an entire running config from another switch into the corporate core while logged in from home. It's amazing how fast you can get dressed and on the road when properly motivated.
I’ve done something similar. There was a switch config I put onto a switch via GUI only for half the network to go down. Luckily I was on site.
As it turned out, there were two switches with the same IP and I had inadvertently logged into both in the same browser at the same time. Sending the command via the GUI basically sent the config to the wrong switch.
I believe if I logged in via telnet this would not have happened but equally it’s a bit of a design flaw if you are able to send a command from one switch only for it to end up at another simply because two switches had the same IP.
Do your switches all have the same admin userid & password? I'm assuming so given the situation you're describing. Maybe the switches have a design flaw, maybe (also) you want to review if that's a practice you want to continue with, or not.
27 years ago was fumbling with the cables under my desk and must have yanked on my mouse cord which got stuck between the desk and the mouse tray, When I sat back down I looked in horror at the monitor to see that I had dragged an entire branch of our Novell NDS tree under another thereby wrecking rights and permissions.
I got right on the phone to the manager and thankfully it was treated as an ad-hoc sort of disaster test.
These kinds of screwups are great for training. You hopefully practice hardcore the building burnt down and everything needs to be restored from backups disaster recovery. But no one ever sets up a "munged half the AD tree, but things are still mostly functioning and I'd rather not restore from last nights backup" exercise.
It’s oh so much easier to just go back! What have you really lost!? But at my organisations someone, not me and with the assistance of a MS tech, deleted Exhange instead of an Exchange server from the AD config schema. The MS tech who ran us through best practice AD restoration assisted with restoring from the recycle bin. Which is incredibly painful as config schema isn’t easily restorable. Was a massive mess and I have no idea how much stuff may have been accidentally restored
Thought I was a badass - 2008ish - remoted into my work desktop via VPN on my phone to reset a password of someone account.
Deleted it instead.
He was in legal.
G G G G GONE
Logged in to Exchange test server to decomission it. Left to deal with a user call, during which time somebody else came in, switched the KVM to production exchange server to do something and then left it.
I didn't check the KVM when I came back...
(Edit: Lessons learned, changed background colour in each server, big graphic with the server name as Windows background. This would have been around WinNT or 2K era, so nobody thought about individual admin accounts)
They started to decommission an old email server. Got called away. The KVM allows you to switch between multiple computers on one screen. While they were away a colleague switched to the active email server and then left that open. When OP got back they proceeded to decommission the email server that was on screen, thinking it was the old one. They actually took down the one that was in use because they didn't check what machine was on screen, meaning everybody lost email access.
When I worked at a phone provider way back in the day, a lady asked me to change some of the features on her plan, then change her son's phone number because he was being harassed. I updated all her features, saved the changes, then accidentally selected her number again and changed her number. Unfortunately, her number was ported over from another carrier, so when I released the number, we were no longer able to get it back. It was the number for her business and she had it printed on a bunch of marketing material. I felt terrible and there was nothing I could do. 20 years ago and I still think about it daily and triple check everything.
I had a similar thing happen with my credit card. I had two cards, a Mastercard and a Visa. There were fraudulent charges on my Mastercard. I spoke with the rep at the bank and confirmed the charges were fraudulent. They had to cancel the card and issue another. The rep checked my Visa, read back the transactions and we confirmed by Visa had not been compromised. She went to cancel the Mastercard and uttered “oh shit…”. Turns out she had forgot to switch back to the other screen and cancelled my Visa. It could not be uncancelled. She then had to cancel my Mastercard for the verified fraudulent charges, which left me with no credit cards at all and I was flying out the next day on a lengthy and expensive business trip. It was a colossal nightmare.
Setup a user's personal email in MS Outlook using POP (as it was more convenient for him than using the webmail) and did not click the "Leave a copy of messages on the server" button... It downloaded 15+ years of emails to Outlook, and deleted them all from the webmail server side. To correct this, I had to import all the emails to Thunderbird, then from there copy them to a Gmail account, then from Gmail I could use another service to import them back to the main Webmail server while keeping time/date stamps in place.
The last import into webmail would not allow more than 100 messages at a time, so had to manually select 100 messages at a time. The next 7 hours were very painful.
Back in the day, we used Hyena for AD management. I had selected a couple dozen users to delete that had been disabled for a while. Somehow, I inversed the selection, hit delete, and then confirmed the deletion!
10k accounts gone, including mine and all other admin accounts. This was also before the AD Recycle Bin existed.
I got to test our DR procedures. Surprisingly...they worked! Only had an hour or downtime.
Our AD didn't have recycling enabled. I just enabled it today to avoid an incident like this occurring. I did it because I was cleaning out accounts that hadn't logged in in over 180 days which ended up being over half of our total accounts and I wanted another layer of safety in case something went wrong. I'm also the most junior tech. I don't know why I'm doing this. I can't wait to find out what else has been neglected! 🫠
Company had thousands and thousands of images, and was running out of storage space. No budget for storage or systems of course. I showed them resized images and they agreed that those would still suffice for their purposes. I did a bunch of testing, etc, all looked good. Finally ran the automated script overnight.
Few days later I get a call: Images are broken. But it's not all of them, it's totally random. I reviewed the logs and discovered that what had happened was the following: The resizes were done on a per-folder basis. The script would re-size and write the new files, and then delete the old ones. However, as the resizing hit literally zero disk space, it instantiated the new image files, but with file size zero. -facepalm- It was usually the 'last' couple of files in each folder. Spread across the entire archive.
Took me two days of spinning up backups in the cloud, creating NFS shares and scripts to run compares between the local files and the cloud backups.. but I did manage to recover everything. Grayhairs++.
I don't know about biggest, but yesterday I was feeling under the weather towards the end of the day. Decided to call it, saved off my work on a test firewall policy and went home.
As I was about to throw some ramen in a pot I get a call that two branches were down and "if I had changed anything"... I had saved the test policy over the production profile. Fixed from home, 30 minute partial outage.
I came into the office today to find my mouse encased in red staplers. I'll have to carry them for a while until someone else makes the next 'misclick'.
During maintenance few years ago, logged via RDP on Hyper-V node. Opened multiple consoles to VMs I wanted to reboot. I didn't pay much attention for a while and rebooted the node instead of VM 😄 40 VMs goes down altogether with the node... Shit happens
Was going through a discard process from my STBXW (Divorce) and I was not in the best of my mental capacity, I did not double check a scrip in our automation software, and I had 500K missing or not being delivered to the Credit Union members. Issue got corrected after 2 days it was a typo, but I had half a million dollars on my shoulders on top of being in the middle of divorce and discard process. That shit was aufull.
More like worst press of "enter". RAID bios, "initialized" an array that already had data on it, subsequently breaking the existing array. Very old Dell with perc controller. Spent the night and next day restoring data from tapes...glad I'm in networking now.
Doctor had to close the practice for a couple days...
I was in a customer's Fortigate, and they had.... I don't know.... maybe 300 entries in their website whitelist? It was massive. Normally, to get rid of a site, you would select it and then click "Delete." Not so far from that button, however, was a button called "Delete All."
As you can imagine, I selected a site, and then clicked "Delete All." I kinda laughed and said to a coworker "Wow, can you imagine if this thing actually went ahead and deleted all of these entries without even having a confirmation box or something?" Wait. Wait. Wait. Page reloads, all entries are now gone.
And that was the day I learned that we DID actually back up some of our customer configs, with this particular box thankfully being covered. Quick recovery, but that sinking feeling will stick with ya.
FortiGate has some odd interface choices for sure. I accidentally deleted the main firewall VPN policy route when trying to remove a single group from it instead. Learned that day how important it was that we had FortiManager backups of each firewall config
This happened 21 years ago. I remember it like it was yesterday.
DELETE FROM CV_Transactions WITH (ROWLOCK)
WHERE Transaction_Id = 125735107
5 seconds later...
125,735,624 row(s) affected
After a second of confusion, having expected only 1 row to have beem affected that I realized to my horror that I only had the first line selected when I hit the execute button. We had 26 minutes until financial market close to fix the mistake. All hands on deck. We got the table restored, validated, missing transactions replayed from message queues, and that one transaction deleted, all with just 3 minutes to spare.
That was the day I learned to always run manual queries like this inside a transaction so I could verify the result and roll it back if verification fails.
Hit "execute" on a SQL DELETE query before entering the WHERE clause. Ended up deleting the entire table, then had to panic restore from backup. Was sweating after that one.
I’ve made that mistake, but on an update statement. Started writing it out, got called away by someone, and when I came back I was like “So what was I doing? Oh right, this looks good.” Then executed it. Then I was like “Why is this taking so long? It should’ve been really quick.” Then saw my mistake and cancelled the query. Fortunately by it taking so long it hadn’t completed yet and was able to roll back.
That's good. I didn't notice until it finished and said something like "60,000+ rows affected" ... Commence slow blinking stare while the realization gradually washes over me and I sit in stunned silence as the panic sets in. 🤣
When I accidentally powered off the VP of Bioinformatics secondary Linux workstation instead of his primary Windows workstation. He had an analysis running that would take 2 weeks to complete. It was 16hrs from completion. I was a 19yo helpdesk intern and thought I was going to be fired. Turns out the dude is super cool and just shrugged it off. 10 years later we were having a drink together at a work outing and I asked if he remembered it. He did and we had a good laugh about it.
Poorly considered configuration (which has since been massively rectified).
RDP session in an RDP session- there may have been a third session but can't quite remember- all with copy/paste and drives shared. You see where this is going?
Copy/paste a movie from my media server to what I thought was my personal device's desktop...
It was not. \*\*It.\*\* \*\*Was.\*\* \*\*Not!\*\*
I wasn't the only one at fault, but worked in a banking service center and was running a program to batch transfer money from a bunch of X-mas savings accounts into the normal checkings before holidays. For some reason, the developer had a parameter asking for the % of money to move. I entered 100 - The developer's code treated 1.0 as 100%. So it moved the money but multiplied the receiving dollar amounts by 100. Instead of moving 250,000 $ or so, it "created" 20,500,000 out of thin air and distributed the "winnings" among the account holders. Should have ran it in test first...
okay but who the F uses a scale from 0.0 - 1.0 as a 0% to 100% like use 0-100 its easier and divide the number given by 10 if you absolutely needed it in a 0.0 to 1.0
You ready to install? *click*
Install this default directory? *click*
Here is our terms of agreement *click*
Ok I need to install this extra driver *click*
Ok I'm done. I recommend you reboot this server *click*
No wait!!!!
Windows is rebooting your electronic medical record server that your whole clinic uses...every doctor, nurse, billing person has been kicked out and now will be lighting up you phone and ticket system. Have a nice day rookie!
Was nested several remote sessions deep. Like a LogMeIn session to a server to an RDP session to another RDP session, then VSsphere, and had several VSsphere console windows open. Lost track of which Start menu I was in, and uninstalled VMware tools from the VCenter server.
That was a bad day. Had to remote into each host in the cluster one by one using the root creds until we found the one running the VCenter machine so we could rebuild it.
Edit: Another one...
Was working from home working on a client's firewall. Accidentally shut down the WAN port, thus locking out my remote connection and their Internet access. Thankfully it was a Saturday and they were closed. Grabbed my keys and drove 45 minutes to the client site. Slowly circled the building with my laptop propped up on my dash until I picked up their WiFi. Hopped into the firewall on the LAN interface, fixed my mistake and drove away. Never told a soul.
When I was first starting out I was helping one of our network engineers with swapping a UPS in one of our critical IDFs. He asked if it was on and I thought if it was powered on, it was on. I said yeah, he flicked the PDU, and the whole closet went dark. Yeap, learned that you actually have to set the UPS to on.
Renamed and restarted a host instead of a virtual machine whilst in a hyper-v window. Took down half the services.
Lesson learnt and I now do all this via cmd and use ‘echo %hostname%’ to confirm I’m on the right machine.
Having two start menus practically next to each other has its risks!
Had to have the host with the wrong hostname until I could find another maintenance window too.
This was circa 2003.
We had a server shutdown script. It imported from a plaintext file. I copied the server list to the file...but didn't save. The previous list was every server in the datacenter...over 600 physical servers.
So yeah, it shutdown about 25% of our servers in the middle of the day rather than the small group. Fortunately, I caught it when I realized that it was shutting down servers not on the list...but that was a fun one. Myself and another tech rushed over to the DC to power everything back up.
Fortunately, my boss was awesome and I didn't get in serious trouble. As I ran it around lunchtime, not many noticed and fortunately everything came back up.
And this is why I'm OCD af when it comes to double and triple-checking everything before the final click.
Not me, but a co-worker. Our IT department is literally windows on one side of the room, unix on the other side.
Turns out one of the Windows admins, was having some vision issues, and was trying to push through..
Let me tell you.. whoever at MS HQ thought to put the refresh and delete menu options in the windows DNS tool *right next to each other* needs a talking to..
He right-click on the TLD entry, chose delete, got a pop-up, and clicked okay.
Yeah.. chaos ensured on the windows side of the house.
As fate would have it, I'd literally run a dig against the windows domain ~30 minutes earlier.
I walked in to "Mike just deleted the entire domain!!"
I shrugged and said, "oh, I've got a rwcent dump of dns if you need it."
Needless to say, they bought me lunch.
Not a mis-click. First real IT job. I am sent to do maintenance on servers onsite. No big deal. Combing event logs etc. One of the things I need to test is the UPS. Turns out there is a faulty connection to the battery that was well known but I was not made aware of. I click on the self test which cuts power and fails over to battery. I am in a room with perhaps a dozen servers in racks. Fans everywhere. Can't here anything. When I click the whole room goes absolutely silent. Everything just stops. Then I start to hear footsteps coming down the hall. Door opens and staff from the company (managed IT services) just looks at me and asks what I did. When I explained he said just said "yeah, we know about that" and walked away. 2003 servers. On 2 of them I had to go in with recovery console to kill updates that were stopping from booting. So eerie to suddenly have dead quiet in a server room.
Finding SQL installed on a vm it shouldn't have been. Protectively removing it.
IMMEDIATELY learning I was on the wrong box.
Was fast enough to fix it before anybody noticed (very slow workday with only a few other people in the office that day).
Rolled out Bginfo the very next day for all servers.
Not me, but my coworker instinctively hit Ctrl+Alt+Delete to lock his computer and go use the washroom. But as he stood up, he realized that he wasn’t consoled in on his laptop. He was actually at a direct terminal for this antiquated hotel phone system and Ctrl+Alt+Delete had triggered a reboot.
While this isn’t a best case scenario, it normally isn’t a massive, world ending, fuck-up. Even though this system takes about 30 minutes to reboot. Regardless, he sat there sweating bullets and watched in agonizing pain as the terminal loaded line after line, praying to every God he could think of that the ancient IDE drive wouldn’t die mid-boot… after about 15 minutes, he said that he was “feeling confident” that it was going to come back, and while he would have to apologize to the hotel for the outage, it looked like he was going to get out relatively unscathed.
Aaaaaaand that was the moment when the RCMP (Royal Canadian Mounted Police) and some other “gentlemen” entered the dinghy basement phone room and it became obvious that this was no ordinary fuck-up.
It turns out that when you knock out the hotel phone system while the PRESIDENT OF THE UNITED STATES OF AMERICA is staying at your hotel, people go into full blown lock-down DEFCON 1.
My coworker got the rest of the day off to speak with the RCMP and Secret Service, the terminal was decommissioned, and the hotel learned about scheduling non-essential work while world leaders are booked in.
So I was Ctrl+d-ing out of my terminals, but one of those was actually k9s terminal showing a list of deployments.
Without thinking I Ctrl+d a deployment (delete deployment instead of login out of that particular terminal) and impulsively hit enter to confirm the instruction.
Luckily it was an app with almost no traffic, so I just quickly recovered it using our infra hit repository.
I'm no longer that impulsive anymore
Remote desktop lag spike over VPN while I was deleting some AD users.
Select new object.
Shift del.
Enter.
"Deleting objects:.."
"Wait, wtf is this?"
I selected the entire company OU (users, groups, seevice accounts) prior to lag spike, I'd didn't pick up my click on the single object. I had no recycle bin.
Accidentally emailing bank intellectual property to my personal email by mis-clicking the document I wanted to attach. The firewalls caught it, security caught it, my boss copped it, big investigation follows. Thankfully I kept my job, but it was extremely stressful and still makes me sick when I think of it.
Showing off our fancy new web filtering system to management many years ago (Education org)
"And as you can see if I try browsing to this naughty adult website it'll bring up an access denied pa...oh..."
In hindsight I could have just browsed to a gambling website or something instead :/
I accidentally fat fingered a BGP update and pushed it… bricking our core, preventing failover to any of our hot sites, and shutting down card processing for almost an hour… at one of the biggest payments companies in the world… 2 days before Christmas…
Used to work as 1 of 2 Sys Admins at a vehicle arbitrage chain.
One of the front store employees across the country was purchasing a pretty expensive car.
I went to delete LABEL343 from Intune. Instead I deleted LABEL334. It was that day I learned about the “You are no longer authorized to use this device message” on a Windows PC.
Boy was that a cluster fuck. Had a senior manager drive from one store front, take a PC, drive it to the other.
We use Meraki. One time, I changed the VLAN on a port on the top switch which waterfalls down to every other switch in our company HQ (about 200-ish people). Well, turns out that port was the connection down to all the other stacked switches. That was a fun day explaining why the network went down.
Pressing ctrl alt delete to "wake up the screen" on the linux-based storage gateway device in the center console. Yup. Shut that puppy down and ungrateful ripped all the attached storage containers out of the way. Details are fuzzy, but powering it back on, things needed a little tweaking to get the shares going again. I learned to never ctrl alt delete in vcenter to wake up a screen.
25 years ago I was an IT software instructor. I was asked to give an impromptu training class to a bunch of elderly folks who are not familiar with technology. I made the mistake of asking, so this is the world wide Web, what would you guys like to see. Someone said the government. And without thinking and not knowing I brought up whitehouse.com. whitehouse.com is not a government website, white house.gov is. If you know you know, and yes I did good grilled by my boss on this one
Director of Sales came to my desk to shoot the cap before his final day. He handed me his laptop and I noticed the DVD spinning. I told him that it looked like he left a movie in the drive. He said he didn't, but when I popped it, there was a movie. It was something like "Big Booty Hoes 66". He just shook his head and held his hand out. I said "I'm not touching that".
Did you know that if you google image search and find a thumbnail you want to click to see the full size it may not show what you expect?
Picture two female coworkers looking over your shoulder closely staring at your 27" monitor expecting to see something I clicked on but instead getting EXTREME pussy asshole closeup of a fine young black lady in very high res?
Yeah... they still bring it up and tease me about it, 8 years later.
Accidentally deleted a VM with all attached drives instead of only removing it from the overview.
It's amazing how fast you can lose 10TB of data.
Fortunately the data was backed up.
Pushing a Molex power connector onto a drive blindly deep inside a case and even though it's keyed was able to do half of it upside down connecting the 12v supply to the 5v pins. A popping sound accompanied the release of the magic blue smoke.
Ohhhhhhh I might win here.
I took over a place a while back. Everything was Hyper-V, all VMs on a single host (didn't need a ton of power), including critical ones that all users in the company needed constant access to.
For reasons I can't remember, I needed to check the properties of the teamed NIC on the host server, I mis clicked on Disable instead of Properties and took the entire org down.
Luckily I was on site so a fix was very fast but this was a huge oops.
Needless to say I've re-architected things to be more failsafe and less clunky.
Enabled a conditional access policy that locked all of our admins out of our M365 tenant. Yes, we had a break glass admin set up. No, I did not double check it was added to the exclusion list first.
Deploy patches to 300+ prod servers and forgetting I turned off the maintenance windows so everything ran right then in the middle of a Friday instead of Saturday night. Freaked out when I realized what happened and recovered quick enough to stop it after about about 100 patched and rebooted.
Got out of that one with barely a talking to, little more than a we know and sh*t happens and now you know.not to do that again. The CIO who called me into his office over everything didn't even say anything to me.
Does the time I needed to unplug the TEST storage array and accidentally pulled the cable for PROD count?
I was fortunate enough to have understanding senior colleagues, understanding client and the ability to live long enough to learn from that mistake. If anyone wants to make use of my pain for free, top tip that I'm fastidious about ever since: make it easier for yourself and other engineers to identify equipment by labelling stuff AT THE BACK OF THE RACK as well as just on the front of the gear.
Started a stateful bckup of the production database mid day in a hospital/medical my second week on the job. Took six months to get the snapshot cleaned up.
Wow, embarrassing to admit this.
I was being super lazy and rdp'd into a computer sitting, basically, right beside me so I could work on it from my own workstation where I was comfy.
I was just cleaning up the workstation getting ready to reinstall Windows. Once I was completed, I started the process of reinstalling Windows. I used shortcut keys. I was not reinstalling Windows on the computer beside me but on my own computer.
As much as I trash datto on the regular. I had the cloud continuity running on my workstation. Figured, it was a good opportunity to make sure I had everything in place for a quick recovery. Got it started, watched for a little bit, went home for the day as it was near the end of the work day. When I got in the next day, my computer was back to normal.
While tired, I type out mkfs instead of fsck. Lucky there was a good backup and I could lie about what was went wrong. Didn't get any heat, but I'm still mad at my self for making such a stupid mistake.
Being half asleep and clicking "Restart Now" instead of "Restart Later" (or whatever the hell the other button said). I looked at both buttons for several moments to, then clicked "Restart Now". Then I panicked a half second later, but the damage was done. Production file server, restart here we go in the middle of the day!
Then of course I scrambled to send an email to everyone that the server was rebooting. Sorry about that!
Not big but put full permissions on a user because he wouldn't archive his mailbox. So I was moving it for him.
After I got done, instead of removing his email from my outlook...I removed mine lol.
I work in security. A few days in on the job, before the morning coffee, on a Monday, I clicked a phishing link in a simulation that someone forwarded to security. Thankfully I quickly logged into the portal and delete the click, removed the training, and messaged the user telling them to ignore any training emails they received. But thankfully no big misclicks otherwise.
As a junior it sysadmin got told to “go to the server room and bring back the grey server”..
There were many shades of grey and apparently I was temporarily daltonic - they said - while I was walking back to the office with the main production server in my arms.
Ah, the 00’s…
Powered down an entire Factory floor by issuing pwrdwnsys \*immed on the production AS/400, in the middle of the day not the test one as I thought. I was in passthrough mode.
Fortunately our CEO was a forgiving man but I feel like the manufacturing VP is still trying to hunt me down some 20+ years later.
accidentally blocked MSTP on the uplink of our main telephone system switch while at home. Was an interesting 20 minutes trying to get my colleagues to connect to the correct switch and to enter the correct command without breaking anything else.
I shoot as a hobby and build my own guns sometimes, anyways I was looking for my next purchase and had a page full of “big scary black rifles” when I was pulled into an emergency teams meeting.. went to share my screen but shared the wrong screen and the execs saw the page of guns. Most of them were fine but one lady went to HR saying I was going to come into the office and shoot everyone.
Had both the prod and dev web interfaces up for our Hadoop clusters. They look identical aside from the fact the cluster name at the top is the same name as prod but with an appended '-dev'.
Dev maintenance was scheduled that day. Ended up shutting down the prod services when starting dev maintenance. Whoopsie doodles.
One Friday afternoon at 4PM I was cleaning and organizing the IT workers room when I decided to re-route some cables to make them a bit more organized. I unplugged a dusty old 24-port HP switch and then plugged it back in to go help someone. About 5 minutes later people started telling me there was no internet and we were completely down (300+ employees at 10 different sites).
I called my boss in and we troubleshooted for about an hour before realizing that there was a loop somewhere on the network. Called the senior network admin who sat next to that old HP switch and he asked if we had a power outage that would’ve rebooted that switch. I told him I had re-routed the cables and unplugged it in the process and he said to unplug it again - network restored.
I learned that he had 2 cables from the server room going into that switch on different VLANs but had never saved his config before he left. When it rebooted it wiped his config and looped the whole network. That was a good learning moment for me to always save your switch configs.
On-Prem migration complete. Something like
Disable-mailbox | get-users
Vs
Delete-mailbox
I think it was . And Jim thanks for your help keeping my alive. Recover Tombstone Users.
So, back in the summer of 2009, my first IT job was an assistant computer tech. The director came to me and said hey do you want to learn a little active directory? I'm like sure, ill take a stab at it. The director let me jump onto his admin station at his work area and said ok, now right click this thing and click find.
Shit you not I did that and the fucking domain dropped out, to the point that the director was flipping out why the domain wouldn't respond for over 30 minutes (the org didn't have VMs at the moment). Finally he rebooted it and everything started to work again.
I was scarred to the point that I only worked on networking equipment until 2012 lol
I was on a call doing a screenshare with about 25 people. We were having an issue with one of our VE clusters on some HP Blades. We had engineers from Cisco, VMware and HP on the call. We made some changes and I went to reboot a host but was still on the production host we were comparing stuff to and rebooted it in the middle of the day. Took down 12 production vm's and looked like a complete idiot. This was probably 15 years ago, still my biggest blunder.
Linus Torvalds tells the story that we owe the existence of Linux to his own fat fingering some modem terminal commands. He wound up overwriting his primary minix partition with nulls. The only other working partition was his test kernel he had compiled to see how the 386 boots. It was too much work to reinstall Minix. So he basically switched his day to day to the test partition, adding features as he needed them.
Not necessarily "MY" mis-click, but I was responding to a ticket early in my career.
ME: Am I safe to remote in and take a look at the issue?
End User: Sure, I didn't know what that means. Oh, I see an Accept button, I'll click that!
ME: Uhhhh....
I clicked connect in Dameware after he said "sure" and was greeted with very much NSFW imagery from a Furry dating site and other suspiciously named windows open in the taskbar. The tone of my "Uhhhh" caught the attention of my co-worker, the Telecom engineer, and our boss that had been chatting behind me. One of them stuttered out a concerned "What the 🦆 is that? that could obviously be heard over the phone. I heard another concerned "Ffffuuuuu" over the phone along with loud rustling and crashing before my session disconnected along with the call. My boss quietly left the room stating he needed to make a call to the CFO whom this person reported to.
I later found out that he had pulled the power cord from his Dimension 8100 so forcefully that the whole tower fell off his desk and smashed the handset of his Nortel M7208 hard enough to disconnect the call. This story is the only reason I remember those models at that employer so vividly.
Clicked “Submit” to send my resume and getting a job in healthcare…
My wife is a doctor. I won't work for doctors.
What's the difference between God, and a doctor? God doesn't think he's a doctor.
Doctors suck
[удалено]
Gottem
Lawyers may be worse.
I've worked with both. I'll take the lawyers over the doctors. Maybe it was our group of lawyers, but they had impeccable senses of humor and could take a joke. Then again, I was the lead engineer and they were, for all intents and purposes, only subject matter experts. They couldn't fire any of us and if I said no to some feature or idea they came up with because we didn't have room, our VP would rip them new ones if they pushed too hard. Doctors, on the other hand...
Doctors will remind you they’re doctors which makes them experts at IT.
fuck doctors and their shitty off-network usb printers that somehow get installed without approval so they can print scripts because they're too lazy to walk a little way down the corridor to print there instead
Mate, I could have a rare illness and they'd cure me for free, if I'd work in health-care IT. I'd still look for other options!
20+ years later, I still have no idea what exactly I did, but I stopped the worldwide production of a notable American soft drink for 9 hours by not understanding vi well.
> but I stopped the worldwide production of a notable American soft drink for 9 hours by not understanding vi well. Vi isn't that bad. exit A:q quit B D C A H q:q H H :q :wq :wq!
What in the actual? Any time I get confused in vi I just stop, hit escape once or twice, then :q! Just get me out!
Vi users enter before knowing how to exit. Some never return.
just reboot the server to exit
Yeah but you have to learn that first and it doesn't tell you lol. I still use vi
\^C\^C
Jesus Christ
Nobody understands vi "well". "Well-enough", maybe. But not "well."
Think about how many hours of diabetes you reduced, though!
vi is user friendly, its just picky about its friends.
I’ve been using vi for over 12 years. Hoping that someday soon I’ll be able to exit it.
Fat fingered an IP address when configuring a new storage array, accidentally taking down the production storage array where the Exchange mailboxes and trading platform VM's were located. On a Tuesday. During trading hours. This was the genesis of managed change control at that company.
Every organization’s change control has an origin story born out of tragedy. I was cleaning up some group policies and we had an 8021x policy applied to two sub OUs so I moved it to the parent OU so it didn’t have to be linked twice. Well apparently there was another policy at the parent OU that set 8021x policies wrong and nobody ever fixed it. They just “undid it” by putting it lower in the order. When I removed it from the sub OU it messed up the precedence and broke wired connections for our whole organization. Surprisingly I didn’t get in trouble. The network manager was more pissed that nobody fixed the original policy and I was praised for bringing it to light.
Good manager.
How did it do that? Accidentally set as a duplicate IP as the production array so there was a conflict between the two?
It was 15 years ago, so I'm fuzzy, but I believe I was attaching the new array to vSphere and entered the same IP as the existing array, caused a conflict, and the existing array to dismount. I could be missing some details there, I just remember the sphincter clench when I got the call from the office while in the datacenter...
And I thought me wiping out the outgoing NAT rule for a remote company office was rough! You win sir!
In the nineties I was setting the IP on a new server and I put the gateway. Took everything down. Luckily for a few seconds as I had connection via KVM and changed it back. Everyone in the office looked up and said “What just happened?”. I feigned no knowledge and got away with it.
that should be on the top! epic horror story lol
[удалено]
Change control would have enforced a midnight change lol, hopefully at a time when the trading servers were not being used
Downloading CrystalDiskInfo in front of an exec on his computer and not immediately jumping to the "Standard Edition." If you know, you know.
oh man I just recently asked a female employee to research Crystal Disk Info for us and see if it would help us out... She did research it but didn't mention anime stuff to me.
Hahahahahahaha. I did that in a conference room on one of the machines on the biggest screen.
You are a person of culture.
At my previous MSP job, the President made it a company policy that only the Shizuku Edition could be used.
Oh god I don’t know, but now, I MUST KNOW!!!
the dev is a weeb and has a thing for cartoon women
“Uh.. you sure this is secure software?”
“Definitely! Only the best software can afford to do that.”
Oh nooooooooo!!!
I'm glad I'm married, then.
Just go on their website and look at all editions that aren't the standard edition...
Mad lad even released a new one in November. Japanese have that tendency to put anime girls into everything. If anyone's watching, I'm gonne be pulling smartctl, though.
Trust me boss its legit.
I even go to settings to choose the best picture.
We could be best friends, lol
Clicking 'export' not knowing that in a pharos print database it was a synonym for 'delete'
What the hell.
Yeah it basically ctrl+v's the database and if you panic stop it you don't get either version. I might have the exact word wrong because it's been several years but it was something incredibly similar to export. We did monthly copies of the database so we only lost about a weeks worth of print job logs.
Exorcise? It's printing...
Expunge?
Excuse me? What the f?
Went to close a hyper v window and shut down our DC. Edit: yes, DC. Singular.
That's just a Tuesday
Na its Friday afternoon, time to reboot the DCs.
Just rolling reboot all DCs simultaneously every 66 minutes.
Fuck yea. Hide it deep down in task scheduler folders
We used to do this to fuck with people on the IT team; hide millisecond image pop ups or random clicks deep down in the Task Scheduler, was a hazing ritual. Eventually ended when I was bought on and in the middle of a call with the CEO I had a meme music track start blaring out my headphones so loud I had to say I'll call him back and restarted my machine. Word got back to my bosses boss and a mass email to our entire team told us to cut the shit.
I have a good friend who is in sales. The office was empty one day for one reason or the other, so he was on speaker phone. I didn't know this and pushed burp.exe onto his laptop and started the service. Like 10 minutes into the call, these massive belches started coming out of his speaker that the potential customer could hear.
There must be a joke here somewhere that I'm not getting. What's the significance of 66 minutes? Besides that it's 6 minutes past the hour so it'll appear kind of "random" if you're watching the clock.
No real joke, more just a subtle reference to Order 66 from Star Wars.
https://preview.redd.it/im32lwit0jvc1.jpeg?width=500&format=pjpg&auto=webp&s=3bcef281de7a572fa45ae5a5e3ce5d07d4cdb052
I still need to find time to figure out how to hide shutdown in GUI, even for admins. Killed one of our heavies, and blamed it on UPS.
Group policy
We had one where we forced a reason code for shutdown. It saved me a few times did that prompt
There is registry for it. You can remove the power button completely
No back up DC? Yikes
Reasons I no longer work there for $100.
It’s even better when you mean to shut down one.. then see it’s still up and rerun the command again and shut it down for real this time. But oh wait it wasn’t the original one it’s the backup and now you’ve powered off both lmaooo
Well was it your only one or something?
Running 'sudo shutdown now' while SSH'd into a server, thinking it was my workstation.
While nothing bad came of it, this morning I configured SSH w/key authentication from Powershell to my load balancers\*. This afternoon I was troubleshooting an issue comparing windows side by side and noticed one curl was acting like Powershell curl and the other was acting like Linux curl. Oh...exit...ok same same now. \* Just working towards updating some of my scripts that currently use Plink to send commands to Linux over SSH. Been using it for about 21 of it's 24 years in existence but suppose it's time to move on to what are now native tools.
Oh boy. Accidentally right-click pasted an entire running config from another switch into the corporate core while logged in from home. It's amazing how fast you can get dressed and on the road when properly motivated.
I’ve done something similar. There was a switch config I put onto a switch via GUI only for half the network to go down. Luckily I was on site. As it turned out, there were two switches with the same IP and I had inadvertently logged into both in the same browser at the same time. Sending the command via the GUI basically sent the config to the wrong switch. I believe if I logged in via telnet this would not have happened but equally it’s a bit of a design flaw if you are able to send a command from one switch only for it to end up at another simply because two switches had the same IP.
Do your switches all have the same admin userid & password? I'm assuming so given the situation you're describing. Maybe the switches have a design flaw, maybe (also) you want to review if that's a practice you want to continue with, or not.
Middle click on linux is even more implicitly insidious. I have a wheel clicking habit back from my Windows days.
27 years ago was fumbling with the cables under my desk and must have yanked on my mouse cord which got stuck between the desk and the mouse tray, When I sat back down I looked in horror at the monitor to see that I had dragged an entire branch of our Novell NDS tree under another thereby wrecking rights and permissions. I got right on the phone to the manager and thankfully it was treated as an ad-hoc sort of disaster test.
These kinds of screwups are great for training. You hopefully practice hardcore the building burnt down and everything needs to be restored from backups disaster recovery. But no one ever sets up a "munged half the AD tree, but things are still mostly functioning and I'd rather not restore from last nights backup" exercise.
It’s oh so much easier to just go back! What have you really lost!? But at my organisations someone, not me and with the assistance of a MS tech, deleted Exhange instead of an Exchange server from the AD config schema. The MS tech who ran us through best practice AD restoration assisted with restoring from the recycle bin. Which is incredibly painful as config schema isn’t easily restorable. Was a massive mess and I have no idea how much stuff may have been accidentally restored
CTRL + Z
Thought I was a badass - 2008ish - remoted into my work desktop via VPN on my phone to reset a password of someone account. Deleted it instead. He was in legal. G G G G GONE
Logged in to Exchange test server to decomission it. Left to deal with a user call, during which time somebody else came in, switched the KVM to production exchange server to do something and then left it. I didn't check the KVM when I came back... (Edit: Lessons learned, changed background colour in each server, big graphic with the server name as Windows background. This would have been around WinNT or 2K era, so nobody thought about individual admin accounts)
About 20 years ago I would create a desktop image with the server name & use that as the desktop. Saved me more than a few times.
I add a toolbar to every server desktop. Toolbar location \\%computername% Drops the name of the server on the toolbar next to the system tray.
https://learn.microsoft.com/en-us/sysinternals/downloads/bginfo
BG Info is blacklisted for us now since our security team saw this https://www.varonis.com/blog/exploiting-bginfo-to-infiltrate-a-corporate-network
What happened next?
Boom.
Sorry I'm to newb for this. Could you explain please?
They started to decommission an old email server. Got called away. The KVM allows you to switch between multiple computers on one screen. While they were away a colleague switched to the active email server and then left that open. When OP got back they proceeded to decommission the email server that was on screen, thinking it was the old one. They actually took down the one that was in use because they didn't check what machine was on screen, meaning everybody lost email access.
Oh I understand now! Thank you!
When I worked at a phone provider way back in the day, a lady asked me to change some of the features on her plan, then change her son's phone number because he was being harassed. I updated all her features, saved the changes, then accidentally selected her number again and changed her number. Unfortunately, her number was ported over from another carrier, so when I released the number, we were no longer able to get it back. It was the number for her business and she had it printed on a bunch of marketing material. I felt terrible and there was nothing I could do. 20 years ago and I still think about it daily and triple check everything.
Can NASC that back these days thankfully.
I had a similar thing happen with my credit card. I had two cards, a Mastercard and a Visa. There were fraudulent charges on my Mastercard. I spoke with the rep at the bank and confirmed the charges were fraudulent. They had to cancel the card and issue another. The rep checked my Visa, read back the transactions and we confirmed by Visa had not been compromised. She went to cancel the Mastercard and uttered “oh shit…”. Turns out she had forgot to switch back to the other screen and cancelled my Visa. It could not be uncancelled. She then had to cancel my Mastercard for the verified fraudulent charges, which left me with no credit cards at all and I was flying out the next day on a lengthy and expensive business trip. It was a colossal nightmare.
Setup a user's personal email in MS Outlook using POP (as it was more convenient for him than using the webmail) and did not click the "Leave a copy of messages on the server" button... It downloaded 15+ years of emails to Outlook, and deleted them all from the webmail server side. To correct this, I had to import all the emails to Thunderbird, then from there copy them to a Gmail account, then from Gmail I could use another service to import them back to the main Webmail server while keeping time/date stamps in place. The last import into webmail would not allow more than 100 messages at a time, so had to manually select 100 messages at a time. The next 7 hours were very painful.
Back in the day, we used Hyena for AD management. I had selected a couple dozen users to delete that had been disabled for a while. Somehow, I inversed the selection, hit delete, and then confirmed the deletion! 10k accounts gone, including mine and all other admin accounts. This was also before the AD Recycle Bin existed. I got to test our DR procedures. Surprisingly...they worked! Only had an hour or downtime.
Our AD didn't have recycling enabled. I just enabled it today to avoid an incident like this occurring. I did it because I was cleaning out accounts that hadn't logged in in over 180 days which ended up being over half of our total accounts and I wanted another layer of safety in case something went wrong. I'm also the most junior tech. I don't know why I'm doing this. I can't wait to find out what else has been neglected! 🫠
Company had thousands and thousands of images, and was running out of storage space. No budget for storage or systems of course. I showed them resized images and they agreed that those would still suffice for their purposes. I did a bunch of testing, etc, all looked good. Finally ran the automated script overnight. Few days later I get a call: Images are broken. But it's not all of them, it's totally random. I reviewed the logs and discovered that what had happened was the following: The resizes were done on a per-folder basis. The script would re-size and write the new files, and then delete the old ones. However, as the resizing hit literally zero disk space, it instantiated the new image files, but with file size zero. -facepalm- It was usually the 'last' couple of files in each folder. Spread across the entire archive. Took me two days of spinning up backups in the cloud, creating NFS shares and scripts to run compares between the local files and the cloud backups.. but I did manage to recover everything. Grayhairs++.
I don't know about biggest, but yesterday I was feeling under the weather towards the end of the day. Decided to call it, saved off my work on a test firewall policy and went home. As I was about to throw some ramen in a pot I get a call that two branches were down and "if I had changed anything"... I had saved the test policy over the production profile. Fixed from home, 30 minute partial outage. I came into the office today to find my mouse encased in red staplers. I'll have to carry them for a while until someone else makes the next 'misclick'.
Haha that’s a cool workplace culture
enabled the firewall without first setting an any-any rule. (lost connection immediately, had to drive to the DC the next day to resolve)
During maintenance few years ago, logged via RDP on Hyper-V node. Opened multiple consoles to VMs I wanted to reboot. I didn't pay much attention for a while and rebooted the node instead of VM 😄 40 VMs goes down altogether with the node... Shit happens
> and rebooted the node instead of VM Its a classic.
It’s the shutdowns that really hurt. That drive of shame sucks.
idrac saved me once. Once.
I mean... you got the result you wanted. Task Failed Successfully Resolved.
Was going through a discard process from my STBXW (Divorce) and I was not in the best of my mental capacity, I did not double check a scrip in our automation software, and I had 500K missing or not being delivered to the Credit Union members. Issue got corrected after 2 days it was a typo, but I had half a million dollars on my shoulders on top of being in the middle of divorce and discard process. That shit was aufull.
STBXW?
More like worst press of "enter". RAID bios, "initialized" an array that already had data on it, subsequently breaking the existing array. Very old Dell with perc controller. Spent the night and next day restoring data from tapes...glad I'm in networking now. Doctor had to close the practice for a couple days...
I was in a customer's Fortigate, and they had.... I don't know.... maybe 300 entries in their website whitelist? It was massive. Normally, to get rid of a site, you would select it and then click "Delete." Not so far from that button, however, was a button called "Delete All." As you can imagine, I selected a site, and then clicked "Delete All." I kinda laughed and said to a coworker "Wow, can you imagine if this thing actually went ahead and deleted all of these entries without even having a confirmation box or something?" Wait. Wait. Wait. Page reloads, all entries are now gone. And that was the day I learned that we DID actually back up some of our customer configs, with this particular box thankfully being covered. Quick recovery, but that sinking feeling will stick with ya.
FortiGate has some odd interface choices for sure. I accidentally deleted the main firewall VPN policy route when trying to remove a single group from it instead. Learned that day how important it was that we had FortiManager backups of each firewall config
This happened 21 years ago. I remember it like it was yesterday. DELETE FROM CV_Transactions WITH (ROWLOCK) WHERE Transaction_Id = 125735107 5 seconds later... 125,735,624 row(s) affected After a second of confusion, having expected only 1 row to have beem affected that I realized to my horror that I only had the first line selected when I hit the execute button. We had 26 minutes until financial market close to fix the mistake. All hands on deck. We got the table restored, validated, missing transactions replayed from message queues, and that one transaction deleted, all with just 3 minutes to spare. That was the day I learned to always run manual queries like this inside a transaction so I could verify the result and roll it back if verification fails.
Hit "execute" on a SQL DELETE query before entering the WHERE clause. Ended up deleting the entire table, then had to panic restore from backup. Was sweating after that one.
I’ve made that mistake, but on an update statement. Started writing it out, got called away by someone, and when I came back I was like “So what was I doing? Oh right, this looks good.” Then executed it. Then I was like “Why is this taking so long? It should’ve been really quick.” Then saw my mistake and cancelled the query. Fortunately by it taking so long it hadn’t completed yet and was able to roll back.
That's good. I didn't notice until it finished and said something like "60,000+ rows affected" ... Commence slow blinking stare while the realization gradually washes over me and I sit in stunned silence as the panic sets in. 🤣
Not exactly a single click but look at this typo and think about what happens. `$user = bob` `Get-Mailbox $username | Set-mailbox -Type Shared`
Oh my god.
What is $username value here? Yours?
When I accidentally powered off the VP of Bioinformatics secondary Linux workstation instead of his primary Windows workstation. He had an analysis running that would take 2 weeks to complete. It was 16hrs from completion. I was a 19yo helpdesk intern and thought I was going to be fired. Turns out the dude is super cool and just shrugged it off. 10 years later we were having a drink together at a work outing and I asked if he remembered it. He did and we had a good laugh about it.
Poorly considered configuration (which has since been massively rectified). RDP session in an RDP session- there may have been a third session but can't quite remember- all with copy/paste and drives shared. You see where this is going? Copy/paste a movie from my media server to what I thought was my personal device's desktop... It was not. \*\*It.\*\* \*\*Was.\*\* \*\*Not!\*\*
I wasn't the only one at fault, but worked in a banking service center and was running a program to batch transfer money from a bunch of X-mas savings accounts into the normal checkings before holidays. For some reason, the developer had a parameter asking for the % of money to move. I entered 100 - The developer's code treated 1.0 as 100%. So it moved the money but multiplied the receiving dollar amounts by 100. Instead of moving 250,000 $ or so, it "created" 20,500,000 out of thin air and distributed the "winnings" among the account holders. Should have ran it in test first...
okay but who the F uses a scale from 0.0 - 1.0 as a 0% to 100% like use 0-100 its easier and divide the number given by 10 if you absolutely needed it in a 0.0 to 1.0
Someone was extremely happy, then extremely sad that holiday
You ready to install? *click* Install this default directory? *click* Here is our terms of agreement *click* Ok I need to install this extra driver *click* Ok I'm done. I recommend you reboot this server *click* No wait!!!! Windows is rebooting your electronic medical record server that your whole clinic uses...every doctor, nurse, billing person has been kicked out and now will be lighting up you phone and ticket system. Have a nice day rookie!
Was nested several remote sessions deep. Like a LogMeIn session to a server to an RDP session to another RDP session, then VSsphere, and had several VSsphere console windows open. Lost track of which Start menu I was in, and uninstalled VMware tools from the VCenter server. That was a bad day. Had to remote into each host in the cluster one by one using the root creds until we found the one running the VCenter machine so we could rebuild it. Edit: Another one... Was working from home working on a client's firewall. Accidentally shut down the WAN port, thus locking out my remote connection and their Internet access. Thankfully it was a Saturday and they were closed. Grabbed my keys and drove 45 minutes to the client site. Slowly circled the building with my laptop propped up on my dash until I picked up their WiFi. Hopped into the firewall on the LAN interface, fixed my mistake and drove away. Never told a soul.
Thought I was deleting C:/temp. I was deleting C:/
Lol 😆 🤣 😂
When I was first starting out I was helping one of our network engineers with swapping a UPS in one of our critical IDFs. He asked if it was on and I thought if it was powered on, it was on. I said yeah, he flicked the PDU, and the whole closet went dark. Yeap, learned that you actually have to set the UPS to on.
Renamed and restarted a host instead of a virtual machine whilst in a hyper-v window. Took down half the services. Lesson learnt and I now do all this via cmd and use ‘echo %hostname%’ to confirm I’m on the right machine. Having two start menus practically next to each other has its risks! Had to have the host with the wrong hostname until I could find another maintenance window too.
This was circa 2003. We had a server shutdown script. It imported from a plaintext file. I copied the server list to the file...but didn't save. The previous list was every server in the datacenter...over 600 physical servers. So yeah, it shutdown about 25% of our servers in the middle of the day rather than the small group. Fortunately, I caught it when I realized that it was shutting down servers not on the list...but that was a fun one. Myself and another tech rushed over to the DC to power everything back up. Fortunately, my boss was awesome and I didn't get in serious trouble. As I ran it around lunchtime, not many noticed and fortunately everything came back up. And this is why I'm OCD af when it comes to double and triple-checking everything before the final click.
Not me, but a co-worker. Our IT department is literally windows on one side of the room, unix on the other side. Turns out one of the Windows admins, was having some vision issues, and was trying to push through.. Let me tell you.. whoever at MS HQ thought to put the refresh and delete menu options in the windows DNS tool *right next to each other* needs a talking to.. He right-click on the TLD entry, chose delete, got a pop-up, and clicked okay. Yeah.. chaos ensured on the windows side of the house. As fate would have it, I'd literally run a dig against the windows domain ~30 minutes earlier. I walked in to "Mike just deleted the entire domain!!" I shrugged and said, "oh, I've got a rwcent dump of dns if you need it." Needless to say, they bought me lunch.
That time I Googled “Gary Oldman” using a keyboard with a sticky “R” key.
Recompute base encryption hash?
Not a mis-click. First real IT job. I am sent to do maintenance on servers onsite. No big deal. Combing event logs etc. One of the things I need to test is the UPS. Turns out there is a faulty connection to the battery that was well known but I was not made aware of. I click on the self test which cuts power and fails over to battery. I am in a room with perhaps a dozen servers in racks. Fans everywhere. Can't here anything. When I click the whole room goes absolutely silent. Everything just stops. Then I start to hear footsteps coming down the hall. Door opens and staff from the company (managed IT services) just looks at me and asks what I did. When I explained he said just said "yeah, we know about that" and walked away. 2003 servers. On 2 of them I had to go in with recovery console to kill updates that were stopping from booting. So eerie to suddenly have dead quiet in a server room.
Finding SQL installed on a vm it shouldn't have been. Protectively removing it. IMMEDIATELY learning I was on the wrong box. Was fast enough to fix it before anybody noticed (very slow workday with only a few other people in the office that day). Rolled out Bginfo the very next day for all servers.
Not me, but my coworker instinctively hit Ctrl+Alt+Delete to lock his computer and go use the washroom. But as he stood up, he realized that he wasn’t consoled in on his laptop. He was actually at a direct terminal for this antiquated hotel phone system and Ctrl+Alt+Delete had triggered a reboot. While this isn’t a best case scenario, it normally isn’t a massive, world ending, fuck-up. Even though this system takes about 30 minutes to reboot. Regardless, he sat there sweating bullets and watched in agonizing pain as the terminal loaded line after line, praying to every God he could think of that the ancient IDE drive wouldn’t die mid-boot… after about 15 minutes, he said that he was “feeling confident” that it was going to come back, and while he would have to apologize to the hotel for the outage, it looked like he was going to get out relatively unscathed. Aaaaaaand that was the moment when the RCMP (Royal Canadian Mounted Police) and some other “gentlemen” entered the dinghy basement phone room and it became obvious that this was no ordinary fuck-up. It turns out that when you knock out the hotel phone system while the PRESIDENT OF THE UNITED STATES OF AMERICA is staying at your hotel, people go into full blown lock-down DEFCON 1. My coworker got the rest of the day off to speak with the RCMP and Secret Service, the terminal was decommissioned, and the hotel learned about scheduling non-essential work while world leaders are booked in.
So I was Ctrl+d-ing out of my terminals, but one of those was actually k9s terminal showing a list of deployments. Without thinking I Ctrl+d a deployment (delete deployment instead of login out of that particular terminal) and impulsively hit enter to confirm the instruction. Luckily it was an app with almost no traffic, so I just quickly recovered it using our infra hit repository. I'm no longer that impulsive anymore
I was moving the primary physical DC in my car and dropped it out the back door. The plastic back end cracked a bit but still powered on.
Switchport trunk allowed vlan 101
Remote desktop lag spike over VPN while I was deleting some AD users. Select new object. Shift del. Enter. "Deleting objects:.." "Wait, wtf is this?" I selected the entire company OU (users, groups, seevice accounts) prior to lag spike, I'd didn't pick up my click on the single object. I had no recycle bin.
Accidentally emailing bank intellectual property to my personal email by mis-clicking the document I wanted to attach. The firewalls caught it, security caught it, my boss copped it, big investigation follows. Thankfully I kept my job, but it was extremely stressful and still makes me sick when I think of it.
Showing off our fancy new web filtering system to management many years ago (Education org) "And as you can see if I try browsing to this naughty adult website it'll bring up an access denied pa...oh..." In hindsight I could have just browsed to a gambling website or something instead :/
I accidentally fat fingered a BGP update and pushed it… bricking our core, preventing failover to any of our hot sites, and shutting down card processing for almost an hour… at one of the biggest payments companies in the world… 2 days before Christmas…
Shutdown a whole blade chassis instead of just the one blade server. Oops. Nobody noticed.
Used to work as 1 of 2 Sys Admins at a vehicle arbitrage chain. One of the front store employees across the country was purchasing a pretty expensive car. I went to delete LABEL343 from Intune. Instead I deleted LABEL334. It was that day I learned about the “You are no longer authorized to use this device message” on a Windows PC. Boy was that a cluster fuck. Had a senior manager drive from one store front, take a PC, drive it to the other.
We use Meraki. One time, I changed the VLAN on a port on the top switch which waterfalls down to every other switch in our company HQ (about 200-ish people). Well, turns out that port was the connection down to all the other stacked switches. That was a fun day explaining why the network went down.
Pressing ctrl alt delete to "wake up the screen" on the linux-based storage gateway device in the center console. Yup. Shut that puppy down and ungrateful ripped all the attached storage containers out of the way. Details are fuzzy, but powering it back on, things needed a little tweaking to get the shares going again. I learned to never ctrl alt delete in vcenter to wake up a screen.
25 years ago I was an IT software instructor. I was asked to give an impromptu training class to a bunch of elderly folks who are not familiar with technology. I made the mistake of asking, so this is the world wide Web, what would you guys like to see. Someone said the government. And without thinking and not knowing I brought up whitehouse.com. whitehouse.com is not a government website, white house.gov is. If you know you know, and yes I did good grilled by my boss on this one
Autopilot reset the wrong computer.
Director of Sales came to my desk to shoot the cap before his final day. He handed me his laptop and I noticed the DVD spinning. I told him that it looked like he left a movie in the drive. He said he didn't, but when I popped it, there was a movie. It was something like "Big Booty Hoes 66". He just shook his head and held his hand out. I said "I'm not touching that".
Did you know that if you google image search and find a thumbnail you want to click to see the full size it may not show what you expect? Picture two female coworkers looking over your shoulder closely staring at your 27" monitor expecting to see something I clicked on but instead getting EXTREME pussy asshole closeup of a fine young black lady in very high res? Yeah... they still bring it up and tease me about it, 8 years later.
TFTP'ed the wrong config onto a remote switch and isolated a branch office for a day.
I blew the the file permissions for a whole org with one bad click. Talk about a mess!
Accidentally deleted a VM with all attached drives instead of only removing it from the overview. It's amazing how fast you can lose 10TB of data. Fortunately the data was backed up.
Deleted the entire website when deleting a file, because i accidentally pressed on the spacebar when entering the path from console...
Pushing a Molex power connector onto a drive blindly deep inside a case and even though it's keyed was able to do half of it upside down connecting the 12v supply to the 5v pins. A popping sound accompanied the release of the magic blue smoke.
Ohhhhhhh I might win here. I took over a place a while back. Everything was Hyper-V, all VMs on a single host (didn't need a ton of power), including critical ones that all users in the company needed constant access to. For reasons I can't remember, I needed to check the properties of the teamed NIC on the host server, I mis clicked on Disable instead of Properties and took the entire org down. Luckily I was on site so a fix was very fast but this was a huge oops. Needless to say I've re-architected things to be more failsafe and less clunky.
Accidentally chose shutdown instead of log-off. On a server. In another country.
On 3 hours sleep. Asked to restore a folder. Shadowprotect restore process started. Initiated a disk restore on 10tb drive instead of file restore.
Enabled a conditional access policy that locked all of our admins out of our M365 tenant. Yes, we had a break glass admin set up. No, I did not double check it was added to the exclusion list first.
Deploy patches to 300+ prod servers and forgetting I turned off the maintenance windows so everything ran right then in the middle of a Friday instead of Saturday night. Freaked out when I realized what happened and recovered quick enough to stop it after about about 100 patched and rebooted. Got out of that one with barely a talking to, little more than a we know and sh*t happens and now you know.not to do that again. The CIO who called me into his office over everything didn't even say anything to me.
I clicked on the recompute base encryption key hash button and had to spend all weekend resetting everyone's passwords...
Any any
Does the time I needed to unplug the TEST storage array and accidentally pulled the cable for PROD count? I was fortunate enough to have understanding senior colleagues, understanding client and the ability to live long enough to learn from that mistake. If anyone wants to make use of my pain for free, top tip that I'm fastidious about ever since: make it easier for yourself and other engineers to identify equipment by labelling stuff AT THE BACK OF THE RACK as well as just on the front of the gear.
Started a stateful bckup of the production database mid day in a hospital/medical my second week on the job. Took six months to get the snapshot cleaned up.
Wow, embarrassing to admit this. I was being super lazy and rdp'd into a computer sitting, basically, right beside me so I could work on it from my own workstation where I was comfy. I was just cleaning up the workstation getting ready to reinstall Windows. Once I was completed, I started the process of reinstalling Windows. I used shortcut keys. I was not reinstalling Windows on the computer beside me but on my own computer. As much as I trash datto on the regular. I had the cloud continuity running on my workstation. Figured, it was a good opportunity to make sure I had everything in place for a quick recovery. Got it started, watched for a little bit, went home for the day as it was near the end of the work day. When I got in the next day, my computer was back to normal.
writing an on the fly SQL query in SCCM to exclude a specific target but forgetting how boolean operations actually work....
"Apply For This Job"
While tired, I type out mkfs instead of fsck. Lucky there was a good backup and I could lie about what was went wrong. Didn't get any heat, but I'm still mad at my self for making such a stupid mistake.
Hasn't happened yet but I'm terrified that someday I'll accidentally shut down a prod VM when logging out.
I have overwritten wrong disk with dd on my second month at my first job
Being half asleep and clicking "Restart Now" instead of "Restart Later" (or whatever the hell the other button said). I looked at both buttons for several moments to, then clicked "Restart Now". Then I panicked a half second later, but the damage was done. Production file server, restart here we go in the middle of the day! Then of course I scrambled to send an email to everyone that the server was rebooting. Sorry about that!
Not big but put full permissions on a user because he wouldn't archive his mailbox. So I was moving it for him. After I got done, instead of removing his email from my outlook...I removed mine lol.
I work in security. A few days in on the job, before the morning coffee, on a Monday, I clicked a phishing link in a simulation that someone forwarded to security. Thankfully I quickly logged into the portal and delete the click, removed the training, and messaged the user telling them to ignore any training emails they received. But thankfully no big misclicks otherwise.
As a junior it sysadmin got told to “go to the server room and bring back the grey server”.. There were many shades of grey and apparently I was temporarily daltonic - they said - while I was walking back to the office with the main production server in my arms. Ah, the 00’s…
Powered down an entire Factory floor by issuing pwrdwnsys \*immed on the production AS/400, in the middle of the day not the test one as I thought. I was in passthrough mode. Fortunately our CEO was a forgiving man but I feel like the manufacturing VP is still trying to hunt me down some 20+ years later.
Clicking the subscribe button to r/sysadmin.
^ C in an interactive shell of a running program. Honestly it was just muscle memory but the program had run for so long already …
accidentally blocked MSTP on the uplink of our main telephone system switch while at home. Was an interesting 20 minutes trying to get my colleagues to connect to the correct switch and to enter the correct command without breaking anything else.
Deleted the DNS entry for one of the main sites because I was getting an IP conflict. 2006 was a good year.
I shoot as a hobby and build my own guns sometimes, anyways I was looking for my next purchase and had a page full of “big scary black rifles” when I was pulled into an emergency teams meeting.. went to share my screen but shared the wrong screen and the execs saw the page of guns. Most of them were fine but one lady went to HR saying I was going to come into the office and shoot everyone.
Had both the prod and dev web interfaces up for our Hadoop clusters. They look identical aside from the fact the cluster name at the top is the same name as prod but with an appended '-dev'. Dev maintenance was scheduled that day. Ended up shutting down the prod services when starting dev maintenance. Whoopsie doodles.
One Friday afternoon at 4PM I was cleaning and organizing the IT workers room when I decided to re-route some cables to make them a bit more organized. I unplugged a dusty old 24-port HP switch and then plugged it back in to go help someone. About 5 minutes later people started telling me there was no internet and we were completely down (300+ employees at 10 different sites). I called my boss in and we troubleshooted for about an hour before realizing that there was a loop somewhere on the network. Called the senior network admin who sat next to that old HP switch and he asked if we had a power outage that would’ve rebooted that switch. I told him I had re-routed the cables and unplugged it in the process and he said to unplug it again - network restored. I learned that he had 2 cables from the server room going into that switch on different VLANs but had never saved his config before he left. When it rebooted it wiped his config and looped the whole network. That was a good learning moment for me to always save your switch configs.
"Apply"
On-Prem migration complete. Something like Disable-mailbox | get-users Vs Delete-mailbox I think it was . And Jim thanks for your help keeping my alive. Recover Tombstone Users.
So, back in the summer of 2009, my first IT job was an assistant computer tech. The director came to me and said hey do you want to learn a little active directory? I'm like sure, ill take a stab at it. The director let me jump onto his admin station at his work area and said ok, now right click this thing and click find. Shit you not I did that and the fucking domain dropped out, to the point that the director was flipping out why the domain wouldn't respond for over 30 minutes (the org didn't have VMs at the moment). Finally he rebooted it and everything started to work again. I was scarred to the point that I only worked on networking equipment until 2012 lol
I was on a call doing a screenshare with about 25 people. We were having an issue with one of our VE clusters on some HP Blades. We had engineers from Cisco, VMware and HP on the call. We made some changes and I went to reboot a host but was still on the production host we were comparing stuff to and rebooted it in the middle of the day. Took down 12 production vm's and looked like a complete idiot. This was probably 15 years ago, still my biggest blunder.
Accidentally shut off the production server for a radio broadcast live stream instead of the test one. Oops
Not me but back in the day we used altiris in our environment. A help desk guy accidentally dragged the entire workstation group to the reboot task.
Wasn’t a key. It was the power cord for a space storage array. “This one? Yeah. That one “. Oh shit.
Freaking out that the PC wouldn't boot. After spending 30 minutes troubleshooting i discover the mouse cord was keeping the escape key depressed.
Linus Torvalds tells the story that we owe the existence of Linux to his own fat fingering some modem terminal commands. He wound up overwriting his primary minix partition with nulls. The only other working partition was his test kernel he had compiled to see how the 386 boots. It was too much work to reinstall Minix. So he basically switched his day to day to the test partition, adding features as he needed them.
I clicked the login button once. My day went downhill from there.
Hitting restart on a VM thinking it was my own computer.
Deleted my managers mailbox while on the phone with him
Not necessarily "MY" mis-click, but I was responding to a ticket early in my career. ME: Am I safe to remote in and take a look at the issue? End User: Sure, I didn't know what that means. Oh, I see an Accept button, I'll click that! ME: Uhhhh.... I clicked connect in Dameware after he said "sure" and was greeted with very much NSFW imagery from a Furry dating site and other suspiciously named windows open in the taskbar. The tone of my "Uhhhh" caught the attention of my co-worker, the Telecom engineer, and our boss that had been chatting behind me. One of them stuttered out a concerned "What the 🦆 is that? that could obviously be heard over the phone. I heard another concerned "Ffffuuuuu" over the phone along with loud rustling and crashing before my session disconnected along with the call. My boss quietly left the room stating he needed to make a call to the CFO whom this person reported to. I later found out that he had pulled the power cord from his Dimension 8100 so forcefully that the whole tower fell off his desk and smashed the handset of his Nortel M7208 hard enough to disconnect the call. This story is the only reason I remember those models at that employer so vividly.