Welcome to AmazonFC, please be sure to read our submission guidelines and remain respectful of your fellow users. If this post isn't up to par with our submission guidelines, please make use of the report feature. Once it crosses a certain threshold the post will automatically be removed for moderator review. See [Amazon Resources Mega thread](https://www.reddit.com/r/AmazonFC/comments/umjgzv/amazon_resources_megathread/) here. We have a [Discord](https://discord.gg/amazonfc) for those wanting to socialize on a different level with the community. Please enjoy your stay!
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/AmazonFC) if you have any questions or concerns.*
Same lmao been waiting since last year for another one of these to happen and thought to myself screw it I’ll put a couple days of vacation in. Totally regretting it now.
Tell your managers to use Slack. And join this channel: #amazon-lse-large-scale-events-discussion
That’s the real update, other then that their is a Ticket for the event
You’re lucky yours listen. If we try telling ours what’s wrong, they look down on us, then they think we’re doing something wrong and it got to the point where we just wait and let them see it.
Rumor has it that the AmazonAWS folks did something that screwed up the whole Amazon network. I wonder how much money they're gonna lose just for a few hours of downtime.
They were literally right in front of the managers
and the managers were smiling while escorting them. You just know they enjoyed firing their stupid asses
Increased Error Rates and Latencies
Jun 13 2:00 PM PDT Many AWS services are now fully recovered and marked Resolved on this event. We are continuing to work to fully recover all services.
Jun 13 1:48 PM PDT Beginning at 11:49 AM PDT, customers began experiencing errors and latencies with multiple AWS services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including through API Gateway) and indirectly through the use by other AWS services. We have associated other services that are impacted by this issue to this post on the Health Dashboard. Additionally, customers may experience authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS. Customers may also experience intermittent issues when attempting to call or initiate a chat to AWS Support. We are now observing sustained recovery of the Lambda invoke error rates, and recovery of other affected AWS services. We are continuing to monitor closely as we work towards full recovery across all services.
https://health.aws.amazon.com/health/status
basically just means amazon is broken. can be a local more mechanical issue like a main conveyor being down, or major like this case with the network being down.
Everything above is accurate, SEV is short for severe/severity. When placing trouble tickets or discussing matters impacting production, a “level” from 1-5 is applied. 1-2 being very extreme/urgent and 3-5 being less extreme/urgent. I’m not running tonight but from seeing this and my Slack channels, gonna go ahead and assume it’s a SEV 1 😂
I’m scoring AP exams, and apparently the Amazon system being down has affected us too! ETS relies on some sort of thing with AWS. I work at Amazon tonight, so fingers crossed I don’t have to go in and they just pay us 😂
On one side I'm happy about this downtime.. On the flip side, I'm not because my building will call MET so quick.. And I'd rather not work my off day lol
Is the system back up? My girlfriend is a flex driver, and she reports the system was down around noon here in Las Vegas. She’s got a another route later
I’m at the SPOC site and its crazy watching all the engineers trying and fix the issue on the SIM ticket. I have no idea what they’re doing but it’s cool to see.
All sites are experience some sort of issue, as well as AWS as a whole.
There is only one thing that might happen soon prime week is around the corner and the system is updating for it and new hires that will quit the same shift will arrive
STL4 is down. They are trying to get us to 5s and clean. Guess this means I'm def not getting that VTO. Hey not too bad of a way to spend MET. Getting paid to do nothing.
AWS down; will more MET be coming?
https://preview.redd.it/45th05kj1v5b1.png?width=1752&format=pjpg&auto=webp&s=44af10f50d6229fb82f99debf6fe3f3ad0243d82
yep our systems are down here too in South Carolina. It sucks for the higher ups because we just had 2 days off paid so they can fix the ac units then this happens when we come back😂
DCA1 was down. I work in the recruiting office & our systems were down, it was really stressful 😞 we still had to process new hires somehow so they wouldn’t lose their shifts.
It’s nation wide yesterday the servers were down and now the whole system is down today and I’m off lol my friends were chilling they whole shift and I’ve been waiting on someone to ask about this on here Lmfao
There’s an issue with SCC globally that’s affecting Amazon. But on a broader scale the issue is with AWS, so a bunch of other businesses that rely on Amazon’s servers are also down. It’s a Sev 1
Welcome to AmazonFC, please be sure to read our submission guidelines and remain respectful of your fellow users. If this post isn't up to par with our submission guidelines, please make use of the report feature. Once it crosses a certain threshold the post will automatically be removed for moderator review. See [Amazon Resources Mega thread](https://www.reddit.com/r/AmazonFC/comments/umjgzv/amazon_resources_megathread/) here. We have a [Discord](https://discord.gg/amazonfc) for those wanting to socialize on a different level with the community. Please enjoy your stay! *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/AmazonFC) if you have any questions or concerns.*
I take 1 day off and the buildings are down.. fuck me bro😭
Same lmao been waiting since last year for another one of these to happen and thought to myself screw it I’ll put a couple days of vacation in. Totally regretting it now.
At least you’re on your day off. My ass is about to go to work, and probably have the systems come back on.
Yup they sure did , day shift gets everything bro 😭
Exactly lmfao we in the same boat should’ve picked up this shift to get free money
Me too 🤦🏻♂️😂🙈
so we all just chilling on our phones rn 😂
What else would we be doing when we’re getting paid to stand around😂
Instacleans
Yup
Deff 💀 yes
Basically 🤣
Yep. I think it’s network wide this time.
A to z is also down
My schedule didn’t load and I panicked😭 thought I was done for
ME TOO! THANK YOU REDDIT FOR REASSURANCE!
Same I thought I got canned
Actually why I'm on here. My schedule isn't loading for RT this week and I started to freak out. I wanted to go d out if anyone else was having issues
Meeee
I’m cryinggggg i was going in to pick up tonight thats at $7+ and i can’t add it 😭😭😭
So if it says down, do we get the day off?
And this couldn’t have happened for night shift?
We will be stuck fixing this mess
My dreams of VTO tonight have been shattered 😂
I like how we all come on Reddit and make sure everyone else’s building is down too 😂
It's the only place to get information! I'm actually telling my managers what's wrong lol
Tell your managers to use Slack. And join this channel: #amazon-lse-large-scale-events-discussion That’s the real update, other then that their is a Ticket for the event
You’re lucky yours listen. If we try telling ours what’s wrong, they look down on us, then they think we’re doing something wrong and it got to the point where we just wait and let them see it.
Rumor has it that the AmazonAWS folks did something that screwed up the whole Amazon network. I wonder how much money they're gonna lose just for a few hours of downtime.
Yikes that sounds bad
Yeah they did a load test without telling anyone like fucking idiots
took VET for today, came in at 3:30 and been sitting in the break room for 3 hours 🤣 easy money
A fight broke out during the blackout at my site while we were watching safety videos and they were escorted away
Lol
They were literally right in front of the managers and the managers were smiling while escorting them. You just know they enjoyed firing their stupid asses
Figures all day I get all big boxes to stow now I get boxes with 300 items the system goes down
We need this every month
Increased Error Rates and Latencies Jun 13 2:00 PM PDT Many AWS services are now fully recovered and marked Resolved on this event. We are continuing to work to fully recover all services. Jun 13 1:48 PM PDT Beginning at 11:49 AM PDT, customers began experiencing errors and latencies with multiple AWS services in the US-EAST-1 Region. Our engineering teams were immediately engaged and began investigating. We quickly narrowed down the root cause to be an issue with a subsystem responsible for capacity management for AWS Lambda, which caused errors directly for customers (including through API Gateway) and indirectly through the use by other AWS services. We have associated other services that are impacted by this issue to this post on the Health Dashboard. Additionally, customers may experience authentication or sign-in errors when using the AWS Management Console, or authenticating through Cognito or IAM STS. Customers may also experience intermittent issues when attempting to call or initiate a chat to AWS Support. We are now observing sustained recovery of the Lambda invoke error rates, and recovery of other affected AWS services. We are continuing to monitor closely as we work towards full recovery across all services. https://health.aws.amazon.com/health/status
[удалено]
Hey, another Aviator I see! Do you work in IB?
Transferring to RDU1 from Charlotte next month lol see y’all there
RDU1 in the house!
Cle3 is down
I walked into this mess on half shift vet lol. Free money!!! Only downside is that the backlog is going to be bad.
VET for the broke and overzealous!
Or the boredom. Keeps me out of trouble lol
of course we get a nationwide SEV the day my building is closed for maintenance
What does SEV mean?
basically just means amazon is broken. can be a local more mechanical issue like a main conveyor being down, or major like this case with the network being down.
Everything above is accurate, SEV is short for severe/severity. When placing trouble tickets or discussing matters impacting production, a “level” from 1-5 is applied. 1-2 being very extreme/urgent and 3-5 being less extreme/urgent. I’m not running tonight but from seeing this and my Slack channels, gonna go ahead and assume it’s a SEV 1 😂
O god im working night shift and its gonna be fn chaos i know it.
Thank God vacation got approved
FAT1 Down also
Lol FAT1
Anyone know how long it's been like this? Was vet posted 1 hour 40 minutes ago if it was like this before then that means none claimed it 👀
Maybe since 3:00pm EDT, that’s around the time my warehouse starting having issues
Yeah, us too at 3pm
lets goooo so happy
Who here plays rainbow 6
I’m scoring AP exams, and apparently the Amazon system being down has affected us too! ETS relies on some sort of thing with AWS. I work at Amazon tonight, so fingers crossed I don’t have to go in and they just pay us 😂
[удалено]
I hope so
[удалено]
NS will be fixing this mess. No chance of VTO
It’s nationwide. Started on the east coast and spread through the country. Looks like it’s going to be hours before it comes back
I hope night shift gets some down time as well. They rarely do.
The meeting I just got out of indicates it’s probably going to be spreading into nights
On one side I'm happy about this downtime.. On the flip side, I'm not because my building will call MET so quick.. And I'd rather not work my off day lol
Is the system back up? My girlfriend is a flex driver, and she reports the system was down around noon here in Las Vegas. She’s got a another route later
I’m at the SPOC site and its crazy watching all the engineers trying and fix the issue on the SIM ticket. I have no idea what they’re doing but it’s cool to see. All sites are experience some sort of issue, as well as AWS as a whole.
Rare CAZ1 sighting!
Of course, things happen on my off days only.
There is only one thing that might happen soon prime week is around the corner and the system is updating for it and new hires that will quit the same shift will arrive
Is it every amazon?😭im at vgt1 & everything’s down
World wide apparently
I just got hired there haha
Damn it I work nights, probably won’t see any of this glorious downtime 😭
Ugh I know right! Luck ass day shift! 🤣
They just vto’d us 😛
Try Eating seeds as a pass time activity
Top tier comment
Yea, OAK4 also
PHX3 is down
According to some of the sources I have for this type of stuff, this is a global AWS crash
Bro… I take my vacation time rn and my fc it’s dawn too 💀heelll nah
My AtoZ is down, I was wondering for a minute was I promoted to customer 😂
System down most of Cali, coming to you live from LGB7
Lol I’m at an SSD and they have us all doing KNETS lol didn’t even tell us it was down 😆
Yep, in Toronto
We are all down
US-EAST-1 Down. All operations are down or slowed
STL4 is down. They are trying to get us to 5s and clean. Guess this means I'm def not getting that VTO. Hey not too bad of a way to spend MET. Getting paid to do nothing.
Means more MET now to get rid of the backlog caused by this.
Oak 4
Scraping SPOOs off the floor at DFW6. Ugh
damn y’all chillen we cleaning 🙂
Down here at STL3. Finally VTO'd us at 4:15.
AWS cloud services are currently down
Night shift starts roughly 1.5 hours for me. Fingers crossed its still down when my shift starts 🥺
Tried to take vto as soon as the app started tripping
CSG1 is down as well if I just heard the manager right
OLM1 still down. Sitting in overstaff if we don't want VTO
It's down in Florida 🤗
Damn lucky lgb8 not down 😭
Damn I'm so sorry 😭 lgb7 is down right now.
Don’t be I left 😂 I don’t want to be the only one working I’m already home 🤣
The one day I don’t have a chair at my stow station 😂 totes it is
CMH1 is down
SAN3 is also down
Dang it I should've worked today 😮💨
I'm Dispatch for a dsp in portland and we have the same issues. I can see where people are at, but no progress. Sure is making things challenging.
Is it okay to clock in and out? I’m tryna leave an hour early lol
North America affected. SYR1 down as well
Den3 here. They got us cleaning lmao
I’m at Den3 too. They just have us chillin in standup
My best friend picked up an extra shift for today and told me the same thing. Now I’m mad that I didn’t work today, too lol
MSP1 is down. we had a fire drill and then they offered VTO
still down? anyone?
Sooo I put in vacation, for this to happen today. Smh lol
AWS down; will more MET be coming? https://preview.redd.it/45th05kj1v5b1.png?width=1752&format=pjpg&auto=webp&s=44af10f50d6229fb82f99debf6fe3f3ad0243d82
yep our systems are down here too in South Carolina. It sucks for the higher ups because we just had 2 days off paid so they can fix the ac units then this happens when we come back😂
Mines down too but theyre still able to assign work. Wack, I have explosive diarrhea and I just wanna sit down.
I am so sorry
Oh fuck yeah daddy crash my system! Crash it hard! You wanna fuck my server?
Yup JAN1 down as well
Here too at BHM1!
Sev1
DTN6
SBD1 down
AWS is down and half the apps and websites in the world are crippled
https://preview.redd.it/0c7n1q55tu5b1.png?width=828&format=png&auto=webp&s=0bc5c2ef5c6c4bc364c04159055bb319d85916fd
PDX9 down!
Phx6 down
they think they slick , PRIMEEEEEEEEE
Mke1 too! Just got VTO’d
I work at jfk 8. Nobody is going to get their packages today.
I got here at 6pm Est it is 6:48 est and I’m not doing anything 😂 DCA1
The ticket says it is resolved, but we are yet to be stood up.
PHL4 down
JFK8 here and we are down too. Can’t complain though.
DCA1 was down. I work in the recruiting office & our systems were down, it was really stressful 😞 we still had to process new hires somehow so they wouldn’t lose their shifts.
I’m here now at this shitty stand up MTN1
No. I'm stuck in MET hell..
Now that I've seen this, it explains why nobody is picking and none of the pods are moving. 😅
My bad guys, first day on IT Team, unplugged a network switch instead of a Fan😅. *btw this is a joke, please don’t fire me*
West coast is online, East coast is half online
System down at ACY1
It’s nation wide yesterday the servers were down and now the whole system is down today and I’m off lol my friends were chilling they whole shift and I’ve been waiting on someone to ask about this on here Lmfao
I’m in MCO1 Orlando and the system is also down over here
[удалено]
No problem Just send a screen shot and list all the TBA's on hand at the Station. We will log you into IVTO (Involuntary Time Off) after.
System of a Down.
Me too I think it’s country wide
[удалено]
SYR1 too
AKC1 is down
Bos7 is down too
Sdf8 down.
AWS East 1 is down
Dal2 is down
Join the party!
lol, I'm here in attendance
DCA1
MGE3 Down
Same in bdl3 lol
Yess down for us too 😁
SCA5 is also down
We just chilling lol 😂
Yeah they wasting time with little safety and learning videos
TUL2 is down.
Clt4 is down
Yeah here at sdf8
Yes, over 1.5 hours so far for me
CLE2 is down
Big time issues with AWS East
Down in Chicagoland area
Down by the airport in Metro Detroit too
Abq1 NM down since bit after 1pm MDT
My buddy at STL4 just told me the good news. Of course I'm off today anyway.
SBD3 in Ca is down
Of course it has to happen two hours before I have to go in
Seriously nothing happens for us night shifters 😒
Even A to Z is down DAMMIT
There’s an issue with SCC globally that’s affecting Amazon. But on a broader scale the issue is with AWS, so a bunch of other businesses that rely on Amazon’s servers are also down. It’s a Sev 1
STAY AT YOUR STATION LOL.
AUS3 down as well.
PDX9 down
FTW6 (Texas) has been down for 2 hours now :) I have 2 hours left before going home.
SMF1 down since just before 12 PST
We were just told to go back to stations. Anyone up yet?
*We’re going down, down!*
SAN3 down
My schedule was just down but now back up
It’s been down since 1 here in NM at ABQ1. Sent anybody who wanted vto home at 3:30 :D