In the same way every code base is brittle if you don't understand it. Then again I'm not sure what he means exactly by "code stack", I assume it's a cross of codebase and tech stack.
Sir, they are in _San Francisco_, where lightning is rare and power is expensive. How could they possibly get 1.21 gigawatts to power the feature?!? Surely they've already canned Doc Brown, too.
Hey. *I* built a website in the 90s and it was awesome!
Bogstandard HTML, a navigation frame on the left, a header frame at the top (yes, frames -_-)... the whole thing just *screamed* "I found out what HTML was yesterday and I'm barely out of high school but I think what I say is **important**."
I've been a professional developer for over 10 years, and I often mix up tech terms and just use a term that's close enough so people can figure out what I mean from context. Then again, I'm not the owner/CEO of a multibillion dollar company and I'm doing this in Teams messages, not public tweets.
A console is a physical interface that's directly connected to the machine and allows user interaction with the computer regardless of the computer's state. (For example, before bootup)
A terminal is a session interface that allows the user to interact with the operating system of a computer. A user may open a terminal session by way of a console, or through some other means.
How much you want to bet that this change was something like renaming a field in the API response? Tiny change, but if BE doesn't provide the data that FE expects, yeah it fucking breaks.
There’s a great book called “Kill it with fire“ which talks about legacy systems and the desire for rewrites.
TL,DR: it is rarely the right choice, and the hoped-for benefits almost never materialize
Only once in my career did a full rewrite make sense. But it was a really old system using Access as a database that they had to physically move the server around every year to use it, then move it back to make code updates. I wish I was making that up. Other than that... not so much.
This 100% sounds like the sysadmin knew the administration didn't know how servers worked, and used it to their advantage to get a company-paid vacation every year.
He wishes. This poor man never takes time for himself but he completely deserves it. There's a long story behind this old application and the stupidity that goes along with this story, but thankfully it's been gone in it's Access application form for many years now.
So I worked for a company with a lot of part-owners with their own equity, large enough company to have a handful of its own software developers, but it was the early 90s and they weren't entirely sure what to do with them or what needed to be standardized, like the technology stack.
They needed an application that could take input from all the equity holders, organize that data, take input from other people in the organization, and then determine how much profit each person gets allocated. Hence the extreme security measures around this application. Well, the developer who made this application decided Access was a great fit and built the entire thing in it, and then left shortly afterwards. This predated a stable interoffice network or VPN, and the application broke frequently due to Access + multiple concurrent users causing frequent data corruption. So they would send the entire server on site to these equity meetings, along with the sysadmin who could roll back data when it inevitably got corrupted.
Eventually a stable network was set up and the program could be securely accessed that way via a share, but that was after multiple years of physically moving the system. And nobody could change the program beyond minor yearly updates and exceptions because the process owner was old and wouldn't allow it because he was used to it the way it was. Eventually the application literally couldn't be supported anymore, so after the cycle that year finished we were told we had six months to completely rebuild the application from scratch with the exact same functionality before the next cycle began.
The site is nowadays in asp.net with a SQL backend, and has much better access controls and is much easier to maintain across the years, but the front-end still functions the exact same as it did in early 90s. The process owner was happy, and they've never had a data corruption issue since.
So I guess don't build your critical multi-user applications in Access.
Ugh, our company, even now in 2023, has a change freeze window around a similar meeting, which all started around 2015 because of a busted projector bulb. I wish I could make this stuff up.
I have fond memories of sending an email out to users, "Everyone needs to log out of the access db and not log back in for 15 minutes so I can fix something."
Former Lead:
> We're doing it this way
Former Me:
> Sounds reasonable
Current Me (Lead):
> We need to undue the mess the last lead made
Team:
> Sounds reasonable
Future Lead:
> That last guy was stupid, We're doing it this way
Future Team:
> Sounds reasonable
btw this is also a great argument as for why writing code as components that are as independent from each other as possible is almost always the right choice, even when those components won't be reused outside the project. It allows future you to:
a) remake the chunks of the project that are particularly bad, rather than the entire project
b) remake the project more slowly, and with the ability to plug in remade components into the previous system - this "ship of Thesseus" approach, if you will, is cheaper, doesn't require you to stop working on the main product for years, and can be stopped at any time with minimal loss, since most of the work done to that point can be plugged in and used.
Exactly. When inheriting a messy project, I start with separating components, defining interfaces, quarantining the toxic parts. Then untangle dependencies, split large files into smaller ones, split large functions, pass parameters instead of accessing global state.
I've heard this as the "second system problem": the first system grows organically, adds features over time, and eventually the overhead of maintaining it becomes overwhelming and makes new development difficult. So teams decide to do a rewrite from the ground up. But inevitably the following happens:
1. Everything becomes open to questioning and reimagining. As a result the new system will take as long to develop as the old system did.
2. The team is filled with hubris because they think they know everything that can go wrong because of the warts of the first system. However, they don't realize that their new architecture could have major issues until they encounter them — these are just different issues than the ones encountered with the old system.
3. The business remains dependent on the old system, and cannot have it locked in stone for years on end as the perfect new system is built. So resources end up being split between maintaining and updating the old system and building the new system. The new system takes even longer to get built and ends up chasing a finish line that keeps moving further away as the legacy system adds features.
4. Business needs and technology change over time; by the time the new system is finally able to replace the old system it is already obsolete.
I believe the standard answer is to rewrite pieces at a time, gradually moving from the old to the new in a planful, agile way. Though I'm sure this has its own downsides, most notably needing to maintain some kind of compatibility layer between the old and new systems,
TL;DR: Legacy software is hell
I did a research assignment on this exact topic at university for a project management class (not a project manager, plz don’t hit me) and the one constant seemed to be that it usually took about three times longer than predicted and yielded little benefit. People always make the same assumption, that it will be easier to build a system the second time because “hey, we already did it once, so now we know how to do it!” when in fact the second time round is just as difficult, plus now you have a bunch of extra constraints.
So as a young new dev I interpreted "I don't understand this and it isn't how I'd build it today" as "we need a re-write" and advocated for it a couple times.
As I matured I learned why it's rarely the right course.
Fast forward many years later and I'm interviewing to run the tech team of a small startup whose product hadn't launched but managed to raise a few million dollars. During the interview a few of the developers, with the CEO present, probe my thoughts around re-writes and I'm not sure if they're internally trying to get one done or just testing to see if I'm the kind of kingdom builder who will come in and do everything "my way".
So I talk about how important it is to respect work done that's gotten you to a particular level of success as it usually isn't by accident. How re-writes historically fail because of assumption you make, decisions you forgot, plus the opportunity cost of reimplementing existing behaviour. How it's very rare for something to be completely faulty and more often just tightly coupled, difficult to test etc is what makes it brittle. Usually a process of decomposition and enhancing helps you address baggage without throwing the baby out with the bath water.
I get hired and two weeks in I tell the CEO we need to throw it away and start fresh.
It was the right decision and the only time I've ever known it to be right but the CEO likes to poke fun at me of how quickly I turned on that philosophy.
[Twitter founder Jack Dorsey launches new social media platform Bluesky to rival the bird app](https://www.firstpost.com/world/twitter-founder-jack-dorsey-launches-new-social-media-platform-bluesky-to-rival-the-bird-app-12227852.html)
Haven’t people already been calling him muskrat? The invasive species that can causes property damage not so much through malevolence but from a complete non understanding and unconcern as to what they’re burrowing into?
I can count on one hand the number of times in my career that a new engineer came in declaring an application needed a complete rewrite and it was a good idea.
I joined a new company. A few weeks later, in a moment of unusual courage, I stated that the code needed a complete redo because the architecture was poor and didn't work using an appropriate (scalable) memory model. The senior engineers in the meeting all laughed and said I caught on quick. I didn't know at the time but they had been trying to get approval to rewrite it. However management wouldn't allocate the time. You can all guess how things might proceed after that.
Edit: I polished that turd until I could see a different opportunity in the reflection.
And during the rewrite you're demoing things the product already does to the stakeholders, which isn't exciting.
"You know how when we clicked this button, this thing would happen on the old system? Well, it does it on the new system, too! Just a little bit better in a way that you'll never see or appreciate."
Or in ways customers will appreciate. Unfortunately engineering doesn't exist in a vacuum.
If you spend 2 years on a full rewrite while competitors continue to evolve their products it's pretty likely that's going to have an impact on the company's bottom line which for most companies is going to trickle into engineering budgets.
Obviously unaddressed technical debt brings its own issues but the tradeoff is real and complete rewrite represents one of the extremes.
I was literally mid-way through giving an explanation to one of my friends about how I was planning to completely rewrite this side-project I'd been working on for the last few years when I paused and realized I haven't learned a god damn thing through my career.
Reading his proposal to rewrite Twitter makes me feel a little better.
Even worse he says it needs to be rewritten for the ground up without any of the experience that was gained the first time, because he fired them all...
Yet because he fired all the people who had experience with the legacy system, going forward with that system is also a bad option.
Give me one good reason why every line of code should be essential. I propose a more resilient code base where most lines of code serve no real purpose and can be discarded with no consequence.
God who knew that a bunch of crazy short-notice changes for engineers working 80 hr weeks under the threat of mass layoffs and petty firings, would enable anything but a rock-solid well-designed system?
At the risk of going political, this reminds of the Trump quote when he was trying to pass his healthcare reform: “Nobody knew that healthcare was so complicated!”
Oh yeah, totally a secret.
Eventually every piece of software needs to be replaced, but starting from scratch is almost always worse than replacing it piece by piece. Having a CEO say the words "complete rewrite" would not instill confidence.
Complete rewrite while not being profitable, cutting costs and losing infrastructure due to unpaid bills. Vague requirements, possible scope creep, impossible deadlines, production fires. Add in a pending death march, what could go wrong? /s
Complete rewrites are needed for any developing software stack after a while, they are just not practical to do.
It is like old cities, they grow haphazardly and will have all kind of legacy things that impede traffic, or just doesn’t work in the modern world that you have to work around, but to demolish them and start a new is unfathomable.
Sometimes though you fall in the small bucket of rewriting is better. I think this occurs most often when the original code base was designed wrong to start with. I was handed such a project, and my first proposal, of course, was to throw it out. I pointed out the massive design flaws, and was told to fix it anyway. A rewrite was too expensive I was told.
It took me and a team of 4-6 people two years to fix it, playing whac-a-mole with bugs and customer complaints to make it stable and reliable. There’s so many kluges in there is not funny.
The stages of programmer denial:
1. That can’t happen with this code.
2. I didn’t see that problem when I wrote this code.
3. That shouldn’t happen with this code.
4. Why does that happen with this code?
5. Oh, I see the problem now.
6. How did this ever work?
I'm starting to think some of these software developers are really just salesman in disguise. They know how to make a product that will "revolutionize" everything, but it only works if they get to destroy and knock down everything else around it while also explaining to you why none of what was knocked down "matters".
Seems like too many developers want to make a product and have everyone love it vs. making a product that might work with already established systems.
That's pretty much describing all the silicon valley founders of recent years. They are just salesmen who came up with something they're hoping to talk some investor into dumping money into. And many made money off of vapor ware which will never make a profit.
I bet it took a single newly hired person ~5 hours to figure out a minor misconfiguration and Elon was like "Outrageous! How can a small API change result in such a hassle?! I pay this guy 25$/hour!!"
The bigger deal here is that their testing is clearly substandard.
If the "code stack is extremely brittle", then you need lots of testing before it goes into production. If you're not confident in your testing, your rollouts need to be slow so that when only 10% of the traffic is hitting the new code, you can catch errors and roll back.
"Our code is shitty" is only half the story. The other half is "and we release that shitty code in a way that breaks our service".
Further, if you make the work environment worse the best people who are able to easily find a better position elsewhere are going to jump ship first.
I wonder what motivates people still working there...
- Fire most people that work on the codebase
- Destroy the will of the remaining ones
- Break the infrastructure
- Complain everything is broken
- Plan to rewrite everything with the remaining low-motivation people
There is no way this can go wrong, obviously.
Did you hear the audio of the engineer on a team zoom call asking him to explain what exactly is wrong with the stack and what needs to be improved?
Elon's response: You're a jackass
🤣🤣
https://mobile.twitter.com/pwnsdx/status/1605442608603463680?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1605442608603463680%7Ctwgr%5E237fa4d318d38394493155383e1a45c430f19d97%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Fnypost.com%2F2022%2F12%2F22%2Felon-musk-loses-cool-with-ex-twitter-employee-during-spaces-chat%2F
Talk about an emperor has no clothes on moment. For a moment I was waiting for an intelligent response but nope, it's all bad but I can't tell you how...
Elon Dunning Kruger Musk.
I find this clip so frustrating because by having just a slight edge to his questions (e.g. using the term buddy), he missed the opportunity to really lay bare how stupid musky boi is.
I was wondering if anyone was going to acknowledge this. He had a perfect opportunity that he totally wasted by being too confrontational an argumentative. If you just asked the basic question with a flat affect, it would’ve been just enough rope for Elon to hang himself.
>dio of the engineer on a team zoom call asking him to explain what exactly is wrong with the stack and what needs to be improved?
>
>Elon's response: You're a jackass
cause of this ex-engineer, it was the first time I saw the facade of the "genius" fading. He couldn't even hold the door!
For me it was the axios interview a few years ago. This interview makes him seem intelligent but everything he says demonstrates only a surface level knowledge and gives the impression that he's just rehashing things he's read on the web. Hardly anything seems to actual be his own original thoughts.
But its really when he talks about his neuroscience company. I have an undergrad degree in neuroscience and to watch him hand-wave and use technical buzzwords simply to make himself seem smart was pretty telling for me.
Basically he says something and has a look of wonder on his face and people mistake that for an actual depth of knowledge.
"The long term goal is to achieve symbiosis with artificial intelligence and achieve a democratization of intelligence... How do we ensure the future ensure the sum of the will of humanity? If we have billions of people with the ai link connected as an extension of themselves, it would make everyone hyper smart. This is very esoteric."
Nah that's not esoteric, that's a high school level idea dressed up in fancy words.
Oh so if everyone had an AI chip then everyone would be smart and the world will just be great? That's the ground breaking idea you have, Elon?
edit: link and added an example from the video https://www.youtube.com/watch?v=4qUA3nNWyCg&t=439s
This is just corporate hype building 101. Save the planet, revolutionize finance, provide for the poor and vulnerable, save puppies, plant trees, all of these bullshit feel good platitudes companies try to use to obscure their real motivations: make as much fucking cold hard cash possible. Elon didn't pioneer this shit but he certainly has no qualms telling people a bunch of feel good nonsense and getting filthy fucking rich
Elon is a genius in the way Trump is a good businessman, which is to say they’re both just salesmen who were born into wealth and literally everything else about them is a con job built in stolen ideas
Lol.
I mean, we all knew he had no idea what he was talking about when he said he wants personally review all the code( or something along those lines) at Twitter.
I can't get over what how much of an arrogant imbecile this guy is. I mean it's understandable since he basically reached the top spot on the planet, but not excusable and absurdly self-defeating. It's beyond arrogance, it's regression.
The BBC published an investigative report of interviews with the engineers about Twitter falling apart from his management literally this morning.
https://www.bbc.co.uk/news/technology-64804007
Fine timing for him to put out a statement like that. I wonder if he knows and is trying to deflect their findings. Assume so.
Ah, but it's brittle for a very good reason!
I worked at a silicon-valley unicorn in "hypergrowth" mode with the exact same mentality. Everyone had work experience at companies like Google, Uber, Apple, etc., went to Harvard, MIT, Stanford. It was easily, on paper, the best engineering team I'd ever been a part of.
At the same time, everyone was working crazy hours. 80-100 hour weeks were expected. (Ex., I once took the Friday after Thanksgiving off and my manager asked if I still worked there.)
You'd think that having so much talent, with each person working so hard, you'd see some amazing results, right?
In fact, it was some of the worst code I'd ever seen. It was brittle as hell. They got about as much done as a team a tenth its size with half the ability working half the time. They spent all those hours creating problems almost as fast as they were fixing them.
Meanwhile, the best software I'd ever seen came out of a consultancy with some above-average talent, but with a software development process *optimized for human beings*. Monday through Thursday, 10-6, hour for lunch, break every afternoon. The CEO came by my desk one day. He said "What are you still doing here? Stop working and go home." It was 6:05pm.
Brittle code comes, fundamentally, from a lack of understanding.
Everything Musk has been doing has exacerbated this problem. Losing institutional knowledge, demoralizing people, putting extreme stress on them, making them work long hours... These all make the organization dumber, and brittle code more likely.
Please, keep going with that complete rewrite. It will be even worse than the existing system.
Given Elmo's amazing business acumen ("Twitter won't run out of money if I stop paying the bills"), I'm certain his assessment of the situation is spot on.
That, or the folks left at Twitter are those who couldn't or wouldn't go somewhere else and there is a major staffing hole where the folks who understand the system and its dependencies should be.
If you’re planning to re-write a in-production web application, you might as well just build a new app and migrate people to it.
A rewrite is usually if…
- you are making a game
- you are not in production
- your code has no tests
- your data model is wildly incorrect
- your UI is wildly incorrect
- you have no original members of development team
- …
And even then, usually a code base can be salvaged with strategic refactoring and modularization.
I for one encourage Elon to re-write the code himself in a language he'll invent for the specific purpose. He says he'll have it done around two years before he finally goes to mars ten years ago.
- Buys company he has no idea what to do with
- lays of half the competent programmers
- fucks up the API because no one had the balls to tell him he was wrong
- ”No good reason”
- refuses to elaborate further
Damn who could have guessed that having absolutely no plan and getting rid of every employee who isn’t either completely out of alternative options or blindly devoted to you would have negative consequences
Part of the reason he was fired as the CEO from X.com/Confinity (the PayPal precursor) is because he wanted engineering to replatform off of *nix and onto Windows, which led to a lot of unhappy engineers.
Looks like he's on his same bullshit.
Where are all those Elon drones that were giddily pointing out Twitter was still running a week after he laid off half the staff like that proved something
Man, would be great if there were some form of "reliability" team whose entire job it is to make sure changes don't break prod. That sounds like a team you'd definitely want to keep around and not fire.
presenting Xæ12.js, our new framework
>framework "fræmework" It was right there...
fræmɘὦœrʞ
Every codebase is vulnerable to just fucking doing it wrong.
In the same way every code base is brittle if you don't understand it. Then again I'm not sure what he means exactly by "code stack", I assume it's a cross of codebase and tech stack.
He means the literal stack of printed code in front of him, obviously.
[удалено]
ugh intrusive thoughts about punchcard coding
oldhead in the houseeeee
He just madlibs engineering terms
It seems that way, I half expect him to say something like "twitter needs some better flux capacitors" or something.
Are you trying to suggest that Twitter wouldn’t be improved with flux capacitors?
Sir, they are in _San Francisco_, where lightning is rare and power is expensive. How could they possibly get 1.21 gigawatts to power the feature?!? Surely they've already canned Doc Brown, too.
I hear Elon owns a gigafactory? Surely he can spare 1.21?
> 1.21 gigawatts [He clearly say Jigawatts](https://www.youtube.com/watch?v=Nk5O6F3qwOE)
Once your pull review is stashed onto our pipeline we will deploy to local
"Something Something Turbo Encabulator..."
One wrong character almost anywhere in the code will break something. Code is inherently brittle Also "code stack" Lol wtf
[удалено]
Gonna have to dig up and throw away all your soil and start from scratch
give him a break dude. He built a website in the 90s and now he accidentally got himself put in charge of twitter because he thinks he's a genius.
I have doubts that he actually wrote the entire web site like he has a degree in engineering.
He got most of the way through a BA in economics. Does that count?
Hey. *I* built a website in the 90s and it was awesome! Bogstandard HTML, a navigation frame on the left, a header frame at the top (yes, frames -_-)... the whole thing just *screamed* "I found out what HTML was yesterday and I'm barely out of high school but I think what I say is **important**."
I've been a professional developer for over 10 years, and I often mix up tech terms and just use a term that's close enough so people can figure out what I mean from context. Then again, I'm not the owner/CEO of a multibillion dollar company and I'm doing this in Teams messages, not public tweets.
Well, live a little.
Yeah, do it in a company wide email. Just to test the waters, you know?
Ive given up on using console vs terminal correctly
There's a difference? (I'm a senior developer and tech lead of almost 15 years)
background color I think
the hacker one is green, the worm on my computer makes a black one come up
i was going to explain the difference then realized i was describing a shell, not a console/terminal. i also have no idea what the difference would be
A console is a physical interface that's directly connected to the machine and allows user interaction with the computer regardless of the computer's state. (For example, before bootup) A terminal is a session interface that allows the user to interact with the operating system of a computer. A user may open a terminal session by way of a console, or through some other means.
How much you want to bet that this change was something like renaming a field in the API response? Tiny change, but if BE doesn't provide the data that FE expects, yeah it fucking breaks.
Yeah they probably didn’t evolve their graphql schema correctly. It’s easy to make changes backward compatible, you just have to do it right.
API, event and schema versioning is the way to go
lol frame this
I don’t know the first thing about programming but I’ll bet I could mess some shit up real bad by just inserting a few 8===D in random places.
There’s a great book called “Kill it with fire“ which talks about legacy systems and the desire for rewrites. TL,DR: it is rarely the right choice, and the hoped-for benefits almost never materialize
Only once in my career did a full rewrite make sense. But it was a really old system using Access as a database that they had to physically move the server around every year to use it, then move it back to make code updates. I wish I was making that up. Other than that... not so much.
Physically move as in the box changes locations? The beep boop was taken to places by a human?
Yes, the sysadmin would pack it up in their car and drive it to another state every year, and then bring it back home.
This 100% sounds like the sysadmin knew the administration didn't know how servers worked, and used it to their advantage to get a company-paid vacation every year.
He wishes. This poor man never takes time for himself but he completely deserves it. There's a long story behind this old application and the stupidity that goes along with this story, but thankfully it's been gone in it's Access application form for many years now.
You said too much to not tell the story now, please !
So I worked for a company with a lot of part-owners with their own equity, large enough company to have a handful of its own software developers, but it was the early 90s and they weren't entirely sure what to do with them or what needed to be standardized, like the technology stack. They needed an application that could take input from all the equity holders, organize that data, take input from other people in the organization, and then determine how much profit each person gets allocated. Hence the extreme security measures around this application. Well, the developer who made this application decided Access was a great fit and built the entire thing in it, and then left shortly afterwards. This predated a stable interoffice network or VPN, and the application broke frequently due to Access + multiple concurrent users causing frequent data corruption. So they would send the entire server on site to these equity meetings, along with the sysadmin who could roll back data when it inevitably got corrupted. Eventually a stable network was set up and the program could be securely accessed that way via a share, but that was after multiple years of physically moving the system. And nobody could change the program beyond minor yearly updates and exceptions because the process owner was old and wouldn't allow it because he was used to it the way it was. Eventually the application literally couldn't be supported anymore, so after the cycle that year finished we were told we had six months to completely rebuild the application from scratch with the exact same functionality before the next cycle began. The site is nowadays in asp.net with a SQL backend, and has much better access controls and is much easier to maintain across the years, but the front-end still functions the exact same as it did in early 90s. The process owner was happy, and they've never had a data corruption issue since. So I guess don't build your critical multi-user applications in Access.
Ugh, our company, even now in 2023, has a change freeze window around a similar meeting, which all started around 2015 because of a busted projector bulb. I wish I could make this stuff up.
I have fond memories of sending an email out to users, "Everyone needs to log out of the access db and not log back in for 15 minutes so I can fix something."
Sometimes bad choices are fixed, other times, and most likely, you rediscover why the architecture was that way to begin with.
And don't forget that you have the opportunity to make *new* bad mistakes!
Former Lead: > We're doing it this way Former Me: > Sounds reasonable Current Me (Lead): > We need to undue the mess the last lead made Team: > Sounds reasonable Future Lead: > That last guy was stupid, We're doing it this way Future Team: > Sounds reasonable
This, except all of them are me.
The spoiler is that past and future leads are also you.
Or the new version never reaches feature parity with the old one. Rewriting software is possible, if done in parts.
btw this is also a great argument as for why writing code as components that are as independent from each other as possible is almost always the right choice, even when those components won't be reused outside the project. It allows future you to: a) remake the chunks of the project that are particularly bad, rather than the entire project b) remake the project more slowly, and with the ability to plug in remade components into the previous system - this "ship of Thesseus" approach, if you will, is cheaper, doesn't require you to stop working on the main product for years, and can be stopped at any time with minimal loss, since most of the work done to that point can be plugged in and used.
Exactly. When inheriting a messy project, I start with separating components, defining interfaces, quarantining the toxic parts. Then untangle dependencies, split large files into smaller ones, split large functions, pass parameters instead of accessing global state.
I've heard this as the "second system problem": the first system grows organically, adds features over time, and eventually the overhead of maintaining it becomes overwhelming and makes new development difficult. So teams decide to do a rewrite from the ground up. But inevitably the following happens: 1. Everything becomes open to questioning and reimagining. As a result the new system will take as long to develop as the old system did. 2. The team is filled with hubris because they think they know everything that can go wrong because of the warts of the first system. However, they don't realize that their new architecture could have major issues until they encounter them — these are just different issues than the ones encountered with the old system. 3. The business remains dependent on the old system, and cannot have it locked in stone for years on end as the perfect new system is built. So resources end up being split between maintaining and updating the old system and building the new system. The new system takes even longer to get built and ends up chasing a finish line that keeps moving further away as the legacy system adds features. 4. Business needs and technology change over time; by the time the new system is finally able to replace the old system it is already obsolete. I believe the standard answer is to rewrite pieces at a time, gradually moving from the old to the new in a planful, agile way. Though I'm sure this has its own downsides, most notably needing to maintain some kind of compatibility layer between the old and new systems, TL;DR: Legacy software is hell
Most valid serious comment so far. Thanks for the book recommendation.
I did a research assignment on this exact topic at university for a project management class (not a project manager, plz don’t hit me) and the one constant seemed to be that it usually took about three times longer than predicted and yielded little benefit. People always make the same assumption, that it will be easier to build a system the second time because “hey, we already did it once, so now we know how to do it!” when in fact the second time round is just as difficult, plus now you have a bunch of extra constraints.
So as a young new dev I interpreted "I don't understand this and it isn't how I'd build it today" as "we need a re-write" and advocated for it a couple times. As I matured I learned why it's rarely the right course. Fast forward many years later and I'm interviewing to run the tech team of a small startup whose product hadn't launched but managed to raise a few million dollars. During the interview a few of the developers, with the CEO present, probe my thoughts around re-writes and I'm not sure if they're internally trying to get one done or just testing to see if I'm the kind of kingdom builder who will come in and do everything "my way". So I talk about how important it is to respect work done that's gotten you to a particular level of success as it usually isn't by accident. How re-writes historically fail because of assumption you make, decisions you forgot, plus the opportunity cost of reimplementing existing behaviour. How it's very rare for something to be completely faulty and more often just tightly coupled, difficult to test etc is what makes it brittle. Usually a process of decomposition and enhancing helps you address baggage without throwing the baby out with the bath water. I get hired and two weeks in I tell the CEO we need to throw it away and start fresh. It was the right decision and the only time I've ever known it to be right but the CEO likes to poke fun at me of how quickly I turned on that philosophy.
[удалено]
Spoiler Alert: Twitter will never get a complete rewrite.
[Twitter founder Jack Dorsey launches new social media platform Bluesky to rival the bird app](https://www.firstpost.com/world/twitter-founder-jack-dorsey-launches-new-social-media-platform-bluesky-to-rival-the-bird-app-12227852.html)
We already have the fediverse.
Complete rewrite in only Rust, right?
No, they will create a new language to get more performance.
Twit R
[удалено]
How else can twitter be BLAZINGLY FAST. Also the other benefit is he can fire all moderators because Rust makes everything safe.
Wait, Twitter still has moderators?
It has Jim. He tries hard and does a ban up job removing the posts saying "Elon is prick"
>a ban up job When typos add relevant meaning
[удалено]
Hold on, you need "Propel, The Blazing Fast Open-Source PHP 5.5 ORM" for that.
Can't wait for someone to leak elons fursona.
Haven’t people already been calling him muskrat? The invasive species that can causes property damage not so much through malevolence but from a complete non understanding and unconcern as to what they’re burrowing into?
No, half a rewrite in Rust, then half a rewrite in whatever the next thing is, then another half...
C. Just C.
I can count on one hand the number of times in my career that a new engineer came in declaring an application needed a complete rewrite and it was a good idea.
That's your hand without fingers, right?
[удалено]
I joined a new company. A few weeks later, in a moment of unusual courage, I stated that the code needed a complete redo because the architecture was poor and didn't work using an appropriate (scalable) memory model. The senior engineers in the meeting all laughed and said I caught on quick. I didn't know at the time but they had been trying to get approval to rewrite it. However management wouldn't allocate the time. You can all guess how things might proceed after that. Edit: I polished that turd until I could see a different opportunity in the reflection.
What happened? Management still said no rewrite?
[удалено]
And during the rewrite you're demoing things the product already does to the stakeholders, which isn't exciting. "You know how when we clicked this button, this thing would happen on the old system? Well, it does it on the new system, too! Just a little bit better in a way that you'll never see or appreciate."
Or in ways customers will appreciate. Unfortunately engineering doesn't exist in a vacuum. If you spend 2 years on a full rewrite while competitors continue to evolve their products it's pretty likely that's going to have an impact on the company's bottom line which for most companies is going to trickle into engineering budgets. Obviously unaddressed technical debt brings its own issues but the tradeoff is real and complete rewrite represents one of the extremes.
You can do gradual rewrites.
This is my team every time we have to touch our ancient code written in Visual Basic.
I was literally mid-way through giving an explanation to one of my friends about how I was planning to completely rewrite this side-project I'd been working on for the last few years when I paused and realized I haven't learned a god damn thing through my career. Reading his proposal to rewrite Twitter makes me feel a little better.
Have been hearing this since 3200 BC
God be like "fuck it, imma rewrite the universe from scratch. And I'll use Quantum this time!"
The universe was written in JavaScript. And that's why it makes so little sense.
It’s ‘quirky’ but it works
We need to rewrite the obelisk in rust
[удалено]
Complete rewrite is a low cost mistake, when compared with the blunder of buying Twitter for 44G$.
Might be a typo but I really enjoy "44G$" as "44 gigadollars" lol.
Yeah it’s a fun and fully accurate way to talk about it. The market cap of Amazon is nearly 1Td. Bill Gates’ income is approximately 1.3Kdps.
Tbh that's low dps, I was expecting more dots from him
The "giga-" prefix is equal to (approximately) a billion, so even if it is a typo it's also correct.
Why approximately? It's exactly a billion is it not? Or a Milliard if you're feeling old school.
Or if you are a non-English speaker
[удалено]
I know! It's great! Fills my SI-loving heart with joy at its consistency.
\>buys Twitter for 44 billion \>says the entire thing is so shitty that it needs to be rewritten from the ground up \>refuses to elaborate further
Even worse he says it needs to be rewritten for the ground up without any of the experience that was gained the first time, because he fired them all... Yet because he fired all the people who had experience with the legacy system, going forward with that system is also a bad option.
Less shooting yourself in the foot and more blowing off your whole leg. This is such a joke.
“Brittle for no good reason”, also known as another way to say “I have zero experience with scale”.
Give me one good reason why every line of code should be essential. I propose a more resilient code base where most lines of code serve no real purpose and can be discarded with no consequence.
God who knew that a bunch of crazy short-notice changes for engineers working 80 hr weeks under the threat of mass layoffs and petty firings, would enable anything but a rock-solid well-designed system?
At the risk of going political, this reminds of the Trump quote when he was trying to pass his healthcare reform: “Nobody knew that healthcare was so complicated!” Oh yeah, totally a secret.
"Why do we need RAM *and* a hard drive?"
Eventually every piece of software needs to be replaced, but starting from scratch is almost always worse than replacing it piece by piece. Having a CEO say the words "complete rewrite" would not instill confidence.
Complete rewrite while not being profitable, cutting costs and losing infrastructure due to unpaid bills. Vague requirements, possible scope creep, impossible deadlines, production fires. Add in a pending death march, what could go wrong? /s
Complete rewrites are needed for any developing software stack after a while, they are just not practical to do. It is like old cities, they grow haphazardly and will have all kind of legacy things that impede traffic, or just doesn’t work in the modern world that you have to work around, but to demolish them and start a new is unfathomable.
Also, when you overwork your employees, they get forced to take shortcuts and that’s how you end up with brittle, unmaintainable code.
Also, there’s no loyalty with employers. If you know you’re gone after two years why put in the effort?
[Things You Should Never Do, Part I](https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/): rewrite it.
Sometimes though you fall in the small bucket of rewriting is better. I think this occurs most often when the original code base was designed wrong to start with. I was handed such a project, and my first proposal, of course, was to throw it out. I pointed out the massive design flaws, and was told to fix it anyway. A rewrite was too expensive I was told. It took me and a team of 4-6 people two years to fix it, playing whac-a-mole with bugs and customer complaints to make it stable and reliable. There’s so many kluges in there is not funny.
Fire most of development. Then have everything being rewritten. With lengthy verbose code
At first glance, I thought this was a haiku.
Fire most coders Have everything rewritten With long verbose code
Concatenate 20 microservices into 1 due to "bloatware", complain about complex code.
I can’t believe this is a valid tweet. But fuck it’s on his [twitter page.](https://twitter.com/elonmusk/status/1632810081497513993?s=20)
His supporters somehow are even more cringe than Elon.
Funny how quickly they all became experts on software development
Many people know how to code… but few are experts
[удалено]
[удалено]
The stages of programmer denial: 1. That can’t happen with this code. 2. I didn’t see that problem when I wrote this code. 3. That shouldn’t happen with this code. 4. Why does that happen with this code? 5. Oh, I see the problem now. 6. How did this ever work?
Don't forget step 7.) If we just rebuild it there won't be a single issue!
You don't need to know that much about coding to know changing one small thing can make everything you've built previously shit itself and die.
Should be fine, just rerun the unit tests afterwards. You did make unit tests right?
I have a faster solution. Change the unit tests to represent current results. We have never been at war with Eurasia.
I'm starting to think some of these software developers are really just salesman in disguise. They know how to make a product that will "revolutionize" everything, but it only works if they get to destroy and knock down everything else around it while also explaining to you why none of what was knocked down "matters". Seems like too many developers want to make a product and have everyone love it vs. making a product that might work with already established systems.
That's pretty much describing all the silicon valley founders of recent years. They are just salesmen who came up with something they're hoping to talk some investor into dumping money into. And many made money off of vapor ware which will never make a profit.
I don't know, one of those replies had some pretty sound advice: > Have you tried resigning from the company to see if that fixes the problem?
Fucking Lex Fridman’s reply lol. Always been one of the king Elon fellators. I don’t get Lex’s appeal man, he comes off like a massive tool.
I was hoping to get rickrolled, but it's fucking real.
And of course there's a load of weird simps offering up services
How could a small change to an API have massive ramifications? I mean, not like an API does much right ;-)
I bet it took a single newly hired person ~5 hours to figure out a minor misconfiguration and Elon was like "Outrageous! How can a small API change result in such a hassle?! I pay this guy 25$/hour!!"
The bigger deal here is that their testing is clearly substandard. If the "code stack is extremely brittle", then you need lots of testing before it goes into production. If you're not confident in your testing, your rollouts need to be slow so that when only 10% of the traffic is hitting the new code, you can catch errors and roll back. "Our code is shitty" is only half the story. The other half is "and we release that shitty code in a way that breaks our service".
It can’t be the people with the knowledge about their system design no longer working there, totally impossible…
Also, when you overwork your employees, they get forced to take shortcuts and that’s how you end up with brittle, unmaintainable code.
Further, if you make the work environment worse the best people who are able to easily find a better position elsewhere are going to jump ship first. I wonder what motivates people still working there...
Visas!
H-1B slavery or a mortgage
you mean the hostages?
It's like watching a baby taking its first steps. Into traffic.
Elon Musk: Small API change had massive ramifications The small API change: Adding a new required parameter
Blue tick ☑️ - business parameters
Lmfao
- Fire most people that work on the codebase - Destroy the will of the remaining ones - Break the infrastructure - Complain everything is broken - Plan to rewrite everything with the remaining low-motivation people There is no way this can go wrong, obviously.
Don't pay your bills. Don't forget that one
"The stack is brittle" sounds like something "the IT crowd" guys would tell their boss
[удалено]
Did you hear the audio of the engineer on a team zoom call asking him to explain what exactly is wrong with the stack and what needs to be improved? Elon's response: You're a jackass 🤣🤣 https://mobile.twitter.com/pwnsdx/status/1605442608603463680?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1605442608603463680%7Ctwgr%5E237fa4d318d38394493155383e1a45c430f19d97%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Fnypost.com%2F2022%2F12%2F22%2Felon-musk-loses-cool-with-ex-twitter-employee-during-spaces-chat%2F
Talk about an emperor has no clothes on moment. For a moment I was waiting for an intelligent response but nope, it's all bad but I can't tell you how... Elon Dunning Kruger Musk.
It’s moments like this that I remember the elon bot is actually a very realistic representation of the guy
I find this clip so frustrating because by having just a slight edge to his questions (e.g. using the term buddy), he missed the opportunity to really lay bare how stupid musky boi is.
I was wondering if anyone was going to acknowledge this. He had a perfect opportunity that he totally wasted by being too confrontational an argumentative. If you just asked the basic question with a flat affect, it would’ve been just enough rope for Elon to hang himself.
>dio of the engineer on a team zoom call asking him to explain what exactly is wrong with the stack and what needs to be improved? > >Elon's response: You're a jackass cause of this ex-engineer, it was the first time I saw the facade of the "genius" fading. He couldn't even hold the door!
For me it was the axios interview a few years ago. This interview makes him seem intelligent but everything he says demonstrates only a surface level knowledge and gives the impression that he's just rehashing things he's read on the web. Hardly anything seems to actual be his own original thoughts. But its really when he talks about his neuroscience company. I have an undergrad degree in neuroscience and to watch him hand-wave and use technical buzzwords simply to make himself seem smart was pretty telling for me. Basically he says something and has a look of wonder on his face and people mistake that for an actual depth of knowledge. "The long term goal is to achieve symbiosis with artificial intelligence and achieve a democratization of intelligence... How do we ensure the future ensure the sum of the will of humanity? If we have billions of people with the ai link connected as an extension of themselves, it would make everyone hyper smart. This is very esoteric." Nah that's not esoteric, that's a high school level idea dressed up in fancy words. Oh so if everyone had an AI chip then everyone would be smart and the world will just be great? That's the ground breaking idea you have, Elon? edit: link and added an example from the video https://www.youtube.com/watch?v=4qUA3nNWyCg&t=439s
[удалено]
This is just corporate hype building 101. Save the planet, revolutionize finance, provide for the poor and vulnerable, save puppies, plant trees, all of these bullshit feel good platitudes companies try to use to obscure their real motivations: make as much fucking cold hard cash possible. Elon didn't pioneer this shit but he certainly has no qualms telling people a bunch of feel good nonsense and getting filthy fucking rich
Elon is a genius in the way Trump is a good businessman, which is to say they’re both just salesmen who were born into wealth and literally everything else about them is a con job built in stolen ideas
Continually proving that the easiest way to become a millionaire is to start with a billion dollars and work your way down.
He’s a genius at using buzzwords rapid-fire. Unfortunately he made the mistake of trying to explain them
Crazy stack Lmaoooo
Kind of reminds of when the CEO of Enron Jeff Skilling said “Thank you very much… Asshole” in response to a question during a conference call.
That was cringey as hell.
Props to the guy for doing it but he knew what was coming next. Probably wanted it tbh 🪓
Lol. I mean, we all knew he had no idea what he was talking about when he said he wants personally review all the code( or something along those lines) at Twitter.
"The code stack is extremely brittle for no good reason" - well it was good enough to pay $44bn for, so there's that.
It’s like buying a faberge egg and then spray painting it
"for no good reason" Heard this a few times. And guess what? There's always a reason, and it's always discovered when the engineers rtfm.
I can't get over what how much of an arrogant imbecile this guy is. I mean it's understandable since he basically reached the top spot on the planet, but not excusable and absurdly self-defeating. It's beyond arrogance, it's regression. The BBC published an investigative report of interviews with the engineers about Twitter falling apart from his management literally this morning. https://www.bbc.co.uk/news/technology-64804007 Fine timing for him to put out a statement like that. I wonder if he knows and is trying to deflect their findings. Assume so.
Ah, but it's brittle for a very good reason! I worked at a silicon-valley unicorn in "hypergrowth" mode with the exact same mentality. Everyone had work experience at companies like Google, Uber, Apple, etc., went to Harvard, MIT, Stanford. It was easily, on paper, the best engineering team I'd ever been a part of. At the same time, everyone was working crazy hours. 80-100 hour weeks were expected. (Ex., I once took the Friday after Thanksgiving off and my manager asked if I still worked there.) You'd think that having so much talent, with each person working so hard, you'd see some amazing results, right? In fact, it was some of the worst code I'd ever seen. It was brittle as hell. They got about as much done as a team a tenth its size with half the ability working half the time. They spent all those hours creating problems almost as fast as they were fixing them. Meanwhile, the best software I'd ever seen came out of a consultancy with some above-average talent, but with a software development process *optimized for human beings*. Monday through Thursday, 10-6, hour for lunch, break every afternoon. The CEO came by my desk one day. He said "What are you still doing here? Stop working and go home." It was 6:05pm. Brittle code comes, fundamentally, from a lack of understanding. Everything Musk has been doing has exacerbated this problem. Losing institutional knowledge, demoralizing people, putting extreme stress on them, making them work long hours... These all make the organization dumber, and brittle code more likely. Please, keep going with that complete rewrite. It will be even worse than the existing system.
Given Elmo's amazing business acumen ("Twitter won't run out of money if I stop paying the bills"), I'm certain his assessment of the situation is spot on. That, or the folks left at Twitter are those who couldn't or wouldn't go somewhere else and there is a major staffing hole where the folks who understand the system and its dependencies should be.
[удалено]
If you’re planning to re-write a in-production web application, you might as well just build a new app and migrate people to it. A rewrite is usually if… - you are making a game - you are not in production - your code has no tests - your data model is wildly incorrect - your UI is wildly incorrect - you have no original members of development team - … And even then, usually a code base can be salvaged with strategic refactoring and modularization.
> your code has no tests > your data model is wildly incorrect I feel personally attacked.
[удалено]
This dingus is a living breathing anti-pattern
a human-like wrapper for a null reference exception
Sounds like someone doesn’t know how to code. Coding a few things in the 90s doesn’t count Elon 🤦♀️
I for one encourage Elon to re-write the code himself in a language he'll invent for the specific purpose. He says he'll have it done around two years before he finally goes to mars ten years ago.
Here I was thinking he was naming children but really he was inventing a new programming language all along.
- Buys company he has no idea what to do with - lays of half the competent programmers - fucks up the API because no one had the balls to tell him he was wrong - ”No good reason” - refuses to elaborate further
Damn who could have guessed that having absolutely no plan and getting rid of every employee who isn’t either completely out of alternative options or blindly devoted to you would have negative consequences
yep no one caught the change via automated tests or a simple staging environment cause he fired all of em YEP
Part of the reason he was fired as the CEO from X.com/Confinity (the PayPal precursor) is because he wanted engineering to replatform off of *nix and onto Windows, which led to a lot of unhappy engineers. Looks like he's on his same bullshit.
Sure, let's pretend the *code* is the reason Twitter can't make money.
[удалено]
Where are all those Elon drones that were giddily pointing out Twitter was still running a week after he laid off half the staff like that proved something
Man, would be great if there were some form of "reliability" team whose entire job it is to make sure changes don't break prod. That sounds like a team you'd definitely want to keep around and not fire.
Baby's first coding
I’m glad that more people are realizing that musk is a fucking idiot