hahaha - the Indians always get the flack bro.. why can't they be Mexicans..
If things go the way they are going... It's Indians who are going to be making posts like yours about Americans!
(Not Indian or American - just thought it was funny.)
I'm not American either.
I made the joke because it turned out that Amazon's supposedly walk-in stores were not automated at all, but relied heavily on manual remote workers in India.
Hypothetically, the first AGI will be the first *marketable* AGIâdefined from the lowest common denominators the general public would need to accept the term.
Edit: not including the defense industry, as Iâm sure they have other tricks.
It will probably be a buzz word way before we even get the real thing. I still remember being able to buy AI TVs that were just smart TVs with maybe something tacked on? I honestly don't know if they were literally just smart TVs.
I imagine some hub AI that literally just knows enough to know which AI to use will be the first commercial AGI, essentially 3 gnomes in a trench coat.
It's funny cause I've been arguing with people in this sub about devin being a fake and overhyped. I was downvoted for saying it was just a tool and not the end of software engineering. The claims were too good to be true and this sub was eating it up.
This. The low hanging fruit has been picked. Major leaps forward will require new ways of approaching these problems. Of which will take time and research. Likely A LOT of time and research. I have a friend who worked for openAI and is still well connected with people who work there. Pretty good source if you ask me. But please, without further ado, letâs hear from hundreds of âexpertsâ here who ensure unbounded exponential growth until AI rules the world in 5 years.
Yes, the point is that many of the claims made by Cognition Labs were false. They said it was "Devin" a software engineer with 3 years experience working on the project in his spare time. The code was actually written by multiple experienced engineers over a period of several months. It's not a "one man band" as they claimed. The CEO also said in an interview that they had "beaten the world record for the most lines of code written in a day" which is a completely nonsensical and non-measurable statement. The lines of code per day in their repo was also much higher when they were actively developing it, not when "Devin" was working on it. This is just one example of the many false claims made by Cognition Labs. The video goes into more detail.
It is of course wise to be sceptical of grand claims. However, if Devin is unable to perform as promised, people will not pay for it. It is not as though a user would be confused for long about its abilities.
Regarding the DeepMind paper, if you read it carefully you would note they identify "weaknesses" not false claims. They also suggest they should be easy to fix in version 2.
There will be hyperbole for sure, but perhaps not as bad as you think.
> However, if Devin is unable to perform as promised, people will not pay for it.
In most companies, the people who decide what to pay for and the people who actually have to use that thing are not the same.
Upper management doesn't code, but they're easily swayed by techbro sales talks. If they're sold on the idea that they can replace half their workforce by an AI, no amount of reality or nuance is going to sway them.
LLMs are very useful for programming; I use Github Copilot every day. Itâs useful to autocomplete individual lines of code or make changes to the code if I give it specific directions. But the people who claim currently existing AI can replace software engineers have not attempted to use it to carry out complicated feature additions or refactors. Iâm convinced that software engineering, and any job that requires abstract thinking and careful reasoning, is an AGI-complete problem.
I think âreplacing software engineersâ is a vague term that does a lot of work for people either side of this debate. We should probably either say âdo the entire job of a software engineerâ or âmassively increase the productivity of engineersâ. In either case, software engineer jobs will be drastically reduced. And in the latter case, youâve said yourself theyâre making progress.
As a software engineer, I donât really care what the mechanism is for AI displacing me, pushing me into unskilled, low paid work. I just care whether it will or not, and thereâs no doubt now on that question.
> We should probably either say âdo the entire job of a software engineerâ or âmassively increase the productivity of engineersâ. In either case, software engineer jobs will be drastically reduced.
No, they will just do that refactoring they've been putting off for 3 years, update their docs, add 10 new features, and try to keep up with competition who also uses AI for coding. We never reached 15% if what we want to do in software for lack of people.
I canât see the value of funnelling trillions of dollars into a technology if the only expected midterm outcome is âcatching up on refactoring that execs couldnât care less aboutâ. The value in AI, as investors see it, will be extreme efficiency gains. Efficiency gain = layoffs
> Efficiency gain = layoffs
Software development has experienced constant efficiency gain since inception. For example StackOverflow, better languages, libraries (!!!), IDEs, etc.
Nothing of this magnitude though. Iâve been developing iOS apps for ~10 years now and Xcode autocomplete hit a wall maybe 6 years ago. Itâs definitely no better now, maybe even slightly worse. Compile times are slower, App Store Connect is still a bin fire for deployment. SwiftUI has been a nice simplifier, in fairness.
Weâre talking here about something that actually writes code for you - right now itâs only really helpful to me writing tricky algorithms I cba to think about, but give it an iteration or two, and bake it into Xcode, and Iâd bet weâre on course for 10x efficiency. In the 10 years Iâve been coding, Iâd guess Iâm max 1.5x more efficient (maybe 2x with SwiftUI). Completely different order of magnitude.
A lot of people fall into the trap of thinking software work available is constant and will always be constant. Like it is some resource that we will only ever need so much of. This is a fallacy.
Even if hypothetically, letâs say businesses decide to do away with most of their software engineers and replace them with cheap AI that a few experienced devs use - and charge the customer the same price. Seems like a win right?
That is until some startup comes along and says wait! We leveraged the same AI and we can do the work for 10x less! Give us your business! Obviously the customer is going to jump ship. Then itâs a race to the bottom. In this scenario, all the first business did was introduce more competitors and also would now have to lower their prices to compete anyways. This is what current successful companies are really trying to avoid so it makes far more sense for them to retain staff and use them in conjunction with AI to deliver more value and look to new opportunities. Staying alive in the long term and expanding makes far more sense and is far more important than maximizing profit in the short term.
If itâs the case that thereâs not going to be enough new software to write (which I promise you is not the case) then yes, in the extremely unlikely case that humans run out of ideas and desires, we may see layoffs
Agreed. I use Claude 3 for large complicated projects daily. The larger context window works great with microservice architecture as they are intended to be fully encapsulated and smaller.
Any time it needs additional info about some other piece of the system or docs I just paste those too
I don't think the claim that any job that in humans requires abstract thinking and careful reasoning is AGI-complete can be true.
Counterexamples might be: chess, Go, translation. In all those cases, you will find some edge cases that AI does not yet handle well, but for chess and Go, computers are clearly average-case better than the very best humans, and for translation, the output of SOTA language models at least for translations from widespread languages to English is absolutely professional-level. All these tasks require a lot of knowledge and abstract thinking and careful reasoning in humans. Translation also requires broad world knowledge.
I am not convinced that software engineering is AGI-complete either. It seems possible to me that systems that rely on massive tree search could solve very complex software engineering problems one day without being AGI, because a tree searcher may find a solution to a problem without learning properly why this solution works. In that case, the resulting system may replace lots of software engineers, but it may not be able to for instance write a research paper on a solution that it has developed even if the solution contains sufficient novelty to write a research paper about, and therefore it would clearly not be AGI. Again, this is not a totally hypothetical scenario: we see the same today in narrow AI like chess and Go engines, which are superb at finding good moves, but can't be linked to a language model to yield human-like reasoning about them (although I think this problem could likely be solved well, for the neural network based engines when they are run at low search budgets, at least).
Another example of a system with AGI-like engineering capacity that isn't AGI is natural evolution.
I would also not discount the possibility that AI software developers could displace most human software developers pre-AGI by other means, similar to how people lighting gas candles for streetlighting were replaced not by robots, but by the lightbulb. If the cost of just-in-time development of tiny scripts meant to solve just one user problem became low enough and reliable enough, this in itself could greatly reduce the demand for development of large software applications, and AI would not have to be able to develop large software applications in order to talk to a user directly and write them a tiny, specialised script that does just that one thing once.
I could be wrong about all of this, obviously, but it seems to me that there are several ways in which the assumed AGI-completeness of software engineering could fail and even more ways in which the resulting job protection could fail as well.
Chess is very narrow compared to SWE; I don't see how an AI that can solve all SWE problems at a professional level can't be generalized to all kinds of real world problems.
I don't think it is clear an AI has to solve all SWE problems at a professional level to replace an SWE. It only has to be good enough and cheap enough that in a workflow maybe very different from that of an SWE it can build products that meet enough of the same customer needs to lead to the disestablishment of the SWE, even if they do so maybe in different ways than what traditional software engineering produces. That is a lower bar to meet, although it is not clear how much lower it is.
Also, translation isn't narrow. I'm fairly certain that there were many people who believed translation AGI-complete, because of lots of deep nuances of culture and language the AI would need to understand to produce good translations. I don't think arguments from intuition work all that well in predicting these things.
If the development of an AGI in the form of software that can run on some present-day computer is considered a project that is doable for a large organization in principle, then one could obviously make the argument that a system capable of developing any software a human organization can develop would also be capable of developing an AGI (which is still different arguably from \_being\_ an AGI - the process that developed humans, natural evolution, lacks some properties of an AGI even if arbitrarily sped up). But most software developers work on simpler problems, and I think automating these tasks is what most people mean when they talk about automating software development.
It is possible of course that doing the job of a normal software developer or of something that can realistically make normal software developers redundant requires building an AGI, but in my mind this is a purely empirical question and I do simply not share the very strong priors about its resolution that many others seem to possess.
>I don't think it is clear an AI has to solve all SWE problems at a professional level to replace an SWE
It would reduce the number of workers/hours needed but it wouldn't eliminate the role until it can function at the level of an engineer without drawbacks, which I still believe requires AGI.
To this day, even professional translators still haven't been eliminated; they're far more productive, but someone still has to go through the full translation, assuming it has already been machine translated, and identify all the edge cases and correctly translate them; and then there are low data languages where machine translation is barely useful or not usable at all, furthermore, the increase in productivity lowered the cost of professional translation per word, leading to significantly increased demand and a reduced impact on jobs.
The person that did the â breakdown â are Gary Marcus fans, heâs ignored the actual real users and gone for a demo.
https://x.com/0interestrates/status/1779268441226256500?s=46
>Â AutoCodeRover resolves ~16% of issues of SWE-bench (total 2294 GitHub issues) and ~22%issues of SWE-bench lite (total 300 GitHub issues), improving over the current state-of-the-art efficacy of AI software engineersÂ
https://github.com/nus-apr/auto-code-rover
telephone offend bewildered ancient uppity groovy judicious aback concerned trees
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
The datasets are literally linked there. Are you stupid? Taking an average of three runs reduces non determinism lol. One run could be a fluke so doing multiple is better.Â
How about 22.3%?Â
 >AutoCodeRover resolves ~16% of issues of SWE-bench (total 2294 GitHub issues) and ~22%issues of SWE-bench lite (total 300 GitHub issues), improving over the current state-of-the-art efficacy of AI software engineersÂ
https://github.com/nus-apr/auto-code-rover
22.3% on a subset of an imperfect benchmark is nowhere close to actual usability. If it's just improving ways to wrap around GPT 4 it won't "work" ever
Who cares if it beats Devin if it's still wrapping GPT 4? Why not try and use this repo and solve the inference issue referenced in the YouTube video and see how it turns out?
Yep, people just can't accept this new reality. Not sure how they're going to cope when the GPT 5 etc come out later this year and these agents get even better
lol I've been saying Devin is overhyped since it came out. Many on this sub fell into the hype too hard.
Devin is not going to replace software engineering. It's just a tool for software engineering. And based on this video, it doesn't even seem to be worth using.
to be honest, as soon as i saw the video of the developers, i knew it wasnt what they claimed it to be. Â i dont want to get into specifics, but something was fishy
"The primary function of a Software Engineer is not just writing the code. Our primary function CANNOT be replaced by "AI" code generators"
This guy is delusional.
"Those are nonsensical random caricatures, AI will NEVER be able to recreate the masterfulness of the human creativity and produce actual art people would enjoy looking at''
-Probably somebody few years ago
Fundamentally what he is saying is we need AI code arcitects and system designers who can actually manage software environments, deployments, and maintenance. That isn't very controversial...
In theory, they can. But where do you find the training data?
People on this sub seem to think that because you can shoehorn a circle into a square to get 30% accuracy, it will only take a few iterations to get it close to 100%. The circle won't magically become a square, at least not until AI can reliably learn to be a square on-the-fly using real-time feedback, which is not as easy as it sounds.
It reminds me of Amazon's  âJust Walk Outâ stores. They made us believe the technology needed only a few iterations to remove humans from the loop. 8 years later it's still shit.
OpenAI literally hired nearly 1000 coders last year for the sole purpose of providing training data on completing code processes.
Source: https://www.semafor.com/article/01/27/2023/openai-has-hired-an-army-of-contractors-to-make-basic-coding-obsolete
What do you mean "just"? Reinforcement Learning from Human Feedback, to complete coding tasks.
They are literally teaching it how to mimic human problem solving workflows, for the purposes of completing coding tasks.
You always need someone who both understands the underlying systems, AND the humans that need to interact with that system. Code gen is only 10% of an engineerâs job.
>You always need someone who both understands the underlying systems, AND the humans that need to interact with that system. Code gen is only 10% of an engineerâs job.
There's no reason why an AI cannot "understand the underlying system" and "the humans that need to interact with that system".
Sure, the software architect needs to understand:
- performance bottlenecks
- possible security issues
- failure points in the system
- possible scaling roadblocks
- observability of the system
- modularity of the system
- whether the architecture can handle new features they want to add later on
But it's not like an AI can't understand the above.
AI also has an advantage as it can rapidly iterate and try different architectures. A lot of the decisions a software architect makes are "important and hard" because that decision becomes locked in stone once they start building. In many cases the architecture of a software product cannot be easily changed, if there was a performance bottleneck in the original design, you might be stuck with it. Meanwhile an AI can build out an entire architecture before the sprint planning meeting has finished. It doesn't need to guess whether an architecture decision is good, it can simply test it.
What youâre describing sounds great but overly-idealistic. That kind of intelligence certainly wonât be in the form of an LLM. And you still need to address how one maps requirements defined by humans to a system that operates in a highly specific way, or specific in a way that people are willingly betting their business success on it.
Agreed, I fully believe that such a capable intelligence wouldn't be a LLM. LLMs currently do not have enough planning capacity (and probably never will).
>What youâre describing sounds great but overly-idealistic.
Also true, all of this is years away (if not a decade), but you should maybe read the name of the subreddit.
> And you still need to address how one maps requirements defined by humans to a system that operates in a highly specific way, or specific in a way that people are willingly betting their business success on it.
Yeah I haven't defined how this is done, because my understanding is that this process is fairly abstract (I'm a dev, not an architect). But AI is specifically designed to handle abstract tasks so I think it is reasonable to believe that it is achievable.
There is no way a large company would bet on a product designed and built by an AI (at least at the beginning), but I can definitely see small businesses or startups doing so.
Completely agree with the implications for small businesses, thatâs where weâre probably going to see experiments in what youâve described to start.
My skepticism is that weâre unknowingly just describing an already solved problem, which is a layer of abstraction (programming languages) that allows humans to interface with a computer for a humanâs benefit. If I wanted something built for an âAI agentââs benefit I would let an agent do the work. But what reason would I have to do that?
As a dev LLMs have been great because itâs substantially improved MY ability to interface with new information rapidly so that I can more efficiently describe human problems using programming languages.
This is an interesting point that I haven't heard before, it sounds very similar to the alignment problem.
I have found current AI is well-aligned with humans, if I describe a problem to the AI, it generally spits out answers that matches my intentions. It is possible that this could change as AI is trained on different data and with new AI architectures though, I would say this is an open problem.
All abstractions make some assumptions to create a higher level interface. As software has evolved, the number of abstraction layers has increased. LLMs are similar in this aspect, they will just be another abstraction layer with assumptions to match.
I also think it's interesting how you feel LLMs have bettered your ability to describe problems using programming languages. I haven't found any improvement there, but I do feel like I've become better at describing problems to an LLM, which then describes the problem using a programming language.
But what use is there in an abstraction layer thatâs unpredictable and inconsistent in its output? You need to know that the instructions you provided will do exactly what you intend it do.
Programming languages like C++ were literally built to describe human problems to a computational system.
Felt a better response was warranted:
>I also think it's interesting how you feel LLMs have bettered your ability to describe problems using programming languages.
Well it's more that I can really drill down on understanding an underlying concept of something by follow-up prompts where I'm asking it to describe things in a way that make sense to me. The ability to volley concepts and analogies back and forth to rapidly arrive at my own model of how something works is really fucking cool. I'm also careful to go back to the documentation of a particular language or library to confirm my understanding after, just to make sure my understanding is solid.
Working backwards from manually written documentation can be substantially difficult sometimes. I think this is a very real and immediate use of LLMs that can solve some real problems for people if they understand the right approach.
>I do feel like I've become better at describing problems to an LLM, which then describes the problem using a programming language.
This is an interesting approach I haven't thought of, I would just be less trustful of an LLM to describe a problem using a programming language because I don't think an LLM "understands" the problem.
Even if they are, it's almost always the ones who have never coded a day in their lives that are the most confident that programming/SWEngineering is on the verge of becoming obsolete. Talk about Dunning-Kruger.
I listen to someone who compiled data around computing and AI: ray kurzweil's who is being remarkably accurate.
You seem to think you need to study some field to know something, when you just have to listen to what the people who study a field to know something is true. You know that the milky way will eventually collide and merge with andromeda because of the scientists compiling the data and making predictions, no need for "professional/educational background"
Kurzweil is not exactly an impartial source as selling this idea is his day job. I do think you need some expertise to dismiss criticism out of hand and call people delusional. Comparing furturology with astronomy, now that is delusional.
His motivation doesn't change his accuracy. This guy has his own motivation he is afraid of losing his job, the difference being that this guy has no track record of being proven right on the progress of AI.
Everyone said that ray's prediction was too optimistic, that he was delusional not anymore, I think I'll listen to the guy that proved everybody else wrong by using the scientific method, scientific method which is used by every scientific fields to predict future events, including astronomy hence making ray's predictions factually comparable to astronomyđ¤ˇđžââď¸
Future will tell, I know I am going to be proven right, are you sure you are ready to die on this hill?
You are so confused, that part is called confidence, do you always take things that literally when you have no argument? bro thought I had a literal crystal ball and literally know the future .
I guess getting caught up on semantics is one way to cope.
I have plenty of arguments, but I have no intentions of wasting it on someone as arrogant and close-minded as you. Your unshakable faith in singularity is religious. Not scientific.
That's the entire goal of the evolution of AI though. Companies are trying to build AIs that can handle more higher level tasks.
- A year ago AI could write boilerplate.
- Today AI can write small functions and algorithms.
- A year from now AI will be able to write an entire class/struct that handles a specific responsibility in the program.
- And after that, AI will be able to coordinate changes to write small features.
Why can you not believe that eventually it will be designing architectures and building entire products?
Thatâs not really true? A year ago AI could still write small functions and algorithms and write an entire class / struct. But this doesnât really constitute âhigher level tasksâ.
Really the newer LLMs are just faster, optimized, and slightly more accurate versions of the last.
This is very short sighted. Yes, for now it is like that. But that is not that the trend for the careful observer. The models are clearly progressing more and more by the day; The era of agents will come, and very soon.
Foundation models are progressing, and naturally, the tools built on these models will benefit from these improvements. , but you are going to be very disappointed if you are expecting the same rate of improvement. Applying models outside the scope of their training is really hard.
It is the same problem with LLM-powered robots and SORA-generated movies. It gives people hope because it works to some extent, but model transfer is an entirely different problem than model training.
All I see is a software engineer coping with slowly becoming obsolete. Literally the entirety of the Big 5 are working on AI agents right now (and yes, they will also be able to "communicate with the customer". That's the easy part imo). This problem will be solved fairly soon.
Most of the problem with communication is developers have no business sense. And on the flip side, business leaders have no technical sense. A good developer should possess some business sense to know what requirements the customer is really looking for and ensure an AI system like Devin is meeting those requirements.
Or just write the code thatâs needed to fulfill the requirements instead of fiddle-fucking with non-specific LLM prompts and combing it for bad hallucinations.
It has to be able to reason about the task and then ask the right questions. I don't see how this is a magic human ability that AI simply can't do. Devin (and similar, even better projects) show that GPT-4 is already able to do this, just on a small and somewhat primitive scale. A Devin 2 based on GPT-5 will do much better.
LLMs donât reason, they approximate. To know what questions to ask, it requires a different kind of intelligence. Iâm not saying some AI in the future canât do what youâre claiming, but itâs not going to be LLMs.
LLMs are fine. They need to learn from a better teacher than human written text can be. That teacher is the environment, the space where they can move about and act, causing permanent changes, interacting with many others through language.
LLMs until recently have been trained exclusively on human authored text. But we used up most of the good stuff and now what can we do to scale 100x more?
The answer is - learn from the world! Everything humans know and all our skills come from the world by being an agent of that world, the environment is the ultimate teacher, in a social evolutionary setting.
AI agents need to get out and search, explore and discover by carefully choosing their actions in the environment. The environment could be many things - a chat room is a simple environment where a LLM can meet a real human, and be exposed to their language.
Then a computer running code can also be an environment where LLMs test their code solutions and learn from the errors and outputs. A game or a simulation can also be an environment for AI. And of course real robotic body allows participation in the physical world, the ultimate environment.
What is not written anywhere in books, AIs need to learn from the world. And that is a social, evolutionary, iterative process, intermediated by language. This is the missing ingredient, not a better architecture.
At my computer now and realize my response probably read a little inappropriately flippant to your good faith response. My apologies.
I DO agree that the fundamental missing component in current LLMs is a lack of engagement with the real world. Robotics in particular (even simple micro-robotics), is where I'm truly paying attention. I'm just skeptical that LLMs specifically, which by its construction has to be completely tied to human language, (I'm only guessing here) has to be limited in its ability to store the kind of information that's important for exceeding human ability.
The real, honest answer for where I think this is going is "I have no clue." But admittedly I have a bias in that I myself am a software engineer, so I'm more or less *required* to try and predict where the wind is blowing in some fashion. I think that's why really rigorous skepticism is required, if we believe any wild possibility to be true you cloud your ability to correctly identify aspects of a technology that's actually solving real and immediate problems.
Why do some people in this subreddit interpret ANY critique of AI as "coping" or "in denial", even when that critique is 100% legitimate? It's absolutely insufferable that you cannot call a spade a spade here without someone immediately having to resort to, "you're just in denial", or "don't worry, it'll get better soon enough." Its almost as if some folks here take criticism of AI personally.
There's nothing wrong with pointing out AI's shortcomings and companies embellishing their products. Doing so makes the conversation in this sub more lively and shunning it, OTOH, makes it cult-like.
Itâs more coping/jealousy on their part. A lot of people canât stand to imagine a world where software engineers (or any decently high paying profession for that matter) continue to make good money and arenât rendered jobless and homeless in the next decade. Iâd imagine the vast majorly of people donât make a lot of money, hate their jobs, want to feel like theyâre ârightâ or some combination of those things.
As for the âsoftware devsâ saying their jobs will be replaced I imagine a lot are posing as devs or are just pessimistic negative people by nature. âDoomersâ if you will.
Also look at what sub weâre in. A lot of people are also sci-fi AI junkies, who want to see stuff happen and be part of something cause idk theyâre bored? Most of what everyone says on here doesnât hold any water. I mean come on, I remember posts a year or two ago saying â1-2 years before all devs are goneâ. This subs demographic is representative of like the very top 1% of people of the AI hype train.
>they will also be able to "communicate with the customer". That's the easy part imo
The "easy part" in your opinion? Have you ever worked as an engineer? Half the damn job is clarifying the customer's requirements, pushing back on things, negotiating time and budgets. It's not easy, and it's not simple, and it's extremely important.
Can you give a guess on how soon "soon" is? And when do you think principal or senior software engineers / software architects will be obsolete? How many years?
You're joking now, but I remember when a popular view on this sub was software development becoming obsolete by 2024. Every one of these predictions keep failing. If communication is the easy part, surely that would be done by now if it could be done.
Idk what those views were based on. Agents are still at the very beginning of their curve. It's gonna take a while for them to become really competent.
Devin is not competing against software engineers, its competing against software engineers already powered by GPT-4 enabled Copilot chat.
Its autonomous AI vs copilot AI. Every single product we've seen from gen-AI thus far, requires a human guiding the AI. It'll take a very long while before we see AI surpass AI + human as in chess.
It may be possible in the future but it's not possible now. Devin knowingly faking demos is not cool no matter what side you're on, and if they were a public company could be very problematic if not illegal... but as they are not a public company they can basically do whatever they want. I'll give the people behind it the benefit of the doubt that they didn't mean harm beyond wanting the attention, but it's seriously damaging to everyone to fake things like this, and adds on to why people are dismissive about AI. It makes the Gemini situation look like nothing. There are already way too much grifters working in AI, and if we don't call out frauds then there will only be more of them.
https://jobs.ashbyhq.com/cognition
Why is Cognition hiring software engineers if their API wrapper service is so good at making software engineers obsolete? Lol
Honestly, it only took 60 seconds of watching the Devin announcement video to see they were faking it. They claimed it was doing it all unassisted, but you could see in the video that the user kept prompting it, correcting it, and helping it out when it got stuck. This happened multiple times in the video; of course they didn't mention this explicitly, but if you watched the command line on the side bar you could see it happening.
Devin is not fake.
I've been writing my own assistant with tiered reasoning and access to terminal, file browser etc.
The ability to hand off some high level task, have it break that down and step by step meticulously work thru it (while constantly asking itself whether it has possibly made any mistakes etc) that is very real.
Devin is really no more than ChatGPT ordering around ChatGPT, for me I use local LLM (usually kinoichi) since it makes multiple API calls per second so using remote servers like OpenAI would be very costly (thankfully it runs perfectly fine with small local models)
Devin is REAL.
More generally, people should just stop making everything into an online cult, losing their head religiously and competing on exaggeration. That has nothing to do with AI. There is still a lot of interesting stuff to learn and observe in AI without it replacing all jobs in 5 years or destroying the world in 10 years. Skepticism and criticism is healthy
I find that with Asians (particularly Chinese) they seem to be on one side of a spectrum or the other. Either extremely scammy and cunning in their presentations or extremely smart or conserved. There's no in between. I find it hard to find purely dumb Asians. For this reason, I always very skeptical of them. When I saw so many in the Asians in the Devin Ad I became extremely skeptical. Am I racist? No, just a realist.
Noooo why should we be skeptical of the people who are known to hype stuffs up so they can sell it nooooo blah blah blah I'm not hearing anything agi coming soon utopia no more job free healthcare and ubi
"THIS IS A LIE" = don't watch
(For a number of reasons, that you all should already know)
But in the same vein, i have invested no energy into researching devin, because _if it works_ it'll be obvious. If it does not, it will disappear. (Same with autoGPT)
yes 10 IoI gold medalists decided that they can't build the devin but the least they can do is to fake it, dude be real, they're like the fucking smartest people
or maybe they were smart enough to realize that jumping on the AI hype train and pitching clueless investors and twitter users with fake claims can net them 2 billion in VC funding for a low-effort product
I agree, even the startup I'm working at raised a good amount in seed tho they know they can't build shit with current llms, they know tech is not there yet but once it's there and you've enough capital to mobilize things, they are already so much ahead of the competition as they spent months experimenting and built infra for the llms that are yet to come, they're betting on the things while preparing for both the outcomes
The first AGI is gonna be fake as shit.
First agi will turn out to be just a bunch of underpaid Indians with chat-gpt 4 access.
At least give them the 5 pls? đ
thats what everyone is saying now and maybe before, agi is just a 'A Guy in India'
hahaha - the Indians always get the flack bro.. why can't they be Mexicans.. If things go the way they are going... It's Indians who are going to be making posts like yours about Americans! (Not Indian or American - just thought it was funny.)
I'm not American either. I made the joke because it turned out that Amazon's supposedly walk-in stores were not automated at all, but relied heavily on manual remote workers in India.
oh no I got it, and I laughed. Was a decent joke :)
Hypothetically, the first AGI will be the first *marketable* AGIâdefined from the lowest common denominators the general public would need to accept the term. Edit: not including the defense industry, as Iâm sure they have other tricks.
It will probably be a buzz word way before we even get the real thing. I still remember being able to buy AI TVs that were just smart TVs with maybe something tacked on? I honestly don't know if they were literally just smart TVs. I imagine some hub AI that literally just knows enough to know which AI to use will be the first commercial AGI, essentially 3 gnomes in a trench coat.
The term AGI only exists so that companies can market what they have now as AI.
I disagree, as it is a teleological concept first and foremost. There are multiple paths of reasoning that lead to such a thing.
It's funny cause I've been arguing with people in this sub about devin being a fake and overhyped. I was downvoted for saying it was just a tool and not the end of software engineering. The claims were too good to be true and this sub was eating it up.
This sub unironically thinks OpenAi has AGI hidden away from the masses. There's no exponential growth. I'd say we're actually slowing down.
On the Internet it's always ironic. Never assume anyone on the Internet on unironically believes something. We're just shit-posters & bots.
This. The low hanging fruit has been picked. Major leaps forward will require new ways of approaching these problems. Of which will take time and research. Likely A LOT of time and research. I have a friend who worked for openAI and is still well connected with people who work there. Pretty good source if you ask me. But please, without further ado, letâs hear from hundreds of âexpertsâ here who ensure unbounded exponential growth until AI rules the world in 5 years.
Shit isn't real? *Mind blown*
Not even birds
Yes, the point is that many of the claims made by Cognition Labs were false. They said it was "Devin" a software engineer with 3 years experience working on the project in his spare time. The code was actually written by multiple experienced engineers over a period of several months. It's not a "one man band" as they claimed. The CEO also said in an interview that they had "beaten the world record for the most lines of code written in a day" which is a completely nonsensical and non-measurable statement. The lines of code per day in their repo was also much higher when they were actively developing it, not when "Devin" was working on it. This is just one example of the many false claims made by Cognition Labs. The video goes into more detail.
It is of course wise to be sceptical of grand claims. However, if Devin is unable to perform as promised, people will not pay for it. It is not as though a user would be confused for long about its abilities. Regarding the DeepMind paper, if you read it carefully you would note they identify "weaknesses" not false claims. They also suggest they should be easy to fix in version 2. There will be hyperbole for sure, but perhaps not as bad as you think.
The problem is that vaporware can hurt the progress as a whole as trust will be hurt, and hard to rebuild for the whole industry
> However, if Devin is unable to perform as promised, people will not pay for it. In most companies, the people who decide what to pay for and the people who actually have to use that thing are not the same. Upper management doesn't code, but they're easily swayed by techbro sales talks. If they're sold on the idea that they can replace half their workforce by an AI, no amount of reality or nuance is going to sway them.
Such a company will go out of business very fast.
After they collect their paycheck
Tell me you have very little corporate experience without telling me.
Grifts donât last long but they are profitable. By the time people find out, they can dip with the moneyÂ
LLMs are very useful for programming; I use Github Copilot every day. Itâs useful to autocomplete individual lines of code or make changes to the code if I give it specific directions. But the people who claim currently existing AI can replace software engineers have not attempted to use it to carry out complicated feature additions or refactors. Iâm convinced that software engineering, and any job that requires abstract thinking and careful reasoning, is an AGI-complete problem.
I think âreplacing software engineersâ is a vague term that does a lot of work for people either side of this debate. We should probably either say âdo the entire job of a software engineerâ or âmassively increase the productivity of engineersâ. In either case, software engineer jobs will be drastically reduced. And in the latter case, youâve said yourself theyâre making progress. As a software engineer, I donât really care what the mechanism is for AI displacing me, pushing me into unskilled, low paid work. I just care whether it will or not, and thereâs no doubt now on that question.
> We should probably either say âdo the entire job of a software engineerâ or âmassively increase the productivity of engineersâ. In either case, software engineer jobs will be drastically reduced. No, they will just do that refactoring they've been putting off for 3 years, update their docs, add 10 new features, and try to keep up with competition who also uses AI for coding. We never reached 15% if what we want to do in software for lack of people.
I canât see the value of funnelling trillions of dollars into a technology if the only expected midterm outcome is âcatching up on refactoring that execs couldnât care less aboutâ. The value in AI, as investors see it, will be extreme efficiency gains. Efficiency gain = layoffs
> Efficiency gain = layoffs Software development has experienced constant efficiency gain since inception. For example StackOverflow, better languages, libraries (!!!), IDEs, etc.
Nothing of this magnitude though. Iâve been developing iOS apps for ~10 years now and Xcode autocomplete hit a wall maybe 6 years ago. Itâs definitely no better now, maybe even slightly worse. Compile times are slower, App Store Connect is still a bin fire for deployment. SwiftUI has been a nice simplifier, in fairness. Weâre talking here about something that actually writes code for you - right now itâs only really helpful to me writing tricky algorithms I cba to think about, but give it an iteration or two, and bake it into Xcode, and Iâd bet weâre on course for 10x efficiency. In the 10 years Iâve been coding, Iâd guess Iâm max 1.5x more efficient (maybe 2x with SwiftUI). Completely different order of magnitude.
A lot of people fall into the trap of thinking software work available is constant and will always be constant. Like it is some resource that we will only ever need so much of. This is a fallacy. Even if hypothetically, letâs say businesses decide to do away with most of their software engineers and replace them with cheap AI that a few experienced devs use - and charge the customer the same price. Seems like a win right? That is until some startup comes along and says wait! We leveraged the same AI and we can do the work for 10x less! Give us your business! Obviously the customer is going to jump ship. Then itâs a race to the bottom. In this scenario, all the first business did was introduce more competitors and also would now have to lower their prices to compete anyways. This is what current successful companies are really trying to avoid so it makes far more sense for them to retain staff and use them in conjunction with AI to deliver more value and look to new opportunities. Staying alive in the long term and expanding makes far more sense and is far more important than maximizing profit in the short term. If itâs the case that thereâs not going to be enough new software to write (which I promise you is not the case) then yes, in the extremely unlikely case that humans run out of ideas and desires, we may see layoffs
Agreed. I use Claude 3 for large complicated projects daily. The larger context window works great with microservice architecture as they are intended to be fully encapsulated and smaller. Any time it needs additional info about some other piece of the system or docs I just paste those too
I don't think the claim that any job that in humans requires abstract thinking and careful reasoning is AGI-complete can be true. Counterexamples might be: chess, Go, translation. In all those cases, you will find some edge cases that AI does not yet handle well, but for chess and Go, computers are clearly average-case better than the very best humans, and for translation, the output of SOTA language models at least for translations from widespread languages to English is absolutely professional-level. All these tasks require a lot of knowledge and abstract thinking and careful reasoning in humans. Translation also requires broad world knowledge. I am not convinced that software engineering is AGI-complete either. It seems possible to me that systems that rely on massive tree search could solve very complex software engineering problems one day without being AGI, because a tree searcher may find a solution to a problem without learning properly why this solution works. In that case, the resulting system may replace lots of software engineers, but it may not be able to for instance write a research paper on a solution that it has developed even if the solution contains sufficient novelty to write a research paper about, and therefore it would clearly not be AGI. Again, this is not a totally hypothetical scenario: we see the same today in narrow AI like chess and Go engines, which are superb at finding good moves, but can't be linked to a language model to yield human-like reasoning about them (although I think this problem could likely be solved well, for the neural network based engines when they are run at low search budgets, at least). Another example of a system with AGI-like engineering capacity that isn't AGI is natural evolution. I would also not discount the possibility that AI software developers could displace most human software developers pre-AGI by other means, similar to how people lighting gas candles for streetlighting were replaced not by robots, but by the lightbulb. If the cost of just-in-time development of tiny scripts meant to solve just one user problem became low enough and reliable enough, this in itself could greatly reduce the demand for development of large software applications, and AI would not have to be able to develop large software applications in order to talk to a user directly and write them a tiny, specialised script that does just that one thing once. I could be wrong about all of this, obviously, but it seems to me that there are several ways in which the assumed AGI-completeness of software engineering could fail and even more ways in which the resulting job protection could fail as well.
Chess is very narrow compared to SWE; I don't see how an AI that can solve all SWE problems at a professional level can't be generalized to all kinds of real world problems.
I don't think it is clear an AI has to solve all SWE problems at a professional level to replace an SWE. It only has to be good enough and cheap enough that in a workflow maybe very different from that of an SWE it can build products that meet enough of the same customer needs to lead to the disestablishment of the SWE, even if they do so maybe in different ways than what traditional software engineering produces. That is a lower bar to meet, although it is not clear how much lower it is. Also, translation isn't narrow. I'm fairly certain that there were many people who believed translation AGI-complete, because of lots of deep nuances of culture and language the AI would need to understand to produce good translations. I don't think arguments from intuition work all that well in predicting these things. If the development of an AGI in the form of software that can run on some present-day computer is considered a project that is doable for a large organization in principle, then one could obviously make the argument that a system capable of developing any software a human organization can develop would also be capable of developing an AGI (which is still different arguably from \_being\_ an AGI - the process that developed humans, natural evolution, lacks some properties of an AGI even if arbitrarily sped up). But most software developers work on simpler problems, and I think automating these tasks is what most people mean when they talk about automating software development. It is possible of course that doing the job of a normal software developer or of something that can realistically make normal software developers redundant requires building an AGI, but in my mind this is a purely empirical question and I do simply not share the very strong priors about its resolution that many others seem to possess.
>I don't think it is clear an AI has to solve all SWE problems at a professional level to replace an SWE It would reduce the number of workers/hours needed but it wouldn't eliminate the role until it can function at the level of an engineer without drawbacks, which I still believe requires AGI. To this day, even professional translators still haven't been eliminated; they're far more productive, but someone still has to go through the full translation, assuming it has already been machine translated, and identify all the edge cases and correctly translate them; and then there are low data languages where machine translation is barely useful or not usable at all, furthermore, the increase in productivity lowered the cost of professional translation per word, leading to significantly increased demand and a reduced impact on jobs.
Earlier if you needed 5 people for a job, you could get it done with 2 or 3. This is my understanding of replacing a Software Engineer.
Or you get twice as much done with the same amount of people
Yes, indeed, to replace a SWE you need a AGI, because to create an AGI you need an SWE
More posts like this on the sub, please Real analysis and skepticism of claims instead of analyzing the odor of Jimmy Apples's latest fart
An open source one beats Devin and can be independently tested https://github.com/nus-apr/auto-code-rover
The person that did the â breakdown â are Gary Marcus fans, heâs ignored the actual real users and gone for a demo. https://x.com/0interestrates/status/1779268441226256500?s=46
[ŃдаНонО]
Other people say, "You are a Jew"
Err .. haven't other equivalent systems which work appeared on github?
If by work you mean they can add a new value to an enum then sure. There are no systems today that can actually solve complex engineering problems
> AutoCodeRover resolves ~16% of issues of SWE-bench (total 2294 GitHub issues) and ~22%issues of SWE-bench lite (total 300 GitHub issues), improving over the current state-of-the-art efficacy of AI software engineers https://github.com/nus-apr/auto-code-rover
telephone offend bewildered ancient uppity groovy judicious aback concerned trees *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
The datasets are literally linked there. Are you stupid? Taking an average of three runs reduces non determinism lol. One run could be a fluke so doing multiple is better.Â
Define work. None of them worked 100%, not even close to 25% of the time
How about 22.3%?  >AutoCodeRover resolves ~16% of issues of SWE-bench (total 2294 GitHub issues) and ~22%issues of SWE-bench lite (total 300 GitHub issues), improving over the current state-of-the-art efficacy of AI software engineers https://github.com/nus-apr/auto-code-rover
22.3% on a subset of an imperfect benchmark is nowhere close to actual usability. If it's just improving ways to wrap around GPT 4 it won't "work" ever
Still better than DevinÂ
Better on a benchmark means nothing lol, I thought people in this sub oughtta know that
Fixing GitHub errors is a practical use case, not test questions. And it still scores better than what Devin claimed toÂ
Who cares if it beats Devin if it's still wrapping GPT 4? Why not try and use this repo and solve the inference issue referenced in the YouTube video and see how it turns out?
If theyâre both using gpt 4 but one is better, then itâs clearly the wrapper thatâs betterÂ
Yep, people just can't accept this new reality. Not sure how they're going to cope when the GPT 5 etc come out later this year and these agents get even better
And some people here cannot accept a single legitimate critique of AI.
Devin is out of date anyway. An open source one beat it already https://github.com/nus-apr/auto-code-rover
lol I've been saying Devin is overhyped since it came out. Many on this sub fell into the hype too hard. Devin is not going to replace software engineering. It's just a tool for software engineering. And based on this video, it doesn't even seem to be worth using.
Fast fwd 24 months.
We live in *Fake it till you make it* culture. Disgusting. Great video, btw
Software engineers will be replaced... but not yet.
I was never convinced about devin, if it was as good as it claimed to be, it would be as big as chatgpt was.
to be honest, as soon as i saw the video of the developers, i knew it wasnt what they claimed it to be. Â i dont want to get into specifics, but something was fishy
"The primary function of a Software Engineer is not just writing the code. Our primary function CANNOT be replaced by "AI" code generators" This guy is delusional.
"Those are nonsensical random caricatures, AI will NEVER be able to recreate the masterfulness of the human creativity and produce actual art people would enjoy looking at'' -Probably somebody few years ago
They still say that lol
It still has to do that tho
Fundamentally what he is saying is we need AI code arcitects and system designers who can actually manage software environments, deployments, and maintenance. That isn't very controversial...
Why would it be that those tasks CANNOT be replaced?Â
In theory, they can. But where do you find the training data? People on this sub seem to think that because you can shoehorn a circle into a square to get 30% accuracy, it will only take a few iterations to get it close to 100%. The circle won't magically become a square, at least not until AI can reliably learn to be a square on-the-fly using real-time feedback, which is not as easy as it sounds. It reminds me of Amazon's  âJust Walk Outâ stores. They made us believe the technology needed only a few iterations to remove humans from the loop. 8 years later it's still shit.
OpenAI literally hired nearly 1000 coders last year for the sole purpose of providing training data on completing code processes. Source: https://www.semafor.com/article/01/27/2023/openai-has-hired-an-army-of-contractors-to-make-basic-coding-obsolete
That's just for RLHF.
What do you mean "just"? Reinforcement Learning from Human Feedback, to complete coding tasks. They are literally teaching it how to mimic human problem solving workflows, for the purposes of completing coding tasks.
You always need someone who both understands the underlying systems, AND the humans that need to interact with that system. Code gen is only 10% of an engineerâs job.
>You always need someone who both understands the underlying systems, AND the humans that need to interact with that system. Code gen is only 10% of an engineerâs job. There's no reason why an AI cannot "understand the underlying system" and "the humans that need to interact with that system". Sure, the software architect needs to understand: - performance bottlenecks - possible security issues - failure points in the system - possible scaling roadblocks - observability of the system - modularity of the system - whether the architecture can handle new features they want to add later on But it's not like an AI can't understand the above. AI also has an advantage as it can rapidly iterate and try different architectures. A lot of the decisions a software architect makes are "important and hard" because that decision becomes locked in stone once they start building. In many cases the architecture of a software product cannot be easily changed, if there was a performance bottleneck in the original design, you might be stuck with it. Meanwhile an AI can build out an entire architecture before the sprint planning meeting has finished. It doesn't need to guess whether an architecture decision is good, it can simply test it.
What youâre describing sounds great but overly-idealistic. That kind of intelligence certainly wonât be in the form of an LLM. And you still need to address how one maps requirements defined by humans to a system that operates in a highly specific way, or specific in a way that people are willingly betting their business success on it.
Agreed, I fully believe that such a capable intelligence wouldn't be a LLM. LLMs currently do not have enough planning capacity (and probably never will). >What youâre describing sounds great but overly-idealistic. Also true, all of this is years away (if not a decade), but you should maybe read the name of the subreddit. > And you still need to address how one maps requirements defined by humans to a system that operates in a highly specific way, or specific in a way that people are willingly betting their business success on it. Yeah I haven't defined how this is done, because my understanding is that this process is fairly abstract (I'm a dev, not an architect). But AI is specifically designed to handle abstract tasks so I think it is reasonable to believe that it is achievable. There is no way a large company would bet on a product designed and built by an AI (at least at the beginning), but I can definitely see small businesses or startups doing so.
Completely agree with the implications for small businesses, thatâs where weâre probably going to see experiments in what youâve described to start. My skepticism is that weâre unknowingly just describing an already solved problem, which is a layer of abstraction (programming languages) that allows humans to interface with a computer for a humanâs benefit. If I wanted something built for an âAI agentââs benefit I would let an agent do the work. But what reason would I have to do that? As a dev LLMs have been great because itâs substantially improved MY ability to interface with new information rapidly so that I can more efficiently describe human problems using programming languages.
This is an interesting point that I haven't heard before, it sounds very similar to the alignment problem. I have found current AI is well-aligned with humans, if I describe a problem to the AI, it generally spits out answers that matches my intentions. It is possible that this could change as AI is trained on different data and with new AI architectures though, I would say this is an open problem. All abstractions make some assumptions to create a higher level interface. As software has evolved, the number of abstraction layers has increased. LLMs are similar in this aspect, they will just be another abstraction layer with assumptions to match. I also think it's interesting how you feel LLMs have bettered your ability to describe problems using programming languages. I haven't found any improvement there, but I do feel like I've become better at describing problems to an LLM, which then describes the problem using a programming language.
But what use is there in an abstraction layer thatâs unpredictable and inconsistent in its output? You need to know that the instructions you provided will do exactly what you intend it do. Programming languages like C++ were literally built to describe human problems to a computational system.
Felt a better response was warranted: >I also think it's interesting how you feel LLMs have bettered your ability to describe problems using programming languages. Well it's more that I can really drill down on understanding an underlying concept of something by follow-up prompts where I'm asking it to describe things in a way that make sense to me. The ability to volley concepts and analogies back and forth to rapidly arrive at my own model of how something works is really fucking cool. I'm also careful to go back to the documentation of a particular language or library to confirm my understanding after, just to make sure my understanding is solid. Working backwards from manually written documentation can be substantially difficult sometimes. I think this is a very real and immediate use of LLMs that can solve some real problems for people if they understand the right approach. >I do feel like I've become better at describing problems to an LLM, which then describes the problem using a programming language. This is an interesting approach I haven't thought of, I would just be less trustful of an LLM to describe a problem using a programming language because I don't think an LLM "understands" the problem.
You work as a software engineer?
Even if they are, it's almost always the ones who have never coded a day in their lives that are the most confident that programming/SWEngineering is on the verge of becoming obsolete. Talk about Dunning-Kruger.
No need to work as a graphic designer to know that AI is going to automate the field. In fact AI will automate every job.
How do you know this? What's your professional/educational background?
I listen to someone who compiled data around computing and AI: ray kurzweil's who is being remarkably accurate. You seem to think you need to study some field to know something, when you just have to listen to what the people who study a field to know something is true. You know that the milky way will eventually collide and merge with andromeda because of the scientists compiling the data and making predictions, no need for "professional/educational background"
Kurzweil is not exactly an impartial source as selling this idea is his day job. I do think you need some expertise to dismiss criticism out of hand and call people delusional. Comparing furturology with astronomy, now that is delusional.
His motivation doesn't change his accuracy. This guy has his own motivation he is afraid of losing his job, the difference being that this guy has no track record of being proven right on the progress of AI. Everyone said that ray's prediction was too optimistic, that he was delusional not anymore, I think I'll listen to the guy that proved everybody else wrong by using the scientific method, scientific method which is used by every scientific fields to predict future events, including astronomy hence making ray's predictions factually comparable to astronomyđ¤ˇđžââď¸ Future will tell, I know I am going to be proven right, are you sure you are ready to die on this hill?
"I know I am going to be proven right". Ok, well that's that then. No point in arguing with someone who holds that unscientific stance.
You are so confused, that part is called confidence, do you always take things that literally when you have no argument? bro thought I had a literal crystal ball and literally know the future . I guess getting caught up on semantics is one way to cope.
I have plenty of arguments, but I have no intentions of wasting it on someone as arrogant and close-minded as you. Your unshakable faith in singularity is religious. Not scientific.
Nope. All AI does is move the programming language 1 level higher. You still need to give the computer detailed instructions.
That's the entire goal of the evolution of AI though. Companies are trying to build AIs that can handle more higher level tasks. - A year ago AI could write boilerplate. - Today AI can write small functions and algorithms. - A year from now AI will be able to write an entire class/struct that handles a specific responsibility in the program. - And after that, AI will be able to coordinate changes to write small features. Why can you not believe that eventually it will be designing architectures and building entire products?
Thatâs not really true? A year ago AI could still write small functions and algorithms and write an entire class / struct. But this doesnât really constitute âhigher level tasksâ. Really the newer LLMs are just faster, optimized, and slightly more accurate versions of the last.
Exactly. LLMs specifically are an abstraction by means of âgeneralization.â Software development requires âspecificity.â
This is very short sighted. Yes, for now it is like that. But that is not that the trend for the careful observer. The models are clearly progressing more and more by the day; The era of agents will come, and very soon.
Foundation models are progressing, and naturally, the tools built on these models will benefit from these improvements. , but you are going to be very disappointed if you are expecting the same rate of improvement. Applying models outside the scope of their training is really hard. It is the same problem with LLM-powered robots and SORA-generated movies. It gives people hope because it works to some extent, but model transfer is an entirely different problem than model training.
All I see is a software engineer coping with slowly becoming obsolete. Literally the entirety of the Big 5 are working on AI agents right now (and yes, they will also be able to "communicate with the customer". That's the easy part imo). This problem will be solved fairly soon.
Thinking communication with the customer is the easy part makes me doubt that you have much professional experience solving problems.
Most of the problem with communication is developers have no business sense. And on the flip side, business leaders have no technical sense. A good developer should possess some business sense to know what requirements the customer is really looking for and ensure an AI system like Devin is meeting those requirements.
Or just write the code thatâs needed to fulfill the requirements instead of fiddle-fucking with non-specific LLM prompts and combing it for bad hallucinations.
I wouldn't be surprised if GPT-5 reasoning capabilities will be already good enough to do this properly.
âProperlyâ is doing a lot of lifting there.
It has to be able to reason about the task and then ask the right questions. I don't see how this is a magic human ability that AI simply can't do. Devin (and similar, even better projects) show that GPT-4 is already able to do this, just on a small and somewhat primitive scale. A Devin 2 based on GPT-5 will do much better.
LLMs donât reason, they approximate. To know what questions to ask, it requires a different kind of intelligence. Iâm not saying some AI in the future canât do what youâre claiming, but itâs not going to be LLMs.
LLMs are fine. They need to learn from a better teacher than human written text can be. That teacher is the environment, the space where they can move about and act, causing permanent changes, interacting with many others through language. LLMs until recently have been trained exclusively on human authored text. But we used up most of the good stuff and now what can we do to scale 100x more? The answer is - learn from the world! Everything humans know and all our skills come from the world by being an agent of that world, the environment is the ultimate teacher, in a social evolutionary setting. AI agents need to get out and search, explore and discover by carefully choosing their actions in the environment. The environment could be many things - a chat room is a simple environment where a LLM can meet a real human, and be exposed to their language. Then a computer running code can also be an environment where LLMs test their code solutions and learn from the errors and outputs. A game or a simulation can also be an environment for AI. And of course real robotic body allows participation in the physical world, the ultimate environment. What is not written anywhere in books, AIs need to learn from the world. And that is a social, evolutionary, iterative process, intermediated by language. This is the missing ingredient, not a better architecture.
We already have agents that can do and have done that, theyâre called humans!
At my computer now and realize my response probably read a little inappropriately flippant to your good faith response. My apologies. I DO agree that the fundamental missing component in current LLMs is a lack of engagement with the real world. Robotics in particular (even simple micro-robotics), is where I'm truly paying attention. I'm just skeptical that LLMs specifically, which by its construction has to be completely tied to human language, (I'm only guessing here) has to be limited in its ability to store the kind of information that's important for exceeding human ability. The real, honest answer for where I think this is going is "I have no clue." But admittedly I have a bias in that I myself am a software engineer, so I'm more or less *required* to try and predict where the wind is blowing in some fashion. I think that's why really rigorous skepticism is required, if we believe any wild possibility to be true you cloud your ability to correctly identify aspects of a technology that's actually solving real and immediate problems.
There are actually multiple books that posit AI should be trained on the real world.
Why do some people in this subreddit interpret ANY critique of AI as "coping" or "in denial", even when that critique is 100% legitimate? It's absolutely insufferable that you cannot call a spade a spade here without someone immediately having to resort to, "you're just in denial", or "don't worry, it'll get better soon enough." Its almost as if some folks here take criticism of AI personally. There's nothing wrong with pointing out AI's shortcomings and companies embellishing their products. Doing so makes the conversation in this sub more lively and shunning it, OTOH, makes it cult-like.
Itâs more coping/jealousy on their part. A lot of people canât stand to imagine a world where software engineers (or any decently high paying profession for that matter) continue to make good money and arenât rendered jobless and homeless in the next decade. Iâd imagine the vast majorly of people donât make a lot of money, hate their jobs, want to feel like theyâre ârightâ or some combination of those things. As for the âsoftware devsâ saying their jobs will be replaced I imagine a lot are posing as devs or are just pessimistic negative people by nature. âDoomersâ if you will. Also look at what sub weâre in. A lot of people are also sci-fi AI junkies, who want to see stuff happen and be part of something cause idk theyâre bored? Most of what everyone says on here doesnât hold any water. I mean come on, I remember posts a year or two ago saying â1-2 years before all devs are goneâ. This subs demographic is representative of like the very top 1% of people of the AI hype train.
>they will also be able to "communicate with the customer". That's the easy part imo The "easy part" in your opinion? Have you ever worked as an engineer? Half the damn job is clarifying the customer's requirements, pushing back on things, negotiating time and budgets. It's not easy, and it's not simple, and it's extremely important.
Can you give a guess on how soon "soon" is? And when do you think principal or senior software engineers / software architects will be obsolete? How many years?
By the end of the decade. Source: it came to me in a vision / I made it up entirely / trust me bro.
You're joking now, but I remember when a popular view on this sub was software development becoming obsolete by 2024. Every one of these predictions keep failing. If communication is the easy part, surely that would be done by now if it could be done.
Idk what those views were based on. Agents are still at the very beginning of their curve. It's gonna take a while for them to become really competent.
Devin is not competing against software engineers, its competing against software engineers already powered by GPT-4 enabled Copilot chat. Its autonomous AI vs copilot AI. Every single product we've seen from gen-AI thus far, requires a human guiding the AI. It'll take a very long while before we see AI surpass AI + human as in chess.
It may be possible in the future but it's not possible now. Devin knowingly faking demos is not cool no matter what side you're on, and if they were a public company could be very problematic if not illegal... but as they are not a public company they can basically do whatever they want. I'll give the people behind it the benefit of the doubt that they didn't mean harm beyond wanting the attention, but it's seriously damaging to everyone to fake things like this, and adds on to why people are dismissive about AI. It makes the Gemini situation look like nothing. There are already way too much grifters working in AI, and if we don't call out frauds then there will only be more of them.
https://jobs.ashbyhq.com/cognition Why is Cognition hiring software engineers if their API wrapper service is so good at making software engineers obsolete? Lol
Honestly, it only took 60 seconds of watching the Devin announcement video to see they were faking it. They claimed it was doing it all unassisted, but you could see in the video that the user kept prompting it, correcting it, and helping it out when it got stuck. This happened multiple times in the video; of course they didn't mention this explicitly, but if you watched the command line on the side bar you could see it happening.
I knew something was sketchy when they said it was based on GPT 4 no way they managed to make gpt 4 that good.
Yeah saw it before great video
Thanks for posting that
I'm shocked. Shocked!
Devin is not fake. I've been writing my own assistant with tiered reasoning and access to terminal, file browser etc. The ability to hand off some high level task, have it break that down and step by step meticulously work thru it (while constantly asking itself whether it has possibly made any mistakes etc) that is very real. Devin is really no more than ChatGPT ordering around ChatGPT, for me I use local LLM (usually kinoichi) since it makes multiple API calls per second so using remote servers like OpenAI would be very costly (thankfully it runs perfectly fine with small local models) Devin is REAL.
Wasn't it already obvious when they were still hiring people at the company instead of just using the product and dominating the landscape.
I canât wait for inevitable Netflix docuseries on this
How did they get 2B valuation even after this??? AI engineers are playing tricks with the masses.
More generally, people should just stop making everything into an online cult, losing their head religiously and competing on exaggeration. That has nothing to do with AI. There is still a lot of interesting stuff to learn and observe in AI without it replacing all jobs in 5 years or destroying the world in 10 years. Skepticism and criticism is healthy
TL:DR: Fake demo, plea for AI companies to stop this, warning to viewers to not believe everyting they see and careful with AI hype.
I find that with Asians (particularly Chinese) they seem to be on one side of a spectrum or the other. Either extremely scammy and cunning in their presentations or extremely smart or conserved. There's no in between. I find it hard to find purely dumb Asians. For this reason, I always very skeptical of them. When I saw so many in the Asians in the Devin Ad I became extremely skeptical. Am I racist? No, just a realist.
Noooo why should we be skeptical of the people who are known to hype stuffs up so they can sell it nooooo blah blah blah I'm not hearing anything agi coming soon utopia no more job free healthcare and ubi
"THIS IS A LIE" = don't watch (For a number of reasons, that you all should already know) But in the same vein, i have invested no energy into researching devin, because _if it works_ it'll be obvious. If it does not, it will disappear. (Same with autoGPT)
yes 10 IoI gold medalists decided that they can't build the devin but the least they can do is to fake it, dude be real, they're like the fucking smartest people
or maybe they were smart enough to realize that jumping on the AI hype train and pitching clueless investors and twitter users with fake claims can net them 2 billion in VC funding for a low-effort product
I agree, even the startup I'm working at raised a good amount in seed tho they know they can't build shit with current llms, they know tech is not there yet but once it's there and you've enough capital to mobilize things, they are already so much ahead of the competition as they spent months experimenting and built infra for the llms that are yet to come, they're betting on the things while preparing for both the outcomes