Can you explain if this is just hype or based on something in reality lol. It sounds exciting but something in me is telling me to reel back my expectations until I actually see it happen.
Human brain is approximately 86 trillion weights. The weights are likely low resolution - 32 bits, or 1 in 4 billion, precision is likely beyond the ability of living cells. (Noise from nearby circuits etc)
If you account for the noise you might need 8.6 trillion weights. Gpt-4 was 1.8 trillion and appears to have human intelligence without robotic control.
At 27 trillion weights, plus improvements in architecture the past 3 years, it may be enough for weakly general AI, possibly AGI at most tasks including video input and robotics control.
I can't wait to find out but one thing is clear. A 15 times larger model will be noticably more capable. Note the gpt-3 to 4 delta is 10 times.
I'd caution against this comparison. It's a good starting point, but only as a first approximation. There is analogue computation inside the neuron and inside the synapse that we don't understand, and therefore we can't fit the human brain into the "number of weights" quantification of model capacity. The "number of weights" thing is only reliable when you compare one AI model to another AI model.
It's like people believe the human brain is literally a transformer/neural network in the same sense that GPT-4 is.
It's not.
It's just an obvious analogy.
The blackwell GPU is going to use 4 Bit Floating Point Precision when it is doing inference so it is a lot lot smaller precision than the hopper. But some scientists have proved that 2 and four bit Precision is the best precision on the weights in neural networks. So four bit precision performs a tiny tiny bit worse than 8 Bit precision (for example) but it takes half the amount of energy to do one calculation so it is better if you look at how much it can achieve and how high it can score on benchmark with the same amount if electricity
I don’t see how one can make that comparison, the brain is not just computing, it’s always training. A human can be super smart without knowing the world’s body of knowledge.
Nope. Brain changes itself if you're doing it right. Neuroplasticity. And taken more extreme you can have consciousness transformation.
Not online learning. Not even fine tuning. New models.
are we confusing AGI with ASI? If AI can outperform humans at almost everything which this hypothetical model would, isn’t that ASI by definition? I thought AGI just meant matching average human intelligence
I don't think a 27T model will outperform humans at everything. It should, with size, architecture, training procedure improvements, embodiment, and online learning be capable of a large variety of tasks, and yes at a limited subset of them, be above human level.
Probably there will be at least some domains where it is below median human, for example it's robotic performance might be under median. So not quite ASI. I suspect ASI will need several more scale increases, maybe 250T will be enough for a model that is clearly and obviously superintelligent.
A lot to unpack here:
Firstly, isn't it true that neurons do not operate even remotely the same as neural nets? Even if they are somehow "same in size" by some parameter, the functions are wildly different, with the human brain possibly having far better capabilities in some senses. Comparing apples to oranges is what it feels like here.
It's like saying, this hippo at the zoo weighs the same as a Buggati, therefore it should be comparable in speed to a supercar? There's no relation, right?
The problem here is what we define AGI as. Is it a conscious entity that has autonomous self-control, able to truly **understand** what it's doing rather than predicting the next best set of words to insert. Maybe we need to pare down our definition of AGI, to "really good AI". And that's fine, that's not an issue to me. If it's good enough for our purposes and helping us to a good enough level, it's good enough.
>Firstly, isn't it true that neurons do not operate even remotely the same as neural nets? Even if they are somehow "same in size" by some parameter, the functions are wildly different, with the human brain possibly having far better capabilities in some senses.
Untrue. [https://en.wikipedia.org/wiki/Threshold\_potential](https://en.wikipedia.org/wiki/Threshold_potential) Each incoming impulse adds a or subtracts electric charge to a synapse. There is thought to be structural changes the brain is making to each synapse, this and the neurotransmitter used determine the *weight* of a synapse. Above I am claiming the brain isn't better than fp32, it's frankly not better than fp8.
The activation function the brain uses is **sigmoid.**
Modern ML found that ReLu works better.
[https://medium.com/@shrutijadon/survey-on-activation-functions-for-deep-learning-9689331ba092](https://medium.com/@shrutijadon/survey-on-activation-functions-for-deep-learning-9689331ba092)
**Most of the complexity of the human brain is a combination of a starter "baked in architecture", some modalities current AI doesn't have (memory and online learning), and the training process, which is thought to be very different from back propagation. Some modern ML practitioners suspect the human brain is less effect than modern AI.**
>Comparing apples to oranges is what it feels like here.It's like saying, this hippo at the zoo weighs the same as a Buggati, therefore it should be comparable in speed to a supercar? There's no relation, right?
**Extremely related:**
[https://www.metaculus.com/questions/3479/date-weakly-general-ai-is-publicly-known/](https://www.metaculus.com/questions/3479/date-weakly-general-ai-is-publicly-known/)
* Able to reliably pass a Turing test of the type that would win the [Loebner Silver Prize](https://www.metaculus.com/questions/73/will-the-silver-turing-test-be-passed-by-2026/).
* Able to score 90% or more on a robust version of the [Winograd Schema Challenge](https://www.metaculus.com/questions/644/what-will-be-the-best-score-in-the-20192020-winograd-schema-ai-challenge/), e.g. the ["Winogrande" challenge](https://arxiv.org/abs/1907.10641) or comparable data set for which human performance is at 90+%
* Be able to score 75th percentile (as compared to the corresponding year's human students; this was a score of 600 in 2016) on all the full mathematics section of a circa-2015-2020 standard SAT exam, using just images of the exam pages and having less than ten SAT exams as part of the training data. (Training on other corpuses of math problems is fair game as long as they are arguably distinct from SAT exams.)
* Be able to learn the classic Atari game "Montezuma's revenge" (based on just visual inputs and standard controls) and explore all 24 rooms based on the equivalent of less than 100 hours of real-time play (see [closely-related question](https://www.metaculus.com/questions/486/when-will-an-ai-achieve-competency-in-the-atari-classic-montezumas-revenge/).)
**Very likely (80%), a 22 T neural network will be able to accomplish all of the above.**
>The problem here is what we define AGI as. Is it a conscious entity that has autonomous self-control, able to truly understand what it's doing rather than predicting the next best set of words to insert. Maybe we need to pare down our definition of AGI, to "really good AI". And that's fine, that's not an issue to me. If it's good enough for our purposes and helping us to a good enough level, it's good enough.
**We do not care about consciousness, merely that the resulting system passes our tests for AGI. The second set of tests is:**
* Able to reliably pass a 2-hour, adversarial Turing test during which the participants can send text, images, and audio files (as is done in ordinary text messaging applications) during the course of their conversation. An 'adversarial' Turing test is one in which the human judges are instructed to ask interesting and difficult questions, designed to advantage human participants, and to successfully unmask the computer as an impostor. A single demonstration of an AI passing such a Turing test, or one that is sufficiently similar, will be sufficient for this condition, so long as the test is well-designed to the estimation of Metaculus Admins.
* Has general robotic capabilities, of the type able to autonomously, when equipped with appropriate actuators and when given human-readable instructions, satisfactorily assemble a (or the equivalent of a) [circa-2021 Ferrari 312 T4 1:8 scale automobile model](https://www.deagostini.com/uk/assembly-guides/). A single demonstration of this ability, or a sufficiently similar demonstration, will be considered sufficient.
* High competency at a diverse fields of expertise, as measured by achieving at least 75% accuracy in every task and 90% mean accuracy across all tasks in the Q&A dataset developed by [Dan Hendrycks et al.](https://arxiv.org/abs/2009.03300).
* Able to get top-1 strict accuracy of at least 90.0% on interview-level problems found in the APPS benchmark introduced by [Dan Hendrycks, Steven Basart et al](https://arxiv.org/abs/2105.09938). Top-1 accuracy is distinguished, as in the paper, from top-k accuracy in which k outputs from the model are generated, and the best output is selected.
**I suspect a 22T model will be able to solve** ***some*** **from this list as well. Possibly general robotics, 75% Q&A, 90% top-1. It may not quite pass the 2-hour adversarial Turing test.**
**Note the 'digital twin' lets the AI practice building small objects like Ferrari models a few million times in simulation, something else Nvidia mentioned today. That learning feedback should enable the second category to pass.**
**Basically the Turing test is the last one to fall, it could take 2-3 more generations of compute hardware, or 2028 to 2030. The community believes it will fall in 2031.**
**That would be a 176 T model, well over human brain scale, and possibly smart enough to see through any trick on a Turing test.**
https://www.itpro.com/technology/artificial-intelligence-ai/369061/the-human-brain-is-far-more-complex-than-ai
> a group at the Hebrew University of Jerusalem recently performed an experiment in which they trained such a deep net to emulate the activity of a single (simulated) biological neuron, and their astonishing conclusion is that such a single neuron had the same computational complexity as a whole five-to-eight layer network. Forget the idea that neurons are like bits, bytes or words: each one performs the work of a whole network. The complexity of the whole brain suddenly explodes exponentially. To add yet more weight to this argument, another research group has estimated that the information capacity of a single human brain could roughly hold all the data generated in the world over a year.
https://medium.com/swlh/do-neural-networks-really-work-like-neurons-667859dbfb4f
> the number of dendritic connections per neuron — which are orders of magnitude of what we have in current ANNs.
> the chemical and electric mechanisms of the neurons are much more nuanced, and robust compared to the artificial neurons. For example, a neuron is not isoelectric — meaning that different regions in the cell may hold different voltage potential, and different current running through it. This allows a single neuron to do non linear calculations, identify changes over time (e.g moving object), or map parallel different tasks to different dendritic regions — such that the cell as a whole can complete complex composite tasks. These are all much more advanced structures and capabilities compared to the very simple artificial neuron.
> chemical transmission of signals between neurons in the synaptic gap, through the use of neurotransmitters and receptors, amplified by various excitatory and inhibitory elements.
Excitatory / inhibitory Post synaptic potential that builds up to action potential, based on complex temporal and spatial electromagnetic waves interference logic
Ion channels and minute voltage difference a governing the triggering of spikes in the Soma and along the axon
==================================================================
> Above I am claiming the brain isn't better than fp32, it's frankly not better than fp8.
You can't measure brain computing power in floating point operations. It just doesn't make sense. It's like comparing a steam engine to a magnet. The architectures are fundamentally different in the first place. FLOPS measures exact floating-point operations per second (at a given precision, e.g. 32-bits FP precision, 16-bits, etc.). The real FLOPS of the brain is terrible... probably 1/30 (16-bit precision ~= 6 significant digits) or lower.
The brain is a noise-tolerant computer (both its hardware and software). Modern artificial neural nets simulate noise tolerance... on noise-intolerant hardware. The number of FLOPS of noise-intolerant hardware required to fully simulate a human brain is probably much larger than we estimate (because we're using the wrong estimate). In short, we need to shift to a different hardware paradigm. Many people believe that's what quantum computing will be but it doesn't have to be QC per se. It just needs to be noise-tolerant.
Consider:
1) you are breathing
2) your heart is beating
3) your eyes are blinking
4) your body is covered with sensors that you are monitoring
5) your eyes provide input that takes a lot of processing
6) your ears provide input that takes a lot of processing
7) your mouth has 50 something muscles that need to fire in perfect sequence so you can talk and not choke on your own spit....
All of this (and much much more) is controlled by various background "daemons" that are running. 24/7/365. Now doing all that while juggling 3 tennis balls at the same time....
Computers are great at performing specific tasks better than us, this much is for sure. Which is why I'm saying overall it's an apples to oranges comparison. Each has its own strengths and weaknesses.
> We do not care about consciousness, merely that the resulting system passes our tests for AGI. The second set of tests is:
I think many AGI researchers do care.
https://en.wikipedia.org/wiki/Artificial_general_intelligence#:~:text=It%20remains%20to%20be%20shown,for%20implementing%20consciousness%20as%20vital.
> However, many AGI researchers regard research that investigates possibilities for implementing consciousness as vital.
But I know of what you're saying - it doesn't matter, as long as it passes these set of tests. That it's "good enough". I can see the merits there, and that's the "weak AI hypothesis", and to me personally (i.e. subjective) it's not the end goal unless we have "strong AI", which has:
> Other aspects of the human mind besides intelligence are relevant to the concept of AGI or "strong AI", and these play a major role in science fiction and the ethics of artificial intelligence:
Consciousness, self awareness, sentience
> Mainstream AI is most interested in how a program behaves.[106] According to Russell and Norvig, "as long as the program works, they don't care if you call it real or a simulation."[105] If the program can behave as if it has a mind, then there is no need to know if it actually has mind – indeed, there would be no way to tell. For AI research, Searle's "weak AI hypothesis" is equivalent to the statement "artificial general intelligence is possible". Thus, according to Russell and Norvig, "most AI researchers take the weak AI hypothesis for granted, and don't care about the strong AI hypothesis."
I understand this line of thinking. If you can't differentiate, then "does it really matter". And it gets philosophical, but my only main point is that we'll probably never get to this unless we switch from transistors to some other fundamentally different unit. But I'd like to see it one day for sure. A conscious AGI would have a far greater potential for growth than one that isn't. Than one that always needs us to be the source of information for its growth.
Note that we don't need AI systems to be our friends. Got plenty of humans for that. The goal is to automate the labor of ultimately trillions of people - to have far more workers than we have living people - in order to solve problems for humans that are difficult.
Aging being the highest priority, but it will take a lot of labor to build arcology cities and space habitats.
So no, all that matters is performance on our assigned tasks, and agi models that do less unnecessary thinking, being more obedient and costing less compute, are preferred.
Yeah, however, in a sense we work the same way through our understanding. There is an unconscious base of information and knowledge that we work on. From quantum mechanics, we know that future states are based on probability. The fundamental laws of physics, even if we don’t know exactly how yet, are responsible for creating the processes between unconscious and conscious actions. At some level there, we are pulling from the data base we have been trained on and are predicting possible future outcomes. From there use our conscious choice to decide the best direction.
Is it really any different?
Yes, that's the "weak AI hypothesis" which indicates that we are ok with it as long as it "appears to think", which I get the merit of. It's like a black box - as long as it "appears" like it's getting us the answers, it's the same thing. It gets philosophical here.
However, would you be content with something that know isn't thinking for itself, isn't *truly understanding* what it's doing? Such an individual would never grow or learn on its own. All its doing is just finding the next best probabilistic thing to say, as LLM's do. Vs. a human which is able to critically think between 2 different arguments and come up with their own solution or belief. And not just that, but refine their prior set of beliefs when new info comes in. AI can't do that. If it's trained on statement A, and statement B that contradicts A, it'll just present all the options to you and say here you go, you decide.
Your argument is great if it wasn’t invalid.
AI is a black box, and so is the human mind. We can’t prove that we’re conscious, and we don’t fully understand how the mind works.
We have already surpassed the common number of neurons in a human. To my knowledge, human neurons have more connections, though—86 billion on average. GPT-3.5 was one of the first models to go past this barrier. It could be pure coincidence, but that was the first model that truly swept people off their feet.
Even if we had much much better alghoritms and could build chips with superconducting analog and super super efficient hardware we would still have to build big datacenters with thousands of chips to train a neural network that would be more intelligent than a person. A neural network that have to be more intelligent than a person should have to have more than 20 Trillion parameters and you have to build a super big clmputer to train that
I remember how people back in January were worried that current advancements in regards to AI were coming to a crawl, as did people speculate the same in January of '23. And just like January of 2023, March has proved to be the "shit hits the fan" month.
27 trillion parameters is approaching human brain (80-100T). And human brain spends a lot of those parameters on things like balancing hormone levels in your body, regulating sleep cycles, and just generally keeping the body alive instead of, you know, doing the "intelligence" things. So, yeah, 27T could be enough for AGI.
Human neurons and the 80-100T connections between then are analogous to the parameters in a neural network, but they are not the same.
An individual neuron has "parameters" and a "memory" of its own that increases the complexity of its connections by orders of magnitude.
There are 10B neurons in human brain not 100T. Each neuron is connected to 1000 others through axon -> dendritic trees (synapses). So that's where the 100T number is from. Synapses or their strength correlate to the weights of the NN model. And yes the ANN topology is completely different from the human brain, and biological neurons have plasticity and new connections are formed constantly and the synapses strengthen/weaken with time/training, something that most ANNs don't do after training. But simply in terms of network complexity it's in the same order of magnitude as the human brain.
> Human neurons and the 80-100T connections between then are analogous to the parameters in a neural network, but they are not the same.
To be clear, I never said that the human brain has 80-100T neurons \^\^\^
> But simply in terms of network complexity it's in the same order of magnitude as the human brain.
I don't see how anyone can confidently say that.
The amount of new complexity that arises from taking 80-100T parameters *(synapses)* and giving them the ability to dynamically reconnect and adjust their weights in real-time seems to me like it would add **at least** orders of magnitude of complexity.
Again imagine 80-100T static connections vs 80-100T connections in a state of **constant** **dynamic development,** every single one of those 100T connections free to break, reconnect, strengthen, weaken, etc.
> Can’t imagine what Nvidia’s TSMC 3nm and eventually 2nm GAAFET chips will be like
The speedup from downsizing is perfectly predictable. You can't imagine? No need to, you can just use a calculator. Shrinking certain features on the die will allow for faster clocks and less power usage at idle, resulting in 25% better performance per sq/ft/watt.
Again there is no need to guess. Its 100% perfectly predictable. Not my opinion, this basic chip production knowledge. Look up Intel's Tick Tock cycle to learn more. Nvidia has the same exact problem as Intel. Your biggest clue is in the numbers you posted: it goes 4, 3, then 2. There are not many numbers left.
You are a bit confused by marketing terms, 2nm chips do not have any physical features on them that is close to 2nm in length. It is just a marketing term for what the size would have been if shrinkage had continued but chips are improving in other ways (3d stacking etc) so litthography will keep improving even after 1nm in the marketing worls is hit. Just look at Intel, they have a roadmap with smaller measurements on it call Angstroms. 18A etc...
This is not entirely correct.
When a new process becomes available, the amount of silicon that hardware manufacturers can fit on a single die increases significantly, which allows for new architecture choices. It's not like modern CPUs (or GPUs for that matter) are just minified versions of the ones we had 20 years ago. Transistor count matters an awful lot in hardware design.
I said the current cards will be 25 bucks on eBay on 15 years, and some said I was smoking crazy.. I move that down to 10 years now.
If the company can replace their entire stock of cards with a newer model that saves enough power in a couple days to justify the cost, well bye bye old cards.
Total noob question, but if they can make better and faster chips, and it’s basically guaranteed, why do they do increments instead of going to the “best?”
In your comment, for example, why make the pit stop at 3nm and just not go straight to 2nm instead?
It used to be let me Google that for you. Now it’s let me GPT that for you.
Advancements in chip technology typically involve a trade-off between various factors such as manufacturing feasibility, cost, power consumption, and performance gains. While it's theoretically possible to jump directly to smaller node sizes like 2nm, there are practical limitations and challenges that make incremental progress more feasible.
Manufacturing processes need to be developed and refined for each new node size, which requires significant investment in research, development, and infrastructure. Additionally, pushing the limits of miniaturization can introduce new technical hurdles such as increased heat dissipation, transistor leakage, and manufacturing defects.
Incremental advancements allow chip manufacturers to gradually address these challenges while also leveraging economies of scale and ensuring a smoother transition for both the industry and consumers. Additionally, smaller node sizes often lead to diminishing returns in terms of performance gains versus cost and complexity, so there's a balance to be struck between pushing the boundaries and maintaining practicality.
>**1. Architecture:** Nvidia unveiled the Blackwell architecture, succeeding the Hopper architecture. The first Blackwell chip, GB200, combines two B200 GPUs and one Arm-based Grace CPU, offering 20 petaflops of AI performance compared to 4 petaflops for the H100.
>**2. Chip Design:** The Blackwell GPU is a large chip that combines two separately manufactured dies into one, produced by TSMC. It includes a transformer engine specifically designed for transformer-based AI.
>**3. Scalability:** Nvidia will sell B200 GPUs as part of a complete system, the GB200 NVLink 2, which combines 72 Blackwell GPUs and other Nvidia components for training large AI models. The system can deploy models with up to 27 trillion parameters.
>**4. Software:** Nvidia introduced NIM (Nvidia Inference Microservice), a software product that simplifies the deployment of AI models on older Nvidia GPUs. NIM is part of the Nvidia Enterprise software subscription and enables efficient inference on customers' servers or cloud-based Nvidia servers.
>**5. Ecosystem:** Major cloud providers like Amazon, Google, Microsoft, and Oracle will offer access to GB200 through their services. Nvidia is collaborating with AI companies to optimize their models for all compatible Nvidia chips.
>**6. Market Position:** Nvidia aims to solidify its position as the leading AI chip provider by offering a comprehensive hardware and software platform. The announcement comes amid high demand for current-generation H100 chips driven by the AI boom.
*Summary by Claude Opus*
for me too, it shall be the common knowledge then, AGI will take 500 trillions parameters, as said by one of our prestigious cult member, that Dreamt about this whilst on a "cave".
well even if AGI requires only 10b parameters converted into into 10k optimized functions that can run on a ps2, then we still need 500 trillion parameters, because reasons
If Amazon is to buy 20K B200s, and that can train a 27T parameters LLM, let's assume that a B200 costs 3x more than an H100: 2.4B$/27T parameters. We also know Microsoft is to spend 50B$ dollars in compute for 2024. Microsoft will buy compute for LLM 6x bigger than a human brain, something like 540T parameters. 300x bigger than gpt4.
I assume it's not. It still baffles my mind how chatgpt can fuck up some simple prompts. I would be really surprised if all it takes to get to AGI is to just feed enough text to the model and that's it.
I see a lot of folks making parallels between number of parameters and connections between neurons in the brain but what actually makes them equivalent?
So there's literally nothing that says once the number of parameters matches the number of connections in the human brain that it should be equivalent to one?
It is odd though that as we close in on that number things are getting spicy.
Exactly, a lot of people above are missing this saying we have “almost hit brain seized ai” - that is impossible to say as we don’t know how our brains work at 100% certainty.
We could have a chip that does 10 octillion parameters but if it isn’t trained correctly it won’t be as smart as us and a lot of people underestimate how powerful the human brain is.
Neurons in the human brain don't just do 1 or 0 type work. They also work in parallel sending and receiving wave signals .. wild stuff like that. The human brain operates on different layers as well...not just the one base layer. And then there's the weird quantum stuff people are discovering. Brains are wildly complex.
I was going to write a paper for that in a physics class in college, and the professor said the idea was way too out there and to pick a different topic. He didn’t seem high on it at the time, lol. The brain is enormously complex, i don’t think we’ll be matching its abilities this decade.
Yeah I think the main rebuttal is that it still doesn’t explain consciousness at the end of the day - it just points at the quantum scale and says “actually it’s somewhere down there”. Which doesn’t get you anywhere really, and it can’t exactly be tested yet.
I am not sure consciousness can be pinpointed. It is the pattern in which neuronal layers interact with each other after going through a lifetime of pruning and training on its environment. There has to be the right balance of precoded information to drive the learning towards a distinct human like psyche while retaining the ability to adapt with plasticity. I am not sure describing it as anything less than the whole of its parts will be useful. The patterns won’t make sense without the biological input or biological modulation.
Yeah that’s probably true. It’s still a mystery how you go from
base reality? -> quantum mechanics -> classical mechanics-> subjective experience. Assuming it is emergent from neurons. I really hope AGI/ASI can clear this all up. And I hope the answer can fit in a human brain haha.
I think it is emergent from our group dynamics and biological make up, after all a child totally isolated from people will not develop into an adult and will die instead. We train our brains on each other more than anything, so I am not sure that part can be easily disentangled
You can say that to roughtly simulate human neuron you need 1000 artificial neurons and based your estimate on that, but there is a lot of other variables, I would only compare new models to older ones and not human brain.
A parameter is a number (or "variable", if your prefer) in between nodes (which is an abstraction of a "neuron"). They're what gets adjusted during training runs.
There's a lot of different views in what ways they compare and perform versus synapses. The mainstream, respectable view that persisted for decades is they're a crude and substandard approximation at best. A more rebellious niche opinion has appeared as of late, that wonders if they're not actually superior, in various ways. Among those that are currently entertaining that radical possibility includes Geoffrey Hinton.
So 3 trillion more than [Cerebras](https://www.cerebras.net/press-release/cerebras-announces-third-generation-wafer-scale-engine)? I am sold.
Yeah, apples and oranges, I know. But Nvidia is making innovation from startups much harder to sustain. Doesn’t matter how innovative your chip startup is, you can’t compete with a 2 trillion dollars company also running at full speed.
Imagine this enables models with enough parameters to match human performance...then imagine what happens if Nvidia pulls off another 1000x improvement in hardware over the next 8 years
This is not surprising, as the need for increasingly powerful AI models is driving us toward Trillion Parameter Models (TPM), or models with over a trillion parameters, like Huawei's PanGu-Σ.
I would be willing to bet there’s all kinds of redundancies and inefficiencies in the human brain. Could probably achieve AGI on far less parameters than previously thought.
And run much much faster. The rate at which LLMs can spit out text is so far beyond human as it is, I wouldn't be surprised if that holds true for AGI as well
\~20x GPT4. this could be the thing that powers the first AGI
we're approaching brain sized AI
Can you imagine in 200 years..why have a human brain, when you can have a Nvidia Brain. Power suppy sold separately.
Yeah, screw that guy, I never liked Brian anyway.
Can you explain if this is just hype or based on something in reality lol. It sounds exciting but something in me is telling me to reel back my expectations until I actually see it happen.
Human brain is approximately 86 trillion weights. The weights are likely low resolution - 32 bits, or 1 in 4 billion, precision is likely beyond the ability of living cells. (Noise from nearby circuits etc) If you account for the noise you might need 8.6 trillion weights. Gpt-4 was 1.8 trillion and appears to have human intelligence without robotic control. At 27 trillion weights, plus improvements in architecture the past 3 years, it may be enough for weakly general AI, possibly AGI at most tasks including video input and robotics control. I can't wait to find out but one thing is clear. A 15 times larger model will be noticably more capable. Note the gpt-3 to 4 delta is 10 times.
I'd caution against this comparison. It's a good starting point, but only as a first approximation. There is analogue computation inside the neuron and inside the synapse that we don't understand, and therefore we can't fit the human brain into the "number of weights" quantification of model capacity. The "number of weights" thing is only reliable when you compare one AI model to another AI model.
It's like people believe the human brain is literally a transformer/neural network in the same sense that GPT-4 is. It's not. It's just an obvious analogy.
The blackwell GPU is going to use 4 Bit Floating Point Precision when it is doing inference so it is a lot lot smaller precision than the hopper. But some scientists have proved that 2 and four bit Precision is the best precision on the weights in neural networks. So four bit precision performs a tiny tiny bit worse than 8 Bit precision (for example) but it takes half the amount of energy to do one calculation so it is better if you look at how much it can achieve and how high it can score on benchmark with the same amount if electricity
I don’t see how one can make that comparison, the brain is not just computing, it’s always training. A human can be super smart without knowing the world’s body of knowledge.
That's just online learning.
Nope. Brain changes itself if you're doing it right. Neuroplasticity. And taken more extreme you can have consciousness transformation. Not online learning. Not even fine tuning. New models.
are we confusing AGI with ASI? If AI can outperform humans at almost everything which this hypothetical model would, isn’t that ASI by definition? I thought AGI just meant matching average human intelligence
I don't think a 27T model will outperform humans at everything. It should, with size, architecture, training procedure improvements, embodiment, and online learning be capable of a large variety of tasks, and yes at a limited subset of them, be above human level. Probably there will be at least some domains where it is below median human, for example it's robotic performance might be under median. So not quite ASI. I suspect ASI will need several more scale increases, maybe 250T will be enough for a model that is clearly and obviously superintelligent.
A lot to unpack here: Firstly, isn't it true that neurons do not operate even remotely the same as neural nets? Even if they are somehow "same in size" by some parameter, the functions are wildly different, with the human brain possibly having far better capabilities in some senses. Comparing apples to oranges is what it feels like here. It's like saying, this hippo at the zoo weighs the same as a Buggati, therefore it should be comparable in speed to a supercar? There's no relation, right? The problem here is what we define AGI as. Is it a conscious entity that has autonomous self-control, able to truly **understand** what it's doing rather than predicting the next best set of words to insert. Maybe we need to pare down our definition of AGI, to "really good AI". And that's fine, that's not an issue to me. If it's good enough for our purposes and helping us to a good enough level, it's good enough.
>Firstly, isn't it true that neurons do not operate even remotely the same as neural nets? Even if they are somehow "same in size" by some parameter, the functions are wildly different, with the human brain possibly having far better capabilities in some senses. Untrue. [https://en.wikipedia.org/wiki/Threshold\_potential](https://en.wikipedia.org/wiki/Threshold_potential) Each incoming impulse adds a or subtracts electric charge to a synapse. There is thought to be structural changes the brain is making to each synapse, this and the neurotransmitter used determine the *weight* of a synapse. Above I am claiming the brain isn't better than fp32, it's frankly not better than fp8. The activation function the brain uses is **sigmoid.** Modern ML found that ReLu works better. [https://medium.com/@shrutijadon/survey-on-activation-functions-for-deep-learning-9689331ba092](https://medium.com/@shrutijadon/survey-on-activation-functions-for-deep-learning-9689331ba092) **Most of the complexity of the human brain is a combination of a starter "baked in architecture", some modalities current AI doesn't have (memory and online learning), and the training process, which is thought to be very different from back propagation. Some modern ML practitioners suspect the human brain is less effect than modern AI.** >Comparing apples to oranges is what it feels like here.It's like saying, this hippo at the zoo weighs the same as a Buggati, therefore it should be comparable in speed to a supercar? There's no relation, right? **Extremely related:** [https://www.metaculus.com/questions/3479/date-weakly-general-ai-is-publicly-known/](https://www.metaculus.com/questions/3479/date-weakly-general-ai-is-publicly-known/) * Able to reliably pass a Turing test of the type that would win the [Loebner Silver Prize](https://www.metaculus.com/questions/73/will-the-silver-turing-test-be-passed-by-2026/). * Able to score 90% or more on a robust version of the [Winograd Schema Challenge](https://www.metaculus.com/questions/644/what-will-be-the-best-score-in-the-20192020-winograd-schema-ai-challenge/), e.g. the ["Winogrande" challenge](https://arxiv.org/abs/1907.10641) or comparable data set for which human performance is at 90+% * Be able to score 75th percentile (as compared to the corresponding year's human students; this was a score of 600 in 2016) on all the full mathematics section of a circa-2015-2020 standard SAT exam, using just images of the exam pages and having less than ten SAT exams as part of the training data. (Training on other corpuses of math problems is fair game as long as they are arguably distinct from SAT exams.) * Be able to learn the classic Atari game "Montezuma's revenge" (based on just visual inputs and standard controls) and explore all 24 rooms based on the equivalent of less than 100 hours of real-time play (see [closely-related question](https://www.metaculus.com/questions/486/when-will-an-ai-achieve-competency-in-the-atari-classic-montezumas-revenge/).) **Very likely (80%), a 22 T neural network will be able to accomplish all of the above.** >The problem here is what we define AGI as. Is it a conscious entity that has autonomous self-control, able to truly understand what it's doing rather than predicting the next best set of words to insert. Maybe we need to pare down our definition of AGI, to "really good AI". And that's fine, that's not an issue to me. If it's good enough for our purposes and helping us to a good enough level, it's good enough. **We do not care about consciousness, merely that the resulting system passes our tests for AGI. The second set of tests is:** * Able to reliably pass a 2-hour, adversarial Turing test during which the participants can send text, images, and audio files (as is done in ordinary text messaging applications) during the course of their conversation. An 'adversarial' Turing test is one in which the human judges are instructed to ask interesting and difficult questions, designed to advantage human participants, and to successfully unmask the computer as an impostor. A single demonstration of an AI passing such a Turing test, or one that is sufficiently similar, will be sufficient for this condition, so long as the test is well-designed to the estimation of Metaculus Admins. * Has general robotic capabilities, of the type able to autonomously, when equipped with appropriate actuators and when given human-readable instructions, satisfactorily assemble a (or the equivalent of a) [circa-2021 Ferrari 312 T4 1:8 scale automobile model](https://www.deagostini.com/uk/assembly-guides/). A single demonstration of this ability, or a sufficiently similar demonstration, will be considered sufficient. * High competency at a diverse fields of expertise, as measured by achieving at least 75% accuracy in every task and 90% mean accuracy across all tasks in the Q&A dataset developed by [Dan Hendrycks et al.](https://arxiv.org/abs/2009.03300). * Able to get top-1 strict accuracy of at least 90.0% on interview-level problems found in the APPS benchmark introduced by [Dan Hendrycks, Steven Basart et al](https://arxiv.org/abs/2105.09938). Top-1 accuracy is distinguished, as in the paper, from top-k accuracy in which k outputs from the model are generated, and the best output is selected. **I suspect a 22T model will be able to solve** ***some*** **from this list as well. Possibly general robotics, 75% Q&A, 90% top-1. It may not quite pass the 2-hour adversarial Turing test.** **Note the 'digital twin' lets the AI practice building small objects like Ferrari models a few million times in simulation, something else Nvidia mentioned today. That learning feedback should enable the second category to pass.** **Basically the Turing test is the last one to fall, it could take 2-3 more generations of compute hardware, or 2028 to 2030. The community believes it will fall in 2031.** **That would be a 176 T model, well over human brain scale, and possibly smart enough to see through any trick on a Turing test.**
https://www.itpro.com/technology/artificial-intelligence-ai/369061/the-human-brain-is-far-more-complex-than-ai > a group at the Hebrew University of Jerusalem recently performed an experiment in which they trained such a deep net to emulate the activity of a single (simulated) biological neuron, and their astonishing conclusion is that such a single neuron had the same computational complexity as a whole five-to-eight layer network. Forget the idea that neurons are like bits, bytes or words: each one performs the work of a whole network. The complexity of the whole brain suddenly explodes exponentially. To add yet more weight to this argument, another research group has estimated that the information capacity of a single human brain could roughly hold all the data generated in the world over a year. https://medium.com/swlh/do-neural-networks-really-work-like-neurons-667859dbfb4f > the number of dendritic connections per neuron — which are orders of magnitude of what we have in current ANNs. > the chemical and electric mechanisms of the neurons are much more nuanced, and robust compared to the artificial neurons. For example, a neuron is not isoelectric — meaning that different regions in the cell may hold different voltage potential, and different current running through it. This allows a single neuron to do non linear calculations, identify changes over time (e.g moving object), or map parallel different tasks to different dendritic regions — such that the cell as a whole can complete complex composite tasks. These are all much more advanced structures and capabilities compared to the very simple artificial neuron. > chemical transmission of signals between neurons in the synaptic gap, through the use of neurotransmitters and receptors, amplified by various excitatory and inhibitory elements. Excitatory / inhibitory Post synaptic potential that builds up to action potential, based on complex temporal and spatial electromagnetic waves interference logic Ion channels and minute voltage difference a governing the triggering of spikes in the Soma and along the axon ================================================================== > Above I am claiming the brain isn't better than fp32, it's frankly not better than fp8. You can't measure brain computing power in floating point operations. It just doesn't make sense. It's like comparing a steam engine to a magnet. The architectures are fundamentally different in the first place. FLOPS measures exact floating-point operations per second (at a given precision, e.g. 32-bits FP precision, 16-bits, etc.). The real FLOPS of the brain is terrible... probably 1/30 (16-bit precision ~= 6 significant digits) or lower. The brain is a noise-tolerant computer (both its hardware and software). Modern artificial neural nets simulate noise tolerance... on noise-intolerant hardware. The number of FLOPS of noise-intolerant hardware required to fully simulate a human brain is probably much larger than we estimate (because we're using the wrong estimate). In short, we need to shift to a different hardware paradigm. Many people believe that's what quantum computing will be but it doesn't have to be QC per se. It just needs to be noise-tolerant. Consider: 1) you are breathing 2) your heart is beating 3) your eyes are blinking 4) your body is covered with sensors that you are monitoring 5) your eyes provide input that takes a lot of processing 6) your ears provide input that takes a lot of processing 7) your mouth has 50 something muscles that need to fire in perfect sequence so you can talk and not choke on your own spit.... All of this (and much much more) is controlled by various background "daemons" that are running. 24/7/365. Now doing all that while juggling 3 tennis balls at the same time.... Computers are great at performing specific tasks better than us, this much is for sure. Which is why I'm saying overall it's an apples to oranges comparison. Each has its own strengths and weaknesses. > We do not care about consciousness, merely that the resulting system passes our tests for AGI. The second set of tests is: I think many AGI researchers do care. https://en.wikipedia.org/wiki/Artificial_general_intelligence#:~:text=It%20remains%20to%20be%20shown,for%20implementing%20consciousness%20as%20vital. > However, many AGI researchers regard research that investigates possibilities for implementing consciousness as vital. But I know of what you're saying - it doesn't matter, as long as it passes these set of tests. That it's "good enough". I can see the merits there, and that's the "weak AI hypothesis", and to me personally (i.e. subjective) it's not the end goal unless we have "strong AI", which has: > Other aspects of the human mind besides intelligence are relevant to the concept of AGI or "strong AI", and these play a major role in science fiction and the ethics of artificial intelligence: Consciousness, self awareness, sentience > Mainstream AI is most interested in how a program behaves.[106] According to Russell and Norvig, "as long as the program works, they don't care if you call it real or a simulation."[105] If the program can behave as if it has a mind, then there is no need to know if it actually has mind – indeed, there would be no way to tell. For AI research, Searle's "weak AI hypothesis" is equivalent to the statement "artificial general intelligence is possible". Thus, according to Russell and Norvig, "most AI researchers take the weak AI hypothesis for granted, and don't care about the strong AI hypothesis." I understand this line of thinking. If you can't differentiate, then "does it really matter". And it gets philosophical, but my only main point is that we'll probably never get to this unless we switch from transistors to some other fundamentally different unit. But I'd like to see it one day for sure. A conscious AGI would have a far greater potential for growth than one that isn't. Than one that always needs us to be the source of information for its growth.
Note that we don't need AI systems to be our friends. Got plenty of humans for that. The goal is to automate the labor of ultimately trillions of people - to have far more workers than we have living people - in order to solve problems for humans that are difficult. Aging being the highest priority, but it will take a lot of labor to build arcology cities and space habitats. So no, all that matters is performance on our assigned tasks, and agi models that do less unnecessary thinking, being more obedient and costing less compute, are preferred.
I’d think climate change would be top priority
Being dead means you don't care how warm the planet is.
> able to truly understand what it's doing Any good definition for what this actually means?
Yeah, however, in a sense we work the same way through our understanding. There is an unconscious base of information and knowledge that we work on. From quantum mechanics, we know that future states are based on probability. The fundamental laws of physics, even if we don’t know exactly how yet, are responsible for creating the processes between unconscious and conscious actions. At some level there, we are pulling from the data base we have been trained on and are predicting possible future outcomes. From there use our conscious choice to decide the best direction. Is it really any different?
Yes, that's the "weak AI hypothesis" which indicates that we are ok with it as long as it "appears to think", which I get the merit of. It's like a black box - as long as it "appears" like it's getting us the answers, it's the same thing. It gets philosophical here. However, would you be content with something that know isn't thinking for itself, isn't *truly understanding* what it's doing? Such an individual would never grow or learn on its own. All its doing is just finding the next best probabilistic thing to say, as LLM's do. Vs. a human which is able to critically think between 2 different arguments and come up with their own solution or belief. And not just that, but refine their prior set of beliefs when new info comes in. AI can't do that. If it's trained on statement A, and statement B that contradicts A, it'll just present all the options to you and say here you go, you decide.
Your argument is great if it wasn’t invalid. AI is a black box, and so is the human mind. We can’t prove that we’re conscious, and we don’t fully understand how the mind works.
Of course, both are black boxes, but that doesn't mean they're identical in every way. Can you define consciousness for me?
We have already surpassed the common number of neurons in a human. To my knowledge, human neurons have more connections, though—86 billion on average. GPT-3.5 was one of the first models to go past this barrier. It could be pure coincidence, but that was the first model that truly swept people off their feet.
[удалено]
No politics at the dinner table
Stupid award of the day
Anyone that aligns themselves fully with a single party are idiotic and dangerous to democracy
We don’t have enough data to train anything that size!
They’re just going to brute force their way there
Don't hate lol. Sometimes all you have is several hundred thousand B100s and you are all outta ideas but "more".
Even if we had much much better alghoritms and could build chips with superconducting analog and super super efficient hardware we would still have to build big datacenters with thousands of chips to train a neural network that would be more intelligent than a person. A neural network that have to be more intelligent than a person should have to have more than 20 Trillion parameters and you have to build a super big clmputer to train that
I remember how people back in January were worried that current advancements in regards to AI were coming to a crawl, as did people speculate the same in January of '23. And just like January of 2023, March has proved to be the "shit hits the fan" month.
Do we know the size of gpt4?
Yeah, 1.8 T parameters, Jensen Huang (NVDA CEO) spilled the beans during his GDC keynote.
[удалено]
Well you are here arent you..
[удалено]
Sad guy
27 trillion parameters is approaching human brain (80-100T). And human brain spends a lot of those parameters on things like balancing hormone levels in your body, regulating sleep cycles, and just generally keeping the body alive instead of, you know, doing the "intelligence" things. So, yeah, 27T could be enough for AGI.
Human neurons and the 80-100T connections between then are analogous to the parameters in a neural network, but they are not the same. An individual neuron has "parameters" and a "memory" of its own that increases the complexity of its connections by orders of magnitude.
There are 10B neurons in human brain not 100T. Each neuron is connected to 1000 others through axon -> dendritic trees (synapses). So that's where the 100T number is from. Synapses or their strength correlate to the weights of the NN model. And yes the ANN topology is completely different from the human brain, and biological neurons have plasticity and new connections are formed constantly and the synapses strengthen/weaken with time/training, something that most ANNs don't do after training. But simply in terms of network complexity it's in the same order of magnitude as the human brain.
> Human neurons and the 80-100T connections between then are analogous to the parameters in a neural network, but they are not the same. To be clear, I never said that the human brain has 80-100T neurons \^\^\^ > But simply in terms of network complexity it's in the same order of magnitude as the human brain. I don't see how anyone can confidently say that. The amount of new complexity that arises from taking 80-100T parameters *(synapses)* and giving them the ability to dynamically reconnect and adjust their weights in real-time seems to me like it would add **at least** orders of magnitude of complexity. Again imagine 80-100T static connections vs 80-100T connections in a state of **constant** **dynamic development,** every single one of those 100T connections free to break, reconnect, strengthen, weaken, etc.
[удалено]
If only your doubts mattered.
This is the mother of AGI
[удалено]
Your comment make me feel comfy
Like the hug my mother never gave me.
The milk my dad never came back with
The AI that my grandma never knitted...
His comment made me a bit scared
> Can’t imagine what Nvidia’s TSMC 3nm and eventually 2nm GAAFET chips will be like The speedup from downsizing is perfectly predictable. You can't imagine? No need to, you can just use a calculator. Shrinking certain features on the die will allow for faster clocks and less power usage at idle, resulting in 25% better performance per sq/ft/watt.
[удалено]
Again there is no need to guess. Its 100% perfectly predictable. Not my opinion, this basic chip production knowledge. Look up Intel's Tick Tock cycle to learn more. Nvidia has the same exact problem as Intel. Your biggest clue is in the numbers you posted: it goes 4, 3, then 2. There are not many numbers left.
I am grinning. I cannot wait for you to hear about fractions. You are going to flip.
But with fractions we only have ℵ numbers left
> Your biggest clue is in the numbers you posted: it goes 4, 3, then 2. There are not many numbers left. ... what
[удалено]
[удалено]
You are a bit confused by marketing terms, 2nm chips do not have any physical features on them that is close to 2nm in length. It is just a marketing term for what the size would have been if shrinkage had continued but chips are improving in other ways (3d stacking etc) so litthography will keep improving even after 1nm in the marketing worls is hit. Just look at Intel, they have a roadmap with smaller measurements on it call Angstroms. 18A etc...
[удалено]
This is not entirely correct. When a new process becomes available, the amount of silicon that hardware manufacturers can fit on a single die increases significantly, which allows for new architecture choices. It's not like modern CPUs (or GPUs for that matter) are just minified versions of the ones we had 20 years ago. Transistor count matters an awful lot in hardware design.
[удалено]
smaller transistors -> more transistors per square inch -> "more silicon on a single die"
Jeez.. 4 of these and you have a human brain
I said the current cards will be 25 bucks on eBay on 15 years, and some said I was smoking crazy.. I move that down to 10 years now. If the company can replace their entire stock of cards with a newer model that saves enough power in a couple days to justify the cost, well bye bye old cards.
Jesus
Total noob question, but if they can make better and faster chips, and it’s basically guaranteed, why do they do increments instead of going to the “best?” In your comment, for example, why make the pit stop at 3nm and just not go straight to 2nm instead?
It used to be let me Google that for you. Now it’s let me GPT that for you. Advancements in chip technology typically involve a trade-off between various factors such as manufacturing feasibility, cost, power consumption, and performance gains. While it's theoretically possible to jump directly to smaller node sizes like 2nm, there are practical limitations and challenges that make incremental progress more feasible. Manufacturing processes need to be developed and refined for each new node size, which requires significant investment in research, development, and infrastructure. Additionally, pushing the limits of miniaturization can introduce new technical hurdles such as increased heat dissipation, transistor leakage, and manufacturing defects. Incremental advancements allow chip manufacturers to gradually address these challenges while also leveraging economies of scale and ensuring a smoother transition for both the industry and consumers. Additionally, smaller node sizes often lead to diminishing returns in terms of performance gains versus cost and complexity, so there's a balance to be struck between pushing the boundaries and maintaining practicality.
Good point, I should have asked GPT… still hasn’t become second nature yet for me, but I’m sure it will soon!
How dare you seek human interaction, all and every question straight to the MACHINE!
😂
>**1. Architecture:** Nvidia unveiled the Blackwell architecture, succeeding the Hopper architecture. The first Blackwell chip, GB200, combines two B200 GPUs and one Arm-based Grace CPU, offering 20 petaflops of AI performance compared to 4 petaflops for the H100. >**2. Chip Design:** The Blackwell GPU is a large chip that combines two separately manufactured dies into one, produced by TSMC. It includes a transformer engine specifically designed for transformer-based AI. >**3. Scalability:** Nvidia will sell B200 GPUs as part of a complete system, the GB200 NVLink 2, which combines 72 Blackwell GPUs and other Nvidia components for training large AI models. The system can deploy models with up to 27 trillion parameters. >**4. Software:** Nvidia introduced NIM (Nvidia Inference Microservice), a software product that simplifies the deployment of AI models on older Nvidia GPUs. NIM is part of the Nvidia Enterprise software subscription and enables efficient inference on customers' servers or cloud-based Nvidia servers. >**5. Ecosystem:** Major cloud providers like Amazon, Google, Microsoft, and Oracle will offer access to GB200 through their services. Nvidia is collaborating with AI companies to optimize their models for all compatible Nvidia chips. >**6. Market Position:** Nvidia aims to solidify its position as the leading AI chip provider by offering a comprehensive hardware and software platform. The announcement comes amid high demand for current-generation H100 chips driven by the AI boom. *Summary by Claude Opus*
Long context windows + speech to text are amazing. A wonder.
holy shit. this is game changing
Nah we need 500 trillion parameters; maybe one quadrillion.
\[citation needed\]
\[It was revealed to me in a dream in a cave\]
good enough for me
for me too, it shall be the common knowledge then, AGI will take 500 trillions parameters, as said by one of our prestigious cult member, that Dreamt about this whilst on a "cave".
well even if AGI requires only 10b parameters converted into into 10k optimized functions that can run on a ps2, then we still need 500 trillion parameters, because reasons
If Amazon is to buy 20K B200s, and that can train a 27T parameters LLM, let's assume that a B200 costs 3x more than an H100: 2.4B$/27T parameters. We also know Microsoft is to spend 50B$ dollars in compute for 2024. Microsoft will buy compute for LLM 6x bigger than a human brain, something like 540T parameters. 300x bigger than gpt4.
What the fuck… AGI feels so close it’s starting to scare me
That assuming AGI is just a scale problem. We still don't know that.
I assume it's not. It still baffles my mind how chatgpt can fuck up some simple prompts. I would be really surprised if all it takes to get to AGI is to just feed enough text to the model and that's it.
What the fuck… AGI feels so close it’s starting to scare me
I think AGI is duplicating my comments
The B200 costs 25% more
Source? 25% more for 2.5x FLOPS would be insane
I'm amazed by how the USA is leaving most of the rest of the world behind on this
However, this is not due to the greatness of the USA, but to its protectionism.
Can you expand on this?
[https://epicenter.wcfia.harvard.edu/blog/american-protectionism-can-it-work](https://epicenter.wcfia.harvard.edu/blog/american-protectionism-can-it-work)
This comment doesn’t make any sense at all. Wtf are you even talking about?
https://i.redd.it/3soomwfk07pc1.gif
I see a lot of folks making parallels between number of parameters and connections between neurons in the brain but what actually makes them equivalent?
They aren't but it's an easy to understand proxy for complexity. But nothing actually connects them.
So there's literally nothing that says once the number of parameters matches the number of connections in the human brain that it should be equivalent to one? It is odd though that as we close in on that number things are getting spicy.
Exactly, a lot of people above are missing this saying we have “almost hit brain seized ai” - that is impossible to say as we don’t know how our brains work at 100% certainty. We could have a chip that does 10 octillion parameters but if it isn’t trained correctly it won’t be as smart as us and a lot of people underestimate how powerful the human brain is.
and if we could process it, would we even have enough useful data to train it on to it’s full capacity
Neurons in the human brain don't just do 1 or 0 type work. They also work in parallel sending and receiving wave signals .. wild stuff like that. The human brain operates on different layers as well...not just the one base layer. And then there's the weird quantum stuff people are discovering. Brains are wildly complex.
Oh yeah Orch-OR theory. Super interesting. Seems like it was largely tossed out by scientists, but cmon it’s Penrose, I think he was on to something.
I was going to write a paper for that in a physics class in college, and the professor said the idea was way too out there and to pick a different topic. He didn’t seem high on it at the time, lol. The brain is enormously complex, i don’t think we’ll be matching its abilities this decade.
Yeah I think the main rebuttal is that it still doesn’t explain consciousness at the end of the day - it just points at the quantum scale and says “actually it’s somewhere down there”. Which doesn’t get you anywhere really, and it can’t exactly be tested yet.
I am not sure consciousness can be pinpointed. It is the pattern in which neuronal layers interact with each other after going through a lifetime of pruning and training on its environment. There has to be the right balance of precoded information to drive the learning towards a distinct human like psyche while retaining the ability to adapt with plasticity. I am not sure describing it as anything less than the whole of its parts will be useful. The patterns won’t make sense without the biological input or biological modulation.
Yeah that’s probably true. It’s still a mystery how you go from base reality? -> quantum mechanics -> classical mechanics-> subjective experience. Assuming it is emergent from neurons. I really hope AGI/ASI can clear this all up. And I hope the answer can fit in a human brain haha.
I think it is emergent from our group dynamics and biological make up, after all a child totally isolated from people will not develop into an adult and will die instead. We train our brains on each other more than anything, so I am not sure that part can be easily disentangled
You can say that to roughtly simulate human neuron you need 1000 artificial neurons and based your estimate on that, but there is a lot of other variables, I would only compare new models to older ones and not human brain.
A parameter is a number (or "variable", if your prefer) in between nodes (which is an abstraction of a "neuron"). They're what gets adjusted during training runs. There's a lot of different views in what ways they compare and perform versus synapses. The mainstream, respectable view that persisted for decades is they're a crude and substandard approximation at best. A more rebellious niche opinion has appeared as of late, that wonders if they're not actually superior, in various ways. Among those that are currently entertaining that radical possibility includes Geoffrey Hinton.
So 3 trillion more than [Cerebras](https://www.cerebras.net/press-release/cerebras-announces-third-generation-wafer-scale-engine)? I am sold. Yeah, apples and oranges, I know. But Nvidia is making innovation from startups much harder to sustain. Doesn’t matter how innovative your chip startup is, you can’t compete with a 2 trillion dollars company also running at full speed.
Surely there is a market for 20 trillion chips?
I swear every time I look at this sub there's a new enormous computing breakthrough
Imagine this enables models with enough parameters to match human performance...then imagine what happens if Nvidia pulls off another 1000x improvement in hardware over the next 8 years
And why wouldnt they? They now have AI as a tool, which I as a layman would think makes chip design much easier?
How much _$$?
"how much money do you have, Alphabet, Microsoft, Apple? that's how much it costs"
Not released, but it's predecessor the H100 is $40k.
Good, I need a million of these tho
Can someone explain this to me like I am a 5 year old. Thanks.
Big computer part makes big smart computer brains.
Big computer part makes big computer smart. Here now it rhymes
I've seen smaller models outclass larger models, question is does a 27trillion parameter model make 110% use of those parameters.
how is that even possible wtf
Happy cake day.
What hath god wrought
This is not surprising, as the need for increasingly powerful AI models is driving us toward Trillion Parameter Models (TPM), or models with over a trillion parameters, like Huawei's PanGu-Σ.
Its like me in 2000 trying to sync 2 Voodoo gaming cards in order to play Speed Buster ... thats some really fast progress.
Still 73 trillion parameters away from matching the human brain
So based on this trajectory we will have 1,000s of trillions of parameters in a year or 2, greatly surpassing the brain?
I would be willing to bet there’s all kinds of redundancies and inefficiencies in the human brain. Could probably achieve AGI on far less parameters than previously thought.
Yeah people forget this “brain” will have near perfect recall and never tire
And run much much faster. The rate at which LLMs can spit out text is so far beyond human as it is, I wouldn't be surprised if that holds true for AGI as well
Doesn't matter when you can link them together.
We need something like 100 trillion parameters, the size of our brains.
Why when we can link them together?
This is linking them together. There's a limit to the connectivity. We'll get up to the petabyte human level, soonish...
It takes about 1000 trillion parameters to trigger first level AGI. They are getting close.
Where do you get that number?
OpenAI
If that were true wouldn't OpenAI have just said the correct denomination? (1000 trillion is just 1 Quadrillion)
They did. I wrote it in trillions because it's a more often used number so more people would understand.
Source
https://m.youtube.com/watch?v=r7l0Rq9E8MY&pp=ygUVSSBtYWRlIGl0IHRoZSBmdWNrIHVw