They have a moat defined by being well known in the AI industry, so they have credibility and it wouldn't be hard for anything they make to gain traction. Some unknown player who replicates it, even if it was just as good as what SSI does, will struggle a lot more with gaining attention.
Agreed. But it can be a significant growth boost. Senior partners at high-profile VCs will meet with them. Early key hires they are trying to recruit will be favorably influenced by their reputation. The media will probably cover whatever they launch, accelerating early user adoption. Of course, the product still has to generate meaningful value - but all these 'buffs' do make several early startup challenges significantly easier to overcome. (Source: someone who did multiple tech startups without those buffs and ultimately reached success. Spending 50% of founder time for six months to raise first funding is a significant burden (working through junior partners and early skepticism) vs 20% of founder time for three weeks.)
The impactful innovations in AI these days aren't really from scaling models to be larger. It's more concrete to show higher benchmark scores, and this implies higher intelligence, but this higher intelligence doesn't necessarily translate to all users feeling like the model has significantly improved for their use case. Models sometimes still struggle with simple questions like counting letters in a word, and most people don't have a use case of a model needing phd level research ability.
Research now matters more than scaling when research can fix limitations that scaling alone can't. I'd also argue that we're in the age of product where the integration of product and models play a major role in what they can do combined.
Not necessarily. The problem is that we can't precisely define intelligence (or, at least, haven't so far), and we certainly can't (yet?) measure it directly. And so what we have are certain tests whose scores, we believe, are correlated with that vague thing we call intelligence in humans. Except these test scores can correlate with intelligence (whatever it is) in humans and at the same time correlate with something that's not intelligence in machines. So a high score may well imply high intellignce in humans but not in machines (e.g. perhaps because machine models may overfit more than a human brain does, and so an intelligence test designed for humans doesn't necessarily measure the same thing we think of when we say "intelligence" when applied to a machine).
This is like the following situation: Imagine we have some type of signal, and the only process we know produces that type of signal is process A. Process A always produces signals that contain a maximal frequency of X Hz. We devise a test for classifying signals of that type that is based on sampling them at a frequency of 2X Hz. Then we discover some process B that produces a similar type of signal, and we apply the same test to classify its signals in a similar way. Only, process B can produce signals containing a maximal frequency of 10X Hz and so our test is not suitable for classifying the signals produced by process B (we'll need a different test that samples at 20X Hz).
My definition of intelligence is the capability to process and formalize a deterministic action from given inputs as transferable entity/medium.
In other words knowing how to manipulate the world directly and indirectly via deterministic actions and known inputs and teach others via various mediums.
As example, you can be very intelligent at software programming, but socially very dumb (for example unable to socially influence others).
As example, if you do not understand another person (in language) and neither understand the person's work or it's influence, then you would have no assumption on the person's intelligence outside of your context what you assume how smart humans are.
ML/AI for text inputs is stochastic at best for context windows with language or plain wrong, so it does not satisfy the definition. Well (formally) specified with smaller scope tend to work well from what I've seen so far.
Known to me working ML/AI problems are calibration/optimization problems.
Forming deterministic actions is a sign of computation, not intelligence. Intelligence is probably (I guess) dependent on the nondeterministic actions.
Computation is when you query a standby, doing nothing, machine and it computes a deterministic answer. Intelligence (or at least some sign of it) is when machine queries you, the operator, on it's own volition.
> Intelligence (or at least some sign of it) is when machine queries you, the operator, on it's own volition.
So you think the thing, who holds more control/force at doing arbitrary things as the thing sees fit, is more intelligent? That sounds to me more like the definition of power, not intelligence.
> So you think the thing, who holds more control/force at doing arbitrary things as the thing sees fit, is more intelligent? That sounds to me more like the definition of power, not intelligence.
I want to address this item. I think not about control or comparing something to something. I think intelligence is having at least some/any voluntary thinking. A cat can't do math or write text, but he can think on his own volition and is therefore intelligent being. A CPU running some externally predefined commands, is not intelligent, yet.
I wonder if LLM can be stepping stone to intelligence or not, but it is not clear for me.
> My definition of intelligence is the capability to process and formalize a deterministic action from given inputs as transferable entity/medium.
I don't think that's a good definition because many deterministic processes - including those at the core of important problems, such as those pertaining to the economy - are highly non-linear and we don't necessarily think that "more intelligence" is what's needed to simulate them better. I mean, we've proven that predicting certain things (even those that require nothing but deduction) require more computational resources regardless of the algorithm used for the prediction. Formalising a process, i.e. inferring the rules from observation through induction, may also be dependent on available computational resources.
> What is your definition?
I don't have one except for "an overall quality of the mental processes humans present more than other animals".
> I mean, we've proven that predicting certain things (even those that require nothing but deduction) require more computational resources regardless of the algorithm used for the prediction.
I do understand proofs as formalized deterministic action for given inputs and processing as the solving of various proofs.
> Formalising a process, i.e. inferring the rules from observation through induction, may also be dependent on available computational resources.
Induction is only one way to construct a process and there are various informal processes (social norms etc). It is true, that the overall process depends on various things like available data points and resources.
> I don't have one except for "an overall quality of the mental processes humans present more than other animals".
How would your formalize the process of self-reflection and believing in completely made-up stories of humans often used as example that distinguishes animals from humans? It is hard to make a clear distinction in language and math, since we mostly do not understand animal language and math or other well observable behavior (based on that).
Ok, but the point of a test of this kind is to generalise its result. I.e. the whole point of an intelligence test is that we believe that a human getting a high score on such a test is more likely to do some useful things not on the test better than a human with a low score. But if the problem is that the test results - as you said - don't generalise as we expect them, then the tests are not very meaningful to begin with. If we don't know what to expect from a machine with a high test score when it comes to doing things not on the test, then the only "capacity" we're measuring is the capacity to do well on such tests, and that's not very useful.
Models aren't intelligent, the intelligence is latent in the text (etc) that the model ingests. There is no concrete definition of intelligence, only that humans have it (in varying degrees).
The best you can really state is that a model extracts/reveals/harnesses more intelligence from its training data.
Note that if this is true (and it is!) all the other statements about intelligence and where it is and isn’t found in the post (and elsewhere) are meaningless.
I did notice that, the person you replied to made a categorical statement about intelligence followed immediately with negating that there is anything to make a concrete statement about.
"Scaling" is going to eventually apply to the ability to run more and higher fidelity simulations such that AI can run experiments and gather data about the world as fast and as accurately as possible. Pre-training is mostly dead. The corresponding compute spend will be orders of magnitude higher.
That's true, I expect more inference time scaling and hybrid inference/training time scaling when there's continual learning rather than scaling model size or pretraining compute.
Simulation scaling will be the most insane though. Simulating "everything" at the quantum level is impossible and the vast majority of new learning won't require anything near that. But answers to the hardest questions will require as close to it as possible so it will be tried. Millions upon millions of times. It's hard to imagine.
I don't think so. Serious attempts for producing data specifically for training have not being achieved yet. High quality data I mean, produced by anarcho-capitalists, not corporations like Scale AI using workers, governed by laws of a nation etc etc.
Don't underestimate the determination of 1 million young people to produce within 24 hours perfect data, to train a model to vacuum clean their house, if they don't have to do it themselves ever again, and maybe earn some little money on the side by creating the data.
Counting letters is tricky for LLMs because they operate on tokens, not letters. From the perspective of a LLM, if you ask it "this is a sentence, count the letters in it" it doesn't see a stream of characters like we do, it sees [851, 382, 261, 21872, 11, 3605, 290, 18151, 306, 480].
There is a mapping. An internal, fully learned mapping that's derived from seeing misspellings and words spelled out letter by letter. Some models make it an explicit part of the training with subword regularization, but many don't.
It's hard to access that mapping though.
A typical LLM can semi-reliably spell common words out letter by letter - but it can't say how many of each are in a single word immediately.
But spelling the word out first and THEN counting the letters? That works just fine.
Making a VSCode fork is probably the wrong direction at this point in time. The future of agentic coding should need less support for code editor related functionality, and could eventually primarily support viewing code rather than editing code. There's a lot more flexibility in UI starting from scratch, and personally I want to see a UI that allows flexible manipulation of context and code changes with multiple agents.
GitHub is building a UI like this. I like it. I sometimes need the full IDE, but plenty of times don't. It's nice to be able to easily see what the agent is up to and converse with it in real-time while reviewing it's outputs.
> Briefly, the argument goes that if there is a 0.001% chance of AGI delivering an extremely large amount of value, and 99.999% chance of much less or zero value, then the EV is still extremely large because (0.001% * very_large_value) + (99.999% * small_value) = very_large_value
I haven't heard of that being the argument. The main perspective I'm aware of is that more powerful AI models have a compounding multiplier on productivity, and this trend seems likely to continue at least in the near future considering how much better coding models are at boosting productivity now compared to last year.
> I haven't heard of that being the argument. The main perspective I'm aware of is that more powerful AI models have a compounding multiplier on productivity, and this trend seems likely to continue at least in the near future considering how much better coding models are at boosting productivity now compared to last year.
This is the new line now that LLMs are being commoditized, but in the post-Slate Star Codex AI/Tech Accelerationist era of like '20-'23 the Pascal's wager argument was very much a thing. In my experience it's kind of the "true believer" argument, whereas the ROI/productivity thing is the "I'm in it for the bag" argument.
Right. Nobody makes a Pascal's wager-style argument in _favor_ of investing in AGI. People have sometimes made one against building AGI, on existential risk grounds. The OP author is about as confused on this as the water usage point... But the appetite for arguments against AI (which has legitimate motivations!) is so high that people are willing to drop any critical thinking.
That's not Pascal's Wager unless they're saying AGI has infinitesimal probability but infinite payoff, or something like that. If they think AGI is likely, they may be wrong but it's just technological optimism.
Unironically this is how most of us think about the world. I believe as humans we’re innately selfish and want to feel like we’re usually good or at least justified. Everyone is the protagonist of their own life and sometimes the antagonist of other people’s lives, so it’s worth considering how you could do better to others rather than continuing to justify yourself.
I used to think that Cursor would get killed by the major AI labs owning their coding models, but with the best open source models being on par with the proprietary models, I can realistically see Cursor winning the coding AI race. Once they start shifting most of the AI workload to be on their own custom models and optimize inference for it, their valuation can justifiably go on the same growth trajectory as the major AI labs.
Thanks for the question and feedback. We are hoping to get some early testers on board at first. But yes, those are some good points, and we will be sure to add more details and perhaps some sort of a demo soon!
