weird - even if AI was literally omnipotent and omniscient, you would still be bottlenecked on human's ability to actually evaluate and verify what it is doing and reconciling that with what you wanted it to do. Unless you're of course, willing to YOLO the entire company on output you haven't actually checked yourself.
for that reason alone humans will always need to be in the loop. of course you can debate how many people you need to the above activity, but given that AI isn't omniscient, nor omnipotent I expect that number to be quite high for the foreseeable future.
one example - I've been vibe coding some stuff, and even though a pretty comprehensive set of tests are passing, I still end up reading all of the code. if I'm being honest some of the decisions the AI makes are a bit opaque to me so I end up spending a bunch of time asking it why (of course there's no real ego there, but bare with me...), re-reading the code, thinking about whether that actually makes sense. I personally prefer this activity/mode since the tests pass (which were written by the AI too), and I know anything I manually change can be tested, but it's not something I could just submit to prod right away. this is just a MVP. I can't imagine delegating if real money/customers were on the line without even more scrutiny.
>weird - even if AI was literally omnipotent and omniscient, you would still be bottlenecked on human's ability to actually evaluate and verify what it is doing and reconciling that with what you wanted it to do.
one would hope that one ability of an 'omniscient and omnipotent' AI would be greater understanding.
When speaking of the divine (the only typical example of the omniscient and omnipotent that comes to mind) we never consider what happens when God (or whoever) misunderstands our intent -- we just rely on the fact that an All-Being type thing would just know.
I think the understanding of minute intent is one such trait an omniscient and omnipotent system must have.
p.s. what a bar raise -- we used to just be happy with AGI!
Genies, maybe? They are omnipotent and (generally) sufficiently aware of your desires that they shouldn't actually get "confused". Genies are tricksters that will do their absolute best to fulfill the letter of your wish but not the meaning.
Right, but you have to do a lot of work, and really most of your work is in this area. Less on the actual building stuff.
Figuring out what to build is 80% of the work, building it is maybe 20%. The 20% has never been the bottleneck. We make a lot of software, and most of it is not optimal and requires years if not decades of tweaking to meet the true requirements.
> In reality, even an ASI won’t know your intent unless you communicate it clearly and unambiguously.
I recently came to this realization as well, and it now seems so obvious. I feel dumb for not realizing it sooner. Is there any good writing or podcast on this topic?
Not really a bar raise - many people have assumed that "AGI" would mean essentially omnipotent/omniscient AI since the concept of the technological singularity came into being. Read Kurzweil or Rudy Rucker, there's a reason this sort of thing used to be called the "rapture for nerds."
If anything I've noticed the bar being lowered by the pro-AI set, except for humans, because the prevailing belief is that LLMs must already be AGI but any limitations are dismissed as also being human limitations, and therefore evidence that LLMs are already human equivalent in any way that matters.
And instead of the singularity we have Roko's Basilisk.
The problem goes deeper: verification is harder than generation. When writing an answer yourself, you build the logic chain from scratch. When verifying AI, you have to deconstruct its logic, cross-reference facts, spot hidden hallucinations, and only then approve. For complex cases (which are exactly what the humans were left with), the time for quality verification approaches the time to write from scratch. If the time becomes roughly equal, the AI stops being an accelerator and becomes just a source of noise that yields no productivity gains
Move fast and break things. When a black box can be blamed, why care about quality? What we need is EXTREMELY strict liability on harms done by AIs and other black box processes. If a company adopts a black box, that should be considered reckless behavior until proven otherwise. Taking humans out of the loop is a conscious decision they make therefore they should be fully responsible for any mistakes or harms that result.
>you would still be bottlenecked on human's ability to actually evaluate and verify what it is doing and reconciling that with what you wanted it to do.
this sort of assumes that most humans actually know what they want to do.
It is very untrue in my experience.
Its like most complaints I hear about AI art. yes, it is generic and bland. just like 90% of what human artists produce.
fair. I used to think that too, but I find at least for golang, the sota models write tests way faster than I would be able to. tdd is actually really possible with ai imo. except of course you get the scaffolding implementation (I haven't figured out a way to get models to write tests in a way that ensures the tests actually do something useful without an implementation).
Your final sentence is interesting. I'm not a strict doctrine adherent, but in TDD, don't you write some minimal test, then implement the system to pass the test?
yes, but I find it hard to constrain it to a minimal implementation. what usually happens is it writes some tests, then an implementation, and then according to the thinking, makes some modification. it works with a relatively precise prompt, but starts to go a bit off the rails when you say things in broad terms ("write tests to ensure concurrency works, and the implementation to ensure said tests are correct")
> even if AI was literally omnipotent and omniscient, you would still be bottlenecked on human's ability to actually evaluate and verify what it is doing and reconciling that with what you wanted it to do
no no no you don't get it, you would have ANOTHER AI for that
It's not even about humans "needing" to be in the loop, but that humans "want" to be in the loop. AI is like a genius employee who has no ego and no desire to rise up the ranks, forever a peon while more willful colleagues surpass them in the hierarchy.
Until AI gets ego and will of its own (probably the end of humanity) it will simply be a tool, regardless of how intelligent and capable it is.
Humans need to be in the loop for the same reason other humans peer review humans pull requests: we all fuck up. And AI makes just as many mistakes as humans do. They just do so significantly quicker.
for that reason alone humans will always need to be in the loop. of course you can debate how many people you need to the above activity, but given that AI isn't omniscient, nor omnipotent I expect that number to be quite high for the foreseeable future.
one example - I've been vibe coding some stuff, and even though a pretty comprehensive set of tests are passing, I still end up reading all of the code. if I'm being honest some of the decisions the AI makes are a bit opaque to me so I end up spending a bunch of time asking it why (of course there's no real ego there, but bare with me...), re-reading the code, thinking about whether that actually makes sense. I personally prefer this activity/mode since the tests pass (which were written by the AI too), and I know anything I manually change can be tested, but it's not something I could just submit to prod right away. this is just a MVP. I can't imagine delegating if real money/customers were on the line without even more scrutiny.