Problem is that for strawberries or diamonds or SV startup stuff there is a line where if you make something infinitely available it will also loose the value.
You cannot equate paper clip maximizer with SV companies. Because fitness functions are different.
Not saying that diamonds industry didn’t make world worse off but still somehow it didn’t take over the world to make people eat diamonds for breakfast. So the same there will be no entity that can make all people everywhere eat strawberries for breakfast all year long.
Scary part is finance industry that basically already is self conscious with all the rules baked in and no single person being able to grasp it.
Finance with AGI could already become paper clip optimizer - but it actually needs energy only. It doesn’t need humans anymore. So it would most likely fill in whole world with power plants and erase all other life just to have the electricity.
You cannot equate paper clip maximizer with SV companies. Because fitness functions are different.
Not saying that diamonds industry didn’t make world worse off but still somehow it didn’t take over the world to make people eat diamonds for breakfast. So the same there will be no entity that can make all people everywhere eat strawberries for breakfast all year long.
Scary part is finance industry that basically already is self conscious with all the rules baked in and no single person being able to grasp it.
Finance with AGI could already become paper clip optimizer - but it actually needs energy only. It doesn’t need humans anymore. So it would most likely fill in whole world with power plants and erase all other life just to have the electricity.