"I think without the ability to self-reflect and adjust one's motivations and goals depending on circumstances, you’re just not talking about intelligence anymore. "
This is a pretty bold statement. I believe we can all think of people who are successful in one or more spheres who exhibit neither self-reflection nor any ability to pivot under changing circumstances. But to the larger point that the meerkat is trying to make, the whole issue is that we wouldn't design a system to wipe us out *on purpose*. But that doesn't mean it can't happen. And of course, there are always people who actually do want to wipe everyone out on purpose, so there's that.
There are certainly those who disagree with the definition of intelligence espoused by Llama and Wombat. I think Llama might jump into the discussion and respond: Success depends on a lot more variables than intelligence, and there are other aspects of biological brains besides cognitive capacity that affect one's ability to self-reflect and adjust to circumstances, such as emotions and evolutionary cognitive biases. But when it comes to judging intelligence level, both that of humans as well as other animals, most would consider the intelligence to be greater the more it’s able to self-reflect and adjust to changing circumstances vs. doing either poorly or not at all.
The second point is actually a big one and one to be explored more in future posts. The leading voices on both sides of the discussion don’t deny that an existentially risky system could be implemented accidentally or that people who want to do harm could design such a system deliberately (in Bond supervillain fashion). What Meerkat is arguing, though, is not that people may inadvertently make a system that results in existential risk, but instead that every possible system one could make will lead to existential risk due to the nature of intelligence. This latter claim by Meerkat is the one being made by many of the leading voices representing AI Dystopianism (as in the papers from Yudkowsky, Bostrom, and Omohundro linked to in the post).
"I think without the ability to self-reflect and adjust one's motivations and goals depending on circumstances, you’re just not talking about intelligence anymore. "
This is a pretty bold statement. I believe we can all think of people who are successful in one or more spheres who exhibit neither self-reflection nor any ability to pivot under changing circumstances. But to the larger point that the meerkat is trying to make, the whole issue is that we wouldn't design a system to wipe us out *on purpose*. But that doesn't mean it can't happen. And of course, there are always people who actually do want to wipe everyone out on purpose, so there's that.
Good points!
There are certainly those who disagree with the definition of intelligence espoused by Llama and Wombat. I think Llama might jump into the discussion and respond: Success depends on a lot more variables than intelligence, and there are other aspects of biological brains besides cognitive capacity that affect one's ability to self-reflect and adjust to circumstances, such as emotions and evolutionary cognitive biases. But when it comes to judging intelligence level, both that of humans as well as other animals, most would consider the intelligence to be greater the more it’s able to self-reflect and adjust to changing circumstances vs. doing either poorly or not at all.
The second point is actually a big one and one to be explored more in future posts. The leading voices on both sides of the discussion don’t deny that an existentially risky system could be implemented accidentally or that people who want to do harm could design such a system deliberately (in Bond supervillain fashion). What Meerkat is arguing, though, is not that people may inadvertently make a system that results in existential risk, but instead that every possible system one could make will lead to existential risk due to the nature of intelligence. This latter claim by Meerkat is the one being made by many of the leading voices representing AI Dystopianism (as in the papers from Yudkowsky, Bostrom, and Omohundro linked to in the post).