When I’ve read Stephen M. Omohundro’s article titled “The Basic AI Drives,” several insights suddenly clicked together in my head. He discusses drives or instincts that we can expect to find in all self-aware goal-optimizing systems, including potential AIs we might create, and more importantly human beings.
The most interesting drive that Omohundro proposes is that self-aware goal-optimizing systems will try to prevent counterfeit utility. Each such system necessarily in order to act rationally must have a method to measure utility. In case of human beings, it is a complex neurochemical mechanism that evolved biologically.
This neurochemical mechanism isn’t foolproof. A drug addict might experience several neurochemical aspects of doing really great, but we still have a strong intuition that smoking crack is not the right way to have a good life.
Each goal-optimizing system faces this problem. Utility measurement mechanism can be manipulated by external factors. Rationality fails if your utility measurements are corrupted. This fact about general mind architecture is exploited by many different professions — from drug lords to marketers to pornographers to politicians. It is irrelevant if people using this manipulation to their benefit openly admit to that or rationalize it somehow.
SEO is an example of the same manipulation in case of an AI system as we know it now. Google wants to provide the users with high quality search results, and uses measurement of the pagerank algorithm to judge the quality. This proxy measurement is subject to all kinds of manipulation by the SEO industry, while the Google Search Quality team constantly strives to improve the secret algorithms to detect and stop manipulation.
I believe that the same arms-race is currently happening in human culture. Humans might be good at detecting manipulation in ancestral situations of tribal life on African savanna. Recent developments of (a) communication technology, and (b) psychology and cognitive science suddenly introduced many new ways to induce humans to act irrationally. In order to defend against this tendency we must improve our rationality and get a better explicit understanding of our goals and values.
