Awareness and Understanding

Penrose discusses awareness and understanding briefly and concludes (with no references to the AI literature) that AI researchers have no idea of how to make computer programs with these qualities.

I substantially agree with his characterizations of awareness and understanding and agree that definitions are not appropriate at the present level of understanding of these phenomena. We disagree about whether computers can have awareness and understanding.

Here's how it can be done within the framework of pure logical AI.

Pure logical AI represents all the program's knowledge and belief by sentences in a language of mathematical logic. Purity is inefficient but makes the discussion brief. ([McCarthy, 1989]) is a general discussion of logical AI and has additional references.

We distinguish a part of the robot's memory, which we will call its consciousness. Sentences have to come into consciousness before they are used in reasoning.

Reasoning involves logical deduction and also some nonmonotonic reasoning processes. The results of the reasoning re-enter consciousness. Some old sentences in consciousness get crowded out into the main memory.

Deliberate action in a pure logical robot is a consequence of the robot inferring that it should do the action. The actions include external motor and sensory actions (observations) but also mental actions such as retrieval of sentences from the general memory into consciousness.

Awareness of the program's environment is accomplished by the automatic appearance of certain class of sentences about the program's environment in the program's consciousness. These sentences often appear through actions of observation but should often result from built-in observations, e.g. noticing who comes into the room.

Besides awareness of the environment, there is also self-awareness. Self-awareness is caused by events and actions of self-observation including observations of consciousness and of the memory as a whole. The sentences expressing self-awareness also go into consciousness.

The key question about awareness in the design of logical robots concerns what kinds of sentences can and should appear in consciousness--either automatically or as the result of mental actions. Here are some examples of required mental actions.

• Observing its physical body, recognizing the positions of its effectors, noticing the relation of its body to the environment and noticing the values of important internal variables, e.g. the state of its power supply and of its communication channels.
• Observing whether it knows the telephone number of a certain person. Observing that it does know the number or that it can get it by some procedure is likely to be straightforward logical deduction. Inferring that it doesn't know the number and can't get it by reasoning requires getting around Gödel's theorem, because inferring that any sentence does not follow carries with it an implication that the theory is consistent, and Gödel tells us that this cannot be done entirely within a theory.

  Our approach uses Gödel's [Gödel, 1940] notion of relative consistency which allows inferring that if the theory is consistent, then a certain sentence doesn't follow. In cases of main AI interest, this can be done without the complications that Gödel had to introduce in order to prove the consistency of the continuum hypothesis. See ([McCarthy, 1995]) for a start on details.

• Keeping a journal of physical and intellectual events so it can refer to its past beliefs, observations and actions.
• Observing its goal structure and forming sentences about it.
• Observing its own intentions. The robot may intend to perform a certain action. This would let it later infer that certain possibilities are irrelevant in view of its intentions.
• Observing how it arrived at its current beliefs. Most of the important beliefs of the system will have been obtained by nonmonotonic reasoning, and are therefore uncertain. It will need to maintain a critical view of these beliefs, i.e. believe meta-sentences about them that will aid in revising them when new information warrants doing so.
• Not only pedigrees of beliefs but other auxiliary information should either be represented as sentences or be observable in such a way as to give rise to sentences. Thus a system should be able to answer the question: ``Why don't I believe p?''.
• Regarding its entire mental state up to the present as an object, i.e. a context. [McCarthy, 1993] discusses contexts as formal objects. The ability to transcend one's present context and think about it as an object is an important form of introspection, especially when we compare human and machine intelligence.
• Knowing what goals it can currently achieve and what its choices are for action. Understanding and reasoning about one's own choices constitutes free will.

It seems to me that the notions of awareness and understanding outlined above agree with Penrose's characterizations on p._ 37. However, his ideas about free will strike me as quite confused and not repairable. [McCarthy and Hayes, 1969] discusses free will in deterministic systems, e.g. interacting finite automata.