% This preamble comes from /u/jmc/let/latex-96.tex which is copied into % latex sources by the macro latex-include-96 which is itself called % by the macro mklatex-96 executed in /u/jmc/let/files. It needs the % tab between the file name and the name of the article. % Citations for use with latex2html are in /u/jmc/1/citations.tex % Main citation list is /u/jmc/1/biblio.bib \documentclass[12pt]{article} %\documentclass[a4paper,12pt]{article} % if a4 is wanted % In case we want to use latex2html. \usepackage{html} % for author date style \usepackage{/u/jmc/1/cslibib} % Fri Oct 15 1999 Tom gave me % this version that avoids a bug. \usepackage{varioref} \usepackage{amssymb} %\usepackage{color} % usage: \textcolor{cyan}{ Reasoning about action} % available colors: black, white red, green, blue, yellow, cyan and % magenta % for 3400 display, compile on Powerbook 3400 %\setlength{\textwidth}{8in} %\setlength{\textheight}{5in} % configs on Oztex, select A4 landscape % \usepackage amstex % in case we want full mathematics % \usepackage{chess} % /usr/local/lib/texmf/doc/chess/ boards-2e.ltx % /usr/local/lib/texmf/tex/inputs/chess.sty % \usepackage{a4wide} % auxiliary to chess %\usepackage[dvips]{epsfig} % UNCOMMENT IF FIGURES ARE WANTED % usage: \epsfig{file=blocks1.eps} %***************************** % BEGIN stuff for times and dates \newcount\hours \newcount\minutes \newcount\temp \newtoks\ampm % set the real time of day \def\setdaytime{% \temp\time \divide\temp 60 \hours\temp \multiply\temp -60 \minutes\time \advance\minutes \temp \ifnum\hours =12 \ampm={p.m.} \else \ifnum\hours >12 \advance\hours -12 \ampm={p.m.} \else \ampm={a.m.} \fi \fi } \setdaytime \def\theMonth{\relax \ifcase\month\or Jan\or Feb\or Mar\or Apr\or May\or Jun\or Jul\or Aug\or Sep\or Oct\or Nov\or Dec\fi} \def\theTime{\the\hours:\ifnum \minutes < 10 0\fi \the\minutes\ \the\ampm} \def\jmcdate{ \number\year\space\theMonth\space\number\day} % END stuff for times and dates % \begin{document} % % for author date style \bibliographystyle{/u/jmc/1/cslibib} %\bibliographystyle{alpha} % \title{NOTES ON SELF-AWARENESS} \author{ {\Large\bf John McCarthy} \\ Computer Science Department \\ Stanford University \\ Stanford, CA 94305 \\ {\tt jmc@cs.stanford.edu} \\ {\tt http://www-formal.stanford.edu/jmc/}} \date{\jmcdate ,\ \theTime} \maketitle \begin{abstract} These notes discuss self-awareness in humans and machines. The goal is to determine useful forms of machine self-awareness and also those that are on the road to human-level AI. This is a draft which is to be improved, and suggestions are solicited. There are a few formulas in this version. The final version will have more. \end{abstract} %\tableofcontents %UNCOMMENT IF TABLE OF CONTENTS WANTED % begin here % /u/jmc/f03/self2 Notes on self-awareness \section{Introduction} \label{sec:intro} %Methodological considerations Developing self-aware computer systems will be an interesting and challenging project. It seems to me that the human forms of self-awareness play an important role in humans achieving our goals and will also be important for advanced computer systems. However, I think they will be difficult to implement in present computer formalisms, even in the most advanced logical AI formalisms. The useful forms of computer agent self-awareness will not be identical with the human forms. Indeed many aspects of human self-awareness are bugs and will not be wanted in computer systems. \cite{McC96} includes a discussion of this and other aspects of robot consciousness. Nevertheless, for now, human self-awareness, as observed introspectively is the best clue. Introspection may be more useful than the literature of experimental psychology, because it gives more ideas, and the ideas can be checked for usefulnes by considering programs that implement them. Moreover, at least in the beginning of the study of self-awareness, we should be ontologically promiscuous, e.g. we should not identify intentions with goals Significant differences may become apparent, and we can always squeeze later. \footnote{Some philosophers who emphasize qualia may be inclined to regard self-awareness as a similar phenomenon---in which a person has an undifferentiated awareness of self, like the qualia oriented notion of the pure sensation of red as distinct from blue. This is not at all what is needed for AI. Rather we study the specific aspects of self and its activity which it is useful to be aware.} Some human forms of self-awareness are conveniently and often linguistically expressed and others are not. For example, one rarely has occasion to announce the state of tension in ones muscles. However, something about it can be expressed if useful. How the sensation of blue differs from the sensation of red apparently cannot be verbally expressed. At least the qualia-oriented philosophers have put a lot of effort into saying so. What an artificial agent can usefully express in formulas need not correspond to what humans ordinarily say, or even can say. In general, computer programs can usefully be given much greater powers of self-awareness than humans have, because every component of the state of the machine or its memory can be made accessible to be read by the program. A straightforward way of logically formalizing self-awareness is in terms of a mental situation calculus with certain observable fluents. The agent is aware of the observable mental fluents and their values. A formalism with mental situations and fluents will also have mental events including actions, and their occurrence will affect the values of the observable fluents. I advocate the form of situation calculus proposed in \cite{McC02}. Self-awareness is continuous with other forms of awareness. Awareness of being hot and awareness of the room being hot are similar. A simple fluent of which a person is aware is hunger. We can write $Hungry(s)$ about a mental situation $s$, but we write $Holds(Hungry,s)$, then $Hungry$ can be the value of bound variables.. Anohter advantage is that now $Hungry$ is an object, and the agent can compare $Hungry$ with $Thirsty$ or $Bored$. I'm not sure where the object $Hunger$ comes in, but I'm pretty sure our formalism should have it and not just $Hungry$. We can even use $Holds(Applies(Hunger,I),s)$ but tolerate abbreviations, especially in contexts. % \footnote{The English grammatical form "I am hungry" has no inevitability to it. French has "\emph{J'ai faim}", literally "I have hunger", and German has "\emph{Es hungert mich}", literally "It hungers me". In French the noun "faim" meaning "hunger" is used, whereas in English an adjective ``hungry'' is used. In a logical language we can have both; I don't see a use for a logical version of the German form.} % \footnote{$Holds(Hungry(I),s)$ might be written if our agent needs to compare its hunger with that of other agents. However, if we use formalized contexts (\cite{McCcontext}) we can get by with $Holds(Hungry,s)$ in an inner context in which the sentences are about the agent's self. We won't use formalized contexts in these notes, but an informal notion of context can avoid some worries. For example, some discussions are carried out entirely in contexts in which the fact that John McCarthy is a professor at Stanford is permanent. However, when needed, this context can be transcended. Likewise there are useful time-limited contexts in which George W. Bush is permanently President of the United States. } % \footnote{In spite of the fact that English has an enormous vocabulary, the same word is used with diverse meanings. I don't speak of simple homonyms like "lock on a door" and "lock of hair". These can be ruthlessly eliminated from our computer language, e.g. by having words lock1 and lock2. A more interesting example is that one can speak of knowing a person, knowing a fact, and knowing a telephone number. German uses \emph{kennen} for the first and \emph{wissen} for the second; I don't know about the third. In my \cite{McC79a}, "First order theories of individual concepts and propositions", I use different words for the different concepts. I suspect that it will be useful to tolerate using the same term in related senses, e.g. using the same word for the bank as an institution and as a building, because too many related meanings will arise.} Our goal in this research is an \emph{epistemologically adequate} formalism in the sense of \cite{McCHayes69} for representing what a person or robot can actually learn about the world. In this case, the goal is to represent facts of self-awareness of a system, both as an internal language for the system and as an external language for the use of people or other systems. Basic entities, e.g. automaton states as discussed in \cite{McCHayes69} or neural states may be good for devising theories at present, but we cannot express what a person or robot actually knows about its situation in such terms. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Of what are we aware, and of what should computers be aware?} \label{sec:whataware} Humans are aware of many different aspects of their minds. Here are samples of kinds of self-awareness---alas not a classification. \begin{enumerate} \item Permanent aspects of self and their relations to each other and aspects of other persons. Thus I am human like other humans. [I am a small child, and I am "supposed to do" what the others are doing. This is innate or learned very early on the basis of an innate predisposition to learn it.\footnote{Autistic children may be deficient in this respect.} What might we want an artificial agent to know about the fact that it is one among many agents? It seems to me that the forms in which self-awareness develops in babies and children are likely to be particularly suggestive for what we will want to build into computers. \item I exist in time. This is distinct from having facts about particular times, but what use can we make of the agent knowing this fact---or even how is the fact to be represented? \item Capabilities: I don't know Spanish but can speak Russian and French a little. I can fly a light plane but don't have an instrument rating. It helps to organize as much as possible of a system's knowledge as knowledge of permanent entities. \item Habits: I often think of you. I often have breakfast at Caffe Verona.\footnote{Alas, no more.} %%%%%%%%%%%%%%%%%%%%%%%%%%%%% \item Ongoing processes I am driving to the supermarket. One is aware of the past of the process and also of its future. Awareness of its present depends on some concept of the ``extended now''. %%%%%%%%% Temporary phenomena \item Wants, intentions, hopes, and goals: %Some AI researchers are philosophical minimalists and hastily strive to %reduce the number of AI concepts. In particular, some want to regard %an intention as just a kind of goal. I think one should not be too %hasty to eliminate distinctions made in ordinary language. Even when %an immediate use for a distinction is not obvious, one may turn up. Wants can apply to both states and actions. I want to be healthy, wealthy and wise. I want to marry Yumyum and hope to persuade her guardian Koko to let me. \item I intend to walk home from my office, but if someone offers me a ride, I'll take it. I intend to give X a ride home, but if X doesn't want it, I won't. %Hans Ma\ldots\ doesn't intend to pay the \$50 he owes %me, but he probably will if I pester him. \item If I intend to drive via Pensacola, Florida, I'll think about visiting Pat Hayes. I suppose you can still haggle, and regard intentions as goals, but if you do you are likely to end up distinguishing a particular kind of goal corresponding to what we unsophisticated call an intention. \item Attitudes Attitudes towards the future:\\ hopes, fears, goals, expectations, anti-expectations, intentions action: predict, want to know, promises and commitments. Attitudes toward the past:\\ regrets, satisfactions, counterfactuals\\ I'm aware that I regret having offended him. I believe that if I hadn't done so, he would have supported my position in this matter. It looks like a belief is a kind of weak awareness. Attitudes to the present:\\ satisfaction, I see a dog. I don't see the dog. I wonder where the dog has gone. There are also attitudes toward timeless entities, e.g. towards kinds of people and things. I like strawberry ice cream but not chocolate chip. \item Hopes: A person can observe his hopes. I hope it won't rain tomorrow. Yesterday I hoped it wouldn't rain today. I think it will be advantageous to equip robots with mental qualities we can appropriately call hopes. \item Fears: I fear it will rain tomorrow. Is a fear just the opposite of a hope? Certainly not in humans, because the hormonal physiology is different, but maybe we could design it that way in robots. Maybe, but I wouldn't jump to the conclusion that we should. \item Disappointment: Juliet is disappointed that there is no poison left in Romeo's vial when she tries to kill herself with it. Why are hopes and fears definite mental objects? The human brain is always changing but certain structures can persist. Specific hopes and fears can last for years and can be observed. It is likely to be worthwhile to build such structures into robot minds, because they last much longer than specific neural states. \item An agent may observe that it has incompatible wants. \end{enumerate} %/u/jmc/e03/self %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Mental actions} \label{sec:actions} The companion of observation is action. A theory of self-awareness, i.e. of mental observation, is complemented by a theory of mental action. \cite{McCarthy82} discusses heuristics for coloring maps with four colors. A form of self-awareness is involved. In coloring a map of the United States, the goal of coloring California can be postponed to the very end, because California has only three neighbors and therefore no matter how the rest of the map is colored, there will always be a color for California. Once California is removed from the map, Arizona has only three neighbors. The \emph{postponement} process can be continued as long as possible. In the case of the US, all states get postponed and then can be colored without backtracking. In general it is often possible to observe that in planning a task, certain subtasks can be postponed to the end. Thus postponement of goals is a mental action that is sometimes useful. A human-level agent, and even an agent of considerably lower level, has policies. These are creatable, observable both in their structure and in their actions, and changeable. Useful actions: decide on an intention or a goal. Drop an intention. Clearly there are many more mental actions and we need axioms describing their effects. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Machine self-awareness} \label{sec:machine} Self-awareness is not likely to be a feasible or useful attribute of a program that just computes an answer. It is more likely to be feasible and useful for programs that maintain a persistent activity. What kind of program would be analogous to Pat deciding while on the way to his job that he needed cigarettes? See formula (\ref{eq:b}) below. Here are some possibiliities. It's not clear what event in a computer program might correspond to Pat's sudden need for cigarettes. The following examples don't quite make it. A specialized theorem-proving program $T1$ is being operated as a subprogram by a reasoning program $T$. Assume that the writer of $T$ has only limited knowledge of the details of $T1$, because $T1$ is someone else's program. $T$ might usefully monitor the operation of $T1$ and look at the collection of intermediate results $T1$ has produced. If too many of these are redundant, $T$ may restart $T1$ with different initial conditions and with a restriction that prevents sentences of a a certain form from being generated. An operating system keeps track of the resources a user is using, and check for attempts to use forbidden resources. In particular it might check for generation of password candidates. In its present form this example may be bad, because we can imagine the checking be done by the programs that implement supervisor calls rather than by an inspector operating with clock interrupts. While the programs called by clock interrupts exhibit a simple form of self-awareness, the applications I know about are all trivial. %Cheriton, Hnnessey. The main technical requirement for self-awareness of ongoing processes in computers is an interrupt system, especially a system that allows clock interrupts. Hardware supporting interrupts is standard on all computers today but didn't become standard until the middle 1960s.\footnote{Historical note: The earliest interrupt system I know about was the ``Real time package'' that IBM announced as a special order instigated by Boeing in the late 1950s. Boeing wanted it in order to control a wind tunnel using an IBM 704 computer. At M.I.T. we also needed an interrupt system in order to experiment with time-sharing on the IBM 704. We designed a simple one, but when we heard about the much better ``Real time package'' we started begging for it. Begging from IBM took a while, but they gave it to us. The earliest standard interrupt system was on the D.E.C. PDP-1 and was designed by Ed Fredkin, then at Bolt, Beranek and Newman, who persuaded D.E.C. to include it in the machine design. Again the purpose was time-sharing.} The human brain is not a computer that executes instructions in sequence and therefore doesn't need an interrupt system that can make it take an instruction out of sequence. However, interruption of some kind is clearly a feature of the brain. With humans the boundary between self and non-self is pretty clear. It's the skin. With computer based systems, the boundary may be somewhat arbitrary, and this makes distinguishing self-awareness from other awareness arbitrary. I suppose satisfactory distinctions will become clearer with experience. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Interrupts, programming languages, and self-awarenes} \label{sec:interrupt} Consider a persistent program driving a car that is subject to observation and modification by a higher level program. We mentioned the human example of noticing that cigarettes are wanted and available. The higher level program must observe and modify the state of the driving program. It seems that a clock interrupt activating the higher level program is all we need from the hardware. We need considerably more from the software and from the programming languages. A cursory glance at the interrupt handling facilities of C, Ada, Java, and Forth suggests that they are suitable for handling interrupts of high level processes by low level processes that buffer the transfer of information. Lisp and Smalltalk can handle interrupts, but have no standard facilities. My opinion, subject to correction, is that self-awareness of the kinds proposed in this note will require higher level programming language facilities whose nature may be presently unknown. They will be implemented by the present machine language facilities. However, one feature of Lisp, that programs are data, and their abstract syntax is directly represented, is likely to be necessary for programs that examine themselves and their subprograms. This feature of Lisp hasn't been much used except in macros and has been abandoned in more recent programming languages---in my opinion mistakenly. Indeed human-level AI \cite{McC04} requires self-awareness, and human intelligence is limited by our limited self-awareness. Smarter people are more self-aware than others. %\subsection{Existing self-awareness in computers} %\label{sec:exist} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Formulas} \label{sec:formulas} Formalized contexts as discussed in \cite{McCcontext} will be helpful in expressing self-awareness facts compactly. Pat is aware of his intention to eat dinner at home. \begin{equation} \label{eq:f1} \begin{array}[l]{l} c(Awareness(Pat)): Intend(I, Mod(At(Home),Eat(Dinner))) \\ \mbox{or} \\ Ist(Awareness(Pat),Intend(I, Mod(At(Home),Eat(Dinner)))) \\ \end{array} \end{equation} Here $Awareness(Pat)$ is a certain context. $Eat(Dinner)$ denotes the general act of eating dinner, logically different from eating $Steak7642$.\\ $Mod(At(Home),Eat(Dinner))$ is what you get when you apply the modifier ``at home'' to the act of eating dinner. I don't have a full writeup of this proposal for handling modifiers like adjectives, adverbs, and modifier clauses. $Intend(I,X)$ says that I intend $X$. The use of $I$ is appropriate within the context of a person's (here Pat's) awareness. We should extend this to say that Pat will eat dinner at home unless his intention changes. This can be expressed by formulas like \begin{equation} \label{eq:patintent} \lnot Ab17(Pat,x,s) \land Intends(Pat,Does(Pat,x),s) \rightarrow (\exists s' > s)Occurs(Does(Pat,x),s). \end{equation} % in the notation of \cite{McC02}. Here's an example of awareness leading to action. Pat is driving to his job. Presumably he could get there without much awareness of that fact, since the drive is habitual. However, he becomes aware that he needs cigarettes and that he can stop at Mac's Smoke Shop and get some. Two aspects of his awareness, the driving and the need for cigarettes are involved. That Pat is driving to his job can be expressed with varying degrees of elaboration. Here are some I have considered. \begin{equation} \label{eq:a} \begin{array}[l]{l} Driving(Pat,Job,s) \\ \mbox{} \\ Doing(Pat,Drive(Job),s) \\ \mbox{} \\ Holds(Doing(Pat,Mod(Destination(Job),Drive)),s) \\ \mbox{} \\ Holds(Mod(Ing,Mod(Destination(Job,Action(Drive,Pat))),s) \\ \mbox{}. \\ \end{array} \end{equation} % The last two use a notion like that of an adjective modifying a noun. Here's a simple sentence giving a consequence of Pat's awareness. It uses $Aware$ as a modal operator. This may require repair or it may be ok in a suitably defined context. \begin{equation} \label{eq:b} \begin{array}[l]{l} Aware(Pat,Driving(Job,s),s) \land Aware(Pat,Needs(Cigarettes),s) \\ \land Aware(Pat,About\mbox{-}to\mbox{-}pass(CigaretteStore,s),s) \\ \rightarrow Occurs(StopAt(CigaretteStore),s). \end{array} \end{equation} The machine knows that if its battery is low, it will be aware of the fact. \begin{equation} \label{eq:c} \begin{array}[l]{l} Knows(Machine, (\forall s')(LowBattery(s') \rightarrow Aware(LowBattery(s'))),s) \\ \end{array} \end{equation} The machine knows, perhaps because a sensor is broken, that it will not necessarily be aware of a low battery. \begin{equation} \label{eq:d} \begin{array}[l]{l} Knows(Machine, \lnot (\forall s')(LowBattery(s') \rightarrow Aware(LowBattery(s'))),s) \end{array} \end{equation} The positive sentence ``I am aware that I am aware \ldots '' doesn't seem to have much use by itself, but sentences of the form ``If X happens, I will be aware of Y'' should be quite useful. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Miscellaneous} \label{sec:misc} % B1649 .R93 P65 1997 Here are some examples of awareness and considerations concerning awareness that don't yet fit the framework of the previous sections. I am slow to solve the problem because I waste time thinking about ducks. I'd like Mark Stickel's SNARK to observe, ``I'm slow to solve the problem, because I keep proving equivalent lemmas over and over''. I was aware that I was letting my dislike of the man influence me to reject his proposal unfairly. Here are some general considerations about what fluents should be used in making self-aware systems. 1. Observability. One can observe ones intentions. One cannot observe the state of ones brain at a more basic level. This is an issue of epistemological adequacy as introduced in \cite{McCHayes69}. 2. Duration. Intentions can last for many years, e.g. "I intend to retire to Florida when I'm 65". "I intend to have dinner at home unless something better turns up." 3. Forming a system with other fluents. Thus beliefs lead to other beliefs and eventually actions. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %\section{Remarks and questions} %\label{sec:remarks} Is there a technical difference between observations that constitute self-observations and those that don't? Do we need a special mechanism for \emph{self-observation}? At present I don't think so. If $p$ is a precondition for some action, it may not be in consciousnes, but if the action becomes considered, whether $p$ is true will then come into consciousnes, i.e. short term memory. We can say that the agent is \emph{subaware} of $p$. What programming languages provide for interrupts? \bibliography{/u/jmc/1/biblio} % Its full name is /u/jmc/1/biblio.bib %\bibliography{biblio} % Tom worked some magic. % for use with Alpha on Powerbook 3400 \vfill \begin{latexonly} {\tiny\rm\noindent /@steam.stanford.edu:/u/jmc/f03/selfaware.tex: begun Sun Oct 26 12:45:55 2003, latexed \today\ at \theTime} \end{latexonly} \end{document} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Related files: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Notes: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Leftover material: