Vol. 2, No. 2 (2008)

 

http:/www.eludamos.org

 

A Review of Agent Emotion Architectures

Stuart Slater, Robert Moreton, Kevan Buckley, Andrew Bridges

Eludamos. Journal for Computer Game Culture. 2008; 2; 2; 203-214

 

A Review of Agent Emotion Architectures

Stuart Slater, Robert Moreton, Kevan Buckley, and Andrew Bridges[1]

 

1       Introduction

With successive generations of computer games, development resources in all aspects of the game creation process are increasing to support the growing trend of more photo realistic games. These titles often feature evolving control systems to make them more interactive, including motion sensing systems and force feedback devices. The sound systems in these games continue to improve and currently span multi speaker immersive sound systems to provide a feeling of greater realism for the gamer. With the many advances in games it is sometimes surprising to see that agent A.I. (artificial intelligence) is being developed at a  much slower pace than most other areas, especially in the area of emotionally responsive agents. In previous work (Slater 2006), a lack of developer knowledge (coupled with a reliance on multiplayer gaming) were partly attributed to this slow growth and thus this paper is intended to highlight some of the work in modeling and specifying agent emotions to provide a foundation for game developers who may look to the A.I. research community for the next generation of games.

An investigation into the computational modeling of emotion leads the developer to many publications of independent work conducted over at least the last two decades. Some of this research is fragmented, leading to little further work or potential for subsequent development, but can still add value in the pursuit of computational models of emotions for developers looking to model specific aspects of emotion. Some of this research has shown tremendous potential, such as the successful commercial use of emotion recognition in voices in call center queuing systems to detect a range of emotions, including anger and happiness, facilitated through a two-layer back propagated neural network (Petrushin 2000).

Though examples of contained emotion systems have been proposed and developed, such as the call center queuing system, they are specific to particular application areas, and in the aforementioned system, the use of artificial neural networks (ANN) means that the system requires training prior to use in order to perform optimally, and thus may not be suitable for the adaptation of agent behaviour in the real time systems used in games and simulations. For these systems, other research has been conducted such as using fuzzy logic to represent fear, where the value of a variable increases if an enemy is close, and at a threshold value will cause the agent to move away (Wen et al 2000).

Other research projects are based on more formal theories of emotion such as The Artificial Emotion Engine (Wilson 1999), based on the work of Eysenck known for his work on formalising personality models (Eysenck 1965)  and Gray known predominantly for his study of fear (Gray 1971).

Complementing the research in agent emotional behaviour is a  substantial amount of work in the visual appearance of emotions on agents' faces, such as the Neural Emotion Eliciting System (NEmESys) (Eckschalger et al 2005) which has been used to model the work by Paul Ekman on six emotions that are visible facially (Ekman 2004). This project again uses neural networks that are pre-trained using the 5-factor model of personality (McCrae et al 1996).

Projects that combine behavioral changes with corresponding visual changes to agents have also emerged, including the IST Supporting Affective Interactions for Real-Time Applications (SAFIRA)[2] project. This research, conducted between 1999 and 2001, built upon the foundation of the OCC (Ortony et al 1988) model of emotions (for behaviour) combined with the work of Ekman for visualising facial emotions. XML was used to hold the emotion specifications and both Java and C could be used for coding. According to the research publications, the range of emotions that could be modelled included anger, tranquility, spirituality, insecurity, warmth, happiness and sadness. This reveals that the terminology used to describe the emotions is actually a blend of personality types and emotions, and the merging of these together under the term "emotions" could potentially cause some confusion for the developer looking to implement emotions and personalities in agents. The difference between emotion and personality has been covered in many research publications and it is generally accepted that emotions are what the individual experiences and personality types are associated with the frequency of emotions in an individual (Slater et al 2007).

Researching computational models of emotions provides a broad variation of research publications, but consistently scanning many of these publications and results from search engines reveals consistent recurring reference to a small number of larger research projects including:

The Cognition and Affect Project (CogAff[3]) proposed by Birmingham University with Aaron Sloman being the common research name linked to the project.

The Tok Project[4] developed at Carnegie Mellon University with Joseph Bates, Bryan Loyall and W. Scott Reilly being the common research names linked to the project (specifically Em the aspect of the project dealing with emotion simulation) .

Émile[5] proposed by the University of Southern California with Jonathan Gratch and Stacy Marsella being the common research names linked to the project.

The following sections provide an overview of each of the three research projects with their key features highlighted, followed by a summary of the architectures in relation to suitability for games and simulations. This discussion also includes any discussion of formal psychology theory that underpins the work.

 

2.1     Cognition and Affect Project (CogAff Architecture)

Architecture Overview

The CogAff architecture developed at Birmingham University is a three-layer architecture intended to model behaviour in agents for simulation and games. The three layers are intended to break down the processing into more manageable and self contained blocks that have increased processing the higher the process is dealt with in the architecture, from low in the reactive layer to high processing overheads in the meta-management layer (Sloman et al 1995).

 

The Emotion Aspect

According to later research (Sloman 1999) the architecture can also be used to simulate emotions as shown below:

 

Figure 1

 

The foundation research linking these emotion categories to the stages of the architecture was difficult to ascertain, except for casual implications. Therefore evidence of a firm foundation in formal psychology can only be assumed, in which case it is possible to support the usage of primary, secondary and tertiary emotions for these layers as proposed, though the actual names of the emotions shown in the diagram are under some debate (Slater et al 2007).

Looking at the architecture, and particularly the reactive layer, it could be possible to model a fast responsive basic emotion such as fear, as this layer is intended to be able to interrupt and override other processing if given suitable stimulus, i.e. layers 1 and 2. Though the reactive layer can override the other two layers, in general the three layers are concurrently active.

 

AI Techniques suggested for Implementation

Investigations of the architecture show that it uses typical AI techniques such as finite state machines (FSM's), neural networks (which would seemingly be the reason the architecture is not suitable for real time applications (Sloman et al 1995)) and condition action rules within the layers. Subsequent behaviour is suggested as being schema based to reduce processing (thus more) of a look up facility.

 

The Architecture in Software

Though published research on how exactly the architecture should be implemented, evaluated or its basis on formal psychology is limited, the developers of the architecture have released an implementation of the architecture through the SimAgent toolkit[6]. SimAgent utilizes the Pop-11 programming language to allow developers to create agents for simulated applications, though not in real time (Sloman et al 1995). In early research it was proposed that the architecture could model three distinctly different types of agents: a reactive agent, an affective agent and a deliberative agent based upon the layers of the model:

The Reactive Agents (R-Agents) exhibit simplistic behaviour by simply carrying out instructions such as, "if hungry go straight for food, whether it is safe to do so or not". Modelled using finite state machines implemented using rule based systems (Scheutz et al 2000).

The Affective Agents (A-Agents) differ slightly to the R-Agents as they do not blindly pursue goals; instead, A-Agents can alter their behaviour in pursuing their goals when needed, such as "if there is danger in the path, go a different way or wait".

The Deliberative Agents (D-Agents) - are the most complex agents, and they can plan behaviour to achieve goals such as considering a range of options and outcomes.

 

Conclusions

The usage of Pop-11, a lisp-style programming language, may ease the complications found in implementing agents-based AI for the researcher, but may not be ideal for game developers due to their heavy reliance and familiarity of C and the interpreted nature of Pop-11 potentially affecting performance (Sloman et al 1995). The lack of real time support in the architecture may cause issues in some games that rely on changing agent behaviour based on interaction with the gamer, but it is acknowledged that this could be built into many games beforehand to minimise issues with processing and pre-training. Research conducted as part of this paper could not find the architecture used alongside any game or simulation engine traditionally used in industry and thus it is assumed that developers would need to implement the architecture themselves from scratch, and rely on external advisors for the realism of the underpinning psychology.

 

2.2     Tok Project (featuring HAP & Em)

Architecture Overview

The Tok Project (Bates et al 1992) is an integrated AI architecture designed to work in a specific virtual world called the Oz environment (Reilly et al 1992; Reilly et al 1993). Tok combines a reactivity element called HAP (Loyall et al 1991), goal-directed behaviour; a module called Em (Reilly et al 1992)  that handles emotion (based on the OCC model) and some memory, but has been specifically designed for use in non-real time worlds (Loyall et al 1991b). Tok handles the behaviour aspect of the Oz world inhabitants, and its relationship to Em and HAP is shown in figure 2.

 

Figure 2

 

The Architecture in Software

The Edge of Intention was a project developed as part of the Oz Project (Bates 1994) to demonstrate the architecture. This project involved developing creatures called Woggles that included some goal-based emotions (based on the OCC model) such as anger (activated when goals were failed), and implemented through the Em module. These responses were mapped to personality traits in the Woggles such as when a specific Woggle became alarmed it became angry. HAP was used to assign goals and behaviours to the Woggles, though they could not plan, only react to events within the environment. This aspect of Em means that it is best suited to applications where agents are equipped with task-orientated emotions, i.e. "emotions that arise from the performance of a concrete task" (Gratch et al 2001).

Further software implementations of the architecture include Robbery World, Office Politics and the Playground that were all text-based applications. These applications were part of a research project (Reilly 1996) intended to address more believable agents through social and emotional aspects of agents, with a focus on interactive drama.

 

The Emotion Aspect

Though Em is integrated into the Tok architecture, it is still valuable to investigate the functionality in regards to dealing with emotion representation. As previously mentioned Em is based on a scaled down version of the OCC model of emotions (Reilly et al 1992), figure 3 shows the relevant emotional aspects supported.

 

Emotion	Cause
Joy	Goal success (*)
Distress	Goal failure (*)
Hope	Prospect of Goal success (*)
Fear	Prospect of Goal failure (*)
Pride	Action of self approved according to standards
Shame	Action of self disapproved according to standards
Admiration	Action of other approved according to standards
Reproach	Action of other disapproved according to standards
Love	Attention to liked object
Hate	Attention to disliked object
Gratification	Action of self causes joy and pride
Gratitude	Action of others causes joy and admiration
Remorse	Action of self causes distress and shame
Anger	Action of other causes distress and reproach
(*) Denotes difference from OCC Model

 

Figure 3 (Reilly et al 1992)

 

Where:

Actions are judged to be either pleasing or displeasing according to goals.

Goals are anything the agent wants to do.

Standards are applied to actions as a basis of morality.(pride or shame linked i.e. if agent does something morally wrong then emotion activated probable to be shame).

Attitudes objects (inc agents) can be liked (love) or disliked (hate).

 

The Em architecture has been used in subsequent research to develop architectures for artists to help select character emotions (Reilly 1996). In this research, emotion generators were written in the HAP language[7] to simplify implementation by non-programmers. A questionnaire was used to validate whether or not the project was a success in providing characters with emotions. One interesting aspect of the architecture is the concept of emotions decaying over time, which is consistent with some research in formal emotion psychology (Slater et al 2007). The emotional decay is managed by lowering the intensity of the current emotion each cycle that the agent senses-thinks and acts (Reilly et al 1992).

 

Conclusions

The Em architecture being integrated into Tok potentially limits its application usage outside of observing emotional behaviour in custom simulations, but aspects of the development such as emotional decay have real benefit for inclusion in standalone emotional systems. One issue that seemed to flow throughout the work concerns the vernacular usage of emotion-related terms such as if an agent does not complete a goal, they enter the emotion of distress, but this assumes that distress is an emotion and this is not clearly defined. Other ambiguous "emotions" are joy, hope, satisfaction, disappointment and relief.

A version of Hap was developed for real-time simulations, but research could only uncover small scale applications (Loyall et al 1991).

 

2.3     Émile

Architecture Overview

Émile builds upon both the previously mentioned Em architecture and the Affective Reasoner project (Elliot 1992) and combines fundamentally two aspects of emotion modelling: appraisal and coping. "Appraisal" in this instance is defined as how the environment or current situation has emotional significance to the agent, and "coping" is how an agent deals with this emotional significance by the modification of its actions and goals (Gratch et al 2003).

 

The Emotion Aspect

Émile is based on the emotional appraisal of plans and goals, providing a framework for modelling this appraisal in agents. A significant change from the Em architecture mentioned earlier is that Émile allows agents to observe the emotional states of other agents and alter behaviour accordingly (Gratch 2000). It is ideally suited for applications where agents must plan as well as react, thus allowing a greater scope of realism for agents in simulations.

The system is based upon the OCC model of emotions (a potential legacy from the foundation use of Em) which means that it in part, it features an agent's response to goal-based planning as well as being able to assess external events that may have goal implications.

 

The Architecture in Software

The architecture has been implemented in software by combining it with the IPD architecture (Marsella et al 2000) within the Steve agent behaviour framework (Gratch et al 2001). IPD provides the visual selection of emotions for the agents, and thus complements the behavioural aspects of Émile to allow the developers to create more convincing applications such as the Mission Rehearsal Exercise (MRE[8]) (Gratch et al 2001; Marsella et al 2002).

 

Conclusions

An aspect of Émile that seems beneficial is the ability to include coping mechanisms (Marsella et al 2002) for agents. This is based upon the assumption that human beings generally cope with emotions by doing something about the cause of the emotional trigger, rather than just thinking about the situation that caused the emotion, i.e. a mixture of problem-focused coping, i.e. changing a situation and emotion-focused coping, by changing the individual's view of the situation to change its significance, i.e. shifting responsibility. A second aspect of Émile is the inclusion of emotional intensity linked to how important goals are to agents. Later developments led to Émile becoming EMA (Emotion & Adaption) (Gratch et al 2004).

 

3       Summary

This paper began by highlighting a range of independent research publications that offer solutions to modelling aspects of emotions in software agents. This led to a review of three popular projects that regularly appear in searches and are often quoted in computational research publications related to emotion. Each project was introduced and as already discussed each approach has merit in what it accomplishes in the software implementation of its architecture. From the three projects, two were based on the OCC model of emotions, offering a firm foundation to researchers wishing to observe aspects of emotional behaviour related to goals.

Based on this usage of the OCC model, a refined search of emotion research reveals that it is a popular choice in other projects such as the research conducted by Christoph Bartneck (Bartneck 2002). This paper analyses the OCC model in relation to its suitability as a foundation for computational models of emotion. Bartneck carefully proposes that the OCC model is best approached as a five stage architecture:

Classification - event, action or object is related to which categories it could affect. Discusses need for a classification and storage system to actually do this classification.

Quantification - Intensity of emotion a history mechanism is discussed to allow for regularity to diminish or increase intensity.

Interaction - classification and quantification define an emotional value that affects the agents current emotional state i.e. how the categories interact when different emotions are present; Bartneck indicates that the OCC model lacks this.

Mapping - 22 categories in OCC model, expression mapping accordingly.

Expression - facial expressions.

Conclusions reached in the Bartneck paper are that the OCC model is partially suitable for simulating emotions, but has some drawbacks that need attention such as the need for a history function, emotion interaction function and personality designer and that the OCC model requires simplification for developers.

The OCC model of emotions is a popular choice of goal-based emotional reasoning, with many successful demonstrations of its suitability to many problems facing computational emotional modelers and researchers. These tangible projects have been developed and evaluated in the plethora of research, some of which has been covered in this review paper. One key aspect is apparent from the projects reviewed, and that is the emotional aspect in these implementations is firmly integrated into the agent decision making, and agent behaviour in general, thus making the agent emotion element almost inseparable from the whole agent aspect of the software. Therefore developers using a commercial off-the-shelf game engine for creating a game or simulation may have some challenging issues because most of these technologies already include behaviors for in-game agents, sensing systems and some include goal orientated planning. What developers ideally need is either a middleware emotion module that works with their chosen engine, or an emotion architecture that is both based in sound emotion psychology and geared towards software developers without a background in psychology.

 

4       Conclusions

Games such as World of Warcraft clearly show that some developers have begun to address the lack of emotionally responsive agents through on-screen text and facial animations. But this limited inclusion will need further work in order for games to continue evolving into more immersive experiences for the user, requiring clarity in emotional modeling techniques to meet the demands of a rapidly-growing industry that is more interested in playability than technical accuracy. To this end the research presented here highlights the salient research that is commonly found when investigating the area of computational models of emotion, and clearly shows there is still much confusion for software developers looking at including emotions into agents using their own cognitive architectures rather than emotions built into complex models.

 

Acknowledgements

We would like to thank the referees for their comments which helped improve this paper.

 

References

Bartneck, C. (2002) Integrating the OCC Model of Emotions in Embodied Characters. Workshop on Virtual Conversational Characters, 2002.

Bates, J. (1992) An Architecture for Action, Emotion, and Social Behaviour. 4^th European Workshop on Modelling and Autonomous Agents in a Multi-Agent World. S, Martino al Comino, Italy, July, 1992.

Bates, J. (1994) The Role of Emotion in Believable Agents. Communication of the ACM, Special Issue on Agents 1994.

Eckschlager, M., Nernhaupt, R., Tscheligi, M. (2005) NEmESys - Neural Emotion Eliciting System. VHI 2005, April 2-7, 2005. Portland, Oregon.

Elliott C. D. (1992) The Affective Reasoner: A Process Model of Emotions in a Multi-agent System. Ph.D Thesis, Northwestern University.

Ekman, P. (2004) Emotions Revealed. Phoenix Books, UK. 

Eysenck, H. J. (1965) Fact and Fiction in Psychology. Harmondsworth. Penguin.

Gratch, J. (2000) Émile: Marshalling Passions in Training and Education. Proceedings of the 4^th International Conference on Intelligent Agents, Barcelona, Spain.

Gratch, J., Marsella, S. (2001) Tears and Fears: Modeling emotions and emotional behaviours in synthetic agents. Proceedings of the 5^th International conference on Autonomous Agents, 2001, 278-285.

Gratch, S., Marsella, J. (2003) Fight the Way you Train: The role and limits of Emotions in training for combat. The Brown Journal of World Affairs, 10 (1).

Gratch J., Marsella S. (2004) A Domain-independent Framework for Modeling Emotion. Cognitive Systems Research, 5 (4):269-306.

Gray, J. (1971) The Psychology of Fear and Stress. World University Library.

Loyall, A.B., Bates, J. (1991) Hap: A Reactive, Adaptive Architecture for Agents. Technical report CMU-CS-91-147, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, June 1991.

Loyall, A.B. Bates, J. (1991) Real-Time Control of Broad Agents. Proceedings of AAAI Spring Symposium on Integrated Intelligent Architectures. Stanford, CA, 1991.

Marsella, S. Johnson, W.L. & La Bore, C. (2000) Interactive Pedagogical Drama. Proceedings of the 4^th International Conference on Autonomous Agents.

Marsella, S., Gratch, J. (2002) A Step toward Irrationality: Using emotion to change belief. First International Joint Conference on Autonomous Agents and Multi-agent Systems, Bologna, Italy.

McCrae, R., & Costa, P. T. Jr. (1996) Toward a new generation of personality theories: Theoretical contexts for the five-factor model. In The five-factor model of personality: Theoretical perspectives, J. S. Wiggins (ed.), pp. 51-87. New York: Guilford.

Ortony, A. Clore, G. Collins, A. (1988) The Cognitive Structure of Emotions. Cambridge University Press.

Petrushin, V. (2000) Emotion Recognition Agents in Real World. AAAI Fall Symposium on Socially Intelligent Agents: Human in the Loop.

Reilly, W.S., Bates, J. (1992) Building Emotional Agents. Technical Report CMU-CS-92-143, School of Computer Science, Carnegie.

Reilly, W.S., Bates, J. (1993) Emotion as part of a Broad Agent Architecture. Available at: <http://www.cs.cmu.edu/afs/cs.cmu.edu/user/wsr/Web/research/ waume93.html> [Accessed 26th March 2008].

Reilly, W. S.  (1996) Believable Social and Emotional Agents.  PhD Thesis. School of Computer Science, Carnegie Melon University. Pittsburgh.

Scheutz M, Sloman, A, Logan, B. (2000) Emotional States and Realistic Agent Behaviour. Proceedings of Game-On 2000, 1st International Conference on Intelligent Games & Simulation, London.

Slater,S. (2006) Body Mind and Emotion, an Overview of Agent Implementation in Mainstream Computer Games. ICAPS, 2006. English Lake District.

Slater, S. Bechkoum, K. Buckley, K. (2007) A foundation of Emotion Research for Games & Simulation. AISB, 2007.

Sloman, A. (1999) Why Can't a Goldfish Long For Its Mother? Architectural prerequisites for various types of emotions. British HCI Group One-Day Meeting. University College London. Affective Computing: The Role of Emotion in HCI. 10th April 1999.

Sloman, A. Poli, R. (1995) SIM_AGENT: A Toolkit for exploring agent designs. ATAL-95, Workshop on Agent Theories, Architectures, and Languages, IJCAI-95, Montreal, August 1995.

Wen, Z., Mehdi, Q., Gough, N. and Suliman, H. (2000) Creating Animated Behavioural Game Characters Based on Environmental Effects. Proceedings of Game-On 2000, 1st International Conference on Intelligent Games & Simulation, London.

Wilson, I. (1999) The Artificial Emotion Engine™, Driving Emotional Behaviour. AAAI Technical Report SS-00-02. Available at: <http://www.neon.ai/docs/SS00-02-015.pdf>.

 

Notes