From Emotion AI to Cognitive AI

2.1. Emotion-Cognition Interactions

Survey/review study

From Emotion AI to Cognitive AI

Guoying Zhao ^*, Yante Li , and Qianru Xu

University of Oulu, Pentti Kaiteran Katu 1, Linnanmaa 90570, Finland

^* Correspondence: guoying.zhao@oulu.fi

Received: 22 September 2022

Accepted: 28 November 2022

Published: 22 December 2022

Abstract: Cognitive computing is recognized as the next era of computing. In order to make hardware and software systems more human-like, emotion artificial intelligence (AI) and cognitive AI which simulate human intelligence are the core of real AI. The current boom of sentiment analysis and affective computing in computer science gives rise to the rapid development of emotion AI. However, the research of cognitive AI has just started in the past few years. In this visionary paper, we briefly review the current development in emotion AI, introduce the concept of cognitive AI, and propose the envisioned future of cognitive AI, which intends to let computers think, reason, and make decisions in similar ways that humans do. The important aspect of cognitive AI in terms of engagement, regulation, decision making, and discovery are further discussed. Finally, we propose important directions for constructing future cognitive AI, including data and knowledge mining, multi-modal AI explainability, hybrid AI, and potential ethical challenges.

1. Introduction

Intelligence is what makes us human [1]. Human intelligence is a mental attribute that consists of the ability to learn from experience, adapt to new conditions, manage complex abstract concepts, and interact with and change the environment using knowledge. It is the human intelligence that makes Humans different from other creatures.

Emotional and cognitive functions are inseparable from the human brain and they jointly form human intelligence [2, 3]. Emotions are biological states associated with the nervous system. They are brought on by neurophysiological changes associated with thoughts, feelings, behavioral responses, and pleasure or displeasure. Humans express their feelings or react to external and internal stimuli through emotion. Thus, emotion plays an important role in everyday life. In recent years, research on emotion has increased significantly in different fields such as psychology, neuroscience, and computer science, where the so-called emotion artificial intelligence (AI) is committed to developing systems to recognize, process, interpret and simulate human emotion, leading to various potential applications in human lives including E-teaching, home care, security, and so on.

Cognition refers to the mental action or process related to acquiring knowledge and understanding through thought, experience, and sense [4–7]. Through cognitive processing, we make decisions and produce appropriate responses. Research shows that cognition evolved hand-in-hand with emotion [2, 3]. In other words, the emotion and cognition are two sides of the same system, and separating them leads to anomalous behaviors. However, given the importance of cognition to human intelligence, research on cognitive AI is still very limited. To better understand human needs and advance the development of AI, it is crucial to build computational models with cognition functions to understand human thoughts and behaviors and import the cognition capability to machines for better interaction, communication, and cooperation in human-computer interactions or computer-assisted human-human interactions. In the future, modeling human cognition and reconstructing it with emotion in real-world application systems are the new objectives of AI [ 8].

By their nature, as both emotion AI and cognitive AI are inspired and built to imitate human intelligence, they share a lot in common and require the combined efforts of multiple disciplines. In this sense, the focus of this paper is to discuss the future cognitive AI, which makes computers capable of analyzing, reasoning, and making decisions like human users. We firstly introduce the relationship between the emotion and cognition. Then, the emotion AI and cognitive AI are introduced. The important aspects of cognitive AI are specifically discussed including engagement, regulation, decision making, and discovery. Finally, we share the opinions of research focus for future cognitive AI.

2. Emotion and Cognition

Cognition and emotion appear to be separate from each other, but actually, they are two sides of the same coin, as shown in Figure 1. In fact, for a long historical time dating back to the time of Plato, emotion and cognition have been considered as separate and independent processes [9, 10]. In recent years, however, more and more studies tend to consider emotion and cognition as interrelated and integrated [11]. Numerous psychological studies have found that the processing of salient emotional stimuli and the experience of affective states can have an impact on behavioral and cognitive processing, while in turn cognition can influence and regulate emotion [12, 13]. From a neurobiological perspective, in the human brain, although previous studies have tended to suggest that emotion is responsible for the subcortical processing such as the primitive limbic system, while cognitive-related processes are mostly processed by the cortex areas like the prefrontal cortex, substantial studies in recent years have shown that emotional and cognitive processes are interrelated with each other and shared a large number of overlapping regions [14, 15], for reviews, please see [12, 16]. Therefore, the study of the interaction between emotion and cognition can help us create more trustworthy and human-like AI and enhance human-computer interactions.

Figure 1. Emotion and cognition.

2.2. Emotion AI

Emotion plays an essential role in human communication and cognition. With emotions, humans can express their various feelings or react accordingly to internal and external stimuli. While not all AI or intelligent software systems need to be emotional or have exactly all human-like emotions such as depression and anxiety, using AI to further understand emotions and develop corresponding systems would certainly make AI more user-friendly and more convenient for our daily lives [17].

For this purpose, emotion AI is developed and dedicated to uncovering human emotions and enabling computers to have human-like capabilities to understand, interpret, and even express emotions [18]. Emotion AI can be used in a wide range of areas where emotions play an important role [17]. For instance, in mental health care, AI may help psychologists to detect symptoms or disorders that are hidden or unaware of by patients. In the education field, an AI teacher who can understand students’ emotional states and respond appropriately would enhance the learning process, and thus is utilized in remote or hybrid learning scenarios. Furthermore, emotion AI is coming into use in many other fields such as entertainment, communication, and intelligent agents [17]. To date, the novel methodology developed in affective computing and Emotion AI has significantly contributed to our lives in many areas such as emotional well-being, E-teaching, home care, and security.

Currently, there are three mainstreams in artificial emotional intelligence, namely emotion recognition, emotion synthesis, and emotion augmentation [19]. Emotion recognition refers to applying computer systems to recognize human emotions in affective computing, and is one of the most traditional approaches that has been employed in emotion AI research [20]. Emotion recognition has been widely used in different modalities such as facial expressions [21-23], body gestures [24, 25], acoustic or written linguistic content [26, 27], physiological signals [28, 29], and the fusion of multiple modalities [30–32]. While most previous research has focused on facial expression recognition, given the multi-modal nature of emotions, more and more researchers are tending to refer to other modalities as well.

With these multi-modal cues, AI can better recognize genuine and spontaneous emotions and build more reliable systems accordingly. Besides recognizing human emotions, AI often needs to analyze and process human behaviors and interact with humans, so it is important to equip AI with emotion synthesis capability [33] to improve the human-computer interactive experience. The main development of emotion synthesis is in speech, facial expression, and body behavior synthesis, which can even date back to three decades ago [19]. Exploring emotion synthesis, undoubtedly, can enhance the AI’s affective and social realism, thus improving the reliability of AI and making their interaction with users more natural [34]. However, it is important to note the emotion synthesis we raise here is far from an artificially emotional machine, as emotion synthesis is more about external emotional expressions, i. e., equipping AI with some human-like emotional reactions. There is still a long way to go to let AI generate inner emotions, which involves a lot of ethical issues that cannot be ignored.

In addition, emotion augmentation denotes embedding the principle of emotion in AI and using it in planning, reasoning, or more general goal achievement [19, 35]. In other words, emotion augmentation requires applying emotion concepts to achieve broader goals in AIs. For instance, by introducing two emotional parameters, namely anxiety and confidence, the emotional backpropagation (BP) learning algorithm can reach an outstanding performance compared to the conventional BP-based neural network in the facial recognition task [36]. Apart from that, emotion augmentation can be also used in a wider range of scenarios, such as designing models, computational reinforcement learning, or other cognitive-related tasks [35]. Although still far away, the booming development of studies in sentiment analysis and affective computing (in computer science and other interdisciplinary disciplines) makes the goal of translating emotions into AI less far-fetched than one might think.

2.3. Cognitive AI

Human thinking is beyond imagination. What human think about every day involves: how to act in a dynamic and evolving world; how to juggle multiple goals and preferences; how to face opportunities and threats, etc. Is it possible that a computer develops the ability to think and reason without human intervention? An overarching goal for future AI is to simulate human thought processes in a computerized model. The result is cognitive AI.

Cognitive AI is derived from AI and cognitive science [37]. It develops computer systems simulating human brain functions [38]. The aim of cognitive AI is to mimic human behaviors and solve complex problems. It encompasses the main brain behaviors of human intelligence, including perception, attention, thought, memory, knowledge formation, judgment and evaluation, problem-solving and decision making, etc [39]. Moreover, cognitive processes utilize existing knowledge to discover new knowledge.

By its nature, cognitive AI needs to employ methods from multiple disciplines such as cognitive science, psychology, linguistics, physics, information theory, and mathematics [40, 41]. Based on the knowledge from different disciplines, cognitive AI uses computer theory and techniques to simulate human cognitive tasks in order to make computers think like the human brain [41]. Cognitive AI is fundamentally different from traditional AI. Traditional AI, or traditional computer techniques, are mainly based on the fed data and processed accordingly under human pre-programming [42]. On the contrary, cognitive AI should be able to continuously adapt itself to the context and environment so as to evolve and grow over time. For example, IBM Watson, a typical representative of Cognitive AI, defeated the human champion in the Jeopardy game in 2011 [43]. Unlike its forerunner Deep Blue (known as the first computer that defeated a human chess player), which required an exhaustive search and performed merely quantitative analysis, Watson relies on offline stored information and uses natural language processing combined with appropriate contextual information to perform adaptive reasoning and learning to produce answers [41, 43].

In this sense, the benefits of cognitive AI outweigh traditional AI, as traditional AI prefers to execute optimal algorithms to solve problems, while cognitive AI goes a big step further to replicate human wisdom and intelligence through multifaceted analysis including understanding and modeling human engagement, regulation (self-regulation and shared regulation in interactions), decision making, discovery, etc. Notice that cognitive AI requires complex data learning, knowledge mining, multi-model explainability, and hybrid AI to learn and analyze the patterns and to ensure that the reasoning results are reliable and consistent [44].

3. Important Aspects in Cognitive AI

In this section, we specifically discuss the important aspects of developing a successful cognitive AI system, including engagement, regulation, decision making, and discovery.

3.1. Engagement

Engagement describes the effortful commitment to goals [45]. Cognition, emotion, and motivation are incorporated into the engagement. Multiple studies show that positive emotion and motivation can improve the engagement of humans [46]. These findings have been successfully utilized in various fields, such as education, business dealing and human-computer interaction (HCI) to improve the engagement of users [47–49].

In recent years, besides emotion and motivation, researchers start to study the contribution of cognitive AI to engagement. IBM developed Watson taking advantage of cognitive AI for customer engagement to better identify the wants and needs of the customers [50]. In healthcare, cognitive computing systems collect individual health data from a variety of sources to enhance patient engagement and help professionals treat patients in a customized manner [51]. However, the technology is still at an early stage. How to find what really matters in engaging humans requires further study.

3.2. Regulation

In social cognitive theory, human behavior is motivated and regulated through constant self-influence. The self- regulative mechanism has a strong impact on affect, thought, motivation, and human action [52]. Regulation involves the ability to make adaptive changes in terms of cognition, motivation, and emotions in challenging situations [53]. Emotion changes could be the reason for the regulation, while they also could be the result of regulation [54].

In Bandura’s theory [55], the regulation process contains self-observation, judgment, and self-response. The self-observation is how humans look at themselves and their behaviors. Judgment refers to using a standard to compare what humans see. Self-response is performing different behaviors according to whether meeting the standard or not. Through these steps, humans are able to control their behavior [52]. However, until now, it is almost impossible to evidence the "invisible mental" shared regulation due to methodological limitations [56]. Moreover, it is not clear how humans regulate their internal states in response to internal and external stressors [52]. In the future, the self- and shared-regulative mechanisms should be further investigated and integrated into the AI systems to make AI learn by itself like humans.

3.3. Decision Making

Decision making is regarded as the cognitive process referring to problem-solving activities yielding a solution deemed to be optimal among many alternatives [57]. The decision making process can be based on explicit or tacit knowledge and beliefs. It can be rational or irrational.

Decision making happens every now and then in human’s daily life. A wise decision may make life and the world better. However, human’s decisions may be influenced by multiple factors, especially emotion. Human choices can be influenced by the human’s mood at the time of decision making [58, 59]. For instance, happy people are more likely to be optimistic about the current situation and choose to maintain the current state, whereas sad people are more inclined to be pessimistic about the current situation and choose to change the current state. Fearful people tend to hold pessimistic judgments about future events, while angry people tend to keep optimistic judgments about future events. It is important to study how to model the connections between emotion and decision making and understand the deep process of decision making in human brains.

3.4. Discovery

As a high-level and advanced scope of cognitive AI, discovery refers to finding insights and connections in the vast amount of information and developing skills to extract new knowledge [60]. With the increasing volumes of data, it becomes possible to train models with AI technologies to efficiently discover useful information from massive data. At the same time, the massive complex data increases the difficulty of manually processing by humans. Therefore, AI systems are needed to help exploit information more effectively than humans could do on their own.

In recent years, some systems with discovery capabilities have already emerged. The cognitive information management shell at Louisiana State University is a cognitive solution and has been applied in preventing future oil spills [61]. More specifically, this system built a complex event processing system that can detect problems in mission-critical and generate intelligent decisions to modulate the system environment. The system contains distributed intelligent agents to collect multiple streaming data to build interactive sensing, inspection, and visualization systems. It is able to achieve real-time monitoring and analysis. Through archiving past events and cross-referencing them with current events, the system can discover deeply hidden patterns and then make decisions upon them. Specifically, cognitive information management not only sends alerts but also dynamically re-configures to isolate critical events and repair failures. Currently, the study of system with discovery capabilities are still in the early stages. The value propositions for the future are very compelling. In the future, we should consider more about how to design cognitive systems with discovery ability effectively and efficiently so as to solve different tasks in complex real-life situations.

In general, engagement, self-regulation, decision making, and discovery are considered to be complementary in cognitive AI systems, as shown in Figure 2. Engagement refers to the way how humans and systems interact with each other. Through enhancing the engagement, more effort would be contributed to achieving the final goal. In this process, adaptions referring to regulation are ongoing which further simulates the engagement [62]. Furthermore, engagement and regulation can influence the ability to explore the underlying complex relationships and solve problems that are related to discovery and decision-making, relatively [63]. In turn, just like us human beings, discovery and decision making can also impact engagement and regulation. To develop a successful cognitive AI system, all of the above four aspects should be considered and well-designed.

Figure 2. Four aspects of cognitive AI are complementary to each other.

4. The Future of Cognitive AI

Towards cognitive AI, data and knowledge mining, multi-modal AI explainability, and hybrid AI should be explored, and the potential new ethical issues should also be kept in mind.

4.1. Data and Knowledge Mining

Fueled by advances in AI and the boosting of digital data, the cognitive process could be increasingly delegated to automated processes based on data analysis and knowledge mining [64], which intends to dig hidden patterns and knowledge from vast amounts of information through AI technology [65]. The new AI systems should support analytical engagement, regulations, decision making, and discovery approaches, which involve big data of greater variety, higher volumes and more velocity with the technology development [66]. Most of the current machine learning techniques based on big data are usually based on the vector space and work for specific scenarios [67–69]. It is difficult to store, analyze, and visualize big and complex cognitive data for further processes or results with traditional statistical models. A promising future research focus is to learn from dynamic data of complicated cognitive processes in both Euclidean space and non-Euclidean space with uncertainties for generic scenarios and scenarios with equivocality [70]. Another research direction would be the life-long learning and self-supervised learning that discover new knowledge through digging into the relationships among the data with complex structures and uncertainties. For example, in the field of cybersecurity where dynamic threats and cyber-attacks appear all the time [71], cybersecurity is required to build fast self-reliant cognitive computing systems that can make wiser security decisions based on self-learned knowledge and complex surrounding factors.

4.2. Multi-Modal AI Explainability

Cognitive systems are dedicated to developing advanced human-like AIs. However, as AI becomes more intelligent and impacts our lives in many ways, an insurmountable problem is how we trace back and understand the results of AI algorithms.

Explainable artificial intelligence (XAI) refers to a type of explainable AI that generate details and reasoning processing to help users understand its functions [72]. In terms of cognitive AI, it is often necessary to collect information from different modalities. For example, in building emotion and cognitive AI engines, behavioral information such as facial expressions and body gestures, language and voice information, as well as physiological signals are all important data sources for understanding human emotion and cognition functions, and are collected simultaneously across modalities. However, once we feed all data into the computer, the whole calculation process is seems to be a “black box”, and even the researchers or engineers who develop the algorithms themselves cannot know what happens inside the “box” [72, 73]. Therefore, developing cognitive XAI to help us interpret the results, reveal the reasoning processing, and find causality among multi-modal data variables would be a fruitful area for further work. The study of multi-modal AI explainability could have various applications, especially in health care. On one hand, the cognitive XAI could help doctors better analyze diseases and reduce the rate of misdiagnosis [42, 74]. On the other hand, the patient can better understand their health status which leads to increased trust in treatment options.

4.3. Hybrid AI

Hybrid AI is the future direction, which represents joint intelligence from human beings and algorithms. To date, hybrid AI has gained much more attention and started to be applied to some realistic applications, such as management [75]. However, most of them are simple combinations of human and AI decisions. The way to better take advantage of humans and algorithms should be explored. Moreover, in daily life, efficient interactions and collaborations along with optimal decisions require contributions from multiple people/systems who/which have various strengths. It is valuable to study how to combine the decision of a group of people/systems. In general, when addressing complexity, AI can extend humans’ cognition, while humans offer an intuitive, generic, and creative approach to deal with uncertainties and equivocality. Therefore, hybrid intelligence can strengthen both human intelligence and AI, and get both co-evolving by integrating cognitive AI into the development. The hybrid cognitive AI benefits high-risk situations, such as medical diagnostics, algorithmic trading, and autonomous driving [76].

4.4. Ethical Concerns

If AI algorithms/systems have more emotional and cognitive intelligence, they would be more human-like and play a more important role in various applications, but more ethical concerns might be introduced, which cannot be overlooked in constructing future cognitive computing systems.

AI is usually driven by big data, especially nowadays many researchers emphasize the use of multi-modal data to improve the accuracy of algorithms. However, a large amount of data, especially from the same individual, will also introduce privacy and potential misuse issues. There is also a growing concern about the risks of false interpretation of emotions and false cognitive actions due to inaccurate and unreliable AI algorithms. AI may reveal emotional or cognitive states that people do not want to disclose or be treated unfairly due to the bias of the AI’s decisions. Therefore, it is important to consider the ethical issues and comply with the corresponding principles, e.g., transparency, justice, fairness and equity, non-maleficence, responsibility, and privacy. For a detailed guideline, please see [77] while improving the performance of AI.

5. Conclusion

Emotion AI and cognitive AI simulate human intelligence together. Emotion AI is dedicated to discovering human emotions and enabling computers to have human-like capabilities to understand, interpret, and synthesize emotions. Cognitive AI aims to make computers analyze, reason, and make decisions like a human. In this paper, we review the development and relationship of emotion AI and cognitive AI. Cognitive computing is recognized as the next era of computing. Four important aspects of cognitive AI are discussed including engagement, regulation, decision making, and discovery, which are complementary in cognitive AI. To develop a successful cognitive computing system, all of the above four aspects should be considered. Finally, this paper explores the future of cognitive AI. Possible research directions on future cognitive AI include data and knowledge mining, multi-modal AI explainability, hybrid AI, and potential ethical issues.

Author Contributions: Guoying Zhao, Yante Li and Qianru Xu: conceptualization; Yante Li and Qianru Xu: investigation; Yante Li and Qianru Xu: writing — original draft preparation; Guoying Zhao: writing — review and editing; Guoying Zhao: supervision; Guoying Zhao: project administration; Guoying Zhao: funding acquisition. All authors have read and agreed to the published version of the manuscript.

Funding: This research is funded by the Academy of Finland for Academy Professor project Emotion AI, grant Nos 336116 and 345122, and Ministry of Education and Culture of Finland for AI forum project.

Conflicts of Interest: The authors declare no conflict of interest.

Acknowledgments: Thanks Guoying Zhao for figure design.