SEMINAR ON
ARTIFICIAL INTELLIGENCE
In partial fulfilment of the Assessment Examination of the 8th semester of B.E. in Electronics and Communication at
University Institute of Engineering and Technology Panjab University, Chandigarh
SEMINAR GUIDE:Mrs.SharmeleeThangjam SUBMITTED BY:
Yashpreet Singh (UE85108)
0
UNIVERSITY INSTITUTE OF ENGINEERING AND TECHNOLOGY
CERTIFICATE
Certified that this project report “ARTIFICIAL INTELLIGENCE” is the bonafide work of “Yashpreet Singh (UE85108)” (UE85108)” who carried out the seminar work under my supervision.
MRS. SHARMELEE THANGJAM (SEMINAR GUIDE) DEPARTMENT OF ELECTRONICSAND COMMUNICATION (UIET) PANJAB UNIVERSITY
1
UNIVERSITY INSTITUTE OF ENGINEERING AND TECHNOLOGY
CERTIFICATE
Certified that this project report “ARTIFICIAL INTELLIGENCE” is the bonafide work of “Yashpreet Singh (UE85108)” (UE85108)” who carried out the seminar work under my supervision.
MRS. SHARMELEE THANGJAM (SEMINAR GUIDE) DEPARTMENT OF ELECTRONICSAND COMMUNICATION (UIET) PANJAB UNIVERSITY
1
ACKNOWLEDGEMENT ACKNOWLEDGEMENT I, the students of 8th semester, B.E., University Institute of Engineering and Technology, Panjab University, Chandigarh, am thankful to my teacher Mrs. SHARMELEE THANGJAM for allotting me the seminar on the topic, “Artificial Intelligence”. I am really obliged obli ged for her support and guidance all through the making of the seminar. I thank her for enlightening me with wi th her vast knowledge. I would also like to thank the librarian and the other library staff who helped me by providing the various books and internet access which was inevitable in completion of the project.
Thanking You Yashpreet Singh
(UE85108)
2
ABSTRACT Imagine tools, technologies, environments, data sets, that get better the more you use them, that learn about you and adapt and improve as a result of your interactions and use. Think about being able to create new knowledge based on capturing observed patterns, recognizing behaviours, gleaning and understanding the context of events and actions. This is the promise of Artificial Intelligence to the human kind. Artificial Intelligence (AI) is the key technology in many of today's novel applications, ranging from banking systems that detect attempted credit card fraud, to telephone systems that understand speech, to software systems that notice when you're having problems and offer appropriate advice. These technologies would not exist today without the sustained support of fundamental AI research over the past three decades. AI does not produce stand-alone systems, but instead adds knowledge and reasoning to existing applications, databases, and environments, to make them friendlier, smarter, and more sensitive to user behaviour and changes in their environments. Beyond the myriad of currently deployed applications, ongoing efforts that draw upon these decades of fundamental research point towards even more impressive future capabilities including producing valuable spin-off technologies. AI researchers tend to look very far ahead, crafting powerful tools to help achieve the daunting tasks of building intelligent systems. Artificial Intelligence first conceived and demonstrated such well-known technologies as the mouse, time-sharing, high-level symbolic programming l anguages, computer graphics, the graphical user interface (GUI), computer games, the laser printer, object-oriented programming, the personal computer, email, hypertext, symbolic mathematics systems, and, most recently, the software agents which are now popular on the World Wide Web. There is every reason to believe that AI will continue to produce such spin-off technologies. Surely, we must equally push forward, explore, learn, develop and advance the capabilities presenting themselves today. Just imagine the impact on human life if we had technology that could learn! It describes the concepts of fuzzy sets and operations used in their manipulation, and the study of Neural Network modern neuroscience, artificial neurons, neuron networks and architecture, supervise learning, Statistical Frame work, Statistical learning theory, Associative Memory, Fuzzy principle, applications Application examples, Forecasting, Prediction Rain, temperature, flood, Humidity etc. and neurocomputing and how they are related to each other. The paper gives examples of the fuzzy logic applications, with emphasis on the field of artificial intelligence. The autonomous nature of the actively participating data source is preserved and evolution of the system is allowed by addition and removal of data sources. With these efforts, the proposed architecture supports the technical and organizational problems generally posed by the wide spread use of different technologies and implementation standards. It may improve the faced problems of imposing a single technology throughout the organization. The reason behind this all is the characteristics of the system of hiding the information itself. The autonomous nature of participation to data sources binds it with them. Data mining is a combination of database and artificial intelligence technologies. Although the AI field has taken a major dive in the last decade; this new emerging field has shown that AI can add major contributions to existing fields in computer science. In fact, many experts believe that data mining is the third hottest field in the industry behind the Internet, and data warehousing. Data mining is really just the next step in the process of analyzing data. Instead of getting queries on standard or user-specified relationships, data mining goes a step farther by finding meaningful relationships in data. Relationships that were thought to have not existed or ones that give a more insightful view of the data. For example, a 3
computer generated graph may not give the user any insight; however data mining can find trends in the same data that shows the user more precisely what is going on. Using trends that the end-user would have never thought to query the computer about. Without adding any more data, data mining gives a huge increase in the value added by the database. It allows both technical and non-technical users get better answers, allowing them to make a much more informed decision, Saving their companies millions of dollars. Artificial neural networks have emerged as fast computation tools with learning and adaptively capabilities whereas fuzzy logic has emerged as a mathematical tool to deal with the uncertainties in human perception and reasoning. They also provide a framework for an inference mechanism that allows approximate human reasoning capabilities to be applied to knowledge-based systems. Recently, these two fields have been integrated into a new emerging technology called fuzzy neural networks which combines the benefits of each field.
4
Table of Contents UNIVERSITY INSTITUTE OF ENGINEERING AND TECHNOLOGY ..........................................................1 CERTIFICATE .............................................................................................................................................1 ACKNOWLEDGEMENT .................................................................................................................................2 ABSTRACT......................................................................................................................................................3
List of figures ....................................................................................................................................................9 List of Tables ................................................................................................................................................. 10 Are humans machines? Can machines be like humans? .............................................................................. 11 Introduction to Artificial Intelligence ............................................................................................................ 12 Why Artificial Intelligence? ....................................................................................................................... 12 Defining Artificial Intelligence ................................................................................................................... 13 History ........................................................................................................................................................... 14 The Disciplines of Artificial Intelligence ........................................................................................................ 16 Social intelligence ......................................................................................................................................... 18 Creativity ................................................................................................................................................... 19 General intelligence .................................................................................................................................. 1 9 Machine Learning ......................................................................................................................................... 19 Learning Systems ...................................................................................................................................... 20 Knowledge Representation and Reasoning .................................................................................................. 21 Planning .................................................................................................................................................... 22 Knowledge Acquisition.............................................................................................................................. 22 Intelligent Search ...................................................................................................................................... 22
Logic Programming ................................................................................................................................... 22 Management of Imprecision and Uncertainty.......................................................................................... 22 General Problem Solving Approaches in AI .................................................................................................. 23 Algorithm for solving state-space problems ............................................................................................. 24 Symbolic Approaches ................................................................................................................................ 2 6 Sub-symbolic approaches ......................................................................................................................... 27 Statistical approaches ............................................................................................................................... 27 Integrating the approaches....................................................................................................................... 2 8 Tools .............................................................................................................................................................. 28 Search and optimization ........................................................................................................................... 28 Logic .......................................................................................................................................................... 29 Probabilistic methods for uncertain reasoning ........................................................................................ 30 Classifiers and statistical learning methods .............................................................................................. 30 Data Mining and its applications .................................................................................................................. 30
5
Introduction .............................................................................................................................................. 30 History ....................................................................................................................................................... 32 Fundamentals ........................................................................................................................................... 32 Data, Information, and Knowledge ....................................................................................................... 32 Necessity ................................................................................................................................................... 33 Continuous Innovation.............................................................................................................................. 33 Categories of Data Mining ........................................................................................................................ 33 Data Mining helps extract such information ............................................................................................ 34 Association Rules ...................................................................................................................................... 34 Data Mining Process ................................................................................................................................. 35 The KDD process consists of the following five steps: .......................................................................... 36 Basic Data Mining Model .......................................................................................................................... 36 What can data mining do? ........................................................................................................................ 37 Data Mining Tasks ..................................................................................................................................... 38 1.
Classification ................................................................................................................................. 38
2.
Regression ..................................................................................................................................... 38
3.
Time Series Analysis ...................................................................................................................... 39
4.
Prediction ...................................................................................................................................... 39
5.
Clustering ...................................................................................................................................... 40
6.
Association Rules .......................................................................................................................... 40
Data mining consists of five major elements: ....................................................................................... 41 Different levels of analysis are available: ............................................................................................. 41 What technological infrastructure is required?........................................................................................ 41 Applications .............................................................................................................................................. 41 Artificial Neural Networks and Fuzzy Logic ................................................................................................... 42 Introduction to Artificial Neural Networks ............................................................................................... 42 Types of Artificial Neural Networks (ANN) ............................................................................................... 44 A Neural network with two layers ............................................................................................................ 45 The Biological Model................................................................................................................................. 46 Biological neural network ..................................................................................................................... 46 The Mathematical Model ......................................................................................................................... 47 Activation functions .............................................................................................................................. 48 Neurobiological methods...................................................................................................................... 49 A framework for distributed representation ........................................................................................ 49 Processing units .................................................................................................................................... 50 Neural Network topologies....................................................................................................................... 50 Training of artificial neural networks........................................................................................................ 51 6
Modifying patterns of connectivity of Neural Networks ...................................................................... 51 How the Human Brain Learns? ................................................................................................................. 52 Artificial Neurons and how they work ...................................................................................................... 53 Artificial Neuron:................................................................................................................................... 5 3 Architecture of neural networks............................................................................................................... 54 The Basic Artificial Model ..................................................................................................................... 55 Single-Layer Perceptron Neural Networks............................................................................................ 56 Multilayer perceptron........................................................................................................................... 5 7 Alternative Representations: Neural Nets and Genetic Algorithms ..................................................... 58 Genetic algorithms.................................................................................................................................... 5 9 Transfer Function ...................................................................................................................................... 61 Fuzzy Logic and Neural Network ............................................................................................................... 62 How is FL different from conventional control methods? .................................................................... 62 How does FL work? ............................................................................................................................... 63 Neuro-computing .................................................................................................................................. 63 Fuzzy Logic Analysis and Control:- ........................................................................................................ 64 An engineering approach to fuzzy logic:- .............................................................................................. 65 Artificial Neural Network and Fuzzy Logic ............................................................................................ 65 Applications of ANN and Fuzzy Logic .................................................................................................... 66 Software & Tools................................................................................................................................... 68 Natural Language Processing ........................................................................................................................ 68 Natural Language Understanding and Semantics..................................................................................... 68 Applications for processing large amounts of texts require NLP expertise. ............................................. 69 Languages and Environments for AI ..................................................................................................... 69 Sub-problems ............................................................................................................................................ 70 Where Does It Fix In The Cs Taxonomy? ................................................................................................... 71 Ladder of Transformation ......................................................................................................................... 71 Stages in Text Processing .......................................................................................................................... 71 Natural Language Processing (Nlp) Techniques........................................................................................ 72 Major Tasks in Nlp..................................................................................................................................... 7 3 Distributed Artificial Intelligence .................................................................................................................. 74 Knowledge Acquisition.............................................................................................................................. 74 Cooperation and Explanations .................................................................................................................. 74 Learning .................................................................................................................................................... 75 Single Agent learning ............................................................................................................................ 75 Multiple-Agent Learning ....................................................................................................................... 75 Coordination ............................................................................................................................................. 7 5 7
Soft Computing ............................................................................................................................................. 75 Implications of Soft Computing ................................................................................................................ 76 Cognitive science .......................................................................................................................................... 80 Principles ................................................................................................................................................... 80 Levels of analysis................................................................................................................................... 80 Interdisciplinary nature......................................................................................................................... 81 Knowledge, and Processing, of Language ................................................................................................. 81 Learning and development ....................................................................................................................... 82 Memory..................................................................................................................................................... 82 The Turing Test ............................................................................................................................................. 83 Artificial Intelligence as a Positive and Negative Factor in Global Risk ........................................................ 85 Underestimating the power of intelligence.............................................................................................. 85 Capability and motive ............................................................................................................................... 86 Friendly AI ................................................................................................................................................. 87 Technical failure and philosophical failure ............................................................................................... 88 An example of philosophical failure...................................................................................................... 88 An example of technical failure ............................................................................................................ 89 Hardware .................................................................................................................................................. 91 Threats and promises ............................................................................................................................... 92 AI versus human intelligence .................................................................................................................... 93 Overview of AI Application Areas ................................................................................................................. 95 Game Playing ............................................................................................................................................ 95 Automated Reasoning and Theorem Proving ........................................................................................... 96 Expert Systems.......................................................................................................................................... 97 Image Understanding and Computer Vision............................................................................................. 99 Speech and Natural Language Understanding ....................................................................................... 100 Scheduling ........................................................................................................................................... 100 Intelligent Control ................................................................................................................................... 1 01 Modelling Human Performance.............................................................................................................. 1 02 Planning and Robotics............................................................................................................................. 102 Problems ..................................................................................................................................................... 103 Deduction, reasoning, problem solving .................................................................................................. 103 Knowledge representation ..................................................................................................................... 104 Planning .................................................................................................................................................. 105 Conclusion ................................................................................................................................................... 106 Conclusions ............................................................................................................................................. 1 06 Artificial Intelligence--A Summary .............................................................................................................. 107 8
References Cited ......................................................................................................................................... 1 08
List of figures S. No.
Figure
Title
1
Fig. 1
2
Fig. 2 Fig. 3 Fig. 4 Fig. 5
TOPIO, a humanoid robot, played table tennis at Tokyo International Robot Exhibition (IREX) 2009 AI, its parent disciplines and application areas Pronunciation learning of a child from his mother Kismet, a robot with rudimentary social skills The initial and the final states of the Number Puzzle game, where B denotes the blank space The state-space for the Four-Puzzle problem Brain simulation AI and cybernetics Computational Intelligence Proposed Architecture for the Interpretability of Data Sources Basic WEP encryption: RC4 key-stream XORed with plaintext KDD Process (modified from [FSP2]) Data mining models and tasks [BL] Time Series plots phases of agent-based attack simulator Neuron Artificial neuron Mathematical model Common non-linear functions The style of neural computation Supervised learning situation Components of a neuron The synapse Artificial Neuron A Basic Artificial Neuron A neuron model Multilayer structure Typical multilayered neural network architecture Single Layer Perceptron Capabilities of zero, one, and two hidden neuron layer neural network Sigmoid and derivative of sigmoid function 2 views of the same 2-layer neural network Test function evaluated over interval [0, 10] Crossover operation Example backpropagation neural network with one hidden layer Example backpropagation neural network with two hidden layers Fuzzy system The Biological Neuron Functional structure of brain
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
Fig. 6 Fig. 7 Fig. 8 Fig. 9 Fig. 10 Fig. 11 Fig. 12 Fig. 13 Fig. 14 Fig. 15 Fig. 16 Fig. 17 Fig. 18 Fig. 19 Fig. 20 Fig. 21 Fig. 22 Fig. 23 Fig. 24 Fig. 25 Fig. 26 Fig. 27 Fig. 28 Fig. 29 Fig. 30 Fig. 31 Fig. 32 Fig. 33 Fig. 34 Fig. 35 Fig. 36 Fig. 37 Fig. 38 Fig. 39
9
40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62
Fig. 40 Fig. 41 Fig. 42 Fig. 43 Fig. 44 Fig. 45 Fig. 46 Fig. 47 Fig. 48 Fig. 49 Fig. 50 Fig. 51 Fig. 52 Fig. 53 Fig. 54 Fig. 55 Fig. 56 Fig. 57 Fig. 58 Fig. 59 Fig. 60 Fig. 61 Fig. 62
Representation of 6×4×1 simple network Fuzzy concept Fuzzy logic control analysis method Example of AI systems Noisy channel model Natural Language Processing Example Where Does It Fix In The Cs Taxonomy? Multidimensional model Hierarchy according to cube given in fig43 Object Modelling Aggregation Inheritance Relation Many-to-Many Relationship Object oriented model Phases of warehouse Problem solving technologies Computing models An automated online assistant providing customer service on a web page Basic steps in scene interpretation The Gantt charts for the flowshop scheduling problem with 2 jobs and 3 machines Ontology represents knowledge as a set of concepts within a domain A hierarchical control system Multi layered structure
List of Tables S.No.
Table
Title
1
Table 1 Table 2 Table 3 Table 4
Association Rules History of researchMost commonly used part of speech tags Sample part of speech tags Summary Of Application Of Soft Computing
2 3 4
10
Are humans machines? Can machines be like humans? Such questions were posed first several centuries ago, when thinkers like Descartes and La Mettrie were faced with mechanical clocks and other devices which seemed-on their face-to do things which were, formerly, only within the domain of humans. To the first question, Hobbes and others began, in the 17th century, to use mechanical metaphors for thinking and intelligence. As well, the exploration and explanation of the body-as-machine, the mechanistic approach, emerged as paradigmatic; ancient puzzles such as the circulation of the blood were solved. Anatomy and physiology, at least, used the model of the body-mechanic, to tell us a great deal about the nature of being human. Are machines human? Can they be?-is answered in short: "no" and "yes." No, they are not human; they lack "intelligence." We have witnessed the rise of the age of computers, and the winds are shifting from an "intellectual" or "scientific" approach to the problem of "intelligence" to a technological or engineering ("expert systems") solution, in which very large computational systems will be made to imitate our best human thinkers. We stand on the verge of an extraordinary irony: computers will become "more" human, more intellectual through an approach to intelligence which is it, mostly non-intellectual . The dystopic vision of the movie "2001" -in which thinking computers first rid the world of humanoids, and then mimic us at our most destructive-impinges uncomfortably. But before approaching a description of what is (as far as it can be foretold) happening in the field(s) called "Artificial Intelligence," it is useful to explore why these questions arise in the form that they do; i.e., as "intelligence." Quite probably, the human-machine problem could be considered more broadly, as an aspect of what makes humans, Human. If we open the pages of our history, we see, however, that there are three significant counters-or anti-metaphors in terms of how we think about what is human, which equate human with intelligence. These are: animals, machines, and aliens (non-terrestrial beings, "Martians"). The question of intelligence and machines gains its substance largely in terms not of what we are; but of what we are not! The problem of what is Artificial Intelligence (AI), comes down to us pretty much as the attempt to analyse language. Language, especially the ancient problems of meaning and context, includes much more than the organization of linguistic structures which is the study of syntax and grammar. The concept of language as being unique to humans, and consisting of grammar and syntax, was apparently oversimplified because we had compared humans with a vision of animals which was, itself, much too simple. Second, humans are not machines. Machines were seen by Descartes, as purely mechanical. Like animals, they lacked a soul, a mind. Heirs to this Cartesian analysis of machines, we assumed that they could not think. Machines could not think, essentially, by definition. The current conceptual difficulty is that machines seem to be able to do more and more which overlaps with what we have considered to be thinking. This ranges from being able to play games (e.g., chess), with a fixed and finite (if very large) set of rules, player end domain, to being able to handle symbolic notions, to rewrite themselves (to "learn?"). This poses a dilemma, and places the original questions concerning humans and machines in a now blurred matrix: machines surely seem more human than they used to (they can do "intelligent" things, formerly reserved only to humans); we seem more like machines, reraising the question of what is human, in new and problematic ways. The expert system becomes a kind of human manual in which the very best thinker is a dynamic step-bystep guide to today's doings.This requires very large computers: and we are on the eve of enlarging vastly the capacities of computers by shifting from serial to parallel computing. Serial computing runs serially through a program, each step following the last. Parallel computers (e.g., many small computers linked by a master program which farms out work or micro-chips with specialized functions), are within touch, and 11
will be able to handle the 5,000 or so ways of considering that experts are reputed to be able to do, in close to "real time." The tough part, so far, is not in stating to the computer program how to imitate thinking-in fact, symbolic methods already appear adequate-but in getting persons who know how to interview in depth, the experts: who can get to know them, or get the experts themselves to become good self-analysts, to state how they know and do what they know and do. (Some old-fashioned "field-linguists" used to do approximately this type of work. It requires great patience, and ways of "getting into" another person's head: so-called "cognitive-mapping. ") A model for this approach, already worked out, is the "programmed" self-teaching course, some of which are quite effective. In short, we are on the brink of a powerful, perhaps revolutionary shift in AI, toward the engineering approach. It is "non-intellectual" in the sense that it does not ask deeply about the nature of intelligence or language or human thought. It accepts that experts are, in fact, knowledgeable, and bypasses questions about the nature of that knowledge. It simply models their knowledge about the practical affairs of the world. It responds to the question of whether machines can think in a way that is as surprising as it is powerful: it makes no difference whether machines can actually think, as long as they can model human thought. The question of whether machines can think is, at once, moot, and answered in the affirmative! Now, heading back to the question of what is intelligent remains of great interest. It is not overly strong to state that applied math, philosophy, psychology, and linguistics have joined forces to make computers be able to model what is intelligence. AI, in this context, has been placed within the development of powerful computer programs, which are increasingly geared to act intelligently. Here, AI is principally the ingenuity of thinkers, mostly mathematicians, to concoct programs which act much like humans: which "think." If they "solve" the question of what is intelligence, and can program "it," then the computer would presumably be able to think or learn, change its "mind" (its program), grow in scope from a manipulator of information, toward becoming knowledgeable, and, with great difficulty (my present perception), to become wise. The field was founded on the claim that a central property of humans, intelligence —the sapience of Homo sapiens—can be so precisely described that it can be simulated by a machine.[7] This raises philosophical issues about the nature of the mind and the ethics of creating artificial beings, issues which have been addressed by myth, fiction and philosophy since antiquity.[7] Artificial intelligence has been the subject of optimism,[8] but has also suffered setbacks[9] and, today, has become an essential part of the technology industry, providing the heavy lifting for many of the most difficult problems in computer science.[10]
Introduction to Artificial Intelligence Why Artificial Intelligence?
Computers are fundamentally well suited to performing mechanical computations, using fixed programmed rules. This allows artificial machines to perform simple monotonous tasks efficiently and reliably, which humans are ill-suited to. For more complex problems, things get more difficult... Unlike humans, computers have trouble understanding specific situations, and adapting to new situations. Artificial Intelligence aims to improve machine behaviour in tackling such complex tasks. Together with this, much of AI research is allowing us to understand our intelligent behaviour. Humans have an interesting approach to problem-solving, based on abstract thought, high-level deliberative reasoning and pattern recognition. Artificial Intelligence can help us understand this process by recreating it, then potentially enabling us to enhance it beyond our current capabilities. 12
Defining Artificial Intelligence
The phrase Artificial Intelligence, which was coined by John McCarthy three decades ago, evades a concise and formal definition to date. One representative definition is pivoted around the comparison of intelligence of computing machines with human beings. Another definition is concerned with the performance of machines which "historically have been judged to lie within the domain of intelligence". None of these definitions or the like has been universally accepted, perhaps because of their references to the word "intelligence", which at present is an abstract and immeasurable quantity. A better definition of artificial intelligence, therefore, calls for formalization of the term "intelligence". Psychologist and Cognitive theorists are of the opinion that intelligence helps in identifying the right piece of knowledge at the appropriate instances of decision making. The domain of Artificial Intelligence, in other words the domain of problems that require Intelligence, deals with Bizarre Systems. These are defined as those systems that contain at least one item from each of the four major problem types:
Chaotic Systems Systems that require a Holistic stance Systems that contain (or otherwise have to deal with) ambiguity Systems that exhibit Emergent Properties
Sixteen common kinds of problems have been sorted into four corners corresponding to these four types in this diagram:
Fig 1 bizarre systems The phrase "artificial intelligence" thus can be defined as the simulation of human intelligence on a machine, so make the machine efficient to identify and use the right place of "Knowledge" at a given step of solving a problem . A system capable of planning and executing the right task at the right time is
generally called rational. Thus, AI alternatively may be stated as a subject dealing with computational 13
models that can think and act rationally. A common question then naturally arises: Does rational thinking and acting include all possible characteristics of an intelligent system? If so, how does it represent behavioural intelligence such as machine learning, perception and planning? A little thinking, however, reveals that a system that can reason well must be a successful planner, as planning in many circumstances is part of a reasoning process. Further, a system can act rationally only after acquiring adequate knowledge from the real world. So, perception that stands for building up of knowledge from real world information is a prerequisite feature for rational actions. One step further, thinking envisages that a machine without learning capability cannot possess perception. The rational action of an agent (actor), thus, calls for possession of all the elementary characteristics of intelligence. Relating artificial intelligence with the computational models capable of thinking and acting rationally, therefore, has a pragmatic significance. Artificial Intelligence is a branch of Science which deals with helping machines finds solutions to complex problems in a more human-like fashion. This generally involves borrowing characteristics from human intelligence, and applying them as algorithms in a computer friendly way. A more or less flexible or efficient approach can be taken depending on the requirements established, which influences how artificial the intelligent behaviour appears. Artificial intelligence (AI) is the intelligence of machines and the branch of computer science that aims [4]
to create it. AI textbooks define the field as " the study and design of intelligent agents "
where an intelligent agent is a system that perceives its environment and takes actions that maximize its chances of success.[5]John McCarthy, who coined the term in 1956, defines it as " the science and engineering of [6] making intelligent machines ." AI is generally associated with Computer Science, but it has many important links with other fields such as Maths, Psychology, Cognition, Biology and Philosophy, among many others. Our ability to combine knowledge from all these fields will ultimately benefit our progress in the quest of creating an intelligent artificial being. AI research is highly technical and specialized, and deeply divided into subfields that often fail to communicate with each other.[11] Subfields have grown up around particular institutions, the work of individual researchers, the solution of specific problems, longstanding differences of opinion about how AI should be done and the application of widely differing tools. The central problems of AI include such traits as reasoning, knowledge, planning, learning, communication, perception and the ability to move and manipulate objects.[12] General intelligence (or "strong AI") is still among the field's long term goals.[13]
History Thinking machines and artificial beings appear in Greek myths, such as Talos of Crete, the bronze robot of Hephaestus, and Pygmalion'sGalatea.[14] Human likenesses believed to have intelligence were built in every major civilization: animated cult images were worshipped in Egypt and Greece[15] and humanoid automatons were built by Yan Shi, Hero of Alexandria and Al-Jazari.[16] It was also widely believed that artificial beings had been created by JābiribnHayyān, Judah Loew and Paracelsus.[17] By the 19th and 20th centuries, artificial beings had become a common feature in fiction, as in Mary Shelley's Frankenstein or KarelČapek'sR.U.R. (Rossum's Universal Robots).[18]Pamela McCorduck argues that all of these are examples of an ancient urge, as she describes it, "to forge the gods".[7] Stories of these creatures and their fates discuss many of the same hopes, fears and ethical concerns that are presented by artificial intelligence. Mechanical or "formal" reasoning has been developed by philosophers and mathematicians since antiquity. The study of logic led directly to the invention of the programmable digital electronic computer, 14
based on the work of mathematician Alan Turing and others. Turing's theory of computation suggested that a machine, by shuffling symbols as simple as "0" and "1", could simulate any conceivable act of mathematical deduction.[19][20] This, along with concurrent discoveries in neurology, information theory and cybernetics, inspired a small group of researchers to begin to seriously consider the possibility of building an electronic brain.[21]
Fig 2.TOPIO, a humanoid robot, played table tennis at Tokyo International Robot Exhibition (IREX) 2009 The field of AI research was founded at a conference on the campus of Dartmouth College in the summer of 1956.[22] The attendees, including John McCarthy, Marvin Minsky, Allen Newell and Herbert Simon, became the leaders of AI research for many decades.[23] They and their students wrote programs that were, to most people, simply astonishing: [24] computers were solving word problems in algebra, proving logical theorems and speaking English.[25] By the middle of the 1960s, research in the U.S. was heavily funded by the Department of Defence [26] and laboratories had been established around the world.[27] AI's founders were profoundly optimistic about the future of the new field: Herbert Simon predicted that "machines will be capable, within twenty years, of doing any work a man can do" and Marvin Minsky agreed, writing that "within a generation ... the problem of creating 'artificial intelligence' will substantially be solved".[28] They had failed to recognize the difficulty of some of the problems they faced.[29]In 1974 , in response to the criticism of Sir James Light hill and on-going pressure from the US Congress to fund more productive projects, both the U.S. and British governments cut off all undirected exploratory research in AI. The next few years, when funding for projects was hard to find, would later be called the "AI winter".[30] In the early 1980s , AI research was revived by the commercial success of expert systems,
[31]
a form of AI program that simulated the knowledge and analytical skills of one or more human experts. By 1985 the market for AI had reached over a billion dollars. At the same time, Japan's fifth generation computer project inspired the U.S and British governments to restore funding for academic research in the field .[32] However, beginning with the collapse of the Lisp Machine market in 1987, AI once again fell into disrepute, and a second, longer lasting AI winter began.[33] In the 1990s and early 21st century, AI achieved its greatest successes, albeit somewhat behind the scenes. Artificial intelligence is used for logistics, data mining, medical diagnosis and many other areas throughout the technology industry.[10] The success was due to several factors: the increasing computational power of computers (see Moore's law), a greater emphasis on solving specific sub 15
problems, the creation of new ties between AI and other fields working on similar problems, and a new commitment by researchers to solid mathematical methods and rigorous scientific standards .[34]On 11 May 1997, Deep Blue became the first computer chess-playing system to beat a reigning world chess champion, Garry Kasparov.[35] In 2005, a Stanford robot won the DARPA Grand Challenge by driving autonomously for 131 miles along an unrehearsed desert trail .[36] In February 2011, in a Jeopardy!Quiz show exhibition match, IBM's question answering system, Watson, defeated the two greatest Jeopardy Champions, Brad Rutter and Ken Jennings, by a significant margin.[37] The leading-edge definition of artificial intelligence research is changing over time. One pragmatic definition is: "AI research is that which computing scientists do not know how to do cost-effectively today." For example, in 1956 optical character recognition (OCR) was considered AI, but today, sophisticated OCR software with a context-sensitive spell checker and grammar checker software comes for free with most image scanners. No one would any longer consider already-solved computing science problems like OCR "artificial intelligence" today. Low-cost entertaining chess-playing software is commonly available for tablet computers. DARPA no longer provides significant funding for chess-playing computing system development. The Kinect which provides a 3D body –motion interface for the Xbox 360 uses algorithms that emerged from lengthy AI research, [38] but few consumers realize the technology source. AI applications are no longer the exclusive domain of Department of defense R&D, but are now common place consumer items and inexpensive intelligent toys. In common usage, the term "AI" no longer seems to apply to off-the-shelf solved computing-science problems, which may have originally emerged out of years of AI research. To date, all the traits of human intelligence have not been captured and applied together to spawn an intelligent artificial creature. Currently, Artificial Intelligence rather seems to focus on lucrative domain specific applications, which do not necessarily require the full extent of AI capabilities. This limit of machine intelligence is known to researchers as narrow intelligence. There is little doubt among the community that artificial machines will be capable of intelligent thought in the near future. It's just a question of what and when... The machines may be pure silicon, quantum computers or hybrid combinations of manufactured components and neural tissue. As for the date, expect great things to happen within this century! The subject of artificial intelligence was originated with game-playing and theorem-proving programs and was gradually enriched with theories from a number of parent disciplines. As a young discipline of science, the significance of the topics covered under the subject changes considerably with time. At present, the topics which we find significant and worthwhile to understand the subject are outlined below.
The Disciplines of Artificial Intelligence The subject of artificial intelligence spans a wide horizon. It deals with the various kinds of knowledge representation schemes, different techniques of intelligent search, various methods for resolving uncertainty of data and knowledge, different schemes for automated machine learning and many others. Among the application areas of AI, we have Expert systems, Game-playing, and Theorem-proving, Natural language processing, Image recognition, Robotics and many others. The subject of artificial intelligence has been enriched with a wide discipline of knowledge from Philosophy, Psychology, Cognitive Science, Computer Science, Mathematics and Engineering. Thus in figure, they have been referred to as the parent disciplines of AI. An at-a-glance look at fig. also reveals the subject area of AI and its application areas. Here's a list, but some branches are surely missing, because no-one has identified them yet. Some of these may be regarded as concepts or topics rather than full branches. 16
Fig3: AI, its parent disciplines and application areas. 1. Logical AI: What a program knows about the world in general the facts of the specific situation in which it must act, and its goals are all represented by sentences of some mathematical logical language. The program decides what to do by inferring that certain actions are appropriate for achieving its goals. 2. Search: AI programs often examine large numbers of possibilities, e.g. moves in a chess game or inferences by a theorem proving program. Discoveries are continually made about how to do this more efficiently in various domains. 3. Pattern recognition: When a program makes observations of some kind, it is often programmed to compare what it sees with a pattern. For example, a vision program may try to match a pattern of eyes and a nose in a scene in order to find a face. More complex patterns, e.g. in a natural language text, in a chess position, or in the history of some event are also studied. These more complex patterns require quite different methods than do the simple patterns that have been studied the most. 4. Representation: Facts about the world have to be represented in some way. Usually languages of mathematical logic are used. 5. Inference: From some facts, others can be inferred. Mathematical logical deduction is adequate for some purposes, but new methods of non-monotonic inference have been added to logic since the 1970s. The simplest kind of non-monotonic reasoning is default reasoning in which a conclusion is to be inferred by default, but the conclusion can be withdrawn if there is evidence to the contrary. For 17
6.
7.
8.
9. 10.
11.
12.
example, when we hear of a bird, we can infer that it can fly, but this conclusion can be reversed when we hear that it is a penguin. It is the possibility that a conclusion may have to be withdrawn that constitutes the non-monotonic character of the reasoning. Ordinary logical reasoning is monotonic in that the set of conclusions that can the drawn from a set of premises is a monotonic increasing function of the premises. Circumscription is another form of non-monotonic reasoning. Common sense knowledge and reasoning: This is the area in which AI is farthest from human-level, in spite of the fact that it has been an active research area since the 1950s. While there has been considerable progress, e.g. in developing systems of non-monotonic reasoning and theories of action, yet more new ideas are needed. The Cyc system contains a large but spotty collection of common sense facts. Learning from experience: Programs do that. The approaches to AI based on connectionism and neural nets specialize in that. There is also learning of laws expressed in logic. [Mit97] is a comprehensive undergraduate text on machine learning. Programs can only learn what facts or behaviours their formalisms can represent, and unfortunately learning systems are almost all based on very limited abilities to represent information. Planning: Planning programs start with general facts about the world (especially facts about the effects of actions), facts about the particular situation and a statement of a goal. From these, they generate a strategy for achieving the goal. In the most common cases, the strategy is just a sequence of actions. Epistemology: This is a study of the kinds of knowledge that are required for solving problems in the world. Ontology: Ontology is the study of the kinds of things that exist. In AI, the programs and sentences deal with various kinds of objects, and we study what these kinds are and what their basic properties are. Emphasis on ontology begins in the 1990s. Heuristics: A heuristic is a way of trying to discover something or an idea imbedded in a program. The term is used variously in AI. Heuristic functions are used in some approaches to search to measure how far a node in a search tree seems to be from a goal. Heuristic predicates that compare two nodes in a search tree to see if one is better than the other, i.e. constitutes an advance toward the goal, and may be more useful. Genetic programming: Genetic programming is a technique for getting programs to solve a task by mating random Lisp programs and selecting fittest in millions of generations.
Social intelligence Emotion and social skills [43] play two roles for an intelligent agent. First, it must be able to predict the actions of others, by understanding their motives and emotional states. (This involves elements of game theory, decision theory, as well as the ability to model human emotions and the perceptual skills to detect emotions.) Also, for good human-computer interaction, an intelligent machine also needs to display emotions. At the very least it must appear polite and sensitive to the humans it interacts with. At best, it should have normal emotions itself.
18
Fig 4.Kismet, a robot with rudimentary social skills Creativity
A sub-field of AI addresses creativity both theoretically (from a philosophical and psychological perspective) and practically (via specific implementations of systems that generate outputs that can be considered creative, or systems that identify and assess creativity). A related area of computational research is Artificial Intuition and ArtificialImagination.[44]Intuition is what we use to handle everyday problems such as predicting limb positions and controlling muscle movement, to understand and generate speech, to read, to analyze what we see, to drive a car, etc. In short, all the things we do that we take for granted and which we do "without thinking". Many of these are non-trivial and hard or impossible to do using current computer technology. General intelligence
Most researchers hope that their work will eventually be incorporated into a machine with general intelligence (known as strong AI), combining all the skills above and exceeding human abilities at most or all of them.[45] A few believe that anthropomorphic features like artificial consciousness or an artificial brain may be required for such a project.[46] [47] Many of the problems above are considered AI-complete: to solve one problem, you must solve them all. For example, even a straightforward, specific task like machine translation requires that the machine follow the author's argument (reason), know what is being talked about (knowledge), and faithfully reproduce the author's intention (social intelligence). Machine translation, therefore, is believed to be AIcomplete: it may require strong AI to be done as well as humans can do it.[48]
Machine Learning Learning has remained a challenging area for AI. The importance of learning, however, is beyond question, particularly as this ability is one of the most important components of intelligent behaviour. An expert system may perform extensive and costly computations to solve a problem. Unlike a human being, however, if it is given the same or a similar problem a second time, it usually does not remember the solution. It performs the same sequence of computations again. This is true the second, third, fourth, and every time it solves that problem--hardly the behaviour of an intelligent problem solver. The obvious
19
solution to this problem is for programs to learn on their own, either from experience, analogy, examples, or by being "told" what to do. Although learning is a difficult area, there are several programs that suggest that it is not impossible. One striking program is AM, the Automated Mathematician, designed to discover mathematical laws (Lenat 1977, 1982). Initially given the concepts and axioms of set theory, AM was able to induce such important mathematical concepts as cardinality, integer arithmetic, and many of the results of number theory. AM conjectured new theorems by modifying its current knowledge base and used heuristics to pursue the "best" of a number of possible alternative theorems. More recently, Cotton et al. (2000) designed a program that automatically invents "interesting" integer sequences. Early influential work includes Winston's research on the induction of structural concepts such as "arch" from a set of examples in the blocks world (Winston 1975 a). The ID3 algorithm has proved successful in learning general patterns from examples (Quinlan 1986 a). Meta-DENDRAL learns rules for interpreting mass spectrographic data in organic chemistry from examples of data on compounds of known structure. There are also now many important biological and sociological models of learning. The success of machine learning programs suggests the existence of a set of general learning principles that will allow the construction of programs with the ability to learn in realistic domains. Learning Systems
Among the subject areas covered under artificial intelligence, learning systems needs special mention. The concept of learning is illustrated here with reference to a natural problem of learning of pronunciation by a child from his mother (vide fig 4). The hearing system of the child receives the pronunciation of the character "A" and the voice system attempts to imitate it. The difference of the mother's and the child's pronunciation, hereafter called the error signal, is received by the child's learning system auditory nerve, and an actuation signal is generated by the learning system through a motor nerve for adjustment of the pronunciation of the child. The adaptation of the child's voice system is continued until the amplitude of the error signal is insignificantly low. Each time the voice system passes through an adaptation cycle, the resulting tongue position of the child for speaking "A" is saved by the learning process. The learning problem discussed above is an example of the well-known parametric learning, where the adaptive learning process adjusts the parameters of the child's voice system autonomously to keep its response close enough to the " sample training pattern". The artificial neural networks, which represent the electrical analogue of the biological nervous systems, are gaining importance for their increasing applications in supervised (parametric) learning problems. Besides this type, the other common learning methods, which we do unknowingly, are inductive and analogy-based learning. In inductive learning, the learner makes generalizations from examples. For instance, noting that "cuckoo flies", "parrot flies" and "sparrow flies", the learner generalizes that "birds fly". On the other hand, in analogy-based learning, the learner, for example, learns the motion of electrons in an atom analogously from his knowledge of planetary motion in solar systems.
20
Fig 5: Pronunciation learning of a child from his mother. Machine learning [39] has been central to AI research from the beginning.[40] In 1956, at the original Dartmouth AI summer conference, Ray Solomonoff wrote a report on unsupervised probabilistic [41] machine learning: "An Inductive Inference Machine". Unsupervised learning is the ability to find patterns in a stream of input. Supervised learning includes both classification and numerical regression. Classification is used to determine what category something belongs in, after seeing a number of examples of things from several categories. Regression takes a set of numerical input/output examples and attempts to discover a continuous function that would generate the outputs from the inputs. In [42] reinforcementlearning the agent is rewarded for good responses and punished for bad ones. These can be analysed in terms of decision theory, using concepts like utility. The mathematical analysis of machine learning algorithms and their performance is a branch of theoretical computer science known as computationallearningtheory.
Knowledge Representation and Reasoning In a reasoning problem, one has to reach a pre-defined goal state from one or more given initial states. So, lesser the number of transitions for reaching the goal state, higher the efficiency of the reasoning system. Increasing the efficiency of a reasoning system thus requires minimization of intermediate states, which indirectly calls for an organized and complete knowledge base. A complete and organized storehouse of knowledge needs minimum search to identify the appropriate knowledge at a given problem state and thus yields the right next state on the leading edge of the problem-solving process. Organization of knowledge, therefore, is of paramount importance in knowledge engineering. A variety of knowledge representation techniques are in use in Artificial Intelligence: Production rules, semantic nets, frames, filler and slots, and
21
Predicate logic are only a few to mention.
The selection of a particular type of representational scheme of knowledge depends both on the nature of applications and the choice of users. Planning
Another significant area of artificial intelligence is planning. The problems of r easoning and planning share many common issues, but have a basic difference that originates from their definitions. The reasoning problem is mainly concerned with the testing of the satisfiability of a goal from a given set of data and knowledge. The planning problem, on the other hand, deals with the determination of the methodology by which a successful goal can be achieved from the known initial states. Automated planning finds extensive applications in robotics and navigational problems, some of which will be discussed shortly. Knowledge Acquisition
Acquisition of knowledge is equally hard for machines as it is for human beings. It includes generation of new pieces of knowledge from given knowledge base, setting dynamic data structures for existing knowledge, learning knowledge from the environment and refinement of knowledge. Automated acquisition of knowledge by machine learning approach is an active area of current research in Artificial Intelligence. Intelligent Search
Search problems, which we generally encounter in Computer Science, are of a deterministic nature, i.e., the order of visiting the elements of the search space is known. For example, in depth first and breadth first search algorithms, one knows the sequence of visiting the nodes in a tree. However, search problems, which we will come across in AI, are non-deterministic and the order of visiting the elements in the search space is completely dependent on data sets. The diversity of the intelligent search algorithms will be discussed in detail later. Logic Programming
For more than a century, mathematicians and logicians were used to designing various tools to represent logical statements by symbolic operators. One outgrowth of such attempts is propositional logic, which deals with a set of binary statements (propositions) connected by Boolean operators. The logic of propositions, which was gradually enriched to handle more complex situations of the real world, is called predicate logic. One classical variety of predicate logic-based programs is Logic Program. PROLOG, which is an abbreviation for Programming in Logic, is a typical language that supports logic programs. Logic Programming has recently been identified as one of the prime area of research in AI. The ultimate aim of this research is to extend the PROLOG compiler to handle spatial-temporal models and support a parallel programming environment. Building architecture for PROLOG machines was a hot topic of the last decade. Management of Imprecision and Uncertainty
Data and knowledgebases in many typical AI problems, such as reasoning and planning, are often contaminated with various forms of incompleteness. The incompleteness of data, hereafter called imprecision, generally appears in the database for i) lack of appropriate data, and ii) poor authenticity level of the sources.
22
The incompleteness of knowledge, often referred to as uncertainty, originates in the knowledge base due to lack of certainty of the pieces of knowledge reasoning in the presence of imprecision of data and uncertainty of knowledge is a complex problem. Various tools and techniques have been devised for reasoning under incomplete data and knowledge. Some of these techniques employ i. stochastic ii. fuzzy and iii. Belief network models. In a stochastic reasoning model, the system can have transition from one given state to a number of states, such that the sum of the probability of transition to the next states from the given state is strictly unity. In a fuzzy reasoning system, on the other hand, the sum of the membership value of transition from the given state to the next state may be greater than or equal to one. The belief network model updates the stochastic / fuzzy belief assigned to the facts embedded in the network until a condition of equilibrium is reached; following which there would be no more change in beliefs. Recently, fuzzy tools and techniques have been applied in a specialized belief network, called a fuzzy Petri net, for handling both imprecision of data and uncertainty of knowledge by a unified approach.
General Problem Solving Approaches in AI To understand what exactly artificial intelligence is we illustrate some common problems. Problems dealt with in artificial intelligence generally use a common term called 'state'. A state represents a status of the solution at a given step of the problem solving procedure. The solution of a problem, thus, is a collection of the problem states. The problem solving procedure applies an operator to a state to get the next state. Then it applies another operator to the resulting state to derive a new state. The process of applying an operator to a state and its subsequent transition to the next state, thus, is continued until the goal (desired) state is derived. Such a method of solving a problem is generally referred to as state space approach. We will first discuss the state-space approach for problem solving by a well-known problem, which most of us perhaps have solved in our childhood. Example: Consider a 4-puzzle problem, where in a 4-cell board there are 3 cells filled with digits and 1
blank cell. The initial state of the game represents a particular orientation of the digits in the cells and the final state to be achieved is another orientation supplied to the game player. The problem of the game is to reach from the given initial state to the goal (final) state, if possible, with a minimum of moves. Let the initial and the final state is as shown in figures 5(a) and (b) respectively.
Fig.6: The initial and the final states of the Number Puzzle game, where B denotes the blank space. We now define two operations, blank-up (BU) / blank-down (BD) and blank-left (BL) / blank-right (BR), and the state-space (tree) for the problem is presented below using these operators. The algorithm for the above kind of problems is straightforward. It consists of three steps, described by steps 1, 2(a) and 2(b) below.
23
Algorithm for solving state-space problems
Begin 1. state: = initial-state; existing-state:=state; 2. While state ≠ final state do Begin Apply operations from the set {BL, BR, BU, BD} to each state so as to generate new-states; If new-states ∩ the existing-states ≠ φ Then do Begin state := new-states - existing-states; Existing-states := existing-states - {states} o End; o End while; End.
It is thus clear that the main trick in solving problems by the state-space approach is to determine the set of operators and to use it at appropriate states of the problem. Researchers in artificial intelligence have segregated the AI problems from the non-AI problems. Generally, problems, for which straightforward mathematical / logical algorithms are not readily available and which can be solved by intuitive approach only, are called AI problems. The 4-puzzle problem, for instance, is an ideal AI Problem. There is no formal algorithm for its realization, i.e., given a starting and a goal state, one cannot say prior to execution of the tasks the sequence of steps required to get the goal from the starting state. Such problems are called the ideal AI problems. The wellknown water-jug problem, the Travelling Salesperson Problem (TSP), and the n-Queen problem are typical examples of the classical AI problems. Among the non-classical AI problems, the diagnosis problems and the pattern classification problem need special mention. For solving an AI problem, one may employ both artificial intelligence and non-AI algorithms. An obvious question is: what is an AI algorithm? Formally speaking, an artificial intelligence algorithm generally means a non-conventional intuitive approach for problem solving . The key to artificial intelligence approach is intelligent search and matching. In an intelligent search problem / sub-problem, given a goal (or starting) state, one has to reach that state from one or more known starting (or goal) states.For example, consider the 4-puzzle problem, where the goal state is known and one has to identify the moves for reaching the goal from a pre-defined starting state. Now, the less number of states one generates for reaching the goal, the better is the AI algorithm. The question that then naturally arises is: how to control the generation of states. This, in fact, can be achieved by suitably designing some control strategies, which would filter a few states only from a large number of legal states that could be generated from a given starting intermediate state. As an example, consider the problem of proving a trigonometric identity that children are used to doing during their schooldays. What would they do at the beginning? They would start with one side of the identity, and attempt to apply a number of formulae there to find the possible resulting derivations. But they won't really apply all the formulae there.Rather, they identify the right candidate formula that fits their best, such that the other side of the identity seems to be closer in some sense (outlook). Ultimately, when the decision regarding the selection of the formula is over, they apply it to one side (say the L.H.S) of the identity and derive the new state.
24
Fig.7: The state-space for the Four-Puzzle problem. Thus they continue the process and go on generating new intermediate states until the R.H.S (goal) is reached. But do they always select the right candidate formula at a given state? From our experience, we know the answer is "not always". But what would we do if we find that after generation of a few states, the resulting expression seems to be far away from the R.H.S of the identity. Perhaps we would prefer to move to some old state, which is more promising, i.e., closer to the R.H.S of the identity. The above line of thinking has been realized in many intelligent search problems of AI. Some of these well-known search algorithms are: a) Generate and Test Approach: This approach concerns the generation of the state-space from a known starting state (root) of the problem and continues expanding the reasoning space until the goal node or the terminal state is reached. In fact after generation of each and every state, the generated node is compared with the known goal state. When the goal is found, the algorithm terminates. In case there exist multiple paths leading to the goal, then the path having the smallest distance from the root is preferred. The basic strategy used in this search is only generation of states and their testing for goals but it does not allow filtering of states. (b) Hill Climbing Approach: Under this approach, one has to first generate a starting state and measure the total cost for reaching the goal from the given starting state. Let this cost be f. While f = a predefined utility value and the goal is not reached, new nodes are generated as children of the current node. However, in case the entireneighbourhood nodes (states) yield an identical value of f and the goal is not included in the set of these nodes, the search algorithm is trapped at a hillock or local extreme. One way to overcome this problem is to select randomly a new starting state and then continue the above search process. While proving trigonometric identities, we often use Hill Climbing, perhaps unknowingly. (C) Heuristic Search: Classically heuristics means rule of thumb. In heuristic search, we generally use one or more heuristic functions to determine the better candidate states among a set of legal states that 25
could be generated from a known state. The heuristic function, in other words, measures the fitness of the candidate states. The better the selection of the states, the fewer will be the number of intermediate states for reaching the goal. However, the most difficult task in heuristic search problems is the selection of the heuristic functions. One has to select them intuitively, so that in most cases hopefully it would be able to prune the search space correctly. (d) Means and Ends Analysis: This method of search attempts to reduce the gap between the current state and the goal state. One simple way to explore this method is to measure the distance between the current state and the goal, and then apply an operator to the current state, so that the distance between the resulting state and the goal is reduced. In many mathematical theorem- proving processes, we use Means and Ends Analysis. Symbolic Approaches
When access to digital computers became possible in the middle 1950s, AI research began to explore the possibility that human intelligence could be reduced to symbol manipulation. The research was centred in three institutions: CMU, Stanford and MIT, and each one developed its own style of research. John Haugeland named these approaches to AI "good old fashioned AI" or "GOFAI".[58] Cognitive simulation: EconomistHerbert Simon and Allen Newell studied human problem solving skills
and attempted to formalize them, and their work laid the foundations of the field of artificial intelligence, as well as cognitive science, operations research and management science. Their research team used the results of psychological experiments to develop programs that simulated the techniques that people used to solve problems. This tradition, centred at Carnegie Mellon University would eventually culminate in the development of the Soar architecture in the middle 80s.[59] Logic-based: Unlike Newell and Simon, John McCarthy felt that machines did not need to simulate human
thought, but should instead try to find the essence of abstract reasoning and problem solving, regardless of whether people used the same algorithms .[60] His laboratory at Stanford (SAIL) focused on using formal logic to solve a wide variety of problems, including knowledge representation, planning and learning.[61] Logic was also focus of the work at the University of Edinburgh and elsewhere in Europe which led to the development of the programming language Prolog and the science of logic programming.[62] "Anti-logic" or "scruffy": Researchers at MIT (such as Marvin Minsky and Seymour Papert)
[63]
found that solving difficult problems in vision and natural language processing required ad-hoc solutions – they argued that there was no simple and general principle (like logic) that would capture all the aspects of intelligent behaviour. Roger Schank described their "anti-logic" approaches as "scruffy" (as opposed to the "neat" paradigms at CMU and Stanford).[65]Common-sense knowledge bases (such as Doug Lenat'sCyc) are an example of "scruffy" AI, since they must be built by hand, one complicated concept at a time.[64] Knowledge-based: When computers with large memories became available around 1970, researchers
from all three traditions began to build knowledge into AI applications.[66] This "knowledge revolution" led to the development and deployment of expert systems (introduced by Edward Feigenbaum), the first truly successful form of AI software.[67] The knowledge revolution was also driven by the realization that enormous amounts of knowledge would be required by many simple AI applications.
26
Fig 8.AI and cybernetics Sub-symbolic approaches
During the 1960s, symbolic approaches had achieved great success at simulating high-level thinking in small demonstration programs. Approaches based on cybernetics or neural networks were abandoned or pushed into the background.[68] By the 1980s, however, progress in symbolic AI seemed to stall and many believed that symbolic systems would never be able to imitate all the processes of human cognition, especially perception, robotics, learning and pattern recognition. A number of researchers began to look into "sub-symbolic" approaches to specific AI problems.[69] Bottom-up, embodied, situated, behaviour-based or nouvelle AI: Researchers from the related field of
robotics, such as Rodney Brooks, rejected symbolic AI and focused on the basic engineering problems that would allow robots to move and survive.[70] Their work revived the non-symbolic viewpoint of the early cybernetics researchers of the 50s and reintroduced the use of control theory in AI. This coincided with the development of the embodied mind thesis in the related field of cognitive science: the idea that aspects of the body (such as movement, perception and visualization) are required for higher intelligence. Computational Intelligence: Interest in neural networks and "connectionism" was revived by David
Rumelhart and others in the middle 1980s.[94] These and other sub-symbolic approaches, such as fuzzy systems and evolutionary computation, are now studied collectively by the emerging discipline of computational intelligence. Statistical approaches
In the 1990s, AI researchers developed sophisticated mathematical tools to solve specific sub problems. These tools are truly scientific, in the sense that their results are both measurable and verifiable, and they have been responsible for many of AI's recent successes. The shared mathematical language has also permitted a high level of collaboration with more established fields (like mathematics, economics or operations research). Stuart Russell and Peter Norvig describe this movement as nothing less than a "revolution" and "the victory of the nets."[71] Critiques argue that these techniques are too focussed on particular problems and have failed to address the long term goal of general intelligence.
27
Fig 9 Computational Intelligence Integrating the approaches Intelligent agent paradigm: An intelligent agent is a system that perceives its environment and takes
actions which maximizes its chances of success. The simplest intelligent agents are programs that solve specific problems. More complicated agents include human beings and organizations of human beings (such as firms). The paradigm gives researchers license to study isolated problems and find solutions that are both verifiable and useful, without agreeing on one single approach. An agent that solves a specific problem can use any approach that works — some agents are symbolic and logical, some are subsymbolic neural networks and others may use new approaches. The paradigm also gives researchers a common language to communicate with other fields—such as decision theory and economics—that also use concepts of abstract agents. The intelligent agent paradigm became widely accepted during the 1990s.[72] Agent architectures and cognitive architectures: Researchers have designed systems to build intelligent
systems out of interacting intelligent agents in a multi-agent system.[73] A system with both symbolic and sub-symbolic components is a hybrid intelligent system, and the study of such systems is artificial intelligence systems integration. A hierarchical control system provides a bridge between sub-symbolic AI at its lowest, reactive levels and traditional symbolic AI at its highest levels, where relaxed time constraints permit planning and world modelling.[74]Rodney Brooks' subsumption architecture was an early proposal for such a hierarchical system.[75]
Tools In the course of 50 years of research, AI has developed a large number of tools to solve the most difficult problems in computer science. A few of the most general of these methods are discussed below. Search and optimization
Many problems in AI can be solved in theory by intelligently searching through many possible solutions: [76] Reasoning can be reduced to performing a search. For example, logical proof can be viewed as searching for a path that leads from premises to conclusions, where each step is the application of an inference rule.[77]Planning algorithms search through trees of goals and sub goals, attempting to find a path to a target goal, a process called means-ends analysis.[78]Robotics algorithms for moving limbs and 28
grasping objects use local searches in configuration space.[69] Many learning algorithms use search algorithms based on optimization. Simple exhaustive searches [79] are rarely sufficient for most real world problems: the search space (the number of places to search) quickly grows to astronomical numbers. The result is a search that is too slow or never completes. The solution, for many problems, is to use "heuristics" or "rules of thumb" that eliminate choices that are unlikely to lead to the goal (called "pruning the search tree"). Heuristics supply the program with a "best guess" for what path the solution lies on.[80] A very different kind of search came to prominence in the 1990s, based on the mathematical theory of optimization. For many problems, it is possible to begin the search with some form of a guess and then refine the guess incrementally until no more refinements can be made. These algorithms can be visualized as blind hill climbing: we begin the search at a random point on the landscape, and then, by jumps or steps, we keep moving our guess uphill, until we reach the top. Other optimization algorithms are [81] simulated annealing, beam search and random optimization. Evolutionary computation uses a form of optimization search. For example, they may begin with a population of organisms (the guesses) and then allow them to mutate and recombine, selecting only the fittest to survive each generation (refining the guesses). Forms of evolutionary computation include [82] swarm intelligence algorithms (such as ant colony or particle swarm optimization) and evolutionary [83] algorithms (such as genetic algorithms and genetic programming). Logic
Logic [85] is used for knowledge representation and problem solving, but it can be applied to other problems as well. For example, the sat plan algorithm uses logic for planning[84] and inductive logic programming is a method for learning.[86] Several different forms of logic are used in AI research. Propositional or sentential logic [87] is the logic of statements which can be true or false. First-order logic [88] also allows the use of quantifiers and predicates, and can express facts about objects, their properties, and their relations with each other. [89] Fuzzy logic , is a version of first-order logic which allows the truth o f a statement to be represented as a value between 0 and 1, rather than simply true (1) or False (0). Fuzzy systems can be used for uncertain reasoning and have been widely used in modern industrial and consumer product control systems. [90] Subjective logic models uncertainty in a different and more explicit manner than fuzzy-logic: a given binomial opinion satisfies belief + disbelief + uncertainty = 1 within a Beta distribution. By this method, ignorance can be distinguished from probabilistic statements that an agent makes with high confidence. [51] are forms of logic designed to help with Default logics, non-monotonic logics and circumscription
default reasoning and the qualification problem. Several extensions of logic have been designed to handle specific domains of knowledge, such as: description logics;[45] situation calculus, event calculus and fluent calculus (for representing events and time);[46] causal calculus;[47] belief calculus; and modal logics.[48]
29
Probabilistic methods for uncertain reasoning
Many problems in AI (in reasoning, planning, learning, perception and robotics) require the agent to operate with incomplete or uncertain information. AI researchers have devised a number of powerful tools to solve these problems using methods from probability theory and economics.[91] [92]
are a very general tool that can be used for a large number of problems: reasoning (using the Bayesian inference algorithm),[94] learning (using the expectation-maximization algorithm),[93] planning (using decision networks)[95] and perception (using dynamic Bayesian networks).[96]
Bayesian networks
Probabilistic algorithms can also be used for filtering, prediction, smoothing and finding explanations for
streams of data, helping perception systems to analyse processes that occur over time (e.g., hidden Markov models or Kalman filters).[97] A key concept from the science of economics is "utility": a measure of how valuable something is to an intelligent agent. Precise mathematical tools have been developed that analyse how an agent can make choices and plan, using decision theory, decision analysis,[98]information value theory.[57] These tools include models such as Markov decision processes,[99]dynamic decision networks,[100]game theory and [101] mechanism design. Classifiers and statistical learning methods
The simplest AI applications can be divided into two types: classifiers ("if shiny then diamond") and controllers ("if shiny then pick up"). Controllers do however also classify conditions before inferring actions, and therefore classification forms a central part of many AI systems. Classifiers are functions that use pattern matching to determine a closest match. They can be tuned according to examples, making them very attractive for use in AI. These examples are known as observations or patterns. In supervised learning, each pattern belongs to a certain predefined class. A class can be seen as a decision that has to be made. All the observations combined with their class labels are known as a data set. When a new observation is received, that observation is classified based on previous experience.[102] A classifier can be trained in various ways; there are many statistical and machine learning approaches. The most widely used classifiers are the neural network,[103] kernel methods such as the support vector machine,[104] k-nearest neighbour algorithm,[105] Gaussian mixture model,[106] naive Bayes classifier,[107] and Decision tree.[108]
Theperformance of these classifiers has been compared over a wide range of tasks. Classifier performance depends greatly on the characteristics of the data to be classified. There is no single classifier that works best on all given problems; this is also referred to as the "no free lunch" theorem. Determining a suitable classifier for a given problem is still more an art than science.[109]
Data Mining and its applications Introduction
"Data mining is the process of discovering meaningful new correlations, patterns, and trends by sifting through large amounts of data stored in repositories, using pattern recognition technologies as well as
30
statistical and mathematical techniques". However, really data mining turns databases into knowledge bases which are one of the fundamental components of expert systems. Instead of the computer just
blindly pulling data from a database, the computer is able to take all the data and interpret it, which is a huge step to make. If it was not for existing AI technologies this field could not have emerged as quickly; if at all. Data mining allows companies to focus on the more important information in their data warehouses. Data mining can be broken down into two major categories: Automated prediction of trends and behaviours, and Automated discovery of previously unknown patterns. In the first category, data mining automates the process of finding predictive information in large databases. Questions that traditionally required exhaustive hands-on analysis can now be quickly answered directly from data. In the second category, data mining tools sweep through databases and identify previously hidden patterns in one step. This category is where the major focus of research has been on. "Data mining is a rather new term for a challenge that has been growing for many years: how to scan very large databases to retrieve the high level conceptual information of the greatest interest " (Lindsay). With the advances in data acquisition and storage technologies, the problem of how to turn measured raw data into useful information becomes an important one. Having reached sizes that defy even partial examination by humans, the data volumes are literally swamping users. For example, large US retail chains now mine their data bases with sophisticated data mining programs to look for general trends and geographic clustering in purchases that are not easily visible in the huge multitude of products and sales. Data mining has come from an evolution of searching through data trying to find useful business information. There are four major steps: Data Collection, Data Access, Data Warehousing & Decision Support, and finally Data Mining (Pilot) Data Mining is an analytic process designed to explore data in search of consistent patterns and/or systematic relationships between variables, and then to validate the findings by applying the detected patterns to new subsets of data. The ultimate goal of data mining is prediction - and predictive data mining is the most common type of data mining and one that has the most direct business applications. • • • •
Data Mining (DM) seeks to discover new information or “knowledge” from (very) large databases. Knowledge is represented in the form of statistical rules and patterns. It differs from machine learning in that it deals with large amounts of data stored primarily on disk (rather than in main memory). Knowledge discovered from a database can be represented by a set of rules. Such rules can be discovered using one of two methods: i. User is involved directly in the process of knowledge discovery ii. The DM system is responsible for automatically discovering knowledge from the database, by detecting patterns and correlations in the data.
Example
• Credit ratings/targeted marketing:
31
i.
Given a database of 100,000 names, which persons are the least likely to default on their credit cards? ii. Identify likely responders to sales promotions • Fraud detection i. Which types of transactions are likely to be fraudulent, given the demographics and transactional history of a particular customer? • Customer relationship management: i. Which of my customers are likely to be the most loyal, and which are most likely to leave for a competitor? History
Data Collection started in the 1960s. This is a static data delivery system that came from pulling information from computers, tapes, and disks. For example, what is the total revenue in the last five years? Data Access is the next step and it started in the 1980s. This allowed dynamic data delivery at the record level. Data access mainly uses relational databases using SQL. A Sample Query would be: What were unit sales in Florida last October. Then in the 1990s came Data warehousing and decision support. This allowed dynamic data delivery at multiple levels. This technology came about, because of multidimensional databases and on-line analytic processing (OLAP). This will let the query above go as detailed as city to city in Florida. Finally came data mining, which allowed proactive information delivery . Data mining uses Advanced AI algorithms, multiprocessor computers, and massive databases. With data mining a person ask questions like what is likely to happen to Florida unit sales next month and why (Pilot). Fundamentals
Data, Information, and Knowledge
Data: Data are any facts, numbers, or text that can be processed by a computer. Today,
organizations are accumulating vast and growing amounts of data in different formats and different databases. This includes: • operational or transactional data such as, sales, cost, inventory, payroll, and accounting • non-operational data, such as industry sales, forecast data, and macro-economic data • meta data - data about the data itself, such as logical database design or data dictionary definitions Information: The patterns, associations, or relationships among all this data can provide information. For example, analysis of retail point of sale transaction data can yield information on which products are selling and when. Knowledge: Information can be converted into knowledge about historical patterns and future trends. For example, summary information on retail supermarket sales can be analysed in light of promotional efforts to provide knowledge of consumer buying behaviour. Thus, a manufacturer or retailer could determine which items are most susceptible to promotional efforts. Data Warehouses: A decision support database that is maintained separately from the organization’s operational databases.A data warehouse is a • Subject-oriented, • integrated, • Time-varying, • Non-volatile
32
Collection of data that is used primarily in organizational decision making Dramatic advances in data capture, processing power, data transmission, and storage capabilities are enabling organizations to integrate their various databases into data warehouses. Data warehousing is defined as a process of centralized data management and retrieval. Data warehousing, like data mining, is a relatively new term although the concept itself has been around for years. Data warehousing represents an ideal vision of maintaining a central repository of all organizational data. Centralization of data is needed to maximize user access and analysis. Dramatic technological advances are making this vision a reality for many companies. And, equally dramatic advances in data analysis software are allowing users to access this data freely. The data analysis software is what supports data mining. Necessity
The main reason for the necessity of automated computer systems for intelligent data analysis is the enormous volume of existing and newly appearing data that requires processing. The amount of data accumulated each day by various businesses, scientific, and governmental organizations around the world is appalling. According to GTE research, scientific organizations store about 1 terabyte of new information each day (Mega computer). It is impossible for human analysts to cope with such overwhelming amounts of data. Two problems that surface when human analysts process data are the inadequacy of the human brain when searching for complex dependencies in data, and The lack of objectiveness in their analysis.
Therefore, one of the benefits of using automated data mining systems is that this process has a much lower cost than hiring an army of highly trained and paid professional statisticians. Although data mining does not completely eradicate the need for humans, it allows an analyst who has no programming and statistics skill to extract knowledge from databases (Mega computer). Continuous Innovation
Although data mining is a relatively new term, the technology is not. Companies have used powerful computers to sift through volumes of supermarket scanner data and analyse market research reports for years. However, continuous innovations in computer processing power, disk storage, and statistical software are dramatically increasing the accuracy of analysis while driving down the cost. Example
One Midwest grocery chain used the data mining capacity of Oracle software to analyse local buying patterns. They discovered that when men bought diapers on Thursdays and Saturdays, they also tended to buy beer. Further analysis showed that these shoppers typically did their weekly grocery shopping on Saturdays. On Thursdays, however, they only bought a few items. The retailer concluded that they purchased the beer to have it available for the upcoming weekend. The grocery chain could use this newly discovered information in various ways to increase revenue. For example, they could move the beer display closer to the diaper display. And, they could make sure beer and diapers were sold at full price on Thursdays. Categories of Data Mining
There are two types of categories of data mining. Predictive models use previous customer interactions to predict future events while segmentation techniques are used to place customers with similar behaviours and attributes into distinct groups. This grouping can help marketers to optimize their campaign management and targeting processes. • Police-enforced ANPR in the UK
33
The HUKH has an extensive automatic number plate recognition (ANPR) HCCTVH network. Police and security services use it to track UK vehicle movements in real time. The resulting data are stored for 5 years in the National ANPR Data Centre to be analysed for intelligence and to be used as evidence. A major feature of the National ANPR Data Centre for car numbers is the ability to data mine. Advanced versatile automated data mining software trawls through the vast amounts of data collected, finding patterns and meaning in the data. Data mining can be used on the records of previous sightings to build up intelligence of a vehicle's movements on the road network or can be used to find cloned vehicles by searching the database for impossibly quick journeys. ANPR can be used on investigations. • Stellar wind (code name)
Stellar Wind is the open secret code name for certain information collection activities performed by the United States' National Security Agency. The information collection activities involved data mining electronic data about tens of millions of American citizens within the United States. This data included information about e-mail communications, phone conversations, financial transactions, and internet activity. Data Mining helps extract such information
• Process of semi-automatically analysing large databases to find patterns that are: i. ii. iii. iv.
valid: hold on new data with some certainty novel: non-obvious to the system useful: should be possible to act on the item understandable: humans should be able to interpret the pattern
• Also known as Knowledge Discovery in Databases (KDD) Knowledge Representation using Rules • General form of rules: antecedent =_ consequent • Example: Market Basket Analysis Market Basket ≡ collection of items purchased by customer in a single customertransaction (single visit to a store or through mail-order catalog)Idea: Use DM to identify sets of items that are purchased together _ information can beused to improve layout of goods in a store or catalog. Association Rules
Examining set of transactions yields the rule{pen} =_ {ink},i.e., if a pen is purchased in a transaction, it is likely that ink will also be purchased in thetransaction. Such a rule is called an association rule. General form:LHS =_ RHS, where both RHS and LHS are sets of items Table 1 Association Rules
34
Important measures for an association rule:
Support for a set of items is the percentage of transactions that contain all items in LHS-RHS (75% for the above rule).If support is low (as, e.g., for {diary} =_ {soap} [25%]), then there is not enoughevidence to draw conclusion about correlation between items in LHS and items in RHS. Confidence: Consider transactions that contain all items in LHS. The confidence is thepercentage of transactions that also contain all items of the RHS (75% for the above rule). Data Mining Process
With the enormous amount of data stored in files, databases, and other repositories, it is increasingly important, if not necessary, to develop powerful means in the forms of algorithms for analysis and interpretation of such data and for the extraction of interesting knowledge that could help in decisionmaking. This whole process is termed as machine learning. Machine learning can be defined as programming computers to optimize a performance criterion (prediction or decision to act) using example data or past experiences. Data mining is application of machine learning methods to large databases for inference. The terms knowledge discovery in databases (KDD) and data mining are often used interchangeably. In fact, there have been many other names given to this process of discovering useful (hidden) patterns in data: knowledge extraction, information discovery, exploratory data analysis, information harvesting, and unsupervised pattern recognition [MHD]. Over the last few years KDD has been used to refer to a process consisting of many steps, while data mining is only one of these steps. Knowledge discovery in databases (KDD) is the process of finding useful information and pattern in data. Data mining is the use of algorithms to extract the information and patterns derived by the KDD process. 35
KDD is a process that involves many different steps. The input to this process is the data, and the output is the useful information desired by the users. However, the objective may be unclear or inexact. The process itself is interactive and may require much elapsed time. To ensure the usefulness and accuracy of the results of the process, interaction throughout the process with both domain experts and technical experts might be needed.
Figure 10: KDD Process The KDD process consists of the following five steps: Selection: The data needed for the data mining process may be obtained from many different and
heterogeneous data sources such as spread-sheets, document files and other data warehouses. Data Warehouse is a storehouse; a repository of data collected from multiple data sourcesand is intended to be used as a whole under the same unified schema. Pre-processing: The data to be used by the process may have incorrect or missing data. There may be
anomalous data from multiple sources involving different data types and metrics. Erroneous data may be corrected or removed, whereas missing data can be supplied or predicted (often using data mining tools). Transformation: Data from different sources must be converted into a common format for processing.
Some data may be encoded or transformed into more usable formats. Data reduction may be used to reduce the number of possible data values being considered Initial Target Data Pre-processed Data Transformed MODEL Knowledge Selection Pre-processing Transformation Data Interpretation Data mining: Based on the machine learning tasks being performed, this step applies algorithms to the
transformed data to generate the desired results. Interpretation/Evaluation: How the data mining results are presented to the users is extremely important
because the usefulness of the results is dependent on it. Various visualization and GUI strategies are used at this last step. Basic Data Mining Model
Data mining involves many different algorithms to accomplish different tasks. All these algorithms attempt to fit a model to the data. The algorithms examine the data and determine a model that is closest to the characteristics of the data being examined. Data mining algorithms can be characterized as consisting of three parts.
Model: The purpose of the algorithm is to fit a model to the data.
Preference: Some criteria must be used to fit one model over another.
Search: All algorithms require some technique to search the data.
36
As seen in Figure 13, the model that is created can be either predictive or descriptive in nature. This figure depicts, under each model type, some of the most common data mining tasks which use that type of model.
FIGURE 11: Data mining models and tasks A predictive model makes a prediction about values of data using known results found from different data. Predictive Modelling may be made based on the use of other historical data. For example, a credit card use might be refused not because of the user's Data Mining Predictive Descriptive Model
Classification Time Series analysis Regression Prediction Clustering Summarization Association rule Sequence Discovery own credit history, but because the current purchase is similar to earlier purchases that were subsequently found to be made with stolen cards. Predictive model data mining tasks include classification, regression, time series analysis, and prediction.
A descriptive model identifies patterns or relationships in data. Unlike the predictive model, a descriptive model serves as a way to explore the properties of the data examined, not to predict new properties. Clustering, summarization, association rules, and sequence discovery are some of the tasks for these models. What can data mining do?
Data mining is primarily used today by companies with a strong consumer focus - retail, financial, communication, and marketing organizations. It enables these companies to determine relationships among "internal" factors such as price, product positioning, or staff skills, and "external" factors such as economic indicators, competition, and customer demographics. And, it enables them to determine the impact on sales, customer satisfaction, and corporate profits. Finally, it enables them to "drill down" into summary information to view detail transactional data. With data mining, a retailer could use point-of-sale records of customer purchases to send targeted promotions based on an individual's purchase history. By mining demographic data from comment or warranty cards, the retailer could develop products and promotions to appeal to specific customer segments. 37
For example, Blockbuster Entertainment mines its video rental history database to recommend rentals to individual customers. American Express can suggest products to its cardholders based on analysis of their monthly expenditures. Wal-Mart is pioneering massive data mining to transform its supplier relationships. Wal-Mart captures point-of-sale transactions from over 2,900 stores in 6 countries and continuously transmits this data to its massive 7.5 terabyte Teradata data warehouse. Wal-Mart allows more than 3,500 suppliers, to access data on their products and performs data analyses. These suppliers use this data to identify customer buying patterns at the store display level. They use this information to manage local store inventory and identify new merchandising opportunities. In 1995, Wal-Mart computers processed over 1 million complex data queries. The National Basketball Association (NBA) is exploring a data mining application that can be used in conjunction with image recordings of basketball games. The AdvancedScout software analyses the movements of players to help coaches orchestrate plays and strategies. For example, an analysis of the play-by-play sheet of the game played between the New York Knicks and the Cleveland Cavaliers on January 6, 1995 reveals that when Mark Price played the Guard position, John Williams attempted four jump shots and made each one! Advanced Scout not only finds this pattern, but explains that it is interesting because it differs considerably from the average shooting percentage of 49.30% for the Cavaliers during that game. By using the NBA universal clock, a coach can automatically bring up the video clips showing each of the jump shots attempted by Williams with Price on the floor, without needing to comb through hours of video footage. Those clips show a very successful pick-and-roll play in which Price draws the Knick'sdefence and then finds Williams for an open jump shot. Data Mining Tasks
In this part we will briefly explore some of the data mining functions. We follow the basic outline of tasks shown in Figure 13. This list is not intended to be exhaustive, but rather illustrative. Of course, these individual tasks may be combined to obtain more sophisticated data mining applications 1.
Classification
Classification maps data into predefined groups or classes. It is often referred to as supervised learning because the classes are determined before examining the data. Classification algorithms require that the classes be defined based on data attribute values. They often describe these classes by looking at the characteristics of data already known to belong to the classes. Pattern recognition is a type of classification where an input pattern is classified into one of several classes based on its similarity to these predefined classes. A simple example of pattern recognition is given in Example 1 Example 1
Classification Problem can be defined mathematically as Given a database D= t1, t2…tn- and a set of classes C= c1, c2…cm-, The Classification Problem is to define a mapping f: D_C, such that each ti is assigned to one class cj; where i=1, 2…n, and j=1, 2…m. An airport security screening station is used to determine if passengers are potential terrorists or criminals. To do this, the face of each passenger is scanned and its basic pattern (distance between eyes, size and shape of mouth, shape of head, etc.) is identified. This pattern is compared to entries in a database to see if it matches any patterns that are associated with known offenders. 2.
Regression
Regression is used to map a data item to a real valued prediction variable. In actuality, regression involves the learning of the function that does this mapping. Regression assumes that the target data fit into some known type of function (e.g., linear, logistic, etc.) and then determines the best function of this type that 38
models the given data. Some type of error analysis is used to determine which function is "best" Standard linear regression, as illustrated in Example 2, is a simple example of regression. Example 2
A college professor wishes to reach a certain level of savings before her retirement. Periodically, she predicts what her retirement savings will be, based on its current value and several past values. She uses a simple linear regression formula to predict this value by fitting past behaviour to a linear function and then using this function to predict the values at points in the future. Based on these values, she then alters her investment portfolio. 3.
Time Series Analysis
With time series analysis, the value of an attribute is examined as it varies over time. The values usually are obtained as evenly spaced time points (daily, weekly, hourly, etc.). A time series plot (Figure 1.3) is used to visualize the time series of various events. In this figure you can easily see that the plots for Y and Z have variation in their behaviour but time series for X shows less variation. There are three basic functions performed in time series analysis. In first case, distance measures are used to determine the similarity between different time series. In the second case, the structure of the line is examined to determine (and perhaps classify) its behaviour. In third case it uses the historical time series plot to predict future values. A time series example is given in Example 3 Example 3:
Mr. Smith is trying to determine whether to purchase stock from Companies X, Y, or Z. For a period of one month he charts the daily stock price for each company. Figure 1.3 shows the time series plot that Mr. Smith has generated.
FIGURE 12 Time Series plots [BL] Using this and similar information available from his stockbroker, Mr. Smith decides to purchase stock X because it is less volatile while overall showing a slightly larger relative amount of growth than either of the other stocks. As a matter of fact, the stocks for Y and Z have a similar behaviour. 4.
Prediction
Many real-world data mining applications can be seen as predicting future data states based on past and current data. Prediction can be viewed as a type of classification. The difference is that prediction is predicting a future state rather than a current state. Prediction applications include flooding, speech recognition, machine learning, and pattern recognition. Although future values may be predicted using
39
time series analysis or regression techniques, other approaches may be used as well. Example 4illustrates the process Example 4
Predicting flooding is a difficult problem. One approach uses monitors placed at various points in the river. These monitors collect data relevant to flood prediction: water level, rain amount, time, humidity, and so on. Then the water level at a potential flooding point in the river can be predicted based on the data collected by the sensors upriver from this point. The prediction must be made with respect to the time the data were collected. 5.
Clustering
Clustering is similar to classification except that the groups are not predefined, but rather defined by the data and algorithm. Clustering is alternatively referred to as unsupervised learning or segmentation. It can be thought of as partitioning or segmenting the data into groups that might or might not be disjointed. The clustering is usually accomplished by determining the similarity among the data on predefined attributes. The most similar data are grouped into clusters. Example 5 provides a simple clustering example. Since the clusters are not predefined, a domain expert is often required to interpret the meaning of the created clusters. Example 5
A certain national departmental store chain creates special catalogs targeted to various demographic groups based on attributes such as income, location, and physical characteristics of potential customers (age, height, weight, etc.). To determine the target mailings of the various catalogs and to assist in the creation of new, more specific catalogs, the company performs a clustering of potential customers based on the determined attribute values. The results of the clustering exercise are then used by management to create special catalogs and distribute them to the correct target population based on the cluster for that catalog. 6.
Association Rules
Link analysis, alternatively referred to as affinity analysis or association, refers to the data-mining task of uncovering relationships among data. The best example of this type of application is to determine association rules. An association rule is a model that identifies specific types of data associations. These associations are often used in the retail sales community to identify items that are frequently purchased together. Example 4.6 illustrates the use of association rules in market basket analysis. Here the data analysed consist of information about what items a customer purchases Example 6
A grocery store retailer is trying to decide whether to put bread on sale. To help determine the impact of this decision, the retailer generates association rules that show what other products are frequently purchased with bread. He finds that 60% of the times that bread is sold so are pretzels and that 70% of the time jelly is also sold. Based on these facts, he tries to capitalize on the association between bread, pretzels, and jelly by placing some pretzels and jelly at the end of the aisle where the bread is placed. In addition, he decides not to place either of these items on sale at the same time Users of association rules must be cautioned that these are not causal relationships. They do not represent any relationship inherent in the actual data (as is true with functional dependencies) or in the real world. There probably is no relationship between bread and pretzels that causes them to be purchased together. And there is no guarantee that this association will apply in the future. However, association rules can be used to assist retail store management in effective advertising, marketing, and inventory control.
40
Data mining consists of five major elements:
• • • • •
Extract, transform, and load transaction data onto the data warehouse system. Store and manage the data in a multidimensional database system. Provide data access to business analysts and information technology professionals. Analyse the data by application software. Present the data in a useful format, such as a graph or table.
Different levels of analysis are available:
• • •
•
• •
Artificial neural networks: Non-linear predictive models that learn through training and resemble
biological neural networks in structure. Genetic algorithms: Optimization techniques that use process such as genetic combination, mutation, and natural selection in a design based on the concepts of natural evolution. Decision trees: Tree-shaped structures that represent sets of decisions. These decisions generate rules for the classification of a dataset. Specific decision tree methods include Classification and Regression Trees (CART) and Chi Square Automatic Interaction Detection (CHAID). CART and CHAID are decision tree techniques used for classification of a dataset. They provide a set of rules that you can apply to a new (unclassified) dataset to predict which records will have a given outcome. CART segments a dataset by creating 2-way splits while CHAID segments using chi square tests to create multi-way splits. CART typically requires less data preparation than CHAID. Nearest neighbour method: A technique that classifies each record in a dataset based on a combination of the classes of the k record(s) most similar to it in a historical dataset (where k 1). Sometimes called the k-nearest neighbour technique. Rule induction: The extraction of useful if-then rules from data based on statistical significance. Data visualization: The visual interpretation of complex relationships in multidimensional data. Graphics tools are used to illustrate data relationships.
What technological infrastructure is required?
Today, data mining applications are available on all size systems for mainframe, client/server, and PC platforms. System prices range from several thousand dollars for the smallest applications up to $1 million a terabyte for the largest. Enterprise-wide applications generally range in size from 10 gigabytes to over 11 terabytes. HNCRH has the capacity to deliver applications exceeding 100 terabytes. There are two critical technological drivers: •
Size of the database: the more data being processed and maintained, the more powerful the system
required. •
Query complexity: the more complex the queries and the greater the number of queries being
processed, the more powerful the system required. Relational database storage and management technology is adequate for many data mining applications less than 50 gigabytes. However, this infrastructure needs to be significantly enhanced to support larger applications. Some vendors have added extensive indexing capabilities to improve query performance. Others use new hardware architectures such as Massively Parallel Processors (MPP) to achieve order-ofmagnitude improvements in query time. For example, MPP systems from NCR link hundreds of highspeed Pentium processors to achieve performance levels exceeding those of the largest supercomputers. Applications
Banking: loan/credit card approval; predict good customers based on old customers
Customer relationship management:identify those who are likely to leave for a competitor.
41
Targeted marketing: identify likely responders to promotions
Fraud detection: telecommunications, financial transactions from an online stream of event
identify fraudulent events
Manufacturing and production: automatically adjust knobs when process parameter changes
Medicine: disease outcome, effectiveness of treatments analyse patient disease history: find
relationship between diseases Molecular/Pharmaceutical:identify new drugs Scientific data analysis: identify new galaxies by searching for sub clusters Web site/store design and promotion: find affinity of visitor to pages and modify layout
Other Applications include: • • • • •
Surveillance / Mass surveillance National Security Agency Customer analytics Police-enforced ANPR in the UKH Stellar wind (code name)
Surveillance is the monitoring of the behaviour of a person or group of people, often in a surreptitious manner from a distance by means of electronic equipment (such as CCTV cameras), or interception of electronically transmitted information (such as Internet traffic or phone calls).
Artificial Neural Networks and Fuzzy Logic Introduction to Artificial Neural Networks
An Artificial Neural Network (ANN) is an information processing paradigm that is inspired by the way biological nervous systems, such as the brain, process information. The key element of this paradigm is the novel structure of the information processing system. It is composed of a large number of highly interconnected processing elements (neurons) working in unison to solve specific problems. ANNs, like people, learn by example. An ANN is configured for a specific application, such as pattern recognition or data classification, through a learning process. Artificial Neural Network, as the name implies, is a man-made network formed by neurons and is an effort to replicate the animal brain. Neural networks are a form of multiprocessor computer system, with
• • • •
Simple processing elements A high degree of interconnection Simple scalar messages Adaptive interaction between elements
A biological neuron may have as many as 10,000 different inputs, and may send its output depending on the presence or absence of a short-duration spike to many other neurons. Real brains, however, are orders of magnitude more complex than any artificial neural network. An artificial neural network is a system based on the operation of biological neural networks, in other words, is an emulation of biological neural system. Why would be necessary the implementation of artificial neural networks? Although computing these days is truly advanced, there are certain tasks that a program made for a common microprocessor is unable to perform; even so a software implementation of a neural network can be made with their advantages and disadvantages. Advantages:
42
A neural network can perform tasks that a linear program cannot. When an element of the neural network fails, it can continue without any problem by their parallel nature. A neural network learns and does not need to be reprogrammed. It can be implemented in any application. It can be implemented without any problem.
Disadvantages:
The neural network needs training to operate. The architecture of a neural network is different from the architecture of microprocessors therefore needs to be emulated. Requires high processing time for l arge neural networks.
Another aspect of the artificial neural networks is that there are different architectures, which consequently requires different types of algorithms, but despite to be an apparently complex system, a neural network is relatively simple. Artificial neural networks are among the newest signal processing technologies nowadays. The field of work is very interdisciplinary, but the explanation I will give you here will be restricted to an engineering perspective. In the world of engineering, neural networks have two main functions: Pattern classifiers and as non-linear adaptive filters. As its biological predecessor, an artificial neural network is an adaptive system. By adaptive, it means that each parameter is changed during its operation and it is deployed for solving the problem in matter. This is called the training phase. An artificial neural network is developed with a systematic step-by-step procedure which optimizes a criterion commonly known as the learning rule. The input/output training data is fundamental for these networks as it conveys the information which is necessary to discover the optimal operating point. In addition, a non-linear nature makes neural network processing elements a very flexible system. Basically, an artificial neural network is a system. A system is a structure that receives an input, process the data, and provides an output. Commonly, the input consists in a data array which can be anything such as data from an image file, a WAVE sound or any kind of data that can be represented in an array. Once an input is presented to the neural network, and a corresponding desired or target response is set at the output, an error is composed from the difference of the desired response and the real system output. The error information is fed back to the system which makes all adjustments to their parameters in a systematic fashion (commonly known as the learning rule). This process is repeated until the desired output is acceptable. It is important to notice that the performance hinges heavily on the data. Hence, this is why this data should pre-process with third party algorithms such as DSP algorithms. In neural network design, the engineer or designer chooses the network topology, the trigger function or performance function, learning rule and the criteria for stopping the training phase. So, it is pretty difficult determining the size and parameters of the network as there is no rule or formula to do it. The best we can do for having success with our design is playing with it. The problem with this method is when the system does not work properly it is hard to refine the solution. Despite this issue, neural networks based solution is very efficient in terms of development, time and resources. By experience, I can tell that artificial neural networks provide real solutions that are difficult to match with other technologies. Fifteen years ago, Denker said: “artificial neural networks are the second best way to implement a solution” this motivated by their simplicity, design and universality. Nowadays, neural ne twork technologies are emerging as the technology choice for many applications, such as patter recognition, prediction, system identification and control.
43
Types of Artificial Neural Networks (ANN)
Feedforward Neural Network – The feedforward neural network was the first and arguably most
simple type of artificial neural network devised. In this network the information moves in only one direction — forwards: From the input nodes data goes through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network. Radial Basis Function (RBF) Neural Network – Radial basis functions are powerful techniques for interpolation in multidimensional space. A RBF is a function which has built into a distance criterion with respect to a centre. RBF neural networks have the advantage of not suffering from local minima in the same way as Multi-Layer Perceptrons. RBF neural networks have the disadvantage of requiring good coverage of the input space by radial basis functions. Kohonen Self-organizing Neural Network – The self-organizing map (SOM) performs a form of unsupervised learning. A set of artificial neurons learn to map points in an input space to coordinates in an output space. The input space can have different dimensions and topology from the output space and the SOM will attempt to preserve these. Learning Vector Quantization Neural Network – Learning Vector Quantization (LVQ) can also be interpreted as a neural network architecture. In LVQ, prototypical representatives of the classes parameterize, together with an appropriate distance measure a distance-based classification scheme. Recurrent Neural Networks – Recurrent neural networks (RNNs) are models with bi-directional data flow. Recurrent neural networks can be used as general sequence processors. Various types of Recurrent neural networks are fully recurrent network (Hopfield network and Boltzmann machine), Simple recurrent networks, Echo state network, Long short term memory network, Bi-directional RNN, Hierarchical RNN, and Stochastic neural networks. Modular Neural Network – Biological studies have shown that the human brain functions not as a single massive network, but as a collection of small networks. This realization gave birth to the concept of modular neural networks, in which several small networks cooperate or compete to solve problems. Physical Neural Network – A physical neural network includes electrically adjustable resistance material to simulate artificial synapses.
Other Special Types of Neural Networks
Holographic associative memory – Holographic associative memory represents a family of
analog, correlation-based, associative, stimulus-response memories, where information is mapped onto the phase orientation of complex numbers operating. Instantaneously Trained Neural Networks – Instantaneously trained neural networks (ITNNs) were inspired by the phenomenon of short-term learning that seems to occur instantaneously. Spiking Neural Networks – Spiking neural networks (SNNs) are models which explicitly take into account the timing of inputs. The network input and output are usually represented as series of spikes (delta function or more complex shapes). SNNs have an advantage of being able to process information in the time domain (signals that vary over time). Dynamic Neural Networks – Dynamic neural networks not only deal with nonlinear multivariate behaviour, but also include (learning of) time-dependent behaviour such as various transient phenomena and delay effects. Cascading Neural Networks – Cascade Correlation is an architecture and supervised learning algorithm. Instead of just adjusting the weights in a network of fixed topology, CascadeCorrelation begins with a minimal network, then automatically trains and adds new hidden units one by one, creating a multi-layer structure.
44
Neuro-Fuzzy Neural Networks – A neuro-fuzzy network is a fuzzy inference system in the body of
an artificial neural network. Depending on the FIS type, there are several layers that simulate the processes involved in a fuzzy inference like fuzzification, inference, aggregation and defuzzification. Embedding an FIS in a general structure of an ANN has the benefit of using available ANN training methods to find the parameters of a fuzzy system. Compositional Pattern-producing Neural Networks – Compositional pattern-producing networks (CPPNs) are a variation of ANNs which differ in their set of activation functions and how they are applied. While typical ANNs often contain only sigmoid functions (and sometimes Gaussian functions), CPPNs can include both types of functions and many others.
A Neural network with two layers
Computational models require a mathematically and logically formal representation of a problem. Computer models are used in the simulation and experimental verification of different specific and general properties of intelligence. Computational Modelling can help us to understand the functional organization of a particular cognitive phenomenon. There are two basic approaches to cognitive Modelling. The first is focused on abstract mental functions of an intelligent mind and operates using symbols, and the second, which follows the neural and associative properties of the human brain, and is called subsymbolic. • Symbolic Modelling evolved from the computer science paradigms using the technologies of Knowledge-based systems, as well as a philosophical perspective, see for example "Good Old-Fashioned Artificial Intelligence" (GOFAI). They are developed by the first cognitive researchers and later used in information engineering for expert systems. Since the early 1990s it was generalized in systemics for the investigation of functional human-like intelligence models, such as personoids, and, in parallel, developed as the SOAR environment. Recently, especially in the context of cognitive decision making, symbolic cognitive Modelling is extended to socio-cognitive approach including social and organization cognition interrelated with a sub-symbolic not conscious layer. • Subsymbolic Modelling includes Connectionist/neural network models. Connectionism relies on the idea that the mind/brain is composed of simple nodes and that the power of the system comes primarily from the existence and manner of connections between the simple nodes. Neural nets are textbook implementations of this approach. Some critics of this approach feel that while these models approach biological reality as a representation of how the system works, they lack explanatory powers because complicated systems of connections with even simple rules are extremely complex a nd often less interpretable than the system they model. Other approaches gaining in popularity include the use of Dynamical systems theory and also techniques putting symbolic models and connectionist models into correspondence (Neural-symbolic integration). Bayesian models, often drawn from machine learning, are also gaining popularity.All the above approaches tend to be generalized to the form of integrated computational models of synthetic/abstract intelligence, in order to be applied to the explanation and improvement of individual and social/organizational decision-making and reasoning.
45
The Biological Model
Fig 13 Neuron Artificial neural networks born after McCulloc and Pitts introduced a set of simplified neurons in 1943. These neurons were represented as models of biological networks into conceptual components for circuits that could perform computational tasks. The basic model of the artificial neuron is founded upon the functionality of the biological neuron. By definition, “ Neurons are basic signalling units of the nervous system of a living being in which each neuron is a discrete cell whose several processes are from its cell body”
Biological neural network
The biological neuron has four main regions to its structure. The cell body, or soma, has two offshoots from it. The dendrites and the axon end in pre-synaptic terminals. The cell body is the heart of the cell. It contains the nucleolus and maintains protein synthesis. A neuron has many dendrites, which look like a tree structure, receives signals from other neurons. A single neuron usually has one axon, which expands off from a part of the cell body. This is called the axon hillock. The axon main purpose is to conduct electrical signals generated at the axon hillock down its length. These signals are called action potentials. The other end of the axon may split into several branches, which end in a pre-synaptic terminal. The electrical signals (action potential) that the neurons use to convey the information of the brain are all identical. The brain can determine which type of information is being received based on the path of the signal. The brain analyses all patterns of signals sent, and from that information it interprets the type of information received. The myelin is a fatty issue that insulates the axon. The non-insulated parts of the axon area are called Nodes of Ranvier. At these nodes, the signal traveling down the axon is regenerated. This ensures that the signal travel down the axon to be fast and constant. The synapse is the area of contact between two neurons. They do not physically touch because they are separated by a cleft. The electric signals are sent through chemical interaction. The neuron sending the signal is called pre-synaptic cell and the neuron receiving the electrical signal is called postsynaptic cell. The electrical signals are generated by the membrane potential which is based on differences in concentration of sodium and potassium ions and outside the cell membrane. Biological neurons can be classified by their function or by the quantity of processes they carry out. When they are classified by processes, they fall into three categories: Unipolar neurons, bipolar neurons and Multipolar neurons.
46
Unipolar neurons have a single process. Their dendrites and axon are located on the same stem. These
neurons are found in invertebrates. Bipolar neurons have two processes. Their dendrites and axon have two separated processes too. Multipolar neurons are commonly found in mammals. Some examples of these neurons are spinal motor neurons, pyramidal cells and purkinje cells. When biological neurons are classified by function they fall into three categories. The first group is sensory neurons. These neurons provide all information for perception and motor coordination. The second group provides information to muscles, and glands. There are called motor neurons. The last group, the interneuronal, contains all other neurons and has two subclasses. One group called relay or protection interneurons . They are usually found in the brain and connect different parts of it. The other group called local interneurons are only used in local circuits. The Mathematical Model
Fig14 artificial neuron Once modelling an artificial functional model from the biological neuron, we must take into account three basic components. First off, the synapses of the biological neuron are modelled as weights. Let’s remember that the synapse of the biological neuron is the one which interconnects the neural network and gives the strength of the connection. For an artificial neuron, the weight is a number, and represents the synapse. A negative weight reflects an inhibitory connection, while positive values designate excitatory connections. The following components of the model represent the actual activity of the neuron cell. All inputs are summed altogether and modified by the weights. This activity is referred as a linearcombination. Finally, an activation function controls the amplitude of the output. For example, an acceptable range of output is usually between 0 and 1, or it could be -1 and 1. Mathematically, this process is described in the figure. From this model the interval activity of the neuron can be shown to be:
The output of the neuron, yk, would therefore be the outcome of some activation function on the value of vk. 47
Fig15 mathematical model Activation functions
As mentioned previously, the activation function acts as a squashing function, such that the output of a neuron in a neural network is between certain values (usually 0 and 1, or -1 and 1). In general, there are three types of activation functions, denoted by Φ (.).
First, there is the Threshold Function which takes on a value of 0 if the summed input is less than a certain threshold value (v), and the value 1 if the summed input is greater than or equal to the threshold value.
Secondly, there is the Piecewise-Linear function. This function agai n can take on the values of 0 or 1, but can also take on values between that depending on the amplification factor in a certain region of linear operation.
Thirdly, there is the sigmoid function. This function can range between 0 and 1, but it is also sometimes useful to use the -1 to 1 range. An example of the sigmoid function is the hyperbolic tangent function.
48
Fig 16 common non-linear functions Neurobiological methods
Research methods borrowed directly from neuroscience and neuropsychology can also help us to understand aspects of intelligence. These methods allow us to understand how intelligent behaviour is implemented in a physical system. • Single-unit recording • Direct brain stimulation • Animal models • Post-mortem studies A framework for distributed representation
An artificial neural network consists of a pool of simple processing units which communicate by sending signals to each other over a large number of weighted connections. A set of major aspects of a parallel distributed model can be distinguished:
a set of processing units ('neurons,' 'cells'); a state of activation yk for every unit, which equivalent to the output of the unit; Connections between the units. Generally each connection is defined by a weight wjk which determines the effect which the signal of unit j has on unit k; a propagation rule, which determines the effective input sk of a unit from its external inputs; 49
an activation function Fk, which determines the new level of activation based on the effective input sk(t) and the current activation yk(t) (i.e., the update); an external input (aka bias, offset) øk for each unit; a method for information gathering (the learning rule); An environment within which the system must operate, providing input signals and ifnecessary error signals.
Processing units
Each unit performs a relatively simple job: receive input from neighbours or external sources and use this to compute an output signal which is propagated to other units. Apart from this processing, a second task is the adjustment of the weights. The system is inherently parallel in the sense that many units can carry out their computations at the same time. Within neural systems it is useful to distinguish three types of units: input units (indicated by an index i) which receive data from outside the neural network, output units (indicated by an index o) which send data out of the neural network, and hidden units (indicated by an index h) whose input and output signals remain within the neural network. During operation, units can be updated either synchronously or asynchronously. With synchronous updating, all units update their activation simultaneously; with asynchronous updating, each unit has a (usually fixed) probability of updating its activation at a time t, and usually only one unit will be able to do this at a time. In some cases the latter model has some advantages.
Fig 17The style of neural computation Neural Network topologies
This section focuses on the pattern of connections between the units and the propagation of data. As for this pattern of connections, the main distinction we can make is between: Feed-forward neural networks, where the data ow from input to output units is strictly feed forward. The
data processing can extend over multiple (layers of) units, but no feedback connections are present, that is, connections extending from outputs of units to inputs of units in the same layer or previous layers. Recurrent neural networks that do contain feedback connections: Contrary to feed-forward networks,
the dynamical properties of the network are important. In some cases, the activation values of the units undergo a relaxation process such that the neural network will evolve to a stable state in which these activations do not change anymore. In other applications, the change of the activation values of the output neurons is significant, such that the dynamical behaviour constitutes the output of the neural network (Pearl mutter, 1990). 50
Classical examples of feed-forward neural networks are the Perceptron and Adaline. Examples of recurrent networks have been presented by Anderson (Anderson, 1977), Kohonen (Kohonen, 1977), and Hopfield (Hopfield, 1982). Training of artificial neural networks
A neural network has to be configured such that the application of a set of inputs produces (either 'direct' or via a relaxation process) the desired set of outputs. Various methods to set the strengths of the connections exist. One way is to set the weights explicitly, using a priori knowledge. Another way is to 'train' the neural network by feeding it teaching patterns and letting it change its weights according to some learning rule. We can categorise the learning situations in two distinct sorts. These are:
Supervised learning or Associative learning in which the network is trained by providing it with
input and matching output patterns. These input-output pairs can be provided by an external teacher, or by the system which contains the neural network (self-supervised).
Fig 18 supervised learning situation
Unsupervised learning or Self-organisation in which an (output) unit is trained to respond to
clusters of pattern within the input. In this paradigm the system is supposed to discover statistically salient features of the input population. Unlike the supervised learning paradigm, there is no a priori set of categories into which the patterns are to be classified; rather the system must develop its own representation of the input stimuli. Reinforcement learning:This type of learning may be considered as an intermediate form of the above two types of learning. Here the learning machine does some action on the environment and gets a feedback response from the environment. The learning system grades its action good (rewarding) or bad (punishable) based on the environmental response and accordingly adjusts its parameters. Generally, parameter adjustment is continued until an equilibrium state occurs, following which there will be no more changes in its parameters. The self-organizing neural learning may be categorized under this type of learning.
Modifying patterns of connectivity of Neural Networks
Both learning paradigms supervised learning and unsupervised learning result in an adjustment of the weights of the connections between units, according to some modification rule. Virtually all learning rules 51
for models of this type can be considered as a variant of the Hebbian learning rule suggested by Hebb in his classic book Organization of Behaviour (1949) (Hebb, 1949). The basic idea is that if two units’ j and k are active simultaneously, their interconnection must be strengthened. If j receives input from k, the simplest version of Hebbian learning prescribes to modify the weight wjk with
Neural network training formula:
Where ϒ is a positive constant of proportionality representing the learning rate. Another common rule uses not the actual activation of unit k but the difference between the actual and desired activation for adjusting the weights: in which dk is the desired activation provided by a teacher. This is often called the Widrow-Hoff rule or the delta rule, and will be discussed in the next chapter. Many variants (often very exotic ones) have been published the last few years. How the Human Brain Learns?
Much is still unknown about how the brain trains itself to process information, so theories abound. In the human brain, a typical neuron collects signals from others through a host of fine structures called dendrites. The neuron sends out spikes of electrical activity through a long, thin strand known as an axon, which splits into thousands of branches. At the end of each branch, a structure called a synapse converts the activity from the axon into electrical effects that inhibit or excite activity from the axon into electrical effects that inhibit or excite activity in the connected neurons. When a neuron receives excitatory input that is sufficiently large compared with its inhibitory input, it sends a spike of electrical activity down its axon. Learning occurs by changing the effectiveness of the synapses so that the influence of one neuron on another changes.
Fig 19: Components of a neuron
52
Fig 20: The synapse Artificial Neurons and how they work
Artificial Neuron:
Our basic computational element (model neuron) is often called a node or unit. It receives input from some other units, or perhaps from an external source. Each input has an associated weight w, which can be modified so as to model synaptic learning. The unit computes some function f of the weighted sum of its inputs:
Its output, in turn, can serve as input to other units.
Fig 21: Artificial Neuron • •
The weighted sum is called the net input to unit i, often written neti. Note that it refers to the weight from unit j to unit i (not the other way around).
The function f is the unit's activation function. In the simplest case, f is the identity function, and the unit's output is just its net input. This is called a linear unit. The fundamental processing element of a neural network is a neuron. This building block of human awareness encompasses a few general capabilities. Basically, a biological neuron receives inputs from other sources, combines them in some way, performs a generally nonlinear operation on the result, and then outputs the final result. The basic unit of neural networks, the artificial neurons, simulates the four basic functions of natural neurons.
53
In Figure 25, various inputs to the network are represented by the mathematical symbol, x (n). Each of these inputs is multiplied by a connection weight. These weights are represented by w (n). In the simplest case, these products are simply summed, fed through a transfer function to generate a result, and then output. This process lends itself to physical implementation on a large scale in a small package. This electronic implementation is still possible with other network structures which utilize different summing functions as well as different transfer functions. Some applications require "black and white," or binary, answers. These applications include the recognition of text, the identification of speech, and the image deciphering of scenes. These applications are required to turn real-world inputs into discrete values.
Figure 22:a Basic Artificial Neuron These potential values are limited to some known set, like the ASCII characters or the most common 50,000 English words. Because of this limitation of output options, these applications don't always utilize networks composed of neurons that simply sum up, and thereby smooth, inputs. These networks may utilize the binary properties of ORing and ANDing of inputs. Architecture of neural networks
Single neuron is insufficient for many practical applications therefore networks with large number of neurons are frequently used. Following are some network structures. a. b. c. d.
Layered network. Acyclic network. Feed forward network. Modular neural network.
Once a network has been structured for a particular application, that network is ready to be trained.
54
The Basic Artificial Model
To capture the essence of biological neural systems, an artificial neuron is defined as follows: •
•
It receives a number of inputs either from original data, or from the output of other neurons. Each input comes via a connection that has a weight; these weights correspond to synaptic efficiency in a biological neuron. Each neuron also has a single threshold value. The weighted sum of the inputs is formed, and the threshold subtracted, to compose the activation of the neuron also known as the post-synaptic potential, or PSP, of the neuron. The activation signal is passed through an activation function also known as a transfer function to produce the output of the neuron.
Fig 23: a neuron model This describes an individual neuron. The next question is: how should neurons is connected together? If a network is to be of any use, there must be inputs which carry the values of variables of interest in the outside world and outputs which form predictions, or control signals. There can be hidden neurons that play an internal role in the network. The input, hidden and output neurons need to be connected together. The key issue here is feedback .
55
Fig 24 multilayer structure A simple network has a feed forward structure: signals flow from inputs, forwards through any hidden units, eventually reaching the output units. Such a structure has stable behaviour. A typical feed forward network has neurons arranged in a distinct layered topology. The input layer is not really neural at all: these units simply serve to introduce the values of the input variables. The hidden and output layer neurons are each connected to all of the units in the preceding layer. When the network is executed, the input variable values are placed in the input units, and then the hidden and output layer units are progressively executed. Each of them calculates its activation value by taking the weighted sum of the outputs of the units in the preceding layer, and subtracting the threshold. The activation value is passed through the activation function to produce the output o utput of the neuron. When the entire network has been executed, the outputs of the output layer act as the output of the entire network.
Fig 25: Typical multilayered neural network architecture Single-Layer Perceptron Neural Networks
The perceptron is the simplest form of a neural network used for the classifications of patterns said to be linearly separable basically it consists of a single neuron with adjustable synaptic weights and bias. The algorithm used to adjust the free parameters of this neural network first approach in learning procedure developed by Rosenblatt (1958, 1962) for this perceptron brain model. A single-layer perceptron network consists of one or more artificial neurons in parallel. The neurons may be of the same type we've seen in the Artificial Neuron Applet.
56
Fig 26, Single Layer Perceptron • •
Each neuron in the layer provides one network output, and is usually connected to all of the external (or environmental) inputs. The applet in this tutorial is an example of a single-neuron, single-layer perceptron network, with just two inputs. The perceptron learning rule, which we study next, provides a simple algorithm for training a perceptron neural network. However, as we will see, single-layer perceptron networks cannot learn everything: they are not computationally complete. As mentioned in the introduction, two-input networks cannot approximate the XOR (or XNOR) functions. Of the (122) n or 16 possible functions, a two-input perceptron can only perform 14 functions. As the number of inputs, n, increases, the proportion of functions that can be computed decreases rapidly.
Multilayer perceptron
Multi-layered Perceptron have been applied successfully to solve difficult and diverse problems by training them in a supervised manner with highly popular algorithms known as the error backpropagation algorithm. This algorithm is based on the error correction learning rule. As such, it may be viewed as a generalization of an equally popular adaptive filtering algorithm. Multi-layer perceptron is the most widely used type of neural network. It is both simple and based on solid mathematical grounds. Input quantities are processed through successive layers of "neurons". There is always an input layer, with a number of neurons equal to the number of variables of the problem, and an output layer, where the perceptron response is made available, with a number of neurons equal to the desired number of quantities computed from the inputs (very often only one).The layers in between are called "hidden" layers. With no hidden layer, the perceptron can only perform linear tasks (for example a linear discriminant analysis, which is already useful). All problems which can be solved by a perceptron can be solved with only one hidden layer, but it is sometimes more efficient to use 2 hidden layers. Each neuron of a layer other than the input layer computes first a linear combination of the outputs of the neurons of the previous layer, plus a bias.The coefficients of the linear combinations plus the biases are called the weights. They are usually determined from examples to minimize, on the set of examples, the (Euclidian) norm of the desired output - net output vector.Neurons in the hidden layer then compute a non-linear function of their input. In MLPfit, the non-linear function is the sigmoid function y(x) = 1/ (1+exp (-x))).
57
The output neuron(s) has its output equal to the linear combination. Thus, a Multi-Layer Perceptron with 1 hidden layer basically performs a linear combination of sigmoid function of the inputs. A linear combination of sigmoids is useful because of o f 2 theorems: •
•
A linear function of sigmoids can approximate any continuous function of 1 or more variable(s). This is useful to obtain a continuous function fitting a finite set of points when no underlying model is available. Trained with a desired answer = 1 for signal and 0 for background, the approximated function is the probability of signal knowing the input values. This second theorem is the basic ground for all classification applications.
Alternative Alternative Representations: Representations: Neural Nets Nets and Genetic Genetic Algorithms Algorithms
Most of the techniques presented in this AI book use explicitly represented knowledge and carefully designed search algorithms to implement intelligence. A very different approach seeks to build intelligent programs using models that parallel the structure of neurons in the human brain or the evolving patterns found in genetic algorithms and artificial life.
58
A simple schematic of a neuron (Figure 22) consists of a cell body that has a number of branched protrusions, called dendrites, and a single branch called the axon. Dendrites receive signals from other neurons. When these combined impulses exceed a certain threshold, the neuron fires and an impulse, or spike, passes down the axon. Branches at the end of the axon form synapses with the dendrites of other neurons. The synapse is the point of contact between neurons; synapses may be either excitatory or inhibitory, either adding to the total of signals reaching the neuron or subtracting from that total. This description of a neuron is excessively simple, but it captures those features that are relevant to neural models of computation. In particular, each computational unit computes some function of its inputs and passes the result along to connected units in the network: the final results are produced by the parallel and distributed processing of this network of neural connections and threshold weights. Neural architectures are appealing mechanisms for implementing intelligence for a number of reasons. Traditional AI programs can be brittle and overly sensitive to noise. Human intelligence is much more flexible and good at interpreting noisy input, such as a face in a darkened room or a conversation at a noisy party. Neural architectures, because they capture knowledge i n a large number of fine-grained units distributed about a network, seem to have more potential for partially matching noisy and incomplete data. Genetic algorithms
GAs are typically used to search very large and possibly very high dimensional search spaces. If we want to find a solution as a single point in an N dimensional space where a fitness function has a near maximum value, then we have N parameters to encode in each chromosome. In this chapter we will be solving a simple problem that is one-dimensional so we only need to encode a single number (a floating point number for this example) in each chromosome. Using a GA toolkit, like the one developed in Section 6.2, requires two problem-specific customizations: Characterize the search space by a set of parameters that can be encoded in a chromosome (more on this later). GAs work with the coding of a parameter set, not the parameters themselves (Genetic Algorithms in Search, Optimization, and Machine Learning, David Goldberg, 1989).
59
Provide a numeric fitness function that allows us to rate the fitness of each chromosome in a population. We will use these fitness values to determine which chromosomes in the population are most likely to survive and reproduce using genetic crossover and mutation operations. The GA toolkit developed in this chapter treats genes as a single bit; while you can consider a gene to be an arbitrary data structure, the approach of using single bit genes and specifying the number of genes (or bits) in a chromosome is very flexible. A population is a set of chromosomes. A generation is defined as one reproductive cycle of replacing some elements of the chromosome population with new chromosomes produced by using a genetic crossover operation followed by optionally mutating a few chromosomes in the population. We will describe a simple example problem in this section, write a general purpose library in Section 6.2, and finish the chapter in Section 6.3 by solving the problem posed in this section. For a sample problem, suppose that we want to find the maximum value of the function F with one independent variable x in Equation 6.1 and as seen in Figure 6.1: F(x) = sin(x) sin(0:4x) sin(3 x)
(6.1)
The problem that we want to solve is finding a good value of x to find a near to possible maximum value of F(x). To be clear: we encode a floating point number as a chromosome made up of a specific number of bits so any chromosome with randomly set bits will represent some random number in the interval [0, 10]. The fitness function is simply the function in Equation 6.1.
Figure 6.2 shows an example of a crossover operation. A random chromosome bit index is chosen, and two chromosomes are “cut” at this this index and swap cut parts. The two original chromosomes in generation are shown on the left of the figure and after the crossover operation they produce two new chromosomes in generationn+1 shown on the right of the figure. In addition to using crossover operations 60
to create new chromosomes from existing chromosomes, we will also use genetic mutation: randomly flipping bits in chromosomes. A fitness function that rates the fitness value of each chromosome allows us to decide which chromosomes to discard and which to use for the next generation: we will use the fit chromosomes in the population for producing the next generation using crossover and mutation. With genetic algorithms and artificial life we evolve new problem solutions from components of previous solutions. The genetic operators, such as crossover and mutation, much like their genetic equivalents in the natural world, work to produce, for each new generation, ever better potential problem solutions. Artificial life produces its new generation as a function of the "quality" of its neighbours in previous generations. Both neural architectures and genetic algorithms provide a natural model for parallelism, because each neuron or segment of a solution is an independent unit. Hillis (1985) has commented on the fact that humans get faster at a task as they acquire more knowledge, while computers tend to slow down. This slowdown is due to the cost of sequentially searching a knowledge base; a massively parallel architecture like the human brain would not suffer from this problem. Finally, something is intrinsically appealing about approaching the problems of intelligence from a neural or genetic point of view. After all, the evolved brain achieves intelligence and it does so using a neural architecture. Transfer Function
The behaviour of a Neural Network depends on both the weights and the input-output function (transfer function) that is specified for the units. This function typically falls into one of three categories:
Linear (or ramp): The output activity is proportional to the total weighted output Threshold: The output is set at one of two levels, depending on whether the total input is greater than or less than some threshold value. Sigmoid: The output varies continuously but not linearly as the input changes. Sigmoid units bear a greater resemblance to real neurons than do linear or threshold units, but all three must be considered rough approximations. To make a neural network that performs some specific task, we must choose how the units are connected to one another and we must set the weights on the connections appropriately. The connections determine whether it is possible for one unit to influence another. The weights specify the strength of the influence.
61
Fuzzy Logic and Neural Network
As the complexity of a system increases, it becomes more difficult and eventually impossible to make a precise statement about its behaviour, eventually arriving at a point of complexity where the fuzzy logic method born in humans is the only way to get at the problem. Fuzzy logic is a superset of Boolean logic that has been extended to handle the concept of partial truth -- truth values between "Completely true" and "completely false". It was introduced by Dr.LotfiZadeh of UC/Berkeley in the 1960's as a means
to model the uncertainty of natural language. FL is a problem-solving control system methodology that lends itself to implementation in systems ranging from simple, small, embedded micro-controllers to large, networked, multi-channel PC or workstation-based data acquisition and control systems. It can be implemented in hardware, software, or a combination of both.
Fig33. fuzzy system FL provides a simple way to arrive at a definite conclusion based upon vague, ambiguous, imprecise, noisy, or missing input information. FL's approach to control problems mimics how a person would make decisions, only much faster. In a sense, fuzzy logic resembles human decision making with its ability to work from approximate data and find precise solutions. Fuzzy Logic has been gaining increasing acceptance during the past few years. There are over two thousand commercially available products using Fuzzy Logic, ranging from washing machines to high speed trains. Nearly every application can potentially realize some of the benefits of Fuzzy Logic, Logi c, such as performance, simplicity, lower cost, co st, and productivity. How is FL different from conventional control methods?
FL incorporates a simple, rule-based IF X AND Y THEN Z approach to a solving control problem rather than attempting to model a system mathematically. The FL model is empirically-based, relying on an operator's experience rather than their technical understanding of the system. For example, rather than dealing with temperature control in terms such as "SP =500F", "T <1000F", or "210C
62
How does FL work?
FL requires some numerical parameters in order to operate such as what is considered significant error and significant rate-of-change-of-error, but exact values of these numbers are usually not critical unless very responsive performance is required in which case empirical tuning would determine them. These values don't have to be symmetrical and can be "tweaked" once the system is operating in order to optimize performance. Generally, FL is so forgiving that the system will probably work the first time without any tweaking. For example, a simple temperature control system could use a single temperature feedback sensor whose data is subtracted from the command signal to compute "error" and then timedifferentiated to yield the error slope or rate-of-change-of-error, hereafter called "error-dot". Error might have units of degrees F and a small error considered to be 2F while a large error is 5F. The "error-dot" might then have units of degrees/min with a small error-dot being 5F/min and a large one being 15F/min. Neuro-computing
Very loosely based on how the brain is thought to work. It is an emulation of primitive neural processes in software (or hardware). It attempts to mimic (it cannot yet copy) the workings of a biological brain. Best neural net is currently equivalent to a brain damaged worm but astonishingly this is good enough for many practical problems...
Fig.36 Representation of 6×4×1 simple network Nodes have input signals Dendrites carry an impulse to the neuronin figure 36 Nodes have one output signal Axons carry signal out of neuron and synapses are local regions where signals are transmitted from the axon of one neuron to dendrites of another. Input signal weights are summed at each node Nerve impulses are binary; they are “go” or “no go”. Neurons sum up the incoming signal and fire if a threshold value is reached.[124]
63
Fig 35 functional structure of brain Fuzzy Logic Analysis and Control:-
Human beings have the ability to take in and evaluate all sorts of information from the physical world they are in contact with and to mentally analyse, average and summarize all this input data into an optimum course of action. If you think about it, much of the information you take in is not very precisely defined, such as the speed of a vehicle coming up from behind. We call this fuzzyinput. However, some of your "input" is reasonably precise and non-fuzzy such as the speedometer reading. Your processing of all this information is not very precisely definable. We call this fuzzyprocessing. Fuzzy logic theorists would call it using fuzzyalgorithms is another word for procedure or program, as in a computer program. The fuzzy logic analysis and control method is, therefore:
Receiving of one, or a large number, of measurement or other assessment of conditions existing in some system we wish to analyse or control. Processing all these inputs according to human based, fuzzy "If-Then" rules, which can be expressed in plain language words, in combination with traditional non-fuzzy processing. Averaging and weighting the resulting outputs from all the individual rules into one single output decision or signal which decides what to do or tells a controlled system what to do. The output signal eventually arrived at is a precise appearing, defuzzified, "crisp" value.
64
Fig 37 Fuzzy concept An engineering approach to fuzzy logic:Fuzzy Sets – It is a group of anything that can’t be precisely defined and the where the condition can be
given a value between 0 and 1. Example: A woman is 6 feet, 3 inches tall. In my experience, I think she is one of the tallest women I have ever met, so I rate her height at .98. This line of reasoning can go on indefinitely rating a great number of things between 0 and 1. Universe of Discourse :- a way to say all the objects in the universe of a particular kind, usually designated
by one word, that we happen to be talking about or working with in a fuzzy logic solution.
Fig 38 Fuzzy logic control analysis method Fuzzy control:-The purpose of control is to influence the behaviour of a system by changing an input or
inputs to that system according to a rule or set of rules that model how the system operates. The system being controlled may be mechanical, electrical, chemical or any combination of these.
Artificial Neural Network and Fuzzy Logic Difference: 65
Fuzzy logic: Logicality NN: Learning function
Similarity: Output characteristics of NN and membership function Multiply-add operation of neuron and MAX-MIN operation of approximate reasoning.
The first fusion pattern is a method combining individual advantages. Fuzzy logic can express logic explicitly taking a form of rule. NN is helpful when it is employed for pattern identification because of its learning function. From these advantages of view, (a) a method to endow learning function to fuzzy logic or to conduct pattern processing before fuzzy logic is applied, and (b) a method to incorporate logics in NN structure etc. could be possible for combining these two techniques. The second fusion method is to superpose similarities. The first similarity shown above is to give a membership function to NN without causing a crisp boundary between classes formed by a pattern classification type NN. The reason is that threshold function of its neuron have sigmoid characteristics to attain continuous values of 0, 1. The second similarity is that (a) the MIN operation of input and fuzzy variables conducted at each proposition of IF parts of fuzzy inference rule corresponds to a product of input to the neuron and synaptic weights, and (b) the MAX operation to obtain a final inference value from the THEN part of these plural inference rules corresponds to the input sum within neuron. The endowment of learning function to the fuzzy logic is one of the major particular characteristics of NN only. The degree of association between keywords is expressed in a form of matrix, and the learning is conducted by using the steepest decent method. Although the matrix in this case could be interpreted as a two-layered NN in a way, it could apart from the general accepted image of NN. Applications of ANN and Fuzzy Logic
Language Processing
Language processing encompasses a wide variety of applications. These applications include text-tospeech conversion, auditory input for machines, automatic language translation, secure voice keyed locks, automatic transcription, aids for the deaf, aids for the physically disabled which respond to voice commands, and natural language processing. Many companies and universities are researching how a computer, via ANNs, could be programmed to respond to spoken commands. If this capability could be shrunk to a chip, that chip could become part of almost any electronic device sold today. Literally hundreds of millions of these chips could be sold.
Finance and banking
Banking, credit card companies, and lending institutions deal with decisions that are not clear-cut. They involve learning and statistical trends. The loan approval process involves filling out forms which hopefully can enable a loan officer to make a decision. The data from these forms is now being used by neural networks which have been trained on the data from past decisions. Indeed, to meet government requirements as to why applications are being denied, these packages are providing information on what input, or combination of inputs, weighed heaviest on the decision.
Character Recognition
Neural network based product that can recognize hand printed characters through a scanner. This product can take cards, like a credit card application form, and put those recognized characters into a database. This product has been out for two and a half years. It is 98% to 99% accurate for numbers, a little less for alphabetical characters. Currently, the system is built to highlight characters below a certain percent probability of being right so that a user can manually fill in what the computer could not. This product is in use by banks, financial institutions, and credit card companies.
66
Since neural networks are best at identifying patterns or trends in data, in fusion with fuzzy logic they are well suited for prediction or forecasting needs including:
Sales forecasting Industrial process control Customer research Data validation Risk management Target marketing
Machine learning
Having a computer program itself from a set of examples so you don't have to program it yourself. This will be a strong focus of this course: neural networks that learn from a set of examples. Optimization: given a set of constraints and a cost function, how do you find an optimal solution? E.g. travelling salesman problem. Classification: grouping patterns into classes: i.e. handwritten characters into letters. Associative memory: recalling a memory based on a partial match. Regression: function mapping Neurobiology
Modelling models of how the brain works. Neuron-level higher levels: vision, hearing, etc. Overlap with cognitive folks.
Mathematics
Nonparametric statistical analysis and regression
Philosophy
Can human souls/behaviours be explained in terms of symbols, or does it require something lower level, like a neurally based model?
Medicine
Analysis of electrocardiogram data, Improved prosthetic devices, Pap smear, detection of cancerous cells to drastically reduce errors, RNA & DNA sequencing in proteins, Medical image enhancement, Drug development without animal testing
Pattern Recognition
Signature validation, OCR scanning for machine printed character recognition; also used at Post Office to sort mail, Hand printed character recognition (i.e. insurance forms) to reduce clerical data entry costs, Cursive handwriting recognition (i.e. for pen-based computing),Airport bomb detection (1989 JFK International in NY) analyses gamma ray patterns of various objects after being struck with neutrons.
Telecommunication
Network line Fault detection
Real Estate
Real Estate appraisal
Weather Forecasting
Neural network used in Weather Forecasting to predict the Temperature, Rain, Flood Humidity etc.
Monitoring
Networks have been used to monitor the state of aircraft engines. By monitoring vibration levels and sound, early warning of engine problems can be given. British Rail has also been testing a similar application monitoring diesel engines. Neural networks are applicable in virtually every situation in which a relationship between the predictor variables and predicted variables exists, even when that relationship is very complex and not easy to articulate in the usual terms of "correlations" or "differences between groups."
67
• •
Detection of medical phenomena: - A variety of health-related indices (e.g., a combination of heart rate, levels of various substances in the blood, respiration rate) can be monitored. Electronic noses:-ANNs are used experimentally to implement electronic noses. Electronic noses have several potential applications in telemedicine.
Software & Tools
The Neural Network Toolbox extends MATLAB with tools for designing, implementing, visualizing, and simulating neural networks. Neural networks are invaluable for applications where formal analysis would be difficult or impossible, such as pattern recognition and nonlinear system identification and c ontrol. The Neural Network Toolbox provides comprehensive support for many proven network paradigms, as well as graphical user interfaces (GUIs) that enable you to design and manage your networks. The modular, open, and extensible design of the toolbox simplifies the creation of customized functions and networks.
Natural Language Processing
Fig 43A parse tree represents the syntactic structure of a sentence according to some formal grammar. Natural language processing [129] gives machines the ability to read and understand the languages that humans speak. Many researchers hope that a sufficiently powerful natural language processing system would be able to acquire knowledge on its own, by reading the existing text available over the internet. Some straightforward applications of natural language processing include information retrieval (or text mining) and machine translation.[130] Natural Language Understanding and Semantics
One of the long-standing goals of artificial intelligence is the creation of programs that are capable of understanding and generating human language. Not only does the ability to use and understand natural language seem to be a fundamental aspect of human intelligence, but also its successful automation would have an incredible impact on the usability and effectiveness of computers themselves. Much effort has been put into writing programs that understand natural language. Although these programs have achieved success within restricted contexts, systems that can use natural language with the flexibility and generality that characterize human speech are beyond current methodologies. Understanding natural language involves much more than parsing sentences into their individual parts of speech and looking those words up in a dictionary. Real understanding depends on extensive background knowledge about the domain of discourse and the idioms used in that domain as well as an ability to apply general contextual knowledge to resolve the omissions and ambiguities that are a normal part of human speech. 68
Consider, for example, the difficulties in carrying on a conversation about baseball with an individual who understands English but knows nothing about the rules, players, or history of the game. Could this person possibly understand the meaning of the sentence: "With none down in the top of the ninth and the goahead run at second, the manager called his relief from the bull pen"? Even though all of the words in the sentence may be individually understood, this sentence would be gibberish to even the most intelligent non-baseball fan. The task of collecting and organizing this backgroundknowledge in such a way that it may be applied to language comprehension forms the major problem in automating natural language understanding. Responding to this need, researchers have developed many of the techniques for structuring semantic meaning used throughout artificial intelligence. Language, ability to speak, write and communicate is one of the most fundamental aspects of human behaviour. As the study of human-languages developed the concept of communicating with non-human devices was investigated. This is the origin of natural language processing (NLP). The idea of natural language processing is to design and build a computer system that will analyse, understand and generate natural human-languages. Natural language communication with computers has long been a major goal of artificial intelligence, both for the information it can give about intelligence in general, and for practical utility. Applications for processing large amounts of texts require NLP expertise.
There are many applications of natural language processing developed over the years. They can be mainly divided into two parts as follows. •
•
Text-based applications: This involves applications such as searching for a certain topic or a
keyword in a data base, extracting information from a large document, translating one language to another or summarizing text for different purposes. Dialogue based applications: Some of the typical examples of this are answering systems that can answer questions, services that can be provided over a telephone without an operator, teaching systems, voice controlledmachines (that take instructions by speech) and general problem solving systems.
Languages and Environments for AI
Some of the most important by-products of artificial intelligence research have been advances in programming languages and software development environments. For a number of reasons, including the size of many AI application programs, the importance of a prototyping methodology, the tendency of search algorithms to generate huge spaces, and the difficulty of predicting the behaviour of heuristically driven programs, AI programmers have been forced to develop a powerful set of programming methodologies. Programming environments include knowledge-structuring techniques such as object-oriented programming. High-level languages, such as LISP and PROLOG, which support modular development, help manage program size and complexity. Trace packages allow a programmer to reconstruct the execution of a complex algorithm and make it possible to unravel the complexities of heuristic search. Without such tools and techniques, it is doubtful that many significant AI systems could have been built. Many of these techniques are now standard tools for software engineering and have little relationship to the core of AI theory. Others, such as object-oriented programming, are of significant theoretical and practical interest. Finally, many AI algorithms are also now built in more traditional computing languages, such as C++ and Java. 69
The languages developed for artificial intelligence programming are intimately bound to the theoretical structure of the field. We cover both LISP and PROLOG in this book and prefer to remain apart from religious debates over their relative merits. Rather, we adhere to the adage "a good worker knows all her tools." Sub-problems
•
•
• •
•
Speech segmentation in most spoken languages, the sounds representing successive letters blend
into each other, so the conversion of the analog signal to discrete characters can be a very difficult process. Also, in natural speech there are hardly any pauses between successive words; the location of those boundaries usually must take into account grammatical and semantic constraints, as well as the context. Text segmentation: Some written languages like Chinese, Japanese and Thai do not have singleword boundaries either, so any significant text parsing usually requires the identification of word boundaries, which is often a non-trivial task. Word sense disambiguation: Many words have more than one meaning; we have to select the meaning which makes the most sense in context. Syntactic ambiguity: The grammar for natural languages is ambiguous, i.e. there are often multiple possible parse trees for a given sentence. Choosing the most appropriate one usually requires semantic and contextual information. Specific problem components of syntactic ambiguity include sentence boundary disambiguation. Imperfect or irregular input Foreign or regional accents and vocal impediments in speech; typing or grammatical errors, HOCRH errors in texts.
A sentence can often be considered an action by the speaker. The sentence structure alone may not contain enough information to define this action. For instance, a question is actually the speaker requesting some sort of response from the listener. The desired response may be verbal, physical, or some combination. For example, "Can you pass the class?" is a request for a simple yes-or-no answer, while "Can you pass the salt?" is requesting a physical action to be performed. It is not appropriate to respond with "Yes, I can pass the salt," without the accompanying action (although "No" or "I can't reach the salt" would explain a lack of action).
70
Where Does It Fix In The Cs Taxonomy?
Fig 44 Ladder of Transformation
• • • • • •
Morphological Lexical Syntactic Semantic Discourse Pragmatic
Stages in Text Processing
Natural Language processing systems that process text documents (typically unstructured text) involve a number of stages of processing.
Cleaning removes unwanted control characters, etc.
Tokenization adds spaces to separate text at boundary points between words and surrounding
punctuation, or between different punctuation marks. End-of -sentencedetection identifies and marks sentence boundaries. Part-of-speech tagging adds a tag indicating the part of speech for each token. Phrasedetection identifies and marks units that consist of multiple words – typically they are noun phrases of some type, but need not be. Entitydetection identifies and marks entities, which usually consist of person names, place names, organization or company names and other proper nouns. Categorization identifies and marks what category something belongs to; typically categorization is used primarily for named entities (i.e. proper nouns). Eventdetection identifies and marks events, which generally correspond to verbs
71
Relationdetection identifies and marks relations, which are connections between two or more
entities or between entities and events. XMLorSGMLtagging applies the designated tagging scheme used to markup the document for sentences, phrases, entities, categories, events, relations, etc. Extraction the identified entities, events, relations, and any other identified concepts (like dates) are extracted from the document and stored externally
Natural Language Processing (Nlp) Techniques
Natural language processing (NLP) is a field of computer science and linguistics concerned with the interactions between computers and human (natural) languages. Natural language generation systems convert information from computer databases into readable human language. Natural language understanding systems convert samples of human language into more formal representations such as 72
parse trees or first order logic that are easier for computer programs to manipulate. Many problems within NLP apply to both generation and understanding; for example, a computer must be able to model morphology (the structure of words) in order to understand an English sentence, and a model of morphology is also needed for producing a grammatically correct English sentence. There are several main techniques used in analysing natural language processing. Some of them can be briefly described as follows. Pattern matching: The idea here is an approach to natural language processing is to interpret input
utterances as a whole father than building up their interpretation by combining the structure and meaning of words or other lower level constituents. That means the interpretations are obtained by matching patterns of words against the input utterance. For a deep level of analysis in pattern matching a large number of patterns are required even for a restricted domain. This problem can be ameliorated by hierarchical pattern matching in which the input is gradually canonicalized through pattern matching against subphrases. Another way to reduce the number of patterns is by matching with semantic primitives instead of words. Syntactically driven Parsing: Syntax means ways that words can fit together to form higher level units
such as phrases, clauses and sentences. Therefore syntactically driven parsing means interpretation of larger groups of words are built up out of the interpretation of their syntactic constituent words or phrases. In a way this is the opposite of pattern matching as here the interpretation of the input is done as a whole. Syntactic analyses are obtained by application of a grammar that determines what sentences are legal in the language that is being parsed. Semantic Grammars: Natural language analysis based on semantic grammar is bit similar to syntactically
driven parsing except that in semantic grammar the categories used are defined semantically and syntactically. There here semantic grammar is also involved.[137] Case frame instantiation: Case frame instantiation is one of the major parsing techniques under active
research today. They have some very useful computational properties such as its recursive nature and its ability to combine bottom-up recognition of key. [136] Major Tasks inNlp
• • • • • • •
• • • • •
Automatic summarization Foreign language reading aid Foreign language writing aid Information extraction Information retrieval (IR) - IR is concerned with storing, searching and retrieving information. Machine translation - Automatically translating from one human language to another. Named entity recognition (NER) - Given a stream of text, determining which items in the text map to proper names, such as people or places. Although in English, named entities are marked with capitalized words, many other languages do not use capitalization to distinguish named entities. Natural language generation Natural language understanding Optical character recognition anaphora resolution Question answering - Given a human language question, the task of producing a humanlanguage answer. The question may be a closed-ended (such as "What is the capital of Canada?") or openended (such as "What is the meaning of life?").
73
• • • • •
Speech recognition - Given a sound clip of a person or people speaking, the task of producing a text dictation of the speaker(s). Spoken dialogue system Text simplification Text-to-speech Text-proofing
Distributed Artificial Intelligence Distributed Artificial Intelligence (DAI) systems can be defined as cooperative systems where a set of agents act together to solve a given problem. These agents are often heterogeneous. Its metaphor of intelligence is based upon social behaviour (as opposed to the metaphor of individual human behaviour in classical AI) and its emphasis is on actions and interactions, complementing knowledge representation and inference methods in classical AI. This approach is well suited to face and solve large and complex problems, characterized by physically distributed reasoning, knowledge and data managing. An agent is a real or virtual entity which is emerged in an environment where it can take some actions, which is able to perceive and represent partially this environment, which is able to communicate with the other agents and which possesses an autonomous behaviour that is a consequence of its observations, its knowledge and its interactions with the other agents. DAI systems are based on different technologies like, e.g., distributed expert systems, planning systems or blackboard systems. Different languages are now advocated for describing DAI systems at a conceptual level. As such, DAI systems are therefore recognized as composite systems. A common feature of all these languages relies on the introduction of agents. [146] Knowledge Acquisition
DAI can be exploited in knowledge acquisition for Modelling the experts (their cooperation, their conflicts), for Modelling the knowledge acquisition process and specially the cooperation during knowledge acquisition from a group of experts. The process of knowledge acquisition can be described as the behaviour of a composite system made of several human agents (experts, knowledge engineers and users) and software agents (the knowledge acquisition tool, the final system and the software where the final system will be integrated). This notion of composite system made of heterogeneous, interacting agents can help to model the main relations between such agents (knowledge transfer, explanation, validation, assistance to problem solving...) and to analyse the cooperation underlying the process of knowledge acquisition. The notion of agent should allow modelling the end-user as an agent and easing the description of a knowledge acquisition methodology involving several human agents. The problems of design of a multi-agent system with multiple interaction levels require protocols for interaction among such agents. Such a model can be extended in order to guide the construction of a knowledge acquisition tool which is based on a multiagent architecture and involves several heterogeneous agents: human agents such as the experts, the knowledge engineers, the users, and artificial agents such as the knowledge acquisition too and the cognitive agents composing it.[147] The logics of intentions and of capabilities can be used for specifying the interactions among multiple cooperating agents that may be either human expert from whom knowledge must be acquired or cognitive agents that will represent them in a knowledge acquisition tool. The use of such logics may enable to formalize research on knowledge acquisition from multiple experts. [148] Cooperation and Explanations
The type of cooperation depends on the organization of the agents: horizontal and vertical organizational structures can be distinguished. In non-hierarchical societies, cooperation is based on sharing of tasks and of results, while in a hierarchical society; it relies on commands, bids and competition. In addition, the 74
class of problems studied can influence the kind of cooperation: for example, the study of cooperation among designers influenced the multi-agent architecture implementing such collaboration. In the same way, conflicts among designers are analysed and exploited for a tool supporting cooperative design. Cooperation relies on the notion of goal adoption, which implies a goal common to the different agents. Different types of cooperation are presented: accidental cooperation, unilaterally intended cooperation, mutual cooperation. Cooperation is considered as a function of mutual dependency among the agents. Learning
Single Agent learning
Single agent learning involves improving the performance or increasing the knowledge of a single agent. Single agent learning systems may be classified according to their underlying learning strategies. These strategies are ordered according to the amount of inferencing or the degree of knowledge transformation required by the learning system. This order also reflects the increasing amount of effort required by the learning system and the decreasing effort required by the teacher. [149] Multiple-Agent Learning
Distributed artificial intelligence (DAI) systems solve problems using multiple, cooperative agents. In these systems, control and information are often distributed among the agents. This reduces the complexity of each agent and allows agents to work in parallel and increases problem solving speed. In addition, a DAI system can continue to operate even if some of its agents cease to operate. This behaviour allows the system to degrade gracefully in the event of failure of any of its parts. Also, each agent has resource limitations which could limit the ability of a single agent system to solve large, complex problems. Allowing multiple agents to work on these types of problems may be the only way to realistically solve them. Coordination
Coordination is the process by which an agent reasons about its local actions and the (anticipated) actions of others to try and ensure the community acts in a coherent manner, is perhaps the key problem of the discipline of Distributed Artificial Intelligence (DAI). In order to make advances it is important that the theories and principles which guide this central activity are uncovered and analysed in a systematic and rigorous manner. Traditional AI Traditional AI has devoted considerable attention to problems of manufacturing scheduling and control (Smith 91). By taking into account semantic information about the domain that does not lend itself to numerical computation; by applying heuristics judiciously and selectively (rather than globally as with dispatch rules), and by adopting a "satisficing" approach that does not insist on a theoretically perfect optimum, symbolic computation has led to systems that are somewhat faster than numerical programming and are more flexible and able to accommodate richer constraints, while yielding results superior to dispatch rules. However, these systems still tend to be large, complex, and specific to a particular installation, thus making them expensive to construct and difficult to maintain and reconfigure. Furthermore, while they are faster than some numerical programming codes, they are not fast enough for a facility whose configuration and load changes daily.[150]
Soft Computing Soft computing, according to Prof. Zadeh, is "an emerging approach to computing, which parallels the remarkable ability of the human mind to reason and learn in an environment of uncertainty and imprecision”. It, in general, is a collection of computing tools and techniques, shared by closely related 75
disciplines that include fuzzy logic, artificial neural nets, genetic algorithms, belief calculus, and some aspects of machine learning like inductive logic programming. These tools are used independently as well as jointly depending on the type of the domain of applications. The concept of soft computing was introduced by Dr.LotfiZadeh in 1991. Soft computing infect differs from the Hard conventional computing in that, unlike hard computing it is tolerant of impression, uncertainty and partial truth. In effect, the role model for soft computing is the human mind. The guiding principle of soft computing is; exploit the tolerance for imprecision, uncertainty and partial truth to achieve tractability, robustness and low solution cost. The basic ideas underlying soft computing in its current incarnation have links to many earlier influences like fuzzy sets; analysis of complex systems and decisions processes; and possibility theory and soft data analysis. Loosely speaking that computing has been categorized as the form of computing based upon binary logic, crisp sets (exact reasoning) and rigorous mathematical Modelling. At this juncture, the principal constituents of soft computing are
fuzzy logic (FL), Neural network theory (NN). Genetic algorithms (GA) and Probabilistic reasoning (PR), with the later subsuming belief networks, genetic algorithms, chaos theory and parts of learning theory.
What is important to note is that SC is not a mélange of FL, NN, GA and PR. Rather; it is a partnership in which each of the partners contributes a distinct methodology for addressing problems in its domain. In this perspective, the principal contribution of FL, NN and PR are complementary rather than competitive. Implications of Soft Computing
The methodologies in (SC) soft computing are complementary and synergistic rather than competitive. The complementarity of FL, NN,GA and PR has an important consequence: in many cases a problem can be solved most effectively by using FL, NN, GA and PR rather than exclusively. Within the soft computing, the main concerns of fuzzy logic, Neuro computing and probabilistic computing centre on:
FL: appropriate reasoning, information granulation, computing with words NC: learning adaptation, classification, system Modelling and identification GC: synthesis, tuning an optimization through systematized random search and evolution PC: management of uncertainty belief networks, prediction, chaotic system
76
Fig 45: problem solving technologies It has been proved that non-linear mapping obtained by neural networks can be approximated to any desired accuracy by the use of fuzzy systems [154]. As neural networks have flexible learning capabilities, it is possible to develop non-linear models using only input output data. However it is often cumbersome to fine-tune the modelling accuracy of neural networks, because it may be difficult to explain logically the cause and result in the excitation –response relationship. On the other hand, fuzzy systems provide clear advantages in a knowledge representation and acquisition. For example, knowledge is easily introduced in parallel to an adaptive fuzzy neural network by constructing a hierarchal diagnosis structure and modifying rules by available structured knowledge or modifying and adjusting fuzzy interference for pattern recognition with lack of input data by some complementary knowledge [155]. Fuzzy systems, however, have been missing adaptation capabilities for a long time [156]. It has been shown that under the condition of minor restrictions, functional behaviours of radial bases function networks and fuzzy interference systems are the same [157]. On the other hand, local models in blended multiple models structures for non-linear systems (fast fuzzy neural networks) have been recently investigated [158] [159] [160].
77
Fig 46 computing models Also type-I fuzzy systems implemented using Gaussian radial basis function neural networks as local models in blended model structures for non-linear systems. This fuzzy neural network is obtained by replacing the output layer weights with a linear function of the network inputs. Each neuron represents a local linear model with its corresponding validity function (Membership function). Furthermore, the radial basis function network is normalized like fuzzy membership functions. The side effects of normalizing should be considered, as all validity functions for a specific input combination sum up to one [161]. The Gaussian validity function determines the regions of the input space where each neuron is active. The input space becomes larger when these networks represent dynamic systems. A fast fuzzy neural network with general parameter learning is developed. It is especially suitable for real time fault diagnosis since what we have to do is to observe only changes in a general parameter. Recurrent fuzzy neural networks are recommended as a means to reduce the size of the input space [162]. They are able to yield adaptive selftuning; self-organizing and automated design functions for non-linear systems and system for each suitable mathematical model are not obtained. They are also used for cognitive (fuzzy decision tree etc.) and reactive (multiagent system coordination etc.) decision making. DSP’s and advanced computer systems are at present utilized to implement soft computing. Neuro computing and evolutionary computation usually need to a lot of computational time, which is the disadvantage of the implementation of soft computing. Recently developed fuzzy neural networks enable solutions to be obtained for problems that have not been able to be solved by traditional analytical methods (hard computing) [163], since function approximation is used rather than parameter optimization (TR). Tractability enables industrial systems to become increasingly innovative. Evolutionary computation has been developed and modified for application of optimization for large scale and complex system as shown in this paper. Fogel proposed intelligence based on Bioinformatics. Data mining for which soft computing is an effective and promising approach has been attracting the attention of researchers. Data mining is accepted to be applied large-scale process plants and electric power system for decision support and optimization (TR).
78
Soft computing has recently been playing an important role in advanced knowledge processing. An advanced learning method using a combination of perception and motion has been introduced. Emergent, self-organizing, reflective and interactive (among human beings, environment and artificial intelligence) knowledge processing is considered by using soft computing and by borrowing ideas from bio information processing. Soft computing provides rich knowledge representation (symbol and pattern), flexible knowledge acquisition (by learning from data and by interviews with experts), and knowledge processing (interface by interface between symbolic and pattern knowledge). Therefore it is straightforward to construct low cost intelligent system. The various kinds of artificial intelligence (cognitive and reactive AI) make industrial system intelligent. Such intelligent system has adaptive, autonomous, decision support, optimization and emergent functions (HMIQ). This HMIQ enables innovations in industry fields. Table 4Summary Of Application Of Soft Computing
79
Cognitive science
Fig 47 interdisciplinary nature of cognitive science Cognitive science is the interdisciplinary scientific study of mind and its processes. It examines what cognition is, what it does and how it works. It includes research on how information is processed (in faculties such as perception, language, memory, reasoning, and emotion), represented, and transformed in behaviour, (human or other animal) nervous system or machine (e.g., computer). Cognitive science consists of multiple research disciplines, including psychology, artificial intelligence, philosophy, neuroscience, linguistics, anthropology, sociology, and education. [183]It spans many levels of analysis, from low-level learning and decision mechanisms to high-level logic and planning; from neural circuitry to modular brain organization. The term cognitive science was coined by Christopher Longuet-Higgins in his 1973 commentary on the Lighthill report, which concerned the then-current state of Artificial Intelligence research. [184]In the same decade, the journal Cognitive Science and the Cognitive Science Society were founded.[185]In 1982, Vassar College became the first institution in the world to grant an undergraduate degree in Cognitive Science. [186] Principles
Levels of analysis
A central tenet of cognitive science is that a complete understanding of the mind/brain cannot be attained by studying only a single level. An example would be the problem of remembering a phone number and recalling it later. One approach to understanding this process would be to study behaviour through direct observation. A person could be presented with a phone number, asked to recall it after some delay. Then the accuracy of the response could be measured. Another approach would be to study the firings of individual neurons while a person is trying to remember the phone number. Neither of these experiments on their own would fully explain how the process of remembering a phone number works. 80
Even if the technology to map out every neuron in the brain in real-time was available, and it was known when each neuron was firing, it would still be impossible to know how a particular firing of neurons translates into the observed behaviour. Thus an understanding of how these two levels relate to each other is needed. This can be provided by a functional level account of the process. Studying a particular phenomenon from multiple levels creates a better understanding of the processes that occur in the brain to give rise to a particular behaviour. Marr [188] gave a famous description of three levels of analysis: 1. the computational theory, specifying the goals of the computation; 2. representation and algorithm, giving a representation of the input and output and the algorithm which transforms one into the other; and 3. The hardware implementation, how algorithm and representation may be physically realized. Interdisciplinary nature
Cognitive science is an interdisciplinary field with contributors from various fields, including psychology, neuroscience, and linguistics, philosophy of mind, computer science, anthropology, sociology, and biology. Cognitive science tends to view the world outside the mind much as other sciences do. Thus it too has an objective, observer-independent existence. The field is usually seen as compatible with the physical sciences, and uses the scientific method as well as simulation or Modelling, often comparing the output of models with aspects of human behaviour. Some doubt whether there is a unified cognitive science and prefer to speak of the cognitive sciences in plural. [189] Many, but not all, who consider themselves cognitive scientists have a functionalist view of the mind —the view that mental states are classified functionally, such that any system that performs the proper function for some mental state is considered to be in that mental state. According to some versions of functionalism, even non-human systems, such as other animal species, alien life forms, or advanced computers can, in principle, have mental states. Knowledge, and Processing, of Language
Fig48A well-known example of a Phrase structure tree. This is one way of representing human language that shows how different components are organized hierarchically. The ability to learn and understand language is an extremely complex process. Language is acquired within the first few years of life, and all humans under normal circumstances are able to acquire language proficiently. A major driving force in the theoretical linguistic field is discovering the nature that language must have in the abstract in order to be learned in such a fashion. Some of the driving research questions in studying how the brain itself processes language include: (1) to what extent is linguistic knowledge innate or learned? (2) Why is it more difficult for adults to acquire a second-language than it is for infants to acquire their first-language?And (3) how are humans able to understand novel sentences?
81
The study of language processing ranges from the investigation of the sound patterns of speech to the meaning of words and whole sentences. Linguistics often divides language processing into orthography, phonology and phonetics, morphology, syntax, semantics, and pragmatics. Many aspects of language can be studied from each of these components and from their interaction. The study of language processing in cognitive science is closely tied to the field of linguistics. Linguistics was traditionally studied as a part of the humanities, including studies of history, art and literature. In the last fifty years or so, more and more researchers have studied knowledge and use of language as a cognitive phenomenon, the main problems being how knowledge of language can be acquired and used, and what precisely it consists of. Linguists have found that, while humans form sentences in ways apparently governed by very complex systems, they are remarkably unaware of the rules that govern their own speech. Thus linguists must resort to indirect methods to determine what those rules might be, if indeed rules as such exist. In any event, if speech is indeed governed by rules, they appear to be opaque to any conscious consideration. Learning and development
Learning and development are the processes by which we acquire knowledge and information over time. Infants are born with little or no knowledge (depending on how knowledge is defined), yet they rapidly acquire the ability to use language, walk, and recognize people and objects. Research in learning and development aims to explain the mechanisms by which these processes might take place. A major question in the study of cognitive development is the extent to which certain abilities are innate or learned. This is often framed in terms of the nature versus nurture debate. The nativist view emphasizes that certain features are innate to an organism and are determined by its genetic endowment. The empiricist view, on the other hand, emphasizes that certain abilities are learned from the environment. Although clearly both genetic and environmental input is needed for a child to develop normally, considerable debate remains about how genetic information might guide cognitive development. In the area of language acquisition, for example, some (such as Steven Pinker)[10] have argued that specific information containing universal grammatical rules must be contained in the genes, whereas others (such as Jeffrey Elman and colleagues in Rethinking Innateness) have argued that Pinker's claims are biologically unrealistic. They argue that genes determine the architecture of a learning system, but that specific "facts" about how grammar works can only be learned as a result of experience. Memory
Memory allows us to store information for later retrieval. Memory is often thought of consisting of both a long-term and short-term store. Long-term memory allows us to store information over prolonged periods (days, weeks, years). We do not yet know the practical limit of long-term memory capacity. Shortterm memory allows us to store information over short time scales (seconds or minutes). Memory is also often grouped into declarative and procedural forms. Declarative memory--grouped into subsets of semantic and episodic forms of memory--refers to our memory for facts and specific knowledge, specific meanings, and specific experiences (e.g., who was the first president of the U.S.A.? or "What did I eat for breakfast four days ago?). Procedural memory allows us to remember actions and motor sequences (e.g. how to ride a bicycle) and is often dubbed implicit knowledge or memory. Cognitive scientists study memory just as psychologists do, but tend to focus in more on how memory bears on cognitive processes, and the interrelationship between cognition and memory. One example of this could be, what mental processes does a person go through to retrieve a long-lost memory? Or, what
82
differentiates between the cognitive process of recognition (seeing hints of something before remembering it, or memory in context) and recall (retrieving a memory, as in "fill-in-the-blank")?
The Turing Test The Turing test measures the performance of an allegedly intelligent machine against that of a human being, arguably the best and only standard for intelligent behaviour. The test, which Turing called the imitation game, places the machine and a human counterpart in rooms apart from a second human being, referred to as the interrogator. The interrogator is not able to see or speak directly to either of them, does not know which entity is actually the machine, and may communicate with them solely by use of a textual device such as a terminal. The interrogator is asked to distinguish the computer from the human being solely on the basis of their answers to questions asked over this device. If the interrogator cannot distinguish the machine from the human, then, Turing argues, the machine may be assumed to be intelligent. By isolating the interrogator from both the machine and the other human participant, the test ensures that the interrogator will not be biased by the appearance of the machine or any mechanical property of its voice. The interrogator is free, however, to ask any questions, no matter how devious or indirect, in an effort to uncover the computer's identity. For example, the interrogator may ask both subjects to perform a rather involved arithmetic calculation, assuming that the computer will be more likely to get it correct than the human; to counter this strategy, the computer will need to know when it should fail to get a correct answer to such problems in order to seem like a human. To discover the human's identity on the basis of emotional nature, the interrogator may ask both subjects to respond to a poem or work of art; this strategy will require that the computer have knowledge concerning the emotional makeup of human beings.
Fig 50 Weakness of turning test The important features of Turing's test are:
1. It attempts to give an objective notion of intelligence, i.e., the behaviour of a known intelligent being in response to a particular set of questions. This provides a standard for determining intelligence that avoids the inevitable debates over its "true" nature. 2. It prevents us from being sidetracked by such confusing and currently unanswerable questions as whether or not the computer uses the appropriate internal processes or whether or not the machine is actually conscious of its actions. 83
3. It eliminates any bias in favor of living organisms by forcing the interrogator to focus solely on the content of the answers to questions. Because of these advantages, the Turing test provides a basis for many of the schemes actually used to evaluate modern AI programs. A program that has potentially achieved intelligence in some area of expertise may be evaluated by comparing its performance on a given set of problems to that of a human expert. This evaluation technique is just a variation of the Turing test : a group of humans are asked to blindly compare the performance of a computer and a human being on a particular set of problems. As we will see, this methodology has become an essential tool in both the development and verification of modern expert systems. The Turing test, in spite of its intuitive appeal, is vulnerable to a number of justifiable criticisms. One of the most important of these is aimed at its bias toward purely symbolic problem-solving tasks. It does not test abilities requiring perceptual skill or manual dexterity, even though these are important components of human intelligence. Conversely, it is sometimes suggested that the Turing test needlessly constrains machine intelligence to fit a human mould. Perhaps machine intelligence is simply different from human intelligence and trying to evaluate it in human terms is a fundamental mistake. Do we really wish a machine would do mathematics as slowly and inaccurately as a human? Shouldn't an intelligent machine capitalize on its own assets, such as a large, fast, reliable memory, rather than trying to emulate human cognition? In fact, a number of modern AI practitioners (e.g., Ford and Hayes 1995) see responding to the full challenge of Turing's test as a mistake and a major distraction to the more important work at hand: developing general theories to explain the mechanisms of intelligence in humans and machines and applying those theories to the development of tools to solve specific, practical problems. Although we agree with the Ford and Hayes concerns in the large, we still see Turing's test as an important component in the verification and validation of modern AI software. Turing also addressed the very feasibility of constructing an intelligent program on a digital computer. By thinking in terms of a specific model of computation (an electronic discrete state computing machine), he made some well-founded conjectures concerning the storage capacity, program complexity, and basic design philosophy required for such a system. Finally, he addressed a number of moral, philosophical, and scientific objections to the possibility of constructing such a program in terms of an actual technology. Two of the objections cited by Turing are worth considering further. Lady Lovelace's Objection, first stated by Ada Lovelace, argues that computers can only do as they are told and consequently cannot perform original (hence, intelligent) actions . This objection has become a reassuring if somewhat dubious part of contemporary technological folklore. Expert systems (Section 1.2.3 and Chapter 8), especially in the area of diagnostic reasoning, have reached conclusions unanticipated by their designers. Indeed, a number of researchers feel that human creativity can be expressed in a computer program. The other related objection, the Argument from Informality of Behaviour, asserts the impossibility of creating a set of rules that will tell an individual exactly what to do under every possible set of circumstances. Certainly, the flexibility that enables a biological intelligence to respond to an almost infinite range of situations in a reasonable if not necessarily optimal fashion is a hallmark of intelligent behaviour. While it is true that the control structure used in most traditional computer programs does
not demonstrate great flexibility or originality, it is not true that all programs must be written in this fashion. Nobel laureate Herbert Simon has argued that much of the originality and variability of behaviour s hown by living creatures is due to the richness of their environment rather than the complexity of their own internal programs. In The Sciences of the Artificial, Simon (1981) describes an ant progressing circuitously
along an uneven and cluttered stretch of ground. Although the ant's path seems quite complex, Simon 84
argues that the ant's goal is very simple: to return to its colony as quickly as possible. The twists and turns in its path are caused by the obstacles it encounters on its way. Simon concludes that: An ant, viewed as a behaving system, is quite simple. The apparent complexity of its behaviour over time is largely a reflection of the complexity of the environment in which it finds itself.
Artificial Intelligence as a Positive and Negative Factor in Global Risk By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it. Of course this problem is not limited to the field of AI. Jacques Monod wrote: "A curious aspect of the theory of evolution is that everybody thinks he understands it." (Monod 1974.) My father, a physicist, complained about people making up their own theories of physics; he wanted to know why people did not make up their own theories of chemistry. (Answer: They do.) Nonetheless the problem seems to be unusually acute in Artificial Intelligence. The field of AI has a reputation for making huge promises and then failing to deliver on them. Most observers conclude that AI is hard; as indeed it is. But the embarrassment does not stem from the difficulty. It is difficult to build a star from hydrogen, but the field of stellar astronomy does not have a terrible reputation for promising to build stars and then failing. The critical inference is not that AI is hard, but that, for some reason, it is very easy for people to think they know far more about Artificial Intelligence than they actually do. Underestimating the power of intelligence
We tend to see individual differences instead of human universals. Thus when someone says the word "intelligence", we think of Einstein, instead of humans. Individual differences of human intelligence have a standard label, Spearman's g aka g-factor, a controversial interpretation of the solid experimental result that different intelligence tests are highly correlated with each other and with real-world outcomes such as lifetime income. (Jensen 1999.) Spearman's g is a statistical abstraction from individual differences of intelligence between humans, who as a species are far more intelligent than lizards. Spearman's g is abstracted from millimetre height differences among a species of g iants. We should not confuse Spearman's g with human general intelligence, our capacity to handle a wide range of cognitive tasks incomprehensible to other species. General intelligence is a between-species difference, a complex adaptation, and a human universal found in all known cultures. There may as yet be no academic consensus on intelligence, but there is no doubt about the existence, or the power, of the thing-to-be-explained. There is something about humans that let us set our footprints on the Moon. But the word "intelligence" commonly evokes pictures of the starving professor with an IQ of 160 and the billionaire CEO with an IQ of merely 120. Indeed there are differences of individual ability apart from "book smarts" which contribute to relative success in the human world: enthusiasm, social skills, education, musical talent, rationality. Note that each factor I listed is cognitive. Social skills reside in the brain, not the liver. And jokes aside, you will not find many CEOs, nor yet professors of academia, who are chimpanzees. You will not find many acclaimed rationalists, or artists, or poets, or leaders, or engineers, or skilled networkers, or martial artists, or musical composers who are mice. Intelligence is the foundation of human power, the strength that fuels our other arts. The danger of confusing general intelligence with g-factor is that it leads to tremendously underestimating the potential impact of Artificial Intelligence. (This applies to underestimating potential good impacts, as well as potential bad impacts.) Even the phrase "transhuman AI" or "artificial super intelligence" may still evoke images of book-smarts-in a- box: an AI that's really good at cognitive tasks stereotypically associated with "intelligence", like chess or abstract mathematics. But not superhumanly persuasive; or far better than humans at predicting and manipulating human social situations; or inhumanly clever in formulating long-term strategies. So instead 85
of Einstein, should we think of, say, the 19th-century political and diplomatic genius Otto von Bismarck? But that's only the mirror version of the error. The entire range from village idiot to Einstein, or from village idiot to Bismarck, fits into a small dot on the range from amoeba to human. If the word "intelligence" evokes Einstein instead of humans, then it may sound sensible to say that intelligence is no match for a gun, as if guns had grown on trees. It may sound sensible to say that intelligence is no match for money, as if mice used money. Human beings didn't start out with major assets in claws, teeth, armor, or any of the other advantages that were the daily currency of other species. If you had looked at humans from the perspective of the rest of the ecosphere, there was no hint that the squishy things would eventually clothe themselves in armored tanks. We invented the battleground on which we defeated lions and wolves. We did not match them claw for claw, tooth for tooth; we had our own ideas about what mattered. Such is the power of creativity. Vinge (1993) aptly observed that a future containing smarter-than-human minds is different in kind. Artificial Intelligence is not an amazing shiny expensive gadget to advertise in the latest tech magazines. Artificial Intelligence does not belong in the same graph that shows progress in medicine, manufacturing, and energy. Artificial Intelligence is not something you can casually mix into a lumpenfuturistic scenario of skyscrapers and flying cars and nanotechnological red blood cells that let you hold your breath for eight hours. Sufficiently tall skyscrapers don't potentially start doing their own engineering. Humanity did not rise to prominence on Earth by holding its breath longer than other species. The catastrophic scenario which stems from underestimating the power of intelligence is that someone builds a button, and doesn't care enough what the button does, because they don't think the button is powerful enough to hurt them. Or, since underestimating the power of intelligence implies a proportional underestimate of the potential impact of Artificial Intelligence, the (presently tiny) group of concerned researchers and grant makers and individual philanthropists who handle existential risks on behalf of the human species, will not pay enough attention to Artificial Intelligence. Or the wider field of AI will not pay enough attention to risks of strong AI, and therefore good tools and firm foundations for friendliness will not be available when it becomes possible to build strong intelligences. And one should not fail to mention - for it also impacts upon existential risk - that Artificial Intelligence could be the powerful solution to other existential risks, and by mistake we will ignore our best hope of survival. The point about underestimating the potential impact of Artificial Intelligence is symmetrical around potential good impacts and potential bad impacts. That is why the title of this chapter is "Artificial Intelligence as a Positive and Negative Factor in Global Risk", not "Global Risks of Artificial Intelligence." The prospect of AI interacts with global risk in more complex ways than that; if AI were a pure liability, matters would be simple. Capability and motive
There is a fallacy oft-committed in discussion of Artificial Intelligence, especially AI of superhuman capability. Someone says: "When technology advances far enough we'll be able to build minds far surpassing human intelligence. Now, it's obvious that how large a cheesecake you can make depends on your intelligence. Superintelligence could build enormous cheesecakes - cheesecakes the size of cities - by golly, the future will be full of giant cheesecakes!" The question is whether the superintelligence wants to build giant cheesecakes. The vision leaps directly from capability to actuality, without considering the necessary intermediate of motive. The following chains of reasoning, considered in isolation without supporting argument, all exhibit the Fallacy of the Giant Cheesecake:
86
• • •
A sufficiently powerful Artificial Intelligence could overwhelm any human resistance and wipe out humanity. [And the AI would decide to do so.] Therefore we should not build AI. A sufficiently powerful AI could develop new medical technologies capable of saving millions of human lives. [And the AI would decide to do so.] Therefore we should build AI. Once computers become cheap enough, the vast majority of jobs will be performable by Artificial Intelligence more easily than by humans. A sufficiently powerful AI would even be better than us at math, engineering, music, art, and all the other jobs we consider meaningful. [And the AI will decide to perform those jobs.] Thus after the invention of AI, humans will have nothing to do, and we'll starve or watch television.
Friendly AI
It would be a very good thing if humanity knew how to choose into existence a powerful optimization process with a particular target. Or in more colloquial terms, it would be nice if we knew how to build a nice AI. To describe the field of knowledge needed to address that challenge, I have proposed the term "Friendly AI". In addition to referring to a body of technique, "Friendly AI" might also refer to the product of technique - an AI created with specified motivations. When I use the term Friendly in either sense, I capitalize it to avoid confusion with the intuitive sense of "friendly". One common reaction I encounter is for people to immediately declare that Friendly AI is impossibility, because any sufficiently powerful AI will be able to modify its own source code to break any constraints placed upon it. The first flaw you should notice is a Giant Cheesecake Fallacy. Any AI with free access to its own source would, in principle, possess the ability to modify its own source code in a way that changed the AI's optimization target. This does not imply the AI has the motive to change its own motives. I would not knowingly swallow a pill that made me enjoy committing murder, because currently I prefer that my fellow humans not die. But what if I try to modify myself, and make a mistake? When computer engineers prove a chip valid - a good idea if the chip has 155 million transistors and you can't issue a patch afterward - the engineers use human-guided, machine-verified formal proof. The glorious thing about formal mathematical proof is that a proof of ten billion steps is just as reliable as a proof of ten steps. But human beings are not trustworthy to peer over a purported proof of ten billion steps; we have too high a chance of missing an error. And present-day theorem-proving techniques are not smart enough to design and prove an entire computer chip on their own - current algorithms undergo an exponential explosion in the search space. Human mathematicians can prove theorems far more complex than modern theoremprovers can handle, without being defeated by exponential explosion. But human mathematics is informal and unreliable; occasionally someone discovers a flaw in a previously accepted informal proof. The upshot is that human engineers guide a theoremprover through the intermediate steps of a proof. The human chooses the next lemma, and a complex theorem-prover generates a formal proof, and a simple verifier checks the steps. That's how modern engineers build reliable machinery with 155 million interdependent parts. Proving a computer chip correct requires a synergy of human intelligence and computer algorithms, as currently neither suffices on its own. Perhaps a true AI could use a similar combination of abilities when modifying its own code - would have both the capability to invent large designs without being defeated by exponential explosion, and also the ability to verify its steps with extreme reliability. That is one way a true AI might remain knowably stable in its goals, even after carrying out a large number of selfmodifications. This paper will not explore the above idea in detail. (Though see Schmidhuber 2003 for a related notion.) But one ought to think about a challenge, and study it in the best available technical detail, before declaring it impossible - especially if great stakes depend upon the answer. It is disrespectful to human 87
ingenuity to declare a challenge unsolvable without taking a close look and exercising creativity. It is an enormously strong statement to say that you cannot do a thing - which you cannot build a heavier-thanair flying machine that you cannot get useful energy from nuclear reactions that you cannot fly to the Moon. Such statements are universal generalizations, quantified over every single approach that anyone ever has or ever will think up for solving the problem. It only takes a single counterexample to falsify a universal quantifier. The statement that Friendly (or friendly) AI is theoretically impossible, dares to quantify over every possible mind design and every possible optimization process - including human beings, who are also minds, some of whom are nice and wish they were nicer. At this point there is any number of vaguely plausible reasons why Friendly AI might be humanly impossible, and it is still more likely that the problem is solvable but no one will get around to solving it in time. But one should not so quickly write off the challenge, especially considering the stakes. Technical failure and philosophical failure
Bostrom (2001) defines an existential catastrophe as one which permanently extinguishes Earthoriginating intelligent life or destroys a part of its potential. We can divide potential failures of attempted Friendly AI into two informal fuzzy categories, technical failure and philosophical failure. Technical failure is when you try to build an AI and it doesn't work the way you think it does - you have failed to understand the true workings of your own code. Philosophical failure is trying to build the wrong thing, so that even if you succeeded you would still fail to help anyone or benefit humanity. Needless to say, the two failures are not mutually exclusive. The border between these two cases is thin, since most philosophical failures are much easier to explain in the presence of technical knowledge. In theory you ought to first to say what you want, then figure out how to get it. In practice it often takes a deep technical understanding to figure out what you want. An example of philosophical failure
In the late 19th century, many honest and intelligent people advocated communism, all in the best of good intentions. The people who first invented and spread and swallowed the communist meme were, in sober historical fact, idealists. The first communists did not have the example of Soviet Russia to warn them. At the time, without benefit of hindsight, it must have sounded like a pretty good idea. After the revolution, when communists came into power and were corrupted by it, other motives may have come into play; but this itself was not something the first idealists predicted, however predictable it may have been. It is important to understand that the authors of huge catastrophes need not be evil, or even unusually stupid. If we attribute every tragedy to evil or unusual stupidity, we will look at ourselves, correctly perceive that we are not evil or unusually stupid, and say: "But that would never happen to us." What the first communist revolutionaries thought would happen, as the empirical consequence of their revolution, was that people's lives would improve: labourers would no longer work long hours at backbreaking labor and make little money from it. This turned out not to be the case, to put it mildly. But what the first communists thought would happen, was not so very different from what advocates of other political systems thought would be the empirical consequence of their favorite political systems. They thought people would be happy. They were wrong. Now imagine that someone should attempt to program a "Friendly" AI to implement communism, or libertarianism, or anarcho-feudalism, or favorite political system, believing that this shall bring about utopia. People's favorite political systems inspire blazing suns of positive affect, so the proposal will sound like a really good idea to the proposer. We could view the programmer's failure on a mor al or ethical level - say that it is the result of someone trusting themselves too highly, failing to take into account their own fallibility, refusing to consider the possibility that communism might be mistaken after all. But in the 88
language of Bayesian decision theory, there's a complementary technical view of the problem. From the perspective of decision theory, the choice for communism stems from combining an empirical belief with a value judgment. The empirical belief is that communism, when implemented, results in a specific outcome or class of outcomes: people will be happier, work fewer hours, and possess greater material wealth. This is ultimately an empirical prediction; even the part about happiness is a real property of brain states, though hard to measure. If you implement communism, either this outcome eventuates or it does not. The value judgment is that this outcome satisfices or is preferable to current conditions. Given a different empirical belief about the actual real-world consequences of a communist system, the decision may undergo a corresponding change. We would expect a true AI, an Artificial General Intelligence, to be capable of changing its empirical beliefs (Or its probabilistic world-model, etc.). If somehow Charles Babbage had lived before Nicolaus Copernicus, and somehow computers had been invented before telescopes, and somehow the programmers of that day and age successfully created an Artificial General Intelligence, it would not follow that the AI would believe forever after that the Sun orbited the Earth. The AI might transcend the factual error of its programmers, provided that the programmers understood inference rather better than they understood astronomy. To build an AI that discovers the orbits of the planets, the programmers need not know the math of Newtonian mechanics, only the math of Bayesian probability theory. The folly of programming an AI to implement communism, or any other political system, is that you're programming means instead of ends. You're programming in a fixed decision, without that decision being re-evaluable after acquiring improved empirical knowledge about the results of communism. You are giving the AI a fixed decision without telling the AI how to re-evaluate, at a higher level of intelligence, the fallible process which produced that decision. If I play chess against a stronger player, I cannot predict exactly where my opponent will move against me - if I could predict that, I would necessarily be at least that strong at chess myself. But I can predict the end result, which is a win for the other player. I know the region of possible futures my opponent is aiming for, which is what lets me predict the destination, even if I cannot see the path. When I am at my most creative, that is when it is hardest to predict my actions, and easiest to predict the consequences of my actions (Providing that you know and understand my goals!) If I want a better-than-human chess player, I have to program a search for winning moves. I can't program in specific moves because then the chess player won't be any better than I am. When I launch a search, I necessarily sacrifice my ability to predict the exact answer in advance. To get a really good answer you must sacrifice your ability to predict the answer, albeit not your ability to say what is the question. Such confusion as to program in communism directly, probably would not tempt an AGI programmer who speaks the language of decision theory. I would call it a philosophical failure, but blame it on lack of technical knowledge. An example of technical failure
"In place of laws constraining the behaviour of intelligent machines, we need to give them emotions that can guide their learning of behaviours. They should want us to be happy and prosper, which is the emotion we call love. We can design intelligent machines so their primary, innate emotion is unconditional love for all humans. First we can build relatively simple machines that learn to recognize happiness and unhappiness in human facial expressions, human voices and human body language. Then we can hard-wire the result of this learning as the innate emotional values of more complex intelligent machines, positively reinforced when we are happy and negatively reinforced when we are unhappy. Machines can learn algorithms for approximately predicting the future, as for example investors currently
89
use learning machines to predict future security prices. So we can program intelligent machines to learn algorithms for predicting future human happiness, and use those predictions as emotional values." -- Bill Hibbard (2001), Super-intelligent machines Once upon a time, the US Army wanted to use neural networks to automatically detect camouflaged enemy tanks. The researchers trained a neural net on 50 photos of camouflaged tanks in trees, and 50 photos of trees without tanks. Using standard techniques for supervised learning, the researchers trained the neural network to a weighting that correctly loaded the training set - output "yes" for the 50 photos of camouflaged tanks, and output "no" for the 50 photos of forest. This did not ensure, or even imply, that new examples would be classified correctly. The neural network might have "learned" 100 special cases that would not generalize to any new problem. Wisely, the researchers had originally taken 200 photos, 100 photos of tanks and 100 photos of trees. They had used only 50 of each for the training set. The researchers ran the neural network on the remaining 100 photos, and without further training the neural network classified all remaining photos correctly. Success confirmed! The researchers handed the finished work to the Pentagon, which soon handed it back, complaining that in their own tests the neural network did no better than chance at discriminating photos. It turned out that in the researchers' data set, photos of camouflaged tanks had been taken on cloudy days, while photos of plain forest had been taken on sunny days. The neural network had learned to distinguish cloudy days from sunny days, instead of distinguishing camouflaged tanks from empty forest. A technical failure occurs when the code does not do what you think it does, though it faithfully executes as you programmed it. More than one model can load the same data. Suppose we trained a neural network to recognize smiling human faces and distinguish them from frowning human faces. Would the network classify a tiny picture of a smiley face into the same attractor as a smiling human face? If an AI "hard-wired" to such code possessed the power - and Hibbard (2001) spoke of superintelligence - would the galaxy end up tiled with tiny molecular pictures of smiley-faces? This form of failure is especially dangerous because it will appear to work within a fixed context, then fail when the context changes. The researchers of the "tank classifier" story tweaked their neural network until it correctly loaded the training data, and then verified the network on additional data (without further tweaking). Unfortunately, both the training data and verification data turned out to share an assumption which held over the all data used in development, but not in all the real-world contexts where the neural network was called upon to function. In the story of the tank classifier, the assumption is that tanks are photographed on cloudy days. Suppose we wish to develop an AI of increasing power. The AI possesses a developmental stage where the human programmers are more powerful than the AI - not in the sense of mere physical control over the AI's electrical supply, but in the sense that the human programmers are smarter, more creative, and more cunning than the AI. During the developmental period we suppose that the programmers possess the ability to make changes to the AI's source code without needing the consent of the AI to do so. However, the AI is also intended to possess post developmental stages, including, in the case of Hibbard's scenario, superhuman intelligence. An AI of superhuman intelligence surely could not be modified without its consent. At this point we must rely on the previously laid-down goal system to function correctly, because if it operates in a sufficiently unforeseen fashion, the AI may actively resist our attempts to correct it - and, if the AI is smarter than a human, probably win. Trying to control a growing AI by training a neural network to provide its goal system faces the problem of a huge context change between the AI's developmental stage and post developmental stage. During the developmental stage, the AI may only be able to produce stimuli that fall into the "smiling human faces" category, by solving humanly provided tasks, as its makers intended. Flash forward to a time when the AI 90
is superhumanly intelligent and has built its own nanotech infrastructure, and the AI may be able to produce stimuli classified into the same attractor by tiling the galaxy with tiny smiling faces. Thus the AI appears to work fine during development, but produces catastrophic results after it becomes smarter than the programmers (!). There is a temptation to think, "But surely the AI will know that's not what we meant?" But the code is not given to the AI, for the AI to look over and hand back if it does the wrong thing. The code is the AI. Perhaps with enough effort and understanding we can write code that cares if we have written the wrong code - the legendary DWIM instruction, which among programmers stands for Do-What-I-Mean. (Raymond 2003.) But effort is required to write a DWIM dynamic, and nowhere in Hibbard's proposal is there mention of designing an AI that does what we mean, not what we say. Modern chips don't DWIM their code; it is not an automatic property. And if you messed up the DWIM itself, you would suffer the consequences. For example, suppose DWIM was defined as maximizing the satisfaction of the programmer with the code; when the code executed as superintelligence, it might rewrite the programmers' brains to be maximally satisfied with the code. I do not say this is inevitable; I only point out that Do-What-I-Mean is a major, nontrivial technical challenge of Friendly AI. Hardware
People tend to think of large computers as the enabling factor for Artificial Intelligence. This is, to put it mildly, an extremely questionable assumption. Outside futurists discussing Artificial Intelligence talk about hardware progress because hardware progress is easy to measure - in contrast to understanding of intelligence. It is not that there has been no progress, but that the progress cannot be charted on neat PowerPoint graphs. Improvements in understanding are harder to report on, and therefore less reported. Rather than thinking in terms of the "minimum" hardware "required" for Artificial Intelligence, think of a minimum level of researcher understanding that decreases as a function of hardware improvements. The better the computing hardware, the less understanding you need to build an AI. The extremely case is natural selection, which used a ridiculous amount of brute computational force to create human intelligence using no understanding, only nonchance retention of chance mutations. Increased computing power makes it easier to build AI, but there is no obvious reason why increased computing power would help make the AI Friendly. Increased computing power makes it easier to use brute force; easier to combine poorly understood techniques that work. Moore's Law steadily lowers the barrier that keeps us from building AI without a deep understanding of cognition. It is acceptable to fail at AI and at Friendly AI. It is acceptable to succeed at AI and at Friendly AI. What is not acceptable is succeeding at AI and failing at Friendly AI. Moore's Law makes it easier to do exactly that. "Easier", but thankfully not easy. I doubt that AI will be "easy" at the time it is finally built - simply because there are parties who will exert tremendous effort to build AI, and one of them will succeed after AI first becomes possible to build with tremendous effort. Moore's Law is an interaction between Friendly AI and other technologies, which adds of overlooked existential risk to other technologies. We can imagine that molecular nanotechnology is developed by a benign multinational governmental consortium and that they successfully avert the physical-layer dangers of nanotechnology. They straightforwardly prevent accidental replicator releases, and with much greater difficulty put global defences in place against malicious replicators; they restrict access to "root level" nanotechnology while distributing configurable nanoblocks, et cetera. (See Phoenix and Treder, this volume.) But nonetheless nanocomputers become widely available, either because attempted restrictions are bypassed, or because no restrictions are attempted. And then someone brute-forces an Artificial Intelligence which is nonFriendly; and so the curtain is rung down. This scenario is especially worrying because incredibly powerful nanocomputers would be among the first, the easiest, and the safest-seeming applications of molecular nanotechnology. 91
What of regulatory controls on supercomputers? I certainly wouldn't rely on it to prevent AI from ever being developed; yesterday's supercomputer is tomorrow's laptop. The standard reply to a regulatory proposal is that when nanocomputers are outlawed, only outlaws will have nanocomputers. The burden is to argue that the supposed benefits of reduced distribution outweigh the inevitable risks of uneven distribution. For myself I would certainly not argue in favor of regulatory restrictions on the use of supercomputers for Artificial Intelligence research; it is a proposal of dubious benefit which would be fought tooth and nail by the entire AI community. But in the unlikely event that a proposal made it that far through the political process, I would not expend any significant effort on fighting it, because I don't expect the good guys to need access to the "supercomputers" of their day. Friendly AI is not about bruteforcing the problem. I can imagine regulations effectively controlling a small set of ultra-expensive computing resources that are presently considered "supercomputers". But computers are everywhere. It is not like the problem of nuclear proliferation, where the main emphasis is on controlling plutonium and enriched uranium. The raw materials for AI are already everywhere. That cat is so far out of the bag that it's in your wristwatch, cellphone, and dishwasher. This too is a special and unusual factor in Artificial Intelligence as an existential risk. We are separated from the risky regime, not by large visible installations like isotope centrifuges or particle accelerators, but only by missing knowledge. To use a perhaps over-dramatic metaphor, imagine if subcritical masses of enriched uranium had powered cars and ships throughout the world, before Leo Szilard first thought of the chain reaction. Threats and promises
It is a risky intellectual endeavour to predict specifically how a benevolent AI would help humanity, or an unfriendly AI harm it. There is the risk of conjunction fallacy: added detail necessarily reduces the joint probability of the entire story, but subjects often assign higher probabilities to stories which include strictly added details. (See Yudkowsky, this volume, on cognitive biases). There is the risk - virtually the certainty - of failure of imagination; and the risk of Giant Cheesecake Fallacy that leaps from capability to motive. Nonetheless I will try to solidify threats and promises. The future has a reputation for accomplishing feats which the past thought impossible. Future civilizations have even broken what past civilizations thought (incorrectly, of course) to be the laws of physics. If prophets of 1900 AD - never mind 1000 AD – had tried to bound the powers of human civilization a billion years later, some of those impossibilities would have been accomplished before the century was out; transmuting lead into gold, for example. Because we remember future civilizations surprising past civilizations, it has become cliche that we can't put limits on our great-grandchildren. And yet everyone in the 20th century, in the 19th century, and in the 11th century, was human. We can distinguish three families of unreliable metaphors for imagining the capability of a smarter-than-human Artificial Intelligence: •
•
•
G-factor metaphors: Inspired by differences of individual intelligence between humans. AIs will patent new technologies, publish ground-breaking research papers, make money on the stock market, or lead political power blocs. History metaphors: Inspired by knowledge differences between past and future human civilizations. AIs will swiftly invent the kind of capabilities that cliché would attribute to human civilization a century or millennium from now: molecular nanotechnology; interstellar travel; computers performing 1025 operations per second. Species metaphors: Inspired by differences of brain architecture between species.
92
AI versus human intelligence
I do not think it plausible that Homo sapiens will continue into the indefinite future, thousands or millions of billions of years, without any mind ever coming into existence that breaks the current upper bound on intelligence. If so, there must come a time when humans first face the challenge of smarter-than-human intelligence. If we win the first round of the challenge, then humankind may call upon smarter-thanhuman intelligence with which to confront later rounds. Perhaps we would rather take some other route than AI to smarter-than-human intelligence - say, augment humans instead? To pick one extreme example, suppose the one says: The prospect of AI makes me nervous. I would rather that, before any AI is developed, individual humans are scanned into computers, neuron by neuron, and then upgraded, slowly but surely, until they are super-smart; and that is the ground on which humanity should confront the challenge of superintelligence. We are then faced with two questions: Is this scenario possible? And if so, is this scenario desirable? (It is wiser to ask the two questions in that order, for reasons of rationality: we should avoid getting emotionally attached to attractive options that are not actually options.) Let us suppose an individual human is scanned into computer, neuron by neuron, as proposed in Moravec (1988). It necessarily follows that the computing capacity used considerably exceeds the computing power of the human brain. By hypothesis, the computer runs a detailed simulation of a biological human brain, executed in sufficient fidelity to avoid any detectable high-level effects from systematic low-level errors. Any accident of biology that affects information-processing in any way, we must faithfully simulate to sufficient precision that the overall flow of processing remains isomorphic. To simulate the messy biological computer that is a human brain, we need far more useful computing power than is embodied in the messy human brain itself. The most probable way we would develop the ability to scan a human brain neuron by neuron - in sufficient detail to capture every cognitively relevant aspect of neural structure - would be the invention of sophisticated molecular nanotechnology. Molecular nanotechnology could probably produce a desktop computer with total processing power exceeding the aggregate brainpower of the entire current human population. (Bostrom 1998; Moravec 1999; Merkle and Drexler 1996; Sandberg 1999.) Furthermore, if technology permits us to scan a brain in sufficient fidelity to execute the scan as code, it follows that for some years previously, the technology has been available to obtain extremely detailed pictures of processing in neural circuitry, and presumably researchers have been doing their best to understand it. Furthermore, to upgrade the upload - transform the brain scan so as to increase the intelligence of the mind within - we must necessarily understand the high-level functions of the brain, and how they contribute usefully to intelligence, in excellent detail. Furthermore, humans are not designed to be improved, either by outside neuroscientists, or by recursive self-improvement internally. Natural selection did not build the human brain to be humanly hackable. All complex machinery in the brain has adapted to operate within narrow parameters of brain design. Suppose you can make the human smarter, let alone superintelligent; does the human remain sane? The human brain is very easy to perturb; just changing the balance of neurotransmitters can trigger schizophrenia, or other disorders. Deacon (1997) has an excellent discussion of the evolution of the human brain, how delicately the brain's elements may be balanced, and how this is reflected in modern brain dysfunctions. The human brain is not end-usermodifiable. All of this makes it rather implausible that the first human being would be scanned into a computer and sanely upgraded before anyone anywhere first built an Artificial Intelligence. At the point where technology first becomes capable of uploading, this implies overwhelmingly more computing power, and probably far better cognitive science, than is required to build an AI. Building a 747 from scratch is not easy. But is it easier to: 93
• • • • • • • •
Start with the existing design of a biological bird, and incrementally modify the design through a series of successive stages, each stage independently viable, such that the endpoint is a bird scaled up to the size of a 747, which actually flies, as fast as a 747, and then carry out this series of transformations on an actual living bird, Without killing the bird or making it extremely uncomfortable?
I'm not saying it could never, ever be done. I'm saying that it would be easier to build the 747, and then have the 747, metaphorically speaking, upgrade the bird. "Let's just scale up an existing bird to the size of a 747" is not a clever strategy that avoids dealing with the intimidating theoretical mysteries of aerodynamics. Perhaps, in the beginning, all you know about flight is that a bird has the mysterious essence of flight, and the materials with which you must build a 747 are just lying there on the ground. But you cannot sculpt the mysterious essence of flight, even as it already resides in the bird, until flight has ceased to be a mysterious essence unto you. The above argument is directed at a deliberately extreme case. The general point is that we do not have total freedom to pick a path that sounds nice and reassuring, or that would make a good story as a science fiction novel. We are constrained by which technologies are likely to precede others. I am not against scanning human beings into computers and making them smarter, but it seems exceedingly unlikely that this will be the ground on which humanity first confronts the challenge of smarter-thanhuman intelligence. With various strict subsets of the technology and knowledge required to upload and upgrade humans, one could: • • • •
Upgrade biological brains in-place (for example, by adding new neurons which will be usefully wired in); or usefully interface computers to biological human brains; or usefully interface human brains with each other; Or construct Artificial Intelligence.
Furthermore, it is one thing to sanely enhance an average human to IQ 140, and another to enhance a Nobel Prize winner to something beyond human. (Leaving aside quibbles about the suitability of IQ, or Nobel-Prize-winning, as a measure of fluid intelligence; please excuse my metaphors.) Taking Piracetam (or drinking caffeine) may, or may not, make at least some people smarter; but it will not make you substantially smarter than Einstein. In which case we haven't won any significant new capabilities; we haven't made further rounds of the problem easier; we haven't broken the upper bound on the intelligence available to deal with existential risks. From the standpoint of managing existential risk, any intelligence enhancement technology which doesn't produce a (nice, sane) mind literally smarter than human, begs the question of whether the same time and effort could be more productively spent to find an extremely smart modern-day human and unleash them on the same problem. Furthermore, the farther you go from the "natural" design bounds of the human brain – the ancestral condition represented by the brain itself, to which individual brain components are adapted - the greater the danger of individual insanity. If the augment is substantially smarter than human, this too is a global catastrophic risk. How much damage can an evil augmented human do? Well... how creative are they? The first question that comes to my mind is, "Creative enough to build their own recursively selfimproving AI?" Radical human intelligence enhancement techniques raise their own safety issues. Again, I am not claiming these problems as engineering impossibilities; only pointing out that the problems exist. AI has safety issues; so does human intelligence enhancement. Not everything that clanks is your enemy, 94
and not everything that squishes is your friend. On the one hand, a nice human starts out with all the immense moral, ethical, and architectural complexity that describes what we mean by a "friendly" decision. On the other hand, an AI can be designed for stable recursive self-improvement, and shaped to safety: natural selection did not design the human brain with multiple rings of precautionary measures, conservative decision processes, and orders of magnitude of safety margin. Human intelligence enhancement is a question in its own right, not a subtopic of Artificial Intelligence; and this chapter lacks space to discuss it in detail. It is worth mentioning that I considered both human intelligence enhancement and Artificial Intelligence at the start of my career, and decided to allocate my efforts to Artificial Intelligence. Primarily this was because I did not expect useful, human-transcending intelligence enhancement techniques to arrive in time to substantially impact the development of recursively self-improving Artificial Intelligence. I would be pleasantly surprised to be proven wrong about this. But I do not think that it is a viable strategy to deliberately choose not to work on Friendly AI, while others work on human intelligence enhancement, in hopes that augmented humans will solve the problem better. I am not willing to embrace a strategy which fails catastrophically if human intelligence enhancement takes longer than building AI (Or vice versa, for that matter.) I fear that working with biology will just take too much time – there will be too much inertia, too much fighting of poor design decisions already made by natural selection. I fear regulatory agencies will not approve human experiments. And even human geniuses take years to learn their art; the faster the augment has to learn, the more difficult it is to augment someone to that level. I would be pleasantly surprised if augmented humans showed up and built a Friendly AI before anyone else got the chance. But someone who would like to see this outcome will probably have to work hard to speed up intelligence enhancement technologies; it would be difficult to convince me to slow down. If AI is naturally far more difficult than intelligence enhancement, no harm done; if building a 747 is naturally easier than inflating a bird, then the wait could be fatal. There is a relatively small region of possibility within which deliberately not working on Friendly AI could possibly help, and a large region within which it would be either irrelevant or harmful. Even if human intelligence enhancement is possible, there are real, difficult safety considerations; I would have to seriously ask whether we wanted Friendly AI to precede intelligence enhancement, rather than vice versa. I do not assign strong confidence to the assertion that Friendly AI is easier than human augmentation, or that it is safer. There are many conceivable pathways for augmenting a human. Perhaps there is a technique which is easier and safer than AI, which is also powerful enough to make a difference to existential risk. If so, I may switch jobs. But I did wish to point out some considerations which argue against the unquestioned assumption that human intelligence enhancement is easier, safer, and powerful enough to make a difference.
Overview of AI Application Areas Like most sciences, AI is decomposed into a number of subdisciplines that, while sharing an essential approach to problem solving, have concerned themselves with different applications. In this section we outline several of these major application areas and their contributions to artificial intelligence as a whole. Game Playing
Much of the early research in state space search was done using common board games such as checkers, chess, and the 15-puzzle. In addition to their inherent intellectual appeal, board games have certain properties that made them ideal subjects for this early work. Most games are played using a well-defined set of rules: this makes it easy to generate the search space and frees the researcher from many of the ambiguities and complexities inherent in less structured problems. The board configurations used in playing these games are easily represented on a computer, requiring none of the complex formalisms 95
needed to capture the semantic subtleties of more complex problem domains. As games can be easily played, testing a game-playing program presents no financial or ethical burden. State space search, the paradigm underlying most game-playing research, is presented in Chapters 3 and 4. Games can generate extremely large search spaces. These are large and complex enough to require powerful techniques for determining what alternatives to explore in the problem space. These techniques are called heuristics and constitute a major area of AI research. A heuristic is a useful but potentially fallible problem-solving strategy, such as checking to make sure that an unresponsive appliance is plugged in before assuming that it is broken or to castle in order to try and protect your king from capture in a chess game. Much of what we commonly call intelligence seems to reside in the heuristics used by humans to solve problems. Because most of us have some experience with these simple games, it is possible to devise and test the effectiveness of our own heuristics. We do not need to find and consult an expert in some esoteric problem area such as medicine or mathematics (chess is an obvious exception to this rule). For these reasons, games provide a rich domain for the study of heuristic search. Chapter 4 introduces heuristics using these simple games; Chapter 8 extends their application to expert systems. Game-playing programs, in spite of their simplicity, offer their own challenges, including an opponent whose moves may not be deterministically anticipated, Chapters 5 and 8. This presence of the opponent further complicates program design by adding an element of unpredictability and the need to consider psychological as well as tactical factors in game strategy. Automated Reasoning and Theorem Proving
We could argue that automatic theorem proving is the oldest branch of artificial intelligence, tracing its roots back through Newell and Simon's Logic Theorist (Newell and Simon 1963 a) and General Problem Solver (Newell and Simon 1963 b), through Russell and Whitehead's efforts to treat all of mathematics as the purely formal derivation of theorems from basic axioms, to its origins in the writings of Babbage and Leibniz. In any case, it has certainly been one of the most fruitful branches of the field. Theorem-proving research was responsible for much of the early work in formalizing search algorithms and developing formal representation languages such as the predicate calculus (Chapter 2) and the logic programming language PROLOG (Chapter 15). Most of the appeal of automated theorem proving lies in the rigor and generality of logic. Because it is a formal system, logic lends itself to automation. A wide variety of problems can be attacked by representing the problem description and relevant background information as logical axioms and treating problem instances as theorems to be proved. This insight is the basis of work in automatic theorem proving and mathematical reasoning systems (Chapter 13). Unfortunately, early efforts at writing theorem provers failed to develop a system that could consistently solve complicated problems. This was due to the ability of any reasonably complex logical system to generate an infinite number of provable theorems: without powerful techniques (heuristics) to guide their search, automated theorem provers proved large numbers of irrelevant theorems before stumbling onto the correct one. In response to this inefficiency, many argue that purely formal, syntactic methods of guiding search are inherently incapable of handling such a huge space and that the only alternative is to rely on the informal, ad hoc strategies that humans seem to use in solving problems. This is the approach underlying the development of expert systems (Chapter 8), and it has proved to be a fruitful one. Still, the appeal of reasoning based in formal mathematical logic is too strong to ignore. Many important problems such as the design and verification of logic circuits, verification of the correctness of computer programs, and control of complex systems seem to respond to such an approach. In addition, the 96
theorem-proving community has enjoyed success in devising powerful solution heuristics that rely solely on an evaluation of the syntactic form of a logical expression, and as a result, reducing the complexity of the search space without resorting to the ad hoc techniques used by most human problem solvers. Another reason for the continued interest in automatic theorem provers is the realization that such a system does not have to be capable of independently solving extremely complex problems without human assistance. Many modern theorem provers function as intelligent assistants, letting humans perform the more demanding tasks of decomposing a large problem into subproblems and devising heuristics for searching the space of possible proofs. The theorem prover then performs the simpler but still demanding task of proving lemmas, verifying smaller conjectures, and completing the formal aspects of a proof outlined by its human associate (Boyer and Moore 1979, Bundy 1988, Veroff 1997). Expert Systems
One major insight gained from early work in problem solving was the importance of domain-specific knowledge. A doctor, for example, is not effective at diagnosing illness solely because she possesses some innate general problem-solving skill; she is effective because she knows a lot about medicine. Similarly, a geologist is effective at discovering mineral deposits because he is able to apply a good deal of theoretical and empirical knowledge about geology to the problem at hand. Expert knowledge is a combination of a theoretical understanding of the problem and a collection of heuristic problem-solving rules that experience has shown to be effective in the domain. Expert systems are constructed by obtaining this knowledge from a human expert and coding it into a form that a computer may apply to similar problems. This reliance on the knowledge of a human domain expert for the system's problem solving strategies is a major feature of expert systems. Although some programs are written in which the designer is also the source of the domain knowledge, it is far more typical to see such programs growing out of collaboration between a domain expert such as a doctor, chemist, geologist, or engineer and a separate artificial intelligence specialist. The domain expert provides the necessary knowledge of the problem domain through a general discussion of her problem-solving methods and by demonstrating those skills on a carefully chosen set of sample problems. The AI specialist, or knowledge engineer, as expert systems designers are often known, is responsible for implementing this knowledge in a program that is both effective and seemingly intelligent in its behaviour. Once such a program has been written, it is necessary to refine its expertise through a process of giving it example problems to solve, letting the domain expert criticize its behaviour, and making any required changes or modifications to the program's knowledge. This process is repeated until the program has achieved the desired level of performance. One of the earliest systems to exploit domain-specific knowledge in problem solving was DENDRAL, developed at Stanford in the late 1960s (Lindsay et al. 1980). DENDRAL was designed to infer the structure of organic molecules from their chemical formulas and mass spectrographic information about the chemical bonds present in the molecules. Because organic molecules tend to be very large, the number of possible structures for these molecules tends to be huge. DENDRAL addresses the problem of this large search space by applying the heuristic knowledge of expert chemists to the structure elucidation problem. DENDRAL's methods proved remarkably effective, routinely finding the correct structure out of millions of possibilities after only a few trials. The approach has proved so successful that descendants of the system are used in chemical and pharmaceutical laboratories throughout the world. Whereas DENDRAL was one of the first programs to effectively use domain-specific knowledge to achieve expert level problem-solving performance, MYCIN established the methodology of contemporary expert systems (Buchanan and Shortliff 1984). MYCIN uses expert medical knowledge to diagnose and prescribe treatment for spinal meningitis and bacterial infections of the blood.
97
MYCIN, developed at Stanford in the mid-1970s, was one of the first programs to address the problems of reasoning with uncertain or incomplete information. MYCIN provided clear and logical explanations of its reasoning, used a control structure appropriate to the specific problem domain, and identified criteria to reliably evaluate its performance. Many of the expert system development techniques currently in use were first developed in the MYCIN project (Chapter 8). Other classic expert systems include the PROSPECTOR program for determining the probable location and type of ore deposits based on geological information about a site (Duda et al. 1979 a, 1979 b), the INTERNIST program for performing diagnosis in the area of internal medicine, the Dipmeter Advisor for interpreting the results of oil well drilling logs (Smith and Baker 1983), and XCON for configuring VAX computers. XCON was developed in 1981, and at one time every VAX sold by Digital Equipment Corporation was configured by that software. Numerous other expert systems are currently solving problems in areas such as medicine, education, business, design, and science (Waterman 1986, Durkin 1994). It is interesting to note that most expert systems have been written for relatively specialized, expert level domains. These domains are generally well studied and have clearly defined problem-solving strategies. Problems that depend on a more loosely defined notion of "common sense" are much more difficult to solve by these means. In spite of the promise of expert systems, it would be a mistake to overestimate the ability of this technology. Current deficiencies include:
1. Difficulty in capturing "deep" knowledge of the problem domain. MYCIN, for example, lacks any real knowledge of human physiology. It does not know what blood does or the function of the spinal cord. Folklore has it that once, when selecting a drug for treatment of meningitis, MYCIN asked whether the patient was pregnant, even though it had been told that the patient was male. Whether this actually occurred or not, it does illustrate the potential narrowness of knowledge in expert systems. 2. Lack of robustness and flexibility. If humans are presented with a problem instance that they cannot solve immediately, they can generally return to an examination of first principles and come up with some strategy for attacking the problem. Expert systems generally lack this ability. 3. Inability to provide deep explanations. Because expert systems lack deep knowledge of their problem domains, their explanations are generally restricted to a description of the steps they took in finding a solution. For example, they often cannot tell "why" a certain approach was taken. 4. Difficulties in verification. Though the correctness of any large computer system is difficult to prove, expert systems are particularly difficult to verify. This is a serious problem, as expert systems technology is being applied to critical applications such as air traffic control, nuclear reactor operations, and weapons systems. 5. Little learning from experience. Current expert systems are handcrafted; once the system is completed, its performance will not improve without further attention from its programmers, leading to doubts about the intelligence of such systems. In spite of these limitations, expert systems have proved their value in a number of important applications. It is hoped that these limitations will only encourage the student to pursue this important branch of computer science.
98
Fig 51Anautomated online assistant providing customer service on a web page - one of many applications of artificial intelligence. The potential applications of Artificial Intelligence are abundant. They stretch from the military for
autonomous control and target identification, to the entertainment industry for computer games and robotic pets. Let’s also not forget big establishments dealing with huge amounts of information such as hospitals, banks and insurances, which can use AI to predict customer behaviour and detect trends. As you may expect, the business of Artificial Intelligence is becoming one of the major driving forces for research. With an ever growing market to satisfy, there's plenty of room for more personnel. So if you know what you're doing, there's plenty of money to be made from interested big companies! Artificial intelligence techniques are pervasive and are too numerous to list. Frequently, when a technique reaches mainstream use, it is no longer considered artificial intelligence; this phenomenon is described as the AI effect.[143]Almost every branch of science and engineering currently shares the tools and techniques available in the domain of artificial intelligence. However, for the sake of the convenience of the readers, we mention here a few typical applications, where AI plays a significant and decisive role in engineering automation. Image Understanding and Computer Vision
A digital image can be regarded as a two-dimensional array of pixels containing gray levels corresponding to the intensity of the reflected illumination received by a video camera. For interpretation of a scene, its image should be passed through three basic processes: low, medium and high level vision.
99
The importance of low level vision is to pre-process the image by filtering from noise. The medium level vision system deals with enhancement of details and segmentation (i.e., partitioning the image into objects of interest). The high level vision system includes three steps: recognition of the objects from the segmented image, labelling of the image and interpretation of the scene. Most of the AI tools and techniques are required in high level vision systems. Recognition of objects from its image can be carried out through a process of pattern classification, which at present is realized by supervised learning algorithms. The interpretation process, on the other hand, requires knowledge-based computation.
Fig.52: Basic steps in scene interpretation. Speech and Natural Language Understanding
Understanding of speech and natural languages is basically two classical problems. In speech analysis, the main problem is to separate the syllables of a spoken word and determine features like amplitude, and fundamental and harmonic frequencies of each syllable. The words then could be identified from the extracted features by pattern classification techniques. Recently, artificial neural networks have been employed to classify words from their features. The problem of understanding natural languages like English, on the other hand, includes syntactic and semantic interpretation of the words in a sentence, and sentences in a paragraph. The syntactic steps are required to analyse the sentences by its grammar and are similar with the steps of compilation. The semantic analysis, which is performed following the syntactic analysis, determines the meaning of the sentences from the association of the words and that of a paragraph from the closeness of the sentences. A robot capable of understanding speech in a natural language will be of immense importance, for it could execute any task verbally communicated to it. The phonetic typewriter, which prints the words pronounced by a person, is another recent invention where speech understanding is employed in a commercial application. Scheduling
In a scheduling problem, one has to plan the time schedule of a set of events to improve the time efficiency of the solution. For instance in a class-routine scheduling problem, the teachers are allocated to different classrooms at different time slots, and we want most classrooms to be occupied most of the time. In a flowshop scheduling problem, a set of jobs J1 and J2 (say) are to be allocated to a set of machines M1, M2 and M3. (Say). We assume that each job requires some operations to be done on all these machines in a fixed order say, M1, M2 and M3. Now, what should be the schedule of the jobs (J1-J2) or (J2 -J1), so that the completion time of both the jobs, called the make-span, is minimized? Let the processing time of jobs J1 and J2 on machines M1, M2 and M3 be (5, 8, 7) and (8, 2, 3) respectively. The 100
Gantt charts in fig. (a) And (b) describe the make-spans for the schedule of jobs J1 - J2 and J2 - J1 respectively. It is clear from these figures that J1-J2 schedule requires less make-span and is thus preferred.
Fig.53: The Gantt charts for the flowshop scheduling problem with 2 jobs and 3 machines. Flowshop scheduling problems are a NP complete problem and determination of optimal scheduling (for minimizing the make-span) thus requires an exponential order of time with respect to both machine-size and job-size. Finding a sub-optimal solution is thus preferred for such scheduling problems. Recently, artificial neural nets and genetic algorithms have been employed to solve this problem. The heuristic search, to be discussed shortly, has also been used for handling this problem. Intelligent Control
In process control, the controller is designed from the known models of the process and the required control objective. When the dynamics of the plant is not completely known, the existing techniques for controller design no longer remain valid. Rule-based control is appropriate in such situations. In a rulebased control system, the controller is realized by a set of production rules intuitively set by an expert control engineer. The antecedent (premise) part of the rules in a rule-based system is searched against the dynamic response of the plant parameters. The rule whose antecedent part matches with the plant response is selected and fired. When more than one rule is fireable, the controller resolves the conflict by 101
a set of strategies. On the other hand, there exist situations when the antecedent part of no rules exactly matches with the plant responses. Such situations are handled with fuzzy logic, which is capable of matching the antecedent parts of rules partially/ approximately with the dynamic plant responses. Fuzzy control has been successfully used in many industrial plants. One typical application is the power control in a nuclear reactor. Besides design of the controller, the other issue in process control is to design a plant (process) estimator, which attempts to follow the response of the actual plant, when both the plant and the estimator are jointly excited by a common input signal. The fuzzy and artificial neural network-based learning techniques have recently been identified as new tools for plant estimation. Modelling Human Performance
Although much of the above discussion uses human intelligence as a reference point in considering artificial intelligence, it does not follow that programs should pattern them after the organization of the human mind. Indeed, many AI programs are engineered to solve some useful problem without regard for their similarities to human mental architecture. Even expert systems, while deriving much of their knowledge from human experts, do not really attempt to simulate human internal mental problem solving processes. If performance is the only criterion by which a system will be judged, there may be little reason to attempt to simulate human problem-solving methods; in fact, programs that take nonhuman approaches to solving problems are often more successful than their human counterparts. Still, the design of systems that explicitly model aspects of human performance is a fertile area of research in both artificial intelligence and psychology. Human performance Modelling, in addition to providing AI with much of its basic methodology, has proved to be a powerful tool for formulating and testing theories of human cognition. The problemsolving methodologies developed by computer scientists have given psychologists a new metaphor for exploring the human mind. Rather than casting theories of cognition in the vague language used in early research or abandoning the problem of describing the inner workings of the human mind entirely (as suggested by the behaviorists), many psychologists have adopted the language and theory of computer science to formulate models of human intelligence. Not only do these techniques provide a new vocabulary for describing human intelligence, but also computer implementations of these theories offer psychologists an opportunity to empirically test, critique, and refine their ideas (Luger 1994). Further discussion of the relationship between artificial and human intelligence is found throughout this book and is summarized in Chapter 17. Planning and Robotics
Research in planning began as an effort to design robots that could perform their tasks with some degree of flexibility and responsiveness to the outside world. Briefly, planning assumes a robot that is capable of performing certain atomic actions. It attempts to find a sequence of those actions that will accomplish some higher-level task, such as moving across an obstacle-filled room. Planning is a difficult problem for a number of reasons, not the least of which is the size of the space of possible sequences of moves. Even an extremely simple robot is capable of generating a vast number of potential move sequences. Imagine, for example, a robot that can move forward, backward, right, or left, and consider how many different ways that robot can possibly move around a room. Assume also that there are obstacles in the room and that the robot must select a path that moves around them in some efficient fashion. Writing a program that can intelligently discover the best path under these circumstances, without being overwhelmed by the huge number of possibilities, requires sophisticated techniques for representing spatial knowledge and controlling search through possible environments.
102
One method that human beings use in planning is hierarchical problem decomposition. If you are planning a trip from Albuquerque to London, you will generally treat the problems of arranging a flight, getting to the airport, making airline connections, and finding ground transportation in London separately, even though they are all part of a bigger overall plan. Each of these may be further decomposed into smaller subproblems such as finding a map of the city, negotiating the subway system, and finding a decent pub. Not only does this approach effectively restrict the size of the space that must be searched, but also allows saving of frequently used subplans for future use. While humans plan effortlessly, creating a computer program that can do the same is a difficult challenge. A seemingly simple task such as breaking a problem into independent subproblems actually requires sophisticated heuristics and extensive knowledge about the planning domain. Determining what subplans should be saved and how they may be generalized for future use is an equally difficult problem. A robot that blindly performs a sequence of actions without responding to changes in its environment or being able to detect and correct errors in its own plan could hardly be considered intelligent. Often, a robot will have to formulate a plan based on incomplete information and correct its behaviour as it executes the plan. A robot may not have adequate sensors to locate all obstacles in the way of a projected path. Such a robot must begin moving through the room based on what it has "perceived" and correct its path as other obstacles are detected. Organizing plans in a fashion that allows response to changing environmental conditions is a major problem for planning (Lewis and Luger 2000). Finally, robotics was one of the research areas in AI that produced many of the insights supporting agentoriented problem solving (Section 1.1.4). Frustrated by both the complexities of maintaining the large representational space as well as the design of adequate search algorithms for traditional planning, researchers, including Agre and Chapman (1987) and Brooks (1991a) restated the larger problem in terms of the interaction of multiple semi-autonomous agents. Each agent was responsible for its own portion of the problem task and through their coordination the larger solution would emerge. Planning research now extends well beyond the domains of robotics, to include the coordination of any complex set of tasks and goals. Modern planners are applied to agents (Nilsson 1994) as well as to control of particle beam accelerators (Klein et al. 1999, 2000).
Problems "Can a machine act intelligently?" is still an open problem. Taking "A machine can act intelligently" as a working hypothesis, many researchers have attempted to build such a machine. The general problem of simulating (or creating) intelligence has been broken down into a number of specific sub-problems. These consist of particular traits or capabilities that researchers would like an intelligent system to display. The traits described below have received the most attention.[218] Deduction, reasoning, problem solving
Early AI researchers developed algorithms that imitated the step-by-step reasoning that human were often assumed to use when they solve puzzles play board games or make logical deductions . [219] By the late 1980s and '90s, AI research had also developed highly successful methods for dealing with uncertain or incomplete information, employing concepts from probability and economics.[220] For difficult problems, most of these algorithms can require enormous computational resources — most experience a "combinatorial explosion": the amount of memory or computer time required becomes astronomical when the problem goes beyond a certain size. The search for more efficient problem solving algorithms is a high priority for AI research.[141] 103
Human beings solve most of their problems using fast, intuitive judgments rather than the conscious, step-by-step deduction that early AI research was able to model.[142] AI has made some progress at imitating this kind of "sub-symbolic" problem solving: embodied agent approaches emphasize the importance of sensorimotor skills to higher reasoning; neural net research attempts to simulate the structures inside human and animal brains that give rise to this skill. Knowledge representation
Fig 54: Ontology represents knowledge as a set of concepts within a domain and the relationships between those concepts. Knowledge representation [143] and knowledge engineering [144] are central to AI research. Many of the problems machines are expected to solve will require extensive knowledge about the world. Among the things that AI needs to represent are: objects, properties, categories and relations between objects ; [145] situations, events, states and time;[146] causes and effects;[147] knowledge about knowledge (what we know about what other people know);[148] and many other, less well researched domains. A representation of "what exists" is an ontology (borrowing a word from traditional philosophy), of which the most general are called upper ontologies.[149] Among the most difficult problems in knowledge representations are: Default reasoning and the qualification problem
Many of the things people know take the form of "working assumptions." For example, if a bird comes up in conversation, people typically picture an animal that is fist sized, sings, and flies. None of these things are true about all birds. John McCarthy identified this problem in 1969[50] as the qualification problem: for any common sense rule that AI researchers care to represent, there tend to be a huge number of exceptions. Almost nothing is simply true or false in the way that abstract logic requires. AI research has explored a number of solutions to this problem.[151]
104
The breadth of common sense knowledge: The number of atomic facts that the average person knows is
astronomical. Research projects that attempt to build a complete knowledge base of common sense knowledge (e.g., Cyc) require enormous amounts of laborious ontological engineering — they must be built, by hand, one complicated concept at a time.[152]A major goal is to have the computer understand enough concepts to be able to learn by reading from sources like the internet, and thus be able to add to its own ontology. The sub symbolic form of some common sense knowledge: Much of what people know is not
represented as "facts" or "statements" that they could express verbally. For example, a chess master will avoid a particular chess position because it "feels too exposed"[153] or an art critic can take one look at a statue and instantly realize that it is a fake.[154]These are intuitions or tendencies that are represented in the brain non-consciously and sub-symbolically.[155] Knowledge like this informs supports and provides a context for symbolic, conscious knowledge. As with the related problem of sub-symbolic reasoning, it is hoped that situated AI or computational intelligence will provide ways to represent this kind of knowledge.[155]
Planning Intelligent agents must be able to set goals and achieve them.[56] They need a way to visualize the future (they must have a representation of the state of the world and be able to make predictions about how their actions will change it) and be able to make choices that maximize the utility (or "value") of the available choices.[157] In classical planning problems, the agent can assume that it is the only thing acting on the world and it can be certain what the consequences of its actions may be.[158] However, if this is not true, it must periodically check if the world matches its predictions and it must change its plan as this becomes necessary, requiring the agent to reason under uncertainty.[159] Multi-agent planning uses the cooperation and competition of many agents to achieve a given goal. Emergent behaviour such as this is used by evolutionary algorithms and swarm intelligence.[160]
Fig 55: A hierarchical control system is a form of control system in which a set of devices and governing software is arranged in a hierarchy. 105
Conclusion The computing world has a lot to gain from neuro-fuzzy systems. Their ability to learn by example and processing of unconventional data makes them very flexible and powerful. They are also very well suited for real time systems because of their fast response and computational times. ANN and Fuzzy Logic together as hybrid systems can be used for many practical applications. Yet, the future holds even more promises. Neural networks need faster hardware. They need to become part of hybrid systems which also utilize fuzzy logic and expert systems. It is then that these systems will be able to hear speech, read handwriting, and formulate actions. Perhaps the most exciting aspect of neural networks is the possibility that someday 'conscious' networks might be produced. There are number of scientists arguing that consciousness is a 'mechanical' property and that 'conscious' neural networks are a realistic possibility. According to me, neuro-fuzzy systems cannot produce miracles but when used in right direction they can give very amazing results. They will be able to become the intelligence behind robots that never tire nor become distracted. It is then that they will become the leading edge in an age of "intelligent" machines.
Fig 56 multi layered structure A neural network is an interconnected group of nodes, akin to the vast network of neurons in the human brain. The study of artificial neural networks[124] began in the decade before the field AI research was founded, in the work of Walter Pitts and Warren McCullough. Other important early researchers were Frank Rosenblatt, who invented the perceptron and Paul Werbos who developed the back propagation algorithm.[131] The main categories of networks are acyclic or feedforward neural networks (where the signal passes in only one direction) and recurrent neural networks (which allow feedback). Among the most popular feedforward networks are perceptrons, multi-layer perceptrons and radial basis networks.[132] Among recurrent networks, the most famous is the Hopfield net, a form of attractor network, which was first described by John Hopfield in 1982.[133] Neural networks can be applied to the problem of intelligent control (for robotics) or learning, using such techniques as Hebbian learning and competitive learning.[134] Hierarchical temporal memory is an approach that models some of the structural and algorithmic properties of the neocortex.[135] Conclusions
The Fuzzy logic and Neuron network technique is very useful in problem-solving methodology with applications in embedded control and information processing. Fuzzy provides a remarkably simple way to 106
draw definite conclusions. With the achieved results exhibiting high levels of accuracy, consistency and reliability, with acceptably low computational time. The fuzzy-neural based classifier provides better performance and combines the benefits of both neural networks and fuzzy logic.
Artificial Intelligence--A Summary We have attempted to define artificial intelligence through discussion of its major areas of research and application. This survey reveals a young and promising field of study whose primary concern is finding an effective way to understand and apply intelligent problem solving, planning, and communication skills to a wide range of practical problems. In spite of the variety of problems addressed in artificial intelligence research, a number of important features emerge that seem common to all divisions of the field; these include:
1. The use of computers to do reasoning, pattern recognition, learning, or some other form of inference. 2. A focus on problems that do not respond to algorithmic solutions. This underlies the reliance on heuristic search as an AI problem-solving technique. 3. A concern with problem solving using inexact, missing, or poorly defined information and the use of representational formalisms that enables the programmer to compensate for these problems. 4. Reasoning about the significant qualitative features of a situation. 5. An attempt to deal with issues of semantic meaning as well as syntactic form. 6. Answers that are neither exact nor optimal, but are in some sense "sufficient". This is a result of the essential reliance on heuristic problem-solving methods in situations where optimal or exact results are either too expensive or not possible. 7. The use of large amounts of domain-specific knowledge in solving problems. This is the basis of expert systems. 8. The use of meta-level knowledge to effect more sophisticated control of problem solving strategies. Although this is a very difficult problem, addressed in relatively few current systems, it is emerging as an essential area of research. We hope that this introduction provides some feel for the overall structure and significance of the field of artificial intelligence. We also hope that the brief discussions of such technical issues as search and representation were not excessively cryptic and obscure; they are developed in proper detail throughout the remainder of the book, but included here to demonstrate their significance in the general organization of the field. As we mentioned in the discussion of agent-oriented problem solving, objects take on meaning through their relationships with other objects. This is equally true of the facts, theories, and techniques that constitute a field of scientific study. We have intended to give a sense of those interrelationships, so that when the separate technical themes of artificial intelligence are presented, they will find their place in a developing understanding of the overall substance and directions of the field. We are guided in this process by an observation made by Gregory Bateson (1979), the psychologist and systems theorist: Break the pattern which connects the items of learning and you necessarily destroy all quality.
107
References Cited 1) Keith Gunderson. Mentality and Machines. Anchor Books. Garden City, N.Y.: Doubleday & Co. 1971. 2) Charles Hampden-Turner. Maps of the Mind. New York: Macmillan, 1981. 3) Raymond Kurzweil. "Whate is Artificial Intelligence Anyway? American Scientist, Vol. 73, May-June 1895. 4) Carl Sagan, Ed. Communication with Extraterrestrial Intelligence. CETI. Cambridge: MIT Press 1974. 5) "Knowledge-Based Systems: A Step-by-Step Guide to Getting Started." The Second Artificial Intelligence Satellite Symposium. Symposium Notes. Texas Instruments, Sponsor. 6) Harvey B. Sarles, Language and Human Nature. Minneapolis: University of Minnesota Press, 1985. 7) Robert J. Sternberg: "Human Intelligence: The Model is the Message." Science 230 (4730), 6 Dec 1985. Kurzweil, op cit. 8) Neural Network and Fuzzy Logic. Dr.Valluru B. Rao, Hayagriva V. Rao 9) Neural Network and Fuzzy Systems. Bart Kosko 10) Artificial Neural Network. KishanMehrotra , Chilukuri K. Mohan , Sanjay Ranka 11) Artificial Neural Network Technology (Hwww.dacs.comH) 12) Neural Networks (Hhttp://www.aaai.org/AITopics/html/neural.htmlH) 13) Fuzzy Logic – An Introduction (www.seatlerobotics.com) 14) Fuzzy Logic Research and Life, Japanese Technology Evaluation Center, 1995, (http://itri.loyola.edu/kb/c5_s4.htm) 15) Hochreiter, Sepp and Schmidhuber, Juergen, (1997) "Long Short-Term Memory", Neural Computation, Vol 9 (8), pp: 1735-1780 16) Fausett L., Fundamentals of Neural Networks, Prentice-Hall, 1994. ISBN 0 13 042250 9 17) Neural Computing and Applications, Springer-Verlag. (address: SweetappleHo, Catteshall Rd., Godalming, GU7 3DJ) 18) Staff Development Program on Neurocomputing from 8-20 Jan 2007 by Prof. J.P.Saini H.O.D, ECE Deptt. B.I.E.T. Jhansi 19) IEEEVolume 3, Issue , 2-6 Nov. 2003 Page(s): 2103 - 2107 Vol.3 20) Wikipedia The free encyclopedia 21) Neural Networks A comprehensive Foundamental by Simon Haykin 22) Apache. Apache Project. Hhttp://www.apache.org 23) ASP. Microsoft Active Server Pages. Hhttp://www.microsoft.com/asp 24) Java. Java Java Server Pages/Servlets. Hhttp://java.sun.comH 25) D. Brickley and R.V. Guha. Resource Description Framework (RDF) Schema Specification 1.0, March 2002. Hhttp://www.w3.org/TR/rdf-schema 26) D. Box, D. Ehnebuske, G. Kakivaya, A. Layman, N. Mendelsohn, H. Nielsen, S. Thatte, and D. Winer. Simple Object Access Protocol (SOAP) 1.1. Hhttp://www.w3.org/TR/SOAP. 27) C. Chung. DATAPLEX: An Access to Heterogeneous Distributed Databases. Communications of the ACM, 33(1):70-80, January 1990. 28) CORBA/IIOP. Common Object Request Broker Architecture. Hhttp://www.omg.org/technology/documents/formal/corba-iiop.htm. 29) EDI - Electronic Data Interchange (See Hhttp://www.diffuse.org/edi.htmlH) 30) Net. Microsoft .Net. Hhttp://www.microsoft.com/net 31) D. Heimbigner and D. McLeod. A Federated Architecture for Info. Management. ACM Transaction on Office Information Systems, 3(3):253-278, July 1985. 32) Hhttp://delivery.acm.org/10.1145/1370000/1366341/p159Hbernaschi.pdf?key1=1366341&key2=71 29557421&coll=ACM&dl=ACM&HCFID=45420903 &CFTOKEN=18303323 33) http://delivery.acm.org/10.1145/1520&CFID=45417991&CFTOKEN=935H98617 108
34) 35) 36) 37) 38)
39) 40) 41) 42) 43) 44) 45) 46) 47) 48) 49) 50) 51) 52) 53) 54) 55) 56)
57) 58) 59)
www.HwepHindia.com www.searchsecurity.techtarget.com/.../0,,sid14_gci549087,00.htmH Mega Computer. "Reasons for the growing popularity of data mining." Online. Internet. 3 Oct. 2008. Lindsay, Clark. "Data Mining." Online. Internet. 2007 Available: http://msia02.msi.se/~lindsay/datamine.html Usama Fayyad, "Mining Databases: Towards Algorithms for Knowledge Discovery", Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, vol. 21, no. 1, March2008. http://www.research.microsoft.com/research/db/debull/98mar/issue.html Christopher Matheus, Philip Chan, and Gregory Piatetsky-Shapiro, "Systems for Knowledge Discovery in Databases", IEEE Transactions on Knowledge and Data Engineering, 5(6):9 03-913, November 2006. RakeshAgrawal and Tomasz Imielinski, "Database Mining: A Performance Perspective", IEEE Transactions on Knowledge and Data Engineering, 5(6):914-925, December 2007. D.Alessandri, C.Cachin, M.Dacier, etc., Towards a taxonomy of intrusion detection systems and attacks, MAFTIA deliverable D3, Version 1.01, Project IST-1999-11583. (2001 ). Custom attack simulation language (CASL), Secure Networks. (1998). S.-D.Chi, J.S.Park, K.-C.Jung, J.-S.Lee, Network security modeling and cyber attack simulation methodology, ACISP 2001, Lecture Notes in Computer Science, Vol.2119. (2001). M.Chung, B. Mukherjee, R.A.Olsson, N.Puketza, Simulating concurrent intrusions for testing intrusion detection systems: parallelizing intrusions. Proceedings of the 18 th NISSC. (1995). F.B.Cohen, Information system attacks: a preliminary classification scheme, Computers and Security, Vol. 16, No. 1. (1997). F.Cohen. Simulating cyber attacks, defenses, and consequences, IEEE Symposium on Security and Privacy, Berkeley, CA. (1999). F.Cuppens and R.Ortalo, Lambda: A language to model a database for detection of attacks, Proceedings of RAID'2000. (2000). D.Curry, Intrusion detection message exchange format, extensible markup language (xml) document type definition. draft-ietf-idwg-idmef-xml-02.txt. (2000). J.Dawkins, J. Hale, A Systematic approach to multi-stage network attack analysis, Proceedings of the Second IEEE International Information Assurance Workshop (IWIA’04). (2004). R.Deraison, The nessus attack scripting language reference guide, http://www.nessus.org. (1999). R.Durst, T.Champion, B.Witten, E.Miller, L.Spanguolo, Testing and evaluating computer intrusion detection systems, Communications of ACM, 42(7). (1999). S.T.Eckmann, G.Vigna, R.A.Kemmerer, STATL: An attack language for state-based intrusion detection, Proceedings of the ACM Workshop on Intrusion Detection. (2000). R.Feiertag, C.Kahn, P.Porras, D.Schnackenberg, S.Staniford-Chen, B.Tung, A common intrusion specification language (cisl), Specification draft, http://www.gidos.org. (1999). R.P.Goldman, A Stochastic model for intrusions, Recent Advances in Intrusion Detection. Fifth International Symposium, RAID 2002, Lecture Notes in Computer Science, V.2516. (2002). V.Gorodetski, O.Karsayev, I.Kotenko, A.Khabalov, Software development kit for multi-agent systems design and implementation, Lecture Notes in Artificial Intelligence, Vol.2296. (2002). V.Gorodetski, I.Kotenko, Attacks against computer network: formal grammar-based framework and simulation tool, Recent Advances in Intrusion Detection. Fifth International Symposium. RAID 2002. Lecture Notes in Computer Science, Vol.2516. (2002). S.Hariri, G.Qu, T.Dharmagadda, M.Ramkishore, C. S.Raghavendra, Impact analysis of faults and attacks in large-scale networks, IEEE Security & Privacy, September /October. (2003). J.D.Howard, T.A.Longstaff, A common language for computer security incidents, SANDIA Report, SAND98-8667. (1998). K.Iglun, R.A.Kemmerer, P.A.Porras, State transition analysis: a rule-based intrusion detection system, IEEE Transactions on Software Engineering, 21(3). (1995). 109
60) S.Jha, O.Sheyner, J.Wing, Minimization and reliability analysis of attack graphs, Technical Report CMU-CS-02-109, School of Computer Science, Carnegie Mellon University. (2002 ). 61) R.A.Kemmerer, G.Vigna, NetSTAT: a network-based intrusion detection approach, Proceedings of the 14th Annual Computer Security Applications Conference, Scottsdale, Arizona. (1998). 62) I.Kotenko, Agent-based modeling and simulation of cyber-warfare between malefactors and security agents in Internet, 19th European Simulation Multiconference. ESM’05. (2005). 63) I.Kotenko, M.Stepashkin, Analyzing vulnerabilities and measuring security level at design and exploitation stages of computer network life cycle, MMM-ACNS-05, Lecture Notes in Computer Science, Springer Verlag, Vol.3685. (2005). 64) I.Kotenko, A.Ulanov, Multiagentmodeling and simulation of agents’ competition for network resources availability, Second International Workshop on Safety and Security in Multiagent Systems (SASEMAS '05). Utrecht, The Netherlands. (2005). 65) I.V.Krsul, Software vulnerability analysis, Ph.D. Dissertation, Computer Sciences Department, Purdue University, Lafayette, IN. (1998). 66) S.Kumar, E.H.Spafford, A software architecture to support misuse intrusion detection. Technical Report CSD-TR-95-009. Purdue University. (1995). I. Kotenko et al. / Agent-Based Modeling and Simulation of Malefactors' Attacks 145 67) C.E.Landwehr, A.R.Bull, J.P.McDermott, W.S.Choi, A taxonomy of computer security flaws, ACM Computing Surveys, Vol. 26, No. 3. (1994). 68) U.Lindqvist, E.Jonsson, How to systematically classify computer security intrusions. Proceedings of the 1997 IEEE Symposium on Security and Privacy, Los Alamitos, CA. (1997) . 69) R.Lippmann, J.W.Haines, D.J.Fried, J.Korba, K.Das. The 1999 DARPA off-line intrusion detection evaluation, RAID’2000, Lecture Notes in Computer Science, Vol.1907. (2000). 70) P.Liu, W.Zang, Incentive-based modeling and inference of attacker intent,objectives, and strategies. ACM Transactions on Information and System Security, Vol.8, No. 1. (2005). 71) K.Lye, J.Wing, Game strategies in network security, International Journal of Information Security, February. (2005). 72) J.McDermott, Attack-potential-based survivability modeling for high-consequence systems, Third IEEE International Workshop on Information Assurance, College Park, MD, USA. (2005). 73) J.McHugh, The 1998 Lincoln Laboratory IDS evaluation: a critique, RAID’2000, L ecture Notes in Computer Science, Vol. 1907. (2000). 74) L.Me.Gassata, A genetic algorithm as an alternative tool for security audit trails analysis, Proceedings of the first international workshop on the Recent Advances in Intrusion Detection (RAID'98). (1998). 75) C.Michel, L.Me, ADeLe: an attack description language for knowledge-based intrusion detection, Proceedings of the 16th International Conference on Information Security. Kluwer. (2001). 76) A.P.Moore, R.J.Ellison, R.C.Linger, Attack modeling for information security and survivability, Technical Note CMU/SEI-2001-TN-001. Survivable Systems. (2001). 77) S.D.Moitra, S.L.Konda, A simulation model for managing survivability of networked information systems, Technical Report CMU/SEI-2000-TR-020. (2000). 78) D.M.Nicol, W.H.Sanders, K.S.Trivedi, Model-based evaluation: from dependability to security, IEEE Transactions on Dependable and Secure Computing. Vol.1, N.1. (2004). 79) S.Nikoletseas, G.Prasinos, P.Spirakis, C.Zaroliagis, Attack propagation in networks, Theory of Computing Systems, 36. (2003). 80) P.Ning, D.Xu, C.G.Healey, R.A.St.Amant, Building attack scenarios through integration of complementary alert correlation methods, Proceedings of the 11th Annual Network and Distributed System Security Symposium (NDSS '04). (2004). 81) OMNeT++ homepage. http://www.omnetpp.org 82) V.Paxson, Bro: A system for detecting network intruders in real-time. Proceedings of the 7th Usenix Security Symposium. (1998). 110
83) N.Puketza, M.Chung, R.A.Olsson, B.Mukherjee, A software platform for testing intrusion detection systems, IEEE Software, Vol.14, No 5. (1997). 84) M.Ranum, A Taxonomy of Internet Attacks, Web Security Sourcebook. John Wiley & Sons. (1997). 85) R.W.Ritchey, P.Ammann, Using model checking to analyze network vulnerabilities, Proceedings SOOO IEEE Computer Society Symposium on Security and Privacy. (2000). 86) M.Roesch, Snort - lightweight intrusion detection for networks, Proceedings of the USENIX LISA'99 conference. (1999). 87) S.Rubin, S.Jha, B.P.Miller, Automatic generation and analysis of NIDS attacks, 20 th Annual Computer Security Applications Conference (ACSAC), Tuscon, Arizona. (2004). 88) B.Schneier, Attack trees: modeling security threats, Dr. Do bb’s Journal, December. (1999). 89) B.Shepard, C.Matuszek, C.B.Fraser, etc., A Knowledge-based approach to network security: applying Cyc in the domain of network risk assessment, The Seventeenth Innovative Applications of Artificial Intelligence Conference on Artificial Intelligence (IAAI-05), Pittsburgh, Pennsylvania. (2005). 90) O.Sheyner, J.Haines, S.Jha, R.Lippmann, J.M.Wing, Automated generation and analysis of attack graphs, Proceedings of the IEEE Symposium on Security and Privacy. (2002). 91) S.Singh, J.Lyons, D.M.Nicol, Fast model-based penetration testing, Proceedings of the 2004 Winter Simulation Conference. (2004). 92) A.J.Stewart, Distributed metastasis: a computer network penetration methodology, Phrack Magazine, Vol 9, Issue 55. (1999). 93) L.Swiler, C.Phillips, D.Ellis, S.Chakerian, Computer-attack graph generation tool, Proceedings DISCEX '01: DARPA Information Survivability Conference & ExpositionII. (2001). 94) S.J.Templeton, K.Levitt, A requires/provides model for computer attacks, Proceedings of the New Security Paradigms Workshop. (2000). 95) E.Turner, R.Zachary, Securenet pro software's snp-l scripting system, White paper. http://www.intrusion.com, July. (2000). 96) J.Yuill, F.Wu, J.Settle, F.Gong, R.Forno, M.Huang, J.Asbery, Intrusion-detection for incident-response, using a military battlefield-intelligence process, Computer Networks, No.34. (2000 ). 97) Allen, J., Hunnicutt, M. S., &Klatt, D. H. (1987). From Text to Speech: The MITalk System. Cambridge University Press, New York. 98) Bailly, G. & Benoit, C. (Eds.). (1992). Talking Machines: Theories, Models, and Designs. North Holland, Elvsevier, Amsterdam. 99) Bell, A. G. (1922). Prehistoric Telephone Days. National Geographic Magazine, 41, 223-242. 100) Cahn, J. E. (1990). The generation of affect in synthesized speech. Journal of the American Voice Input/Output Society, 8, 1-19. 101) Carlson, R. &Granström, B. (1976). A text-to-speech system based entirely on rules. Proceedings of the International Conference of Acoustics, Speech, & Signal Processing, ICASSP-76, 686-688. 102) Cater, John P. (1983). Electronically Speaking: Computer Speech Generation. Howard W. Sams& Co., Inc., Indianapolis, Indiana. 103) Cooper, F.S., Liberman, A.M., &Borst, J.M. (1951). The interconversion of audible and visible patterns as a basis for research in the perception of speech. Proceedings of the National Academy of Science, 37, 318-325. 104) Alex Berson, “Data warehousing, Data Mining and OLAP” 105) M.Berry and G. Linoff, “Data Mining Techniques”, john Wi ley 1997. 106) Usama Fayyad, Gregory Piatetsky-Shapiro &Padhraic Smyth, “Knowledge discovery and data mining: Towards a unifying framework”, Proceedings of the International Conference on Know ledge Discovery and Data Mining, pages 82- 88, 1996. 107) Usama Fayyad, Gregory Piatetsky-Shapiro &Padhraic Smyth, “The KDD process for extracting useful knowledge from volumes of data”. Journal of the ACM, 39(11): 27-34, 1996. 111
108) Usama Fayyad, Gregory Piatetsky-Shapiro &Padhraic Smyth, “Advances in knowledge Discovery and Data Mining”, Pages 1-34, AAAI/MIT Press, 1996. 109) Margraret H. Dunham, “Data Mining: Introductory and Advanced Topics”, Pages 8- 10, Pearson Education Ltd, 2003. 110) M.Berry and G. Linoff, “Data Mining Techniques”, john Wiley 1997. 111) Usama Fayyad, Gregory Piatetsky-Shapiro &Padhraic Smyth, “Knowledge discovery and data mining: Towards a unifying framework”, Proceedings of the International Conference on Knowledge Discovery and Data Mining, pages 82- 88, 1996. 112) Usama Fayyad, Gregory Piatetsky-Shapiro &Padhraic Smyth, “The KDD process for extracting useful knowledge from volumes of data”. Journal of the ACM, 39(11): 27-34, 1996. 113) Usama Fayyad, Gregory Piatetsky-Shapiro &Padhraic Smyth, “Advances in knowledge Discovery and Data Mining”, Pages 1-34, AAAI/MIT Press, 1996. 114) Margraret H. Dunham, “Data Mining: Introductory and Advanced Topics”, Pages 8-10, Pearson Education Ltd, 2003. 115) Encyclopaedia of Artificial Intelligence, Many authors. 116) Natural Language Understanding, James Allen. 117) Survey of the Human Language Technology, Ronald A.Cole, Joseph Mariani. 118) Natural Language Theory and Technology, Ron Kaplan,Jeanette Figueroa. 119) Mohamed Frendi, Camille Salinesi: Centre de Recherche en InformatiqueUniversité de Paris 1, Panthéon Sorbonne 120) Paolo Giorgini University of Trento - Italy Stefano Rizzi University of Bologna – Italy MaddalenaGarzetti University of Trento - Italy 121) Finkelstein, C.: An Introduction to Information Engineering. Addison Wesley: Sydney etc. 1989. 122) Brinkkemper, S., Lyytinen K. and Welke R.: Method Engineering, Chapman & Hall, London 1996. 123) Paul Raj Poonia, “Fundamentals of Data Warehousing”, John Wiley & Sons, 2003. 124) Sam Anahony, “Data Warehousing in the real world: A practical guide for building decision support systems”, John Wiley, 2004 125) W. H. Inmon, “Building the operational data store”, 2nd Ed., John Wiley, 1999. 126) Kamber and Han, “Data Mining Concepts and Techniques”, Hartcourt India P. Ltd.,2001 127) BOOCH, G.; RUMBAUGH, J.; JACOBSON, I. UML, 2000. 128) BUZYDLOWSKI, J.; SONG, II-Y.; HASSELL, L. A Framework for object-Oriented On-Line Analytic Processing. In Proc. Of the ACM 1st Int. Workshop on Data warehousing and OLAP (DOLAP). Washington DC, USA, 1998. 129) G.Booch,J.Rumbaugh,and I. Jacobson, The Unified Modeling Language user Guide,Addison Longman Wesley,Reading,Mass.,1998. 130) J.Trujillo, the GOLD Model: An Object Oriented Conceptual model for the design of OLAP Applications, Doctoral dissertation, Languages and Information systems Dept., Alicante University, Spain, June 2001. 131) LUJÁN-MORA, S.; TRUJILLO, J.; SONG, I.Y. Extending UML for Multidimensional Modeling. 5th International Conference on the Unified Modeling Language (UML 2002), p. 290-304. Germany, 2002. 132) SAPIA, C.; BLASCHKA, M.; HÖFLING, G.; DINTER B. Extending the E/R Model for the Multidimensional Paradigm. In: Proc. of the 1st Intl. Workshop on Data Warehouse and Data Mining (DWDM'98), 1998. 133) TRUJILLO, J.; PALOMAR, M. An Object Oriented Approach to Multidimensional Database Conceptual Modeling (OOMD). 1st Int. Workshop on Data Warehousing and OLAP (DOLAP), Washington DC (USA), 1998. 134) TRUJILLO, J.; PALOMAR, M.; GÓMEZ, J. The GOLD definition language (GDL): an object oriented formal specification language for multidimensional databases.Symposium on Applied Computing. Proceedings of the 2000 ACM Symposium on Applied Computing. Italy, p.346-350, 2000. 112
135) TRUJILLO, J. et al. Designing Data warehouse with OO conceptual models. IEEE, p. 66-75, 2001. 136) Carbonell, Jamie G. and Langley, Pat Machine Learning Tutorial from Seventh National Conference on Artificial Intelligence, 1988. 137) Carbonell, Tom M., Michalski, Ryszard S. and Mitchell, Tom M., eds. Machine Learning: Volume 1. Palo Alto, CA: Tioga Pub. Co., 1983. 138) Bond,A.H. and Gasser,L. (1988), "An Analysis of Problems and Research in DAI," in A.H.Bond and L.Gasser, eds., Readings in Distributed Artificial Intelligence, Morgan Kaufmann, 3-36. 139) Applications of Distributed Artificial Intelligence in Industry by Dr. H. Van Dyke PARUNAK, Industrial Technology Institute, eds., Foundations of Distributed Artificial Intelligence. Wiley InterScience, 1994. 140) Coordination Techniques for Distributed Artificial Intelligence by N. R. Jennings, Queen Mary and Westfield College, University of London, Mile End Rd. London E1 4NS UK. 141) Learning for Distributed Artificial Intelligence Systems, by Michael L. Dowell and Ronald D. Bonnell, Department of Electrical & Computer Engineering University of South Carolina, Columbia, SC 29208 142) P. J.Werbos, ”Neuro-control and elastic fuzzy logic: Capabilities, concepts, and applications,” IEEE Trans. on Industrial Electronics, vol. 40, no. 2, pp.170-180, 1993. 143) Y. Dote and R. G. Hoft, Intelligent Control - Power ElectronicsSystems, Oxford, U.K., Oxford University Press, 1998. 144) L. A. Zadeh, ”Fuzzy logic, neural networks and soft computing,”in Proc. of the IEEE Int. Workshop on Neuro Fuzzy Control,Muroran, Japan, p. 1, 1993. 145) L. A. Zadeh,”The role of soft computing and fuzzy logic in theconception, design, and development of intelligent systems,” inProc. of the IEEE Int. Workshop on Soft Computing in Industry, Muroran, Japan, pp. 136-137, 1996.*5+ L. A. Zadeh, ”From computing with numbers to computing withwords -F rom manipulation of measurements to manipulation of perceptions,” in Proc. of the IEEE Int. Workshop on Soft Computingin Industry, Muroran, Japan, pp. 221-222, 1999. 146) L. X. Wang, ”Fuzzy systems are universal approximators,” inProc. of the IEEE Int. Conf. on Fuzzy Systems, San Francisco,CA, pp.1163-1172, 1992. 147) S. Sato, Y. Arai, and K. Hirota, ”Pattern recognition using fuzzyinference with lacked input data,” in Proc. of the IEEE Int. Conf.on Fuzzy Systems, San Antonio, TX, pp.100- 104, 2000. 148) J.-S. R. Jang and C.-T. Sun, ”Functional equivalence betweenradial basis function networks and fuzzy inference systems,” IEEETrans. on Neural Networks, vol. 4, no. 1, pp. 156-159, 1993. 149) R. Shorten, R. Murray-Smith, R. Bjorgan, and H. Gollee, ”On theinterpretation of models in blended multiple model structures,”Int. Journal of Control, vol.72, no. 7/8, pp. 620-628, 1999. 150) T. F. Junge and H. Unbehauen, ”On-line identification of nonlinearsystems using structurally adaptive rectangular local linearmodel networks,” in Proc. of the 3 rd SIMONET Workshop onRecent Results in System Identification and Modeling, Bochum, Germany, pp. 1 -7, 1997. 151) O. Nelles, ”Orthogonal basis functions for nonlinear system identificationwith local linear model trees (LOLIMOTO),” in Proc.of the 11th IFAC Symposium on System Identification, Fukuoka,Japan, pp. 667-672, 1997. 152) R. Murray-Smith and T. A. Johansen (Eds.), Multiple ModelApproaches to Modeling and Control, London, U.K.: Taylor &Francis,1997. 153) C.-H. Lee and C.-C .Teng, ”Identification and control of dynamicsystems using recurrent fuzzy neural networks,” IEEE Trans. OnFuzzy Systems, vol. 8, no. 4, pp. 349- 366, 2000. 154) P. A. Marchi, L. S. Coelho, and A. A. R. Coelho, ”Comparativestudy of parametric and structural methodologies in identificationof an experimental nonlinear process,” in Proc. of the IEEE Int.Conf. on Control Applications, Hawaii, U.S.A., pp. 1062-1067, 1999. A. J. Calise, ”Neural networks in nonlinear aircraft flight control,”IEEE Aerospace and Electronics Systems Magazine, vol.11, no. 7, pp. 5-10, 1996. 113
155) G. Kolumban, M. P. Kennedy, and L. O. Chua, ”The role of synchronization in digital communications using chaos-Part I: Fundamentals of Digital Communications,” IEEE Trans. on Circuitsand Systems-I: Fundamental Theory and Applications, vol. 44, no. 10, pp. 927-936, 1997. 156) S. K. Patra and B. Mulgrew, ”Fuzzy implementation of aBayesian equalizer in the presence of inter-symbol and co-channel interference,” IEE Proceedings- Communications, vol. 145, no. 5, pp. 323-330, 1998. 157) C. E. Cramer and E. Gelenbe, ”Video quality and traffic QoS in learning -based subsampled and receiver-interpolated video sequences,” IEEE Journal on Selected Areas in Communication, vol. 18, no. 2, pp. 150-167, 2000. 158) J. M. Jou and P.-Y. Chen, ”A fast and efficient lossless datacompression method,” IEEE Trans. on Communications, vol. 47, no. 9, pp. 1278-1283, 1999. 159) E. Gelenbe, I. W. Habib, S. Palazzo, and C. Douligeris, ”Guest editorial: Intelligent techniques in high speed networks,” IEEE Journal on Selected Areas in Communication, vol. 18, no. 2, pp. 145-149, 2000. 160) G. Chakraborty and B. Chakraborty, ”A genetic algorithm approach to solve channel as signment problem in cellular radio networks,” in Proc. of the IEEE Midnight-Sun Workshop on Soft Computing Methods in Industrial Applications, Kuusamo, Finland, pp. 34-39, 1999. 161) B. Dengiz, F. Altiparmak, and A. E. Smith, ”Local search genetic algorithm for optimal design of reliable networks,” IEEE Trans. on Evolutionary Computation, vol. 1, no. 3, pp. 179-188,1997. 162) X. M. Gao, X. Z. Gao, J. M. A. Tanskanen, and S. J. Ovaska, ”Power prediction in mobile communication systems using an optimal neural-network structure,” IEEE Trans. on Neural Networks, vol. 8, no. 6, pp. 1446-1455, 1997. 163) J. H. Kim, K. S. Kim, M. S. Sim, K. H. Han, and B. S. Ko, ”An application of fuzzy logic to control the refrigerant distribution for the multi type air conditioner,” in Proc. Of the IEEE Int. Fuzzy Systems Conference, Seoul, Korea, vol. 3, pp. 1350-1354, 1999. 164) R. Zhu, B. Tian, Q. Wang, and G. Dai, ”Application of fuzzy logic in home appliance: Gas heater controller design,” in Proc. Of the IEEE Int. Conf. on Intelligent Processin g Systems, Beijing, China, pp. 373-376, 1997. 165) T. Nitta, ”Applications of neural networks to home appliances,” in Proc. of the IEEE Int. Joint Conf. on Neural Networks, Nagoya, Japan, pp. 1056-1060, 1993. 166) TOPIO: "A Ping-Pong-Playing Terminator". Popular Science. "Best robot 2009". www.gadgetrivia.com.[dead link] 167) Definition of AI as the study of intelligent agents: Poole, Mackworth& Goebel 1998, p. 1, which provides the version that is used in this article. Note that they use the term "computational intelligence" as a synonym for artificial intelligence. Russell &Norvig (2003) (who prefer the term "rational agent") and write "The whole-agent view is now widely accepted in the field" (Russell &Norvig 2003, p. 55). Nilsson 1998 168) The intelligent agent paradigm: Russell &Norvig 2003, pp. 27, 32 –58, 968 –972 Poole, Mackworth& Goebel 1998, pp. 7 –21 Luger & Stubblefield 2004, pp. 235 –240 The definition used in this article, in terms of goals, actions, perception and environment, is due to Russell &Norvig (2003). Other definitions also include knowledge and learning as additional criteria. 169) Although there is some controversy on this point (see Crevier (1993, p. 50)), McCarthy states unequivocally "I came up with the term" in a c|net interview. (Skillings 2006) 170) McCarthy's definition of AI: McCarthy 2007 171) See the Dartmouth proposal, under Philosophy, below. 172) The optimism referred to includes the predictions of early AI researchers (see optimism in the history of AI) as well as the ideas of modern transhumanists such as Ray Kurzweil. 173) The "setbacks" referred to include the ALPAC report of 1966, the abandonment of perceptrons in 1970, the Lighthill Report of 1973 and the collapse of the lisp machine market in 1 987. 114
