CS 473
SOFT COMPUTING
SUBMITTED TO – Dr. Dr. Shailendra Singh
Submitted By Name : Akshit Singla SID : 10103006 Date : 15-10-2013
SC Assignment 2 10103006 Question 1. List atleast 10 applications of feed-forward multi-layer neural networks with a short note on their technical details.
Answer 1. 1. Injection Moulding & Casting Process
ANNs can be employed for casting and IM processes starting from moulding to final inspection stage. Multiple regression equation obtained from DOE fulfill the need of huge training data for ANN. ANNs address the limitations of simulation software by reducing repetitive analysis, computational cost and time, replace need of experts for results interpretation and help in implementation for online process control. Supervised learning, MLP networks, one or two hidden layer(s), BP & LM algorithm, off-line training mode and Matlab simulations are more common in casting and IM process. Use of activation constants in transfer function and bias values for MLP networks reduces the MSE to small values and improves the prediction accuracy. ANN is best used to model complex interaction among several number of input & output parameter and predicts with good accuracy, the only limitation is need of enough training data and ready to sacrifice for computational time and cost. The attempted and proposed empirical equations for optimum hidden layer/neurons selection, however are not generalized and fail in other applications while using those methods. Hence optimum selection for hidden layer(s)/hidden neurons is still under intensive study. Single hidden layer finds maximum applications in MLP networks, reason might be increase the hidden layers increases the computation time or mathematically proven by Simon Hyken, single hidden layer with enough neurons yields better prediction accuracy.
Page 2
SC Assignment 2 10103006 2. Acoustic detection of buried objects
The idea of in-line holography is used to increase the ratio of the signal to noise due to the effect of concealing media that decreases the value of the received signal. The performance is enhanced by using multilayer neural network for noise reduction. The aim of using multilayer neural network is to extract the essential knowledge from a noisy training data. Theoretical and experimental results have showed that preprocessing the noisy data with multilayer neural network will decrease the effect of noise as much as possible. Applying the enhanced data to spectral estimation methods has improved the performance of the model. The results obtained during a research show that during the pre-processing stage, the MLNN was able to enhance the tested recorded signal and produce an output signal that follows the desired model with very good performance. It is demonstrated that the Burg method can be used to detect and image a concealed object of closely separated points. Experimental results show that pre-processing the noisy data with MLNN to decrease the effect of noise as much as possible and then applying the enhanced data to spectral estimation method can improve the performance. Also the use of holography enables an improvement of the signalto- noise ratio by coherently cumulating the acoustic field on the ultrasonic transducers when scanning the field. 3. Vision Problems
A class of feed-forward Artificial Neural Networks (ANNs) is successful in solving two vision problems: recognition and pose estimation of 3D objects from a single 2D perspective view; and handwritten digit recognition. In both cases, a multiMLP classification scheme is developed that combines the decisions of several classifiers. These classifiers operate on the same feature set for the 3D recognition problem whereas different feature types are used for the handwritten digit recognition. The back-propagation learning rule is used to train the MLPs.
Page 3
SC Assignment 2 10103006 4. Pattern Recognition
Many pattern recognition problems, especially character or other symbol recognition and vowel recognition, have been implemented using a multilayer neural network. Note, however, that these networks are not directly applicable for situations where the patterns are deformed or modified due to transformations such as translation, rotation and scale change, although some of them may work well even with large additive uncorrelated noise in the data. Direct applications are successful, if the data is directly presentable to the classification network. The neural network literature is full of pattern recognition applications. Typically one takes pixelated image values as the network input and that maps via layers of hidden units to a set of outputs corresponding to possible classifications of the image. 5. Image Compression
Neural networks offer the potential for providing a novel solution to the problem of data compression by its ability to generate an internal data representation. This network, which is an application of back propagation network, accepts a large amount of image data, compresses it for storage or transmission, and subsequently restores it when desired. 6. Speech recognition
The principle aim of an artificial neural network hybrid (HMM/ANN) system is to combine the efficient discriminative learning capabilities of neural networks and the superior time warping and decoding techniques associated with the HMM approach. The ANN is trained to estimate HMM emission probabilities which are then used by a decoder based on the well-known Viterbi algorithm. Among the advantages in using such an approach is that no assumption about the statistical distribution of the input features is necessary. Due to its classification procedure, an MLP has the ability to de-correlate the input features. Moreover, while in classical HMM based system, the parameters are trained according to a likelihood Page 4
SC Assignment 2 10103006 criterion, an MLP also penalizes the incorrect classes. At every time n, the acoustic vector xn is presented to the network. This generates local probabilities that are used, after division by priors, as local scaled likelihoods in a Viterbi dynamic programming algorithm. Posterior Probability estimation MLP may be used to estimate probabilities. 7. Universal Approximators
Standard multi-layer feedforward networks are capable of approximating any measurable function to any desired degree of accuracy, in a very specific and satisfying sense. This implies that any lack of success in applications must arise from inadequate learning, insufficient numbers of hidden units or the lack of a deterministic relationship between input and target. 8. Predictions
Neural networks have been applied to numerous situations where time series prediction is required – predicting weather, climate, stocks and share prices, currency exchange rates, airline passengers, etc.
We can turn the temporal
problem into a simple input- output mapping by taking the time series data x(t) at k time-slices t, t –1, t –2, …, t–k+1 as the inputs, and the output is the prediction for x(t+1). Such networks can be extended in many ways, e.g. additional inputs for information other than the series x(t), outputs for further time steps into the future, feeding the outputs back through the network to predict further into the future (Weigend & Gershenfeld, 1994). 9. Driving
Pomerleau(1989) constructed a neural network controller ALVINN for driving a car on a winding road. The inputs were a 30 × 32 pixel image from a video camera, and an 8 × 32 image from a range finder. These were fed into a hidden layer of 29 units, and from there to a line of 45 output units corresponding to direction to drive.
Page 5
SC Assignment 2 10103006 The network was originally trained using back-propagation on 1200 simulated road images. After about 40 epochs the network could drive at about 5mph – the speed being limited by the speed of the computer that the neural network was running on. In a later study the network learnt by watching how a human steered, and by using additional views of what the road would look like at positions slightly off course. After about three minutes of training, ALVINN was able to take over and continue to drive. ALVINN has successfully driven at speeds up to 70mph and for distances of over 90 miles on a public highway north of Pittsburgh. (Apparently, actually being inside the vehicle during the test drive was a big incentive for the researchers to develop a good neural network!) 10. Management Formation
The empirical research in strategic planning systems has focused on two areas: the impact of strategic planning on firm performance and the role of strategic planning in strategic decision making Neural networks as efficient tool have utilized for determining and clarifying the relationship between strategic planning and performance and also assessing the decision making. We can say that neural network approaches differ from traditional statistical techniques in many ways and the
differences can
be
exploited
by the
application developer. They are powerful alternative tools and a complement to statistical techniques when data are multivariate with a high degree of interdependence between factors, when the data are noisy or incomplete, or when many hypotheses are to be pursued and high computational rates are required. With their unique features, both methods together can lead to a powerful decision-making tool. Studies and investigations are being made to enhance the applications of ANNs and to achieve the benefits of this new technology [123]. Most frequently quoted advantages of the application of neural networks are:
Neural network models can provide highly accurate results in comparison with regression models.
Page 6
SC Assignment 2 10103006
Neural network models are able to learn any complex non-linear mapping / approximate any continuous function and can handle nonlinearities implicitly and directly.
The significance and accuracy of neural network models can be assessed using the traditional statistical measures of mean squared error and R2.
Neural network models automatically handle variable interactions if they exist [2].
Neural networks as non-parametric methods do not make a prior assumptions about the distribution of the data/input-output mapping
function.
Neural networks are very flexible with respect to incomplete, missing and noisy data/ NNs are fault tolerant.
Neural networks models can be easily updated.
It means they are suitable for dynamic environment.
Page 7
SC Assignment 2 10103006 Question 2. Write a short note on the different variations of the Back Propagation learning method. Answer 2. a) Standard Back propagation (BP)
Drawbacks – o The LMS algorithm is guaranteed to converge to a solution that minimizes the mean squared error, so long as the learning rate is not too large. o Multilayer nonlinear Net → many local minimum points → t he curvature can vary widely in different regions of the parameter space. b) Momentum
The batching form of MOBP, in which the parameters are updated only after the entire example set has been presented. The same initial condition and learning rate. The algorithm now is stable and it tends to accelerate convergence when the trajectory is moving in a consistent direction. c) Delta-bar-delta method
Each weight wjk has its own rate αjk
If ∆wjk remains in the same direction, increase αjk (F has a smooth curve in
the vicinity of current W)
If ∆wjk changes the direction, decrease αjk (F has a rough curve in the
vicinity of current W) rough curve in the vicinity of current W)
delta-bar-delta also involves momentum term
Quickprop algorithm of Fahlman (1988).(It assumes that the error surface is parabolic and concave upward around the minimum point and that the effect of each weight can be considered independently)
SuperSAB algorithm of Tollenaere (1990). (It has more complex rules for adjusting the learning rates).
Drawbacks Page 8
SC Assignment 2 10103006 o
In SDBP we have only one parameter to select, but in heuristic modification sometimes we have six parameters to be selected.
o
Sometimes modifications fail to converge while SDBP will eventually find a solution.
d) Conjugate Gradient
SD is the simplest optimization method but is often slow in converging.
Newton’s method is much faster, but requires that the Hessian matrix and
its inverse be calculated.
The conjugate gradient is a compromise; it does not require the calculation of 2nd derivatives, and yet it still has the quadratic convergence property. o
It’s necessary to choose some initial value for every n ij(0);
o
The “d” parameter is user defined (empirically)
o
The weight update is still a function of the gradient of the error.
e) Levenberg-Marquardt
Variation of Newton’s method Non-linear error function Drawback: matrix inversion Efficient in the number of epochs, but slow within each epoch
Page 9
SC Assignment 2 10103006 Question 3. Answer the following questions – a) How many hidden layers are optimal for a BPN? b) How many training pairs are minimum required to train a BPN? c) For how long a BPN should be trained? d) How can we represent data to a BPN? Answer 3.
a) Problems that require two hidden layers are rarely encountered. However, neural networks with two hidden layers can represent functions with any kind of shape. There is currently no theoretical reason to use neural networks with any more than two hidden layers. In fact, for many practical problems, there is no reason to use any more than one hidden layer. Number of Hidden Layers
none 1 2
Result
Only capable of representing linear separable functions or decisions. Can approximate any function that contains a continuous mapping from one finite space to another. Can represent an arbitrary decision boundary to arbitrary accuracy with rational activation functions and can approximate any smooth mapping to any accuracy.
b) When you back propagate, all it means is that you have changed your weights in such a manner that your neural network will get better at recognizing that particular input (that is, the error keeps decreasing). So if you present pair-1 and then pair-2, it is possible that pair-2 may negate the changes to a certain degree. However in the long run the neural network's weights will tend towards recognizing all inputs properly. The thing is, you cannot look at the result of a particular training attempt for a particular set of inputs/outputs and be concerned that the changes will be negated. As I Page 10
SC Assignment 2 10103006 mentioned before, when you're training a neural network you are traversing an error surface to find a location where the error is the lowest. Think of it as walking along a landscape that has a bunch of hills and valleys. Imagine that you don't have a map and that you have a special compass that tells you in what direction you need to move, and by what distance. The compass is basically trying to direct you to the lowest point in this landscape. Now this compass doesn't know the the landscape well either and so in trying to send you to the lowest point, it may go in a slightly-wrong direction (i.e., send you some way up a hill) but it will try and correct itself after that. In the long run, you will eventually end up at the lowest point in the landscape (unless you're in a local minima i.e., a low-point, but one that is not the lowest point). c) We have time series, i.e., a variable x changing in time x t ( t =1,2,...) and we would like to predict the value of x in time t+h. The prediction of time series using neural network consists of teaching the net the history of the variable in a selected limited time and applying the taught information to the future. Data from past are provided to the inputs of neural network and we expect data from future from the outputs of the network. As we can see, the teaching with teacher is involved. For more exact prediction, additional information can be added for teaching and prediction, for example in the form of interventional variables (intervention indicators). However, more information does not always mean better prediction; sometimes it can make the process of teaching and predicting worse. It is always necessary to select really relevant information, if it is available.
Page 11
SC Assignment 2 10103006
Fig: teaching of time series without interventional variables. The points in graph represent time series obtained by sampling of continuous data.
Page 12
SC Assignment 2 10103006
Fig : Teaching of time series with intervention indicator
d) For reasons of speed and complexity, there is often pressure to minimize the number of variables (input and output) that a neural network has to deal with. The pressure also has a bearing on the resolution at which the data is represented; the finer the data representation, the greater the complexity and consequently the amount of training time and data required. This brings us to second principle of training data representation: explicitness. Given the limit on the number of input variables a network may see, there is pressure to ensure that the variables which are used contain the information required to carry out the task to be learned in a form which is as explicit as possible.
Page 13
