| Author: | Chan, Shun-heng |
| Title: | Novel mutation operators & their application to regularization of artificial neural networks |
| Degree: | Ph.D. |
| Year: | 2003 |
| Subject: | Evolutionary computation Neural networks (Computer science) Hong Kong Polytechnic University -- Dissertations |
| Department: | Department of Electrical Engineering |
| Pages: | xiv, 151 leaves : ill. ; 30 cm |
| Language: | English |
| Abstract: | In recent years, the advances in Artificial Intelligence and the availability of high-speed computers have renewed interest in many system-modeling methods that were previously considered too "heuristic" (non-algorithm) or computationally intensive. In this thesis, we study three of these modern methods: Evolutionary Computation (EC), a derivative-free, global optimization method that consists of the branches of Genetic Algorithm (GA) and Evolutionary Strategies (ES); Markov Chain Monte Carlo (MCMC), a stochastic simulation method for approximating high-dimensional integration and Artificial Neural Networks (ANNs), a connectionist structure capable of learning complex, nonlinear mappings. In contrast to the traditional methods that use the reductionist approaches of linear modeling and local optimization, these modern methods abandon many simplistic assumptions to provide a more complex, yet comprehensive approach to the problem. While these modern methods provide a useful alternative to the traditional methods, they too, have their own weaknesses that hinder the utilization of their full potential. This thesis is on the design of two novel mutation operators that alleviate the weaknesses of GA and ES (GA and ES are branches of Evolutionary Computation) and their application to the hybrid system of GA-ANN and MCMC-ANN to provide the complete parameter estimation - including both conventional (frequentist) and Bayesian parameter estimation - of regularized ANNs. The first novel mutation operator is called MARK, which alleviates the problem of premature convergence of GA. In contrast to the conventional mutation operator that uses probabilistic (soft) operation, MARK uses deterministic (hard) operation. Probabilistic analyses with uni-modal functions and empirical experiments with test functions show that MARK converges faster and achieves higher precision than the conventional mutation operator. To verify its feasibility to complex, engineering problems, it is applied to a hybrid system of GA-ANN that performs conventional parameter estimation, or optimization, of regularized ANN. Experiment involves optimizing a regularized ANN electricity load forecaster that has 507 connection weights and 75 hyperparameters and has shown that the incorporation of MARK significantly improves the performance of the hybrid system. The second novel mutation operator is called Selection Follower (SF), which enhances the covariance-adaptive search in ES. SF replaces the conventional covariance evaluation with linear recombination for producing correlated noise, therefore saving computation intensity and removing computation error simultaneously. The principle of SF is proven with the general probability theory and its performance is shown in the empirical experiments with common test functions. To verify its feasibility to complex problems, SF is incorporated in a multi-chain MCMC algorithm called the Adaptive Metropolis Sampling (AMS) for Bayesian parameter estimation of regularized ANNs. Using EC-based mutation operator in MCMC is itself, a very innovative procedure, involving the identification of similarities between EC and MCMC and establishing theoretical basis for the technical feasibility. Experiments with simple normal distributions and the regularized ANN show that the incorporation of SF enhances the performance, in terms of mixing rate, of the hybrid system. It is concluded that these two novel mutation operators, MARK and SF, are useful in enhancing the performance in EC and their corresponding applications of parameter estimation of regularized ANN. |
| Rights: | All rights reserved |
| Access: | open access |
Files in This Item:
| File | Description | Size | Format | |
|---|---|---|---|---|
| b16974335.pdf | For All Users | 6.56 MB | Adobe PDF | View/Open |
Copyright Undertaking
As a bona fide Library user, I declare that:
- I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
- I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
- I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.
By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.
Please use this identifier to cite or link to this item:
https://theses.lib.polyu.edu.hk/handle/200/3940