Optimization techniques for image restoration, neural networks, and heterogeneous supercomputer system design

Mohamed Mohandes, Purdue University

Abstract

Simulated annealing algorithms for optimization over continuous spaces come in two varieties: Markov chain annealing algorithm (MCAA) and gradient annealing algorithm (GAA). There is a large amount of theoretical analysis and practical methodology developed for MCAA. The practical methodology is closely linked to heuristic procedures developed in Monte Carlo simulation of physical systems, from which the MCAA was originally developed. It has been observed that MCAA is not a viable approach to high-dimensional problems with smooth cost function. Theoretical analysis has been developed for the GAA but no practical methodology exists in the literature. As GAA attempts to exploit smoothness by its use of derivatives, an appropriate implementation should outperform the MCAA for high-dimensional problems with smooth cost functions. In this thesis we propose a practical methodology for the GAA. This methodology is tested and compared with other annealing algorithms on a set of benchmark functions. Also, we use the GAA for the restoration of gray-level images corrupted by multiplicative noise, where we seek the maximum a posteriori probability (MAP) estimate of the original image given the degraded one. In this thesis we show that existing stepsize rules for gradient descent can skip over nearby minima, particularly if the minima are deep and have small regions of attraction. Skipping over nearby minima may be undesirable when a local search is used as part of a global optimization strategy. propose a conservative stepsize rule and prove that it has a linear rate of convergence. This stepsize rule depends on the maximum eigenvalue of the Hessian matrix, which is usually hard to compute. We propose two methods to estimate this value from function and gradient values. Using these methods we develop two adaptive stepsize rules which we use in conjunction with the gradient annealing algorithm and for training feedforward neural networks. Distributed Heterogeneous Supercomputing Systems (DHSS) have been suggested to achieve computational speedup for complex applications comprising numerous tasks with varying computational characteristics. DHSS are expected to outperform homogeneous supercomputing systems, whose performance can be degraded severely by an ill-matched segment of code. In this thesis we propose and evaluate a neural network approach for mapping tasks to a suite of heterogeneous supercomputers.

Degree

Ph.D.

Advisors

Gelfand, Purdue University.

Subject Area

Electrical engineering|Computer science|Artificial intelligence

COinS