Nadezhda SEMENOVA :"General aspects of noise in deep neural networks on an analog hardware architecture"

Abstract : We study and analyze the fundamental aspects of noise propagation in recurrent as well as deep, multilayer neural networks comprising noisy analogue neurons as well as noisy connections. The motivations of our study are therefore neural networks implemented in special purpose analog hardware; yet, the methodology provides insight into noisy networks in general. Considering noisy neurons, we investigate the signal-to-noise ratio at the network’s outputs, which determines the upper limit of computational precision. We study additive and multiplicative as well as correlated and uncorrelated noise and develop analytical methods that predict the noise level in any layer of symmetric feedforward neural networks or feedforward neural networks trained with back propagation. We find that noise accumulation is generally bounded, and adding additional network layers does not worsen the signal to noise ratio beyond this limit. Finally, we propose noise mitigation strategies for deep neural networks working for different noise types. The proposed analytical results are then extended for an application to trained feedforward neural networks. As an example, we show that it is in a good correspondence with networks trained for digit recognition, classification of clothing images and prediction of chaotic time series. In this thesis we also consider recurrent neural networks with linear neurons and apply the obtained analytical results to complex-valued recurrent neural networks with nonlinear neurons, whose topology is based on a hardware realization in integrated photonic circuits. For both networks we identify the parts that are most vulnerable to different kinds of noise, and demonstrate that some noise types can be suppressed by the system itself.

Jury Composition :

M. Laurent LARGER Professeur Université Bourgogne Franche-Comté PhD Director
M. Daniel BRUNNER Chargé de recherche CNRS PhD Co-director
M. Hervé MAILLOTTE Directeur de recherche CNRS Reviewer
M. Miguel C. SORIANO Professeur associé Universitat de les Illes Balears Reviewer
M. Serge MASSAR Professeur Université Libre de Bruxelles Reporter
M. Damien QUERLIOZ Directeur de recherche CNRS Reporter

Localization : Amphithéâtre J-J Gagnepain, Institut FEMTO 15 B, avenue des Montboucons 25000 Besançon