Multi-modal learning using an unsupervised Deep Learning Architecture
LE3 .A278 2015
Master of Science
An unsupervised Deep Learning Architecture (DLA) is developed that is capable of learning multiple modalities and associating them with each other. As humans, we learn by associating the information we receive through di erent sensory channels like vision and hearing. To build an intelligent system that can learn in a way similar to humans, it is necessary for the system to learn multiple modalities and then associate them. The DLA uses several layers of Restricted Boltzmann Machines, an unsuper- vised algorithm, to learn hierarchical features of the data. Deep learning systems have achieved state-of-the-art results in many elds, such as image classi cation and speech recognition. However, most of them are speci c to one modality, like image or audio, and fewer e orts have been made on building a system that can handle multi- ple modalities. Multi-modal systems developed in the past use a supervised learning algorithm called back-propagation, which prevents them from scaling to a larger num- ber of modalities. The multi-modal DLA developed in this thesis uses a variation of the back- tting algorithm which allows the system to scale linearly in the number of modalities. The system tested has four modalities: image, audio, classi cation and motor. The DLA learns to associate features of all four modalities and is able to successfully reconstruct the remaining modalities when presented with as few as one modality. Due to the widely varying nature of the audio signal, the DLA reconstructs other modalities the least well when audio is presented as input. The reconstruction improves when additional modalities are presented along with the audio input. A publicly available website has been developed to demonstrate the multi-modal DLA's ability to generate associated modalities from any one modality.
The author retains copyright in this thesis. Any substantial copying or any other actions that exceed fair dealing or other exceptions in the Copyright Act require the permission of the author.