| Bell, A.J., Sejnowski, T.J. An information maximisation approach to blind separation and blind deconvolution Neural Computation 1995 (7)6:1129-1159 [pdf] |
| We derive a new self-organising learning algorithm which maximises the information transferred in a network of non-linear units. The algo- rithm does not assume any knowledge of the input distributions, and is de ned here for the zero-noise limit. Under these conditions, infor- mation maximisation has extra properties not found in the linear case (Linsker 1989). The non-linearities in the transfer function are able to pick up higher-order moments of the input distributions and perform something akin to true redundancy reduction between units in the out- put representation. This enables the network to separate statistically independent components in the inputs: a higher-order generalisation of Principal Components Analysis. We apply the network to the source separation (or cocktail party) problem, successfully separating unknown mixtures of up to ten speak- ers. We also show that a variant on the network architecture is able to perform blind deconvolution (cancellation of unknown echoes and reverberation in a speech signal). Finally, we derive dependencies of information transfer on time delays. We suggest that information max- imisation provides a unifying framework for problems in `blind' signal processing. |
| |
|
|
| Meinecke, F., Ziehe, A., Kawanabe, M., Müller, K.-R. Assessing Reliability of ICA Projections - a Resampling Approach 2001 [pdf] |
| When applying unsupervised learning techniques like ICA or temporal decorrelation for BSS, a key question is whether the discovered projections are reliable. In other words: can we give error bars or can we assess the quality of our separation? We use resampling methods to tackle these questions and show experimentally that our proposed variance estimations are strongly correlated to the separation error. We demonstrate that this reliability estimation can be used to choose an appropriate ICA-model, to enhance signifi- cantly the separation performance, and, most important, to mark the components that can really have a physical meaning. An application to data from an MEG1-experiment underlines the usefulness of our approach. |
| |
|
|
| Lewicki, M. Efficient coding of natural sounds Nature Neuroscience 2002 (5)4:356-363 [pdf] |
| The auditory system encodes sound by decomposing the amplitude signal arriving at the ear into multiple frequency bands whose center frequencies and bandwidths are approximately exponential functions of the distance from the stapes. This organization is thought to result from the adaptation of cochlear mechanisms to the animal's auditory environment. Here we report that several basic auditory nerve fiber tuning properties can be accounted for by adapting a population of filter shapes to encode natural sounds efficiently. The form of the code depends on sound class, resembling a Fourier transformation when optimized for animal vocalizations and a wavelet transformation when optimized for non-biological environmental sounds. Only for the combined set does the optimal code follow scaling characteristics of physiological data. These results suggest that auditory nerve fibers encode a broad set of natural sounds in a manner consistent with information theoretic principles. |
| |
|
|
| Olshausen, B., Field, D. Emergence of simple-cell receptive field properties by learning a sparse code for natural images Nature 1996 (381):607-609 [html] |
| The receptive fields of simple cells in mammalian primary visual cortex can be characterized as being spatially localized, oriented and bandpass (selective to structure at different spatial scales), comparable to the basis functions of wavelet transforms. One approach to understanding such response properties of visual neurons has been to consider their relationship to the statistical structure of natural images in terms of efficient coding. Along these lines, a number of studies have attempted to train unsupervised learning algorithms on natural images in the hope of developing receptive fields with similar properties, but none has succeeded in producing a full set that spans the image space and contains all three of the above properties. Here we investigate the proposal that a coding strategy that maximizes sparseness is sufficient to account for these properties. We show that a learning algorithm that attempts to find sparse linear codes for natural scenes will develop a complete family of localized, oriented, bandpass receptive fields, similar to those found in the primary visual cortex. The resulting sparse image code provides a more efficient representation for later stages of processing because it possesses a higher degree of statistical independence among its outputs. |
| |
|
|
| Cardoso, J.-F. Equivariant adaptive source separation IEEE Trans. on S.P. 1996 (44)45:3017-3030 [html] |
| Source separation consists in recovering a set of independent signals when only mixtures with unknown coefficients are observed. This paper introduces a class of adaptive algorithms for source separation which implements an adaptive version of equivariant estimation and is henceforth called EASI (Equivariant Adaptive Separation via Independence) . The EASI algorithms are based on the idea of serial updating: this specific form of matrix updates systematically yields algorithms with a simple, parallelizable structure, for both real and complex mixtures. Most importantly, the performance of an EASI algorithm does not depend on the mixing matrix. In particular, convergence rates, stability conditions and interference rejection levels depend only on the (normalized) distributions of the source signals. Close form expressions of these quantities are given via an asymptotic performance analysis. This is completed by some numerical experiments illustrating the effectiveness of the proposed apprach. |
| |
|
|
|
| Hyvärinen, A. Independent Component Analysis in the Presence of Gaussian Noise by Maximizing Joint Likelihood Neurocomputing 1998 (22):49-67 |
| |
|
|
|
| Cardoso, J.-F. Infomax and maximum likelihood for source separation IEEE Letters on Signal Processing 1997 (4)4:112-114 [html] |
| Algorithms for the blind separation of sources can be derived from several different principles. This letter shows that the recently proposed infomax principle is equivalent to maximum likelihood. Introduction. Source separation consists in recovering a set of unobservable signals (sources) from a set of observed mixtures. |
| |
|
|
| MacKay, D.J.C. Maximum Likelihood and Covariant Algorithms for ICA 1996 [html] |
| Bell and Sejnowski (1995) have derived a blind signal processing algorithm for a non-linear feedforward network from an information maximization viewpoint. This paper first shows that the same algorithm can be viewed as a maximum likelihood algorithm for the optimization of a linear generative model. Second, a covariant version of the algorithm is derived. This algorithm is simpler and somewhat more biologically plausible, involving no matrix inversions; and it converges in a smaller number of iterations. Third, this paper gives a partial proof of the `folk-theorem' that any mixture of sources with high-kurtosis histograms is separable by the classic ICA algorithm. Fourth, a collection of formulae are given that may be useful for the adaptation of the non-linearity in the ICA algorithm. |
| |
|
|
|
| Amari, S.-I. Natural Gradient Learning for Over- and Under-Complete Bases in ICA Neural Computation 1999 [pdf] |
| Independent component analysis or blind source separation is a new technique of extracting independent signals from mixtures. It is applicable even when the number of independent sources is unknown and is larger or smaller than the number of observed mixture signals. This article extends the natural gradient learning algorithm to be applicable to these overcomplete and undercomplete cases. Here, the observed signals are assumed to be whitened by preprocessing, so that we use the natural Riemannian gradient in Stiefel manifolds. |
| |
|
|
| Nadal, J.-P. , Parga, N. Non linear neurons in the low noise limit: a factorial code maximizes information transfer Network 1994 [html] |
| We investigate the consequences of maximizing information transfer in a simple neural network (one input layer, one output layer), focussing on the case of non linear transfer functions. We assume that both receptive fields (synaptic efficacies) and transfer functions can be adapted to the environment. The main result is that, for bounded and invertible transfer functions, in the case of a vanishing additive output noise, and no input noise, maximization of information (Linsker's infomax principle) leads to a factorial code - hence to the same solution as required by the redundancy reduction principle of Barlow, or, in the signal processing language, to Independent Component Analysis (ICA). We show also that this result is valid for linear, more generally unbounded, transfer functions, provided optimization is performed under an additive constraint, that is which can be written as a sum of terms, each one being specific to one output neuron. Finally we study the effect of a non zero input noise. We find that, at first order in the input noise, assumed to be small as compared to the - small - output noise, the above results are still valid, provided the output noise is uncorrelated from one neuron to the other. |
| |
|
|
| Murata, N. Properties of the Empirical Characteristic Function and its Application to Testing for Independence 2001 [pdf] |
| In this article, the asymptotic properties of the empirical characteristic function are discussed. The residual of the joint and marginal empirical characteristic functions is studied and the uniform convergence of the residual in the wider sense and the weak convergence of the scaled residual to a Gaussian process are investigated. Taking into account of the result, a statistical test for independence against alternatives is considered. |
| |
|
|
|