| Bell, A.J., Sejnowski, T.J. An information maximisation approach to blind separation and blind deconvolution Neural Computation 1995 (7)6:1129-1159 [pdf] |
| We derive a new self-organising learning algorithm which maximises the information transferred in a network of non-linear units. The algo- rithm does not assume any knowledge of the input distributions, and is de ned here for the zero-noise limit. Under these conditions, infor- mation maximisation has extra properties not found in the linear case (Linsker 1989). The non-linearities in the transfer function are able to pick up higher-order moments of the input distributions and perform something akin to true redundancy reduction between units in the out- put representation. This enables the network to separate statistically independent components in the inputs: a higher-order generalisation of Principal Components Analysis. We apply the network to the source separation (or cocktail party) problem, successfully separating unknown mixtures of up to ten speak- ers. We also show that a variant on the network architecture is able to perform blind deconvolution (cancellation of unknown echoes and reverberation in a speech signal). Finally, we derive dependencies of information transfer on time delays. We suggest that information max- imisation provides a unifying framework for problems in `blind' signal processing. |
| |
|
|
|
|
| Cardoso, J.-F. Infomax and maximum likelihood for source separation IEEE Letters on Signal Processing 1997 (4)4:112-114 [html] |
| Algorithms for the blind separation of sources can be derived from several different principles. This letter shows that the recently proposed infomax principle is equivalent to maximum likelihood. Introduction. Source separation consists in recovering a set of unobservable signals (sources) from a set of observed mixtures. |
| |
|
|
|
| Nadal, J.-P. , Parga, N. Non linear neurons in the low noise limit: a factorial code maximizes information transfer Network 1994 [html] |
| We investigate the consequences of maximizing information transfer in a simple neural network (one input layer, one output layer), focussing on the case of non linear transfer functions. We assume that both receptive fields (synaptic efficacies) and transfer functions can be adapted to the environment. The main result is that, for bounded and invertible transfer functions, in the case of a vanishing additive output noise, and no input noise, maximization of information (Linsker's infomax principle) leads to a factorial code - hence to the same solution as required by the redundancy reduction principle of Barlow, or, in the signal processing language, to Independent Component Analysis (ICA). We show also that this result is valid for linear, more generally unbounded, transfer functions, provided optimization is performed under an additive constraint, that is which can be written as a sum of terms, each one being specific to one output neuron. Finally we study the effect of a non zero input noise. We find that, at first order in the input noise, assumed to be small as compared to the - small - output noise, the above results are still valid, provided the output noise is uncorrelated from one neuron to the other. |
| |
|
|
|
| Csiszar, I. The Method of Types EEETIT: IEEE Transactions on Information Theory 1998 (44)6:2505-2523 [html] |
| The method of types is one of the key technical tools in Shannon Theory, and this tool is valuable also in other fields. In this paper, some key applications will be presented in sufficient detail enabling an interested nonspecialist to gain a working knowledge of the method, and a wide selection of further applications will be surveyed. These range from hypothesis testing and large deviations theory through error exponents for discrete memoryless channels and capacity of arbitrarily varying channels to multiuser problems. While the method of types is suitable primarily for discrete memoryless models, its extensions to certain models with memory will also be discussed. |
| |
|
|