Blind signal separation (BSS) technology originated from the famous “cocktail party” problem (Choi and Cichocki, 1997) and has been a key research issue since then (Bridwell et al., 2018; Yatabe and Kitamura, 2018). It attempts to separate the unknown original signals only by using the observations of their mixture without having any prior information of the source signals and channels. It is important to measure the effectiveness of the blind separation algorithms, since it affects not only the effect of the algorithms, but also the iterative process of the algorithms. Evaluating the quality of the waveforms of the signals and the similarity of the waveforms is the most straightforward way to judge the effectiveness of algorithm. However, this method can only be regarded as a qualitative analysis, which cannot quantitatively reflect the stability and convergence rate of the algorithm. Therefore, researchers have proposed some performance indexes, such as crosstalk error (Cichocki and Amari, 2002; Parra and Sajda, 2003; El-Sankary et al., 2015), correlation coefficient (Li et al., 2012) and signal-to-noise ratio (Laheld and Cardoso, 1993; Li and Sejnowski, 1995; Grellier and Comon, 1998; Nandakumar and Bijoy, 2014; Aroudi et al., 2016; Mirzal, 2017) etc. The consistence between the order of the source signals and the separated signals is needed for calculating most of the performance indexes, but the order of the separated signals is uncertain. The crosstalk error does not need to consider the correspondence, and it can reflect the separation effect directly.

The crosstalk error represents the approximate degree between the global separation matrix C and a diagonal matrix. When C is closer to a diagonal matrix or its permutation matrix, the smaller the crosstalk error, the more similar the separated signals are to the source signals. Theoretically, the crosstalk error should be a good performance index in the field of BSS (Macchi, 1993; Moreau and Macchi, 1996; Amari, 1997; Yang and Amari, 1997; Moreau and Macchi, 1998; Macchi and Moreau, 1999; Yang, 1999; Basak and Amari, 1999). However, it is very sensitive to numerical errors and can not be directly used to evaluate separation effects in many practical situations. On one hand, in order to calculate the crosstalk error, it needs to know C in advance, which means that it cannot directly be applied to real-world experiments. On the other hand, if the maximum value of each row of C is in the same column, the crosstalk error usually does not correctly evaluate the separation effect. In this case, small crosstalk errors do not indicate that the separation effect is good. Since each component of the separated signal is an approximate estimation of the same source signal component, the actual separation effect is poor. The above two shortcomings affect its practical application. In order to overcome the above problems, Li Zong have proposed a method to improve crosstalk error evaluation criterion by introducing correlation coefficients to measure the degree of correlation between the separated signal components and adding them to the calculation formula of crosstalk error (Li et al., 2012). Although this method improves the accuracy of the evaluation when the maximum values of two rows of C are in the same column, C is still unknown when calculating crosstalk error in the practical application. Zhang used the Least Mean Square (LMS) algorithm to estimate C in real time (Zhang et al., 2011). However, the convergence rate of the method is limited by the convergence step size of the algorithm, and once the step size is inappropriate, the algorithm will fail. At the same time, when the value changes subtly in C (Mansour et al., 2002), the crosstalk error may vary greatly. In other words, the crosstalk error is not robust.

In response to the above problems, a fully adaptive crosstalk error estimation model based on the recursive least square (RLS) algorithm is presented in this paper. This new approach consists of sifting the impact of step size on the estimation of crosstalk error. Furthermore, the robustness of the crosstalk error is improved by introducing the position information of the maximum value of each row in the global separation matrix C in the process of calculating the crosstalk error.

The paper is organized as follows. In Section 2, the shortcomings of the original performance index are analysed. The main contributions of the paper can be found in Section 3, where the fully adaptive crosstalk error estimation model is presented firstly, and then the position information of the maximum value of each row is introduced to improve the robustness of the crosstalk error. Finally, the whole process of the algorithm is explained. The validity of the proposed algorithm is verified by experimental analysis in Section 4. Finally, Section 5 contains some conclusions.

Suppose that an antenna array is made up of M units, therefore the instantaneous linear mixing model can be expressed as (1) xt=ASt+Nt, where xt∈Rm stands for the observation vector and St=[s1(t),s2(t),…,sn(t)]T is the source signal vector. Nt=[n1(t),n2(t),…,nm(t)]T is the additive observation noise vector. The subscript in xt , St and Nt stands for the vectors at the moment t. A∈Rm×n is named as mixed matrix. In general, the noise signal ni(t) , i=1,2,…,m satisfies Gaussian distribution, and the literature (Qian et al., 2006) pointed out that the separated signal after the blind source separation is only the superposition of source signal and Gaussian noise.

In order to simplify the problem of BSS and reduce the computational complexity, it is generally necessary to pre-whiten the signal, and the signal after whitening is (2) zt=Vxt, where V is a n×m whitening matrix. After whitening, the problem of BSS changes from seeking for the mixed matrix A to seeking an n×n orthogonal decomposition matrix W (Jiang, 2014). Supposing yt is the estimated vector of St=[s1(t),s2(t),…,sn(t)]T and yt can be modelled as (3) yt=Wzt=WVASt+WVNt. The global separation matrix C is defined as (4) C=WVA, C is a generalized permutation matrix (Cichocki and Amari, 2002), which is equal to the product of a permutation matrix and a full rank diagonal matrix: (5) C=PΛ, where P is a permutation matrix, which represents the uncertainty of the order of the blind separation results. Λ is a full rank diagonal matrix, which represents the uncertainty of the amplitude of the blind separation results. It can be seen that the waveforms of signal can be completely recovered except for the real amplitude and order of the source signals.

In practice, the blind source separation algorithms can only make C as close as possible to a generalized permutation matrix. Based on this, in order to evaluate the performance of the separation algorithms, the literature (Cichocki and Amari, 2002) gave a measurement method by using the difference between C and a generalized permutation matrix, which is named as crosstalk error and expressed as (6) PI(C)=1n(n−1){ ∑i=1n( ∑j=1n|cij|maxk|cik|−1)+ ∑j=1n( ∑i=1n|cij|maxk|ckj|−1)}, where cij stands for the element in row i and column j. The smaller the value of PI , the better the separation effect of the algorithm. If and only if C is a generalized permutation matrix, PI=0 . According to formula (6), it can be seen that the key to compute the crosstalk error is to obtain the matrix C. Another shortcoming of the crosstalk error is that it is not robust. As an example, the following two global separation matrices are presented: (7) C1=10000010124100001000001000080000;C2=10000010124100001000.01001000080000.

In matrices C1 and C2 , the maximum value of every row is in the first column, so the isolated signal is only the same signal. The corresponding separation effect of C1 and C2 is very poor, and the value of PI(C1) and PI(C2) should be very big. However, the crosstalk errors of them are PI(C1)=0.00002 and PI(C2)=0.00057 , which are very small. At the same time, from matrices PI(C1) and PI(C2) , we can also see that the crosstalk error will change greatly even though there is a minor change in C.

Therefore, the maximum value of all the rows in the C should be in different columns. However, from the definition of formula (6), this restriction is not considered. Therefore, in this paper, we add the position information of the maximum value of each row in the C to the definition of the crosstalk error to improve the robustness of the crosstalk error.

For overcoming the shortcomings of the original crosstalk error, firstly a model for estimating the mixed matrix A is established and the corresponding analytic solution is given in this section. Then, the position information constraint is introduced into the definition of crosstalk error to improve its robustness. Finally, the whole process of the algorithm for calculating the crosstalk error is given.

In this section, the proposed algorithm is evaluated by two methods, which are simulation experiment and real world experiment.

At present, the crosstalk error is widely used to verify the validity and stability of the BSS algorithms in the simulation conditions. However, in the real application environment, it is impossible to know the global separation matrix of the signal in advance. Therefore, this criterion cannot be used in practice. In this paper, the crosstalk error calculation model is established based on the RLS algorithm, and the calculation method of the crosstalk error is deduced as an adaptive algorithm without prior knowledge by using real-time estimating of the global separation matrix. It can greatly extend the use scope of the crosstalk error. At the same time, a new crosstalk error definition is proposed to improve the robustness and convergence rate of the original crosstalk error. Finally, the experimental results show that the method proposed in this paper can predict the convergence trend of the crosstalk error well, which indicates the validity and robustness of the method proposed in this paper.