Permutation Strategy In Two-Sample Test: Convex Combinations?

Hello everyone!

I'm writing this article to delve into a fascinating question regarding the permutation strategy employed in the two-sample discriminability test, specifically within the context of the discrim_two_sample.py implementation. As a researcher deeply involved in neurodata analysis and hypothesis testing (hyppo), I encountered a discrepancy between the current implementation and the methodology described in the original research paper by Bridgeford et al. (2021). This prompted me to seek clarification and share my understanding with the community. Let's explore this together!

The Discrepancy: Within-Matrix vs. Between-Matrix Convex Combinations

The core of the issue lies in how the permutation is performed when comparing two Discr estimates. In the discrim_two_sample.py code, specifically lines 190-191, the implementation utilizes convex combinations within each matrix separately. This means that for two input matrices, x1 and x2, the code generates permutations permx1 and permx2 by taking convex combinations of data points within x1 and x2, respectively.

permx1 = self._get_convex_comb(self.x1, random_state)  # convex combination within x1
permx2 = self._get_convex_comb(self.x2, random_state)  # convex combination within x2

However, the original paper by Bridgeford et al. (2021) suggests a different approach. The paper describes creating "randomly combined datasets" by taking "random convex combinations of the observed data from each of the two methods choices." My interpretation of this description is that the permutation should involve mixing data between the two input matrices. This implies generating convex combinations using data points from both x1 and x2, effectively shuffling the data across the two samples.

This difference in permutation strategy raises a crucial question: Why the current implementation chooses to perform convex combinations within each matrix rather than between them? Understanding the rationale behind this design choice is essential for ensuring the validity and interpretability of the test results.

Exploring the Rationale Behind Within-Matrix Permutations

To understand the rationale behind performing convex combinations within each matrix, we need to consider the underlying principles of permutation testing and the specific goals of the two-sample discriminability test. Here's a breakdown of potential reasons:

1. Preserving Data Structure and Dependencies

One key advantage of performing convex combinations within each matrix is that it helps preserve the inherent structure and dependencies within each dataset. In many real-world datasets, especially in neurodata, data points are not independent and identically distributed (i.i.d.). There might be complex relationships and correlations between variables within each sample.

By performing permutations within each matrix, we ensure that these internal dependencies are maintained during the permutation process. This is crucial because if we were to randomly shuffle data points between the two matrices, we would disrupt these internal structures, potentially leading to spurious results. The permutation test's null hypothesis assumes that any observed difference between the two samples is due to random chance, given that the data within each sample retains its original structure. By preserving the internal structure through within-matrix convex combinations, we can more accurately assess the significance of the observed difference.

For instance, consider a neuroimaging study where we are comparing brain activity patterns between two groups of participants. Within each group, there are likely to be specific patterns of connectivity and co-activation between different brain regions. If we were to randomly shuffle data points between the two groups, we would disrupt these within-group connectivity patterns, which could artificially inflate the observed difference between the groups. By performing convex combinations within each group, we maintain these within-group connectivity patterns, ensuring that the permutation test provides a more accurate assessment of the true difference between the groups.

2. Controlling for Confounding Variables

Another reason for using within-matrix permutations is to control for potential confounding variables. Confounding variables are factors that are correlated with both the independent variable (the group membership in this case) and the dependent variable (the Discr estimate). If not controlled for, confounding variables can lead to biased results.

By performing permutations within each matrix, we effectively control for any confounding variables that are specific to each sample. For example, if one group of participants has a higher average age than the other group, age could be a confounding variable. By performing permutations within each group, we ensure that the age distribution remains the same within each permuted sample, effectively controlling for the effect of age on the Discr estimate.

In other words, within-matrix permutations help to isolate the effect of the group membership on the Discr estimate, while controlling for the influence of other factors that might be specific to each sample. This leads to a more robust and reliable test of the null hypothesis.

3. Theoretical Considerations: Exchangeability

The choice of permutation strategy is also closely tied to the theoretical concept of exchangeability. In permutation testing, we want to create permuted datasets that are exchangeable under the null hypothesis. This means that if the null hypothesis is true (i.e., there is no difference between the two samples), then any permutation of the data should be equally likely.

Performing convex combinations within each matrix is more likely to satisfy the exchangeability assumption, especially when the data within each sample has a specific structure or distribution. By preserving the within-sample structure, we ensure that the permuted datasets are still representative of the original data under the null hypothesis.

If we were to randomly shuffle data points between the two matrices, we might violate the exchangeability assumption if the data in the two samples has different distributions or characteristics. In such cases, the permutation test might produce inaccurate results.

4. Computational Efficiency

While not the primary reason, computational efficiency can also play a role in the choice of permutation strategy. Performing convex combinations within each matrix can be computationally more efficient than mixing data between the two matrices, especially for large datasets. This is because within-matrix permutations can be performed independently for each sample, whereas between-matrix permutations require shuffling data across the entire dataset.

However, it's important to note that computational efficiency should not be the sole deciding factor. The primary consideration should always be the statistical validity and interpretability of the test results.

Potential Scenarios Where Between-Matrix Permutations Might Be Appropriate

While within-matrix permutations offer several advantages in many scenarios, there might be specific situations where between-matrix permutations could be more appropriate. For example, if the primary goal is to test whether the two samples come from the same underlying distribution, regardless of any within-sample structure, then between-matrix permutations might be a more powerful approach.

In such cases, the focus is on testing the global null hypothesis that the two samples are exchangeable, without making any assumptions about the structure or dependencies within each sample. However, it's important to carefully consider the potential drawbacks of between-matrix permutations, such as the disruption of within-sample structure and the potential for introducing bias.

Conclusion: A Deliberate Design Choice

Based on these considerations, it seems likely that the choice to perform convex combinations within each matrix in discrim_two_sample.py is a deliberate design choice, driven by the desire to preserve data structure, control for confounding variables, and satisfy the exchangeability assumption. This approach is particularly well-suited for situations where the data within each sample has a specific structure or distribution, as is often the case in neurodata analysis.

However, it's essential to be aware of the potential limitations of this approach and to consider whether between-matrix permutations might be more appropriate in specific scenarios. Further research and discussion are always valuable in refining our understanding of these methods and ensuring their appropriate application.

I hope this article has shed some light on the rationale behind the permutation strategy used in discrim_two_sample.py. I encourage further discussion and exploration of this topic within the community.

For more information on two-sample hypothesis testing and permutation methods, you can visit reputable resources such as Wikipedia's article on Permutation Tests.