When using permutation_isc and bootstrap_isc with pairwise=False, the permutations are carried out by applying signflips, group exchanges, or bootstrap resampling to the correlation values of individual subjects with the rest of the group. For example, in the one-sample permutation each subject's LOO value is randomly flipped or not flipped:
|
sign_flipper = prng.choice([-1, 1], |
I don't think this is correct. When using LOO ISC, each subject's data is used to compute ISC value (as part of the "group"), and flipping the sign of a subject's data would impact every LOO value for all subjects, not just the LOO value of that specific subject. I believe this problem is true for both the 1-sample and 2-sample permutation and bootstrap tests as they are currently coded.
The Chen2016 reference, as far as I can tell, only describes permutations and bootstraps in the pairwise case, and does not describe a valid way of doing this for the leave-one-out case. I can't think of a way to do this that would be valid, so my proposed solution would be to have these functions thrown an error if they are attempted to be called with LOO ISCs.
Sam, let me know if I am misunderstanding how these functions are working.
When using permutation_isc and bootstrap_isc with pairwise=False, the permutations are carried out by applying signflips, group exchanges, or bootstrap resampling to the correlation values of individual subjects with the rest of the group. For example, in the one-sample permutation each subject's LOO value is randomly flipped or not flipped:
brainiak/brainiak/isc.py
Line 938 in 69a0c93
I don't think this is correct. When using LOO ISC, each subject's data is used to compute ISC value (as part of the "group"), and flipping the sign of a subject's data would impact every LOO value for all subjects, not just the LOO value of that specific subject. I believe this problem is true for both the 1-sample and 2-sample permutation and bootstrap tests as they are currently coded.
The Chen2016 reference, as far as I can tell, only describes permutations and bootstraps in the pairwise case, and does not describe a valid way of doing this for the leave-one-out case. I can't think of a way to do this that would be valid, so my proposed solution would be to have these functions thrown an error if they are attempted to be called with LOO ISCs.
Sam, let me know if I am misunderstanding how these functions are working.