Codecov Report

❌ Patch coverage is 93.42105% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.0%. Comparing base (300f8e8) to head (8dc453d).
⚠️ Report is 25 commits behind head on main.

@@          Coverage Diff          @@
##            main    #281   +/-   ##
=====================================
  Coverage   82.0%   82.0%           
=====================================
  Files         24      25    +1     
  Lines       3489    3538   +49     
=====================================
+ Hits        2861    2902   +41     
- Misses       628     636    +8     

OK, done here on the logic & implementation. Thank you @JosiahParry for getting the ball rolling here 😄 Very much appreciated!

I've re-implemented the percentile-based two-sided test from scratch using scipy.stats.scoreatpercentile. This approach finds the percentile for the test statistic in the reference distribution and counts how many replicates are outside of (p, 1-p). Over simulations, these are always 2*directed. Second, I modified your folding approach to fold around the mean of the replicates, rather than zero (since the expected value of local stats generally isn't zero) and kept it as a folded option for testing.

I don't think we should expose the folded variant to the user in the classes, since the power in each direction is dependent on the symmetry of the distribution. For example, in the illustration below, the smallest replicate, when folded, is not "extreme," but this is accounted for in the percentile-based method.

The percentile will always equal the folded version for symmetric distributions, but the folded version becomes a directed test as skew increases. I think that the (over+under)/all is also the intended estimand of the directed approach, after re-reading the Hope paper referred to in #199

If other maintainers approve these four options (greater, lesser, two-sided, and directed) for the user classes and a folded option for this function only (for replication/testing purposes) I can start propagating this across the user classes.

I am still working on this, but I recall now why the implementation of an "alternative" argument was a bit trickier than I expected... because we allow for the user to "discard" the random replicates, rather than store them, we have to push the significance logic all the way down to the conditional randomization numba function. This may have significant performance implications, since we're currently only counting the number of larger random stats in each innermost loop.

It seems clear to me that

1. if the test statistic is as large as k realizations,
2. then there are always 2k simulations outside of the (k/n, (1-(k/n)) interval, plus the test statistic itself.
3. So, the proper p-value for the two-sided test is (2*n_greater+1)/(n_samples + 1),
4. which is off from two times our current p-value by 1/(n_samples+1).

If 1-4 are correct, this means we don't need to change any of the numba code. The correction can be calculated as 2*directed - (1/(n_samples+1)) after the numba calculation. Do I have this right @sjsrey @knaaptime @martinfleis @jGaboardi?

So, if we implement our current test for local stats without flipping (as greater), generate 1-p_sim (as lesser), and implement the above correction for the two-sided test (2*p_sim - (1/(n_samples+1))), none of the numba code needs to change.

Is that OK w/ other maintainers?

This is too much stats for me to say anything useful.

This is too much stats for me to say anything useful.

Same for me.

One further wrinkle as well: some global Moran tests support directed testing a binary option already. Notably, if two_tailed=False, they pick the test direction based on whether the global I is positive or negative. It's also useful to note: this means we currently pick the smallest one-tailed p-value and, if the test is two-tailed, multiply this by two.

For us to roll-out the testing across all the classes, we need to consider if this option should be deprecated in favor of an explicit "alternative" option? Right now, there's no way to force a direction on these tests.

I am still working on this, but I recall now why the implementation of an "alternative" argument was a bit trickier than I expected... because we allow for the user to "discard" the random replicates, rather than store them, we have to push the significance logic all the way down to the conditional randomization numba function. This may have significant performance implications, since we're currently only counting the number of larger random stats in each innermost loop.

It seems clear to me that

1. if the test statistic is as large as k realizations,
2. then there are always 2k simulations outside of the (k/n, (1-(k/n)) interval, plus the test statistic itself.
3. So, the proper p-value for the two-sided test is (2*n_greater+1)/(n_samples + 1),
4. which is off from two times our current p-value by 1/(n_samples+1).

If 1-4 are correct, this means we don't need to change any of the numba code. The correction can be calculated as 2*directed - (1/(n_samples+1)) after the numba calculation. Do I have this right @sjsrey @knaaptime @martinfleis @jGaboardi?

So, if we implement our current test for local stats without flipping (as greater), generate 1-p_sim (as lesser), and implement the above correction for the two-sided test (2*p_sim - (1/(n_samples+1))), none of the numba code needs to change.

Is that OK w/ other maintainers?

I think this is OK.

One thing to check is if:

3. So, the proper p-value for the two-sided test is (2*n_greater+1)/(n_samples + 1),
   Should be
   2(n_greater+1)/(n_samples+1)

One thing to check is if:

Sure, that is what I initially thought & what @JosiahParry suggested.

The reason why I'm thinking it's actually 2*p_sim - (1/(n_permutations + 1)) is because using 2*p_sim amounts to counting the test stat twice: 2*p_directed = 2 * (n_outside + 1)/(n+1) = (2*outside + 2)/(n+1). The difference will be vanishingly small as the number of permutations increases, but it's the principle...

Thinking another way, in the percentile-based version of the test, you compute the percentile p for the test statistic, count how many null statistics are outside of (p,1-p) and add one, since the test stat is always at least at its own percentile. This p-value is smaller than p_sim by 1/(n_samples-1), which is what would happen in the percentile test if you counted the test statistic twice.

the simulation code at the end of esda/simulation.py should illustrate?

@ljwolf I was looking at the discussions in this PR and the other related issue. The correction for the two-sided test 2*p_sim - (1/(n_samples+1)) looks correct to me.

Thank you for the explanation @ljwolf. I think I'm almost there/onboard!

It's worth calling out explicitly this formula can result in a p-value > 1.0 which should also be handled e.g.

p_sim = 0.65
nsim = 999

(p_corrected = (2*p_sim - (1/(nsim + 1))))
#> [1] 1.299

if (p_corrected > 1) {
  1.0
} else {
  p_corrected
}
#> [1] 1

Additionally, would you mind elaborating why it is - (1/(nsim + 1)) as opposed to + (1/(nsim + 1))? To me, it makes more sense to penalize smaller numbers of simulations rather than larger number of simulations. For example subtracting the second term results in smaller p values for smaller numbers of simulations and larger ones for larger numbers of simulations.

calc_p_sim <- function(p_sim, nsim) {
  (p_corrected = (2*p_sim - (1/(nsim + 1))))

  if (p_corrected > 1) {
    1.0
  } else {
    p_corrected
  }

}

calc_p_sim(0.05, 49)
#> [1] 0.08
calc_p_sim(0.05, 99)
#> [1] 0.09
calc_p_sim(0.05, 999)
#> [1] 0.099

@weikang9009 notes correctly that we will also need to update the notebooks where local/global statistics are used before merging this.

the affected notebook has been fixed and #376 has been addressed!

I think this is ready to merge.

Great work, @ljwolf !

Thanks! it seems there's a broadcasting issue that's numba-version dependent. I will squash this issue, and then it's ready.

OK, all tests are passing except windows, which look like a build issue. Could we merge this? Or, can I get some help to identify the issue with the windows build? This touches numba code, but no code is specific to windows, and all tests on linux/macos pass.

Thanks @martinfleis! @sjsrey can you merge?