Some users are running into an issue where many isolates are calling out to C code that will then block. This can cause the dart app to no longer work due to our limitation on maximum number of threads that can be active in an isolate group at a given point in time.

The limitation is there to avoid too many threads executing Dart code at same time. This can lead to situations where X threads all have TLABs which may contain unallocated memory, but the X+1 thread tries to obtain TLAB and fails, which will cause it to trigger GC (despite other thread's TLAB still having unallocated memory)
=> Allowing unbounded number of threads to enter an isolate group can lead to excessive triggering of GCs (despite free memory in other thread's TLAB)

See runtime/vm/heap/scavenger.h for the current calculation of the limit:

  // The maximum number of Dart mutator threads we allow to execute at the same
  // time.
  static intptr_t MaxMutatorThreadCount() {
    // With a max new-space of 16 MB and 512kb TLABs we would allow up to 8
    // mutator threads to run at the same time.
    const intptr_t max_parallel_tlab_usage =
        (FLAG_new_gen_semi_max_size * MB) / Scavenger::kTLABSize;
    const intptr_t max_pool_size = max_parallel_tlab_usage / 4;
    return max_pool_size > 0 ? max_pool_size : 1;
  }

We may consider adding a boolean flag to specify that a FFI call may be blocking / should exit the isolate.

// Static binding
@Native("sleep", exitIsolate: true)
external void sleep(int seconds);

// Dynamic binding
dylib.lookup().asFunction<...>(exitIsolate: true);

to automatically exit and re-enter the isolate to avoid custom C code like this:

auto isolate = Dart_CurrentIsolate();
Dart_ExitIsolate();
<... run blocking C Code, e.g. sleep() ...>
Dart_EnterIsolate(isolate);

See motivating use case: #51254