Share memory (array) sub-system API and optional tight integration with numpy #339

First draft justifications and theoretical underpinning

The main focus of this effort is to provide a simple and low(er) level API for the management of shared memory array/buffers for the purposes of real-time processing.

By leveraging the semantics and constraints of structured concurrency, we can provide an API that avoids the presence of data races in the context of concurrent stream and signal processing and the resource management of shared data structures without the need for special additional "shared resource subsystems" such as the stdlib's ResourceTracker or the public APIs for SharedMemoryManager.

In an SC multi-processing system (which tractor is) such auxiliary subsystems are simply unrequired since the lifetime management of processes and their scheduled tasks are already explicitly defined through nursery constructs 🏄🏼

As the most obvious example, a very simple set of premises provided by our SC actors enables a data race free shm management system without the need for any "special" resource tracking:

• a parent actor is spawned and allocates a shared memory buffer which it, and it alone is given write access
• the parent spawns (multiple) subactor(s) which are given read-only access to the shm

=> due to no child actor being able to outlive its parent, the configuration of processes and the shm buffer already implicitly implements a readers-writer lock; there is zero concern for data races during shm segment management since no child can be left reading after the resource has been unlinked. Further, no special tracking of the shm needs to be conducted as long as the parent/alocator de-allocates the segment on exit.

Outstanding bug-issues to be resolved for this implementation

There is a slew of related piker issues that drive the need to better formalize this API:

• OS X: Shm issue OSError: [Errno 63] File name too long: pikers/piker#306
• OSError: too many files open on linux pikers/piker#332
  • also see patch comment: https://github.com/pikers/piker/pull/302/files#diff-474c4c5fb6faaed9fb4e8fa397ea124d921a4bec1bbdcf185dd2d29f4af2f7a7R551
• FSP engine task re-syncs should not re-attach to shm pikers/piker#359
• fsp chart resyncs are global? pikers/piker#379

Motivating feature enhancements

• Faster y-range sorting pikers/piker#325
• All the tutorials: we need motivating (advanced) examples in our docs #175
• Designing a stream-composable, (soft) real-time ready, chain-time-type-validated FSP cascade system and DSL  pikers/piker#270
• Cross-actor, shm, (tick) ring buffer pikers/piker#107

Near-term ToDo list for first draft landing as a documented public API:

• first implementation factored from the piker project is in Shared memory array API and optional tight integration with numpy #338 and includes:

  • default, native support for the stdlib's ShareableList
  • necessary adjustments to name numpy support optional

• solving the shm allocation bugs in the first 2 bullets of the bug-issues section above:

  • likely we need to write a definitive example/test which demonstrates the buffer leaking / unlinking failures in the stdlib, example was flung in OSError: too many files open on linux pikers/piker#332

• a first advanced example would be nice to have that demonstrates the readers-write lock style mentioned in the first section here

  • potentially the example application could be a streaming search or sorting algo:
    • https://en.wikipedia.org/wiki/Exponential_search
    • https://github.com/lemire/pythonmaxmin/blob/master/maxmin.py#L18