ZMQ::FFI for IPerl

zaki
Wed, 14 Sep 2022 20:06:25 +0000
code, iperl, jupyter, perl programming language, vcpkg, zeromq

Like many FOSS hacking stories, it all started on IRC. Somebody in #perl mentioned that they were having issues with working with ZeroMQ and an event loop. I mentioned that I had a module Net::Async::ZMQ that could help. Turns out that they had been reading the code for that but were trying to make it work with ZMQ::FFI instead of ZMQ::LibZMQ3 and ZMQ::LibZMQ4 which it was originally designed for.

So I added support for ZMQ::FFI. But for my CI build to pass, I needed Windows support and ZMQ::FFI did not support Windows yet.

I then did the next obvious thing which is patch ZMQ::FFI to work on Windows. In my first attempt, I found tests that were using fork(2), Sys::SigAction, and the ZeroMQ ipc:// transport (zmq_ipc(7)) which do not work on Windows, so I left those as is and just decided to have ZMQ::FFI look for .dll in addition to .so and .dylib files. That's not great; those tests should really pass before a release can be made.

As time went by, I went ahead and released Net-Async-ZMQ-0.002. No use in waiting for Windows to pass for ZMQ::FFI when it never did before.

Until one night I acquired a round tuit and went ahead and fixed all the ZMQ::FFI Windows compatibility issues. I also changed the ipc:// transport to inproc:// where possible, however, this changes the tests on Unix-likes. So I went with another attempt that refactored the tests based on OS capability. This and several CI build fixes got rolled into ZMQ-FFI-1.18.

While I was working on all this, I decided that I wanted to port Devel::IPerl to use ZMQ::FFI because it could be easier to install for users on Windows than ZMQ::LibZMQ3 which requires a compiler to link the XS with the ZeroMQ DLL. This is currently a pain point for Devel::IPerl that I work around by providing a script that sets up Alien::ZMQ::latest to install the ZeroMQ DLL and then points the ZMQ::LibZMQ3 install script to use that DLL for linking. Removing that script will make things less complicated overall.

Which is what I did in this PR. It did not take much, just replace functions such as zmq_init() with ZMQ::FFI->new and zmq_socket($zmq_ctx, $type) with $zmq_ctx->socket($type).

This is now available on CPAN as Devel-IPerl-0.012.

Appendix: List of issues and pull requests participated in along the way

Devel::IPerl

Port from ZMQ::LibZMQ3 to ZMQ::FFI by zmughal · Pull Request #113 · EntropyOrg/p5-Devel-IPerl · GitHub

ZMQ::FFI

Add MSWin32 support and cross-platform CI testing by zmughal · Pull Request #44 · zeromq/perlzmq · GitHub

Add Docker test using GitHub Actions by zmughal · Pull Request #45 · zeromq/perlzmq · GitHub

MSWin32 compatibility and test portability by zmughal · Pull Request #47 · zeromq/perlzmq · GitHub

Optional support for Alien::ZMQ::latest by zmughal · Pull Request #48 · zeromq/perlzmq · GitHub

Fix build needed for release by zmughal · Pull Request #49 · zeromq/perlzmq · GitHub

GHA: disable fail-fast by zmughal · Pull Request #50 · zeromq/perlzmq · GitHub

Alien::ZMQ::latest

Only use Alien::gmake for shared install by zmughal · Pull Request #7 · zmughal-CPAN/p5-Alien-ZMQ-latest · GitHub

Update CI to use GHA by zmughal · Pull Request #8 · zmughal-CPAN/p5-Alien-ZMQ-latest · GitHub

Add workflow for Strawberry Perl by zmughal · Pull Request #9 · zmughal-CPAN/p5-Alien-ZMQ-latest · GitHub

Try to get CMake working on Windows · Issue #10 · zmughal-CPAN/p5-Alien-ZMQ-latest · GitHub

Add option for choosing build configuration tool by zmughal · Pull Request #11 · zmughal-CPAN/p5-Alien-ZMQ-latest · GitHub

Set make for autoconf build by zmughal · Pull Request #12 · zmughal-CPAN/p5-Alien-ZMQ-latest · GitHub

Use Download::GitHub by zmughal · Pull Request #13 · zmughal-CPAN/p5-Alien-ZMQ-latest · GitHub

CMake build not working on new Windows Strawberry Perl install · Issue #14 · zmughal-CPAN/p5-Alien-ZMQ-latest · GitHub

Set CMAKE_MAKE_PROGRAM by zmughal · Pull Request #15 · zmughal-CPAN/p5-Alien-ZMQ-latest · GitHub

Net::Async::ZMQ

Look into supporting ZMQ::FFI backend · Issue #3 · zmughal-CPAN/p5-Net-Async-ZMQ · GitHub

Switch from Test::Requires to Test::Needs by zmughal · Pull Request #4 · zmughal-CPAN/p5-Net-Async-ZMQ · GitHub

Add support for ZMQ::FFI by zmughal · Pull Request #5 · zmughal-CPAN/p5-Net-Async-ZMQ · GitHub

Update CI, use GHA by zmughal · Pull Request #6 · zmughal-CPAN/p5-Net-Async-ZMQ · GitHub

Use ZMQ_POLLIN event by zmughal · Pull Request #7 · zmughal-CPAN/p5-Net-Async-ZMQ · GitHub

microsoft/vcpkg

(zeromq) Cherry pick patches to fix build issues by LilyWangLL · Pull Request #22681 · microsoft/vcpkg · GitHub

(zeromq) Download patch files for GitHub PRs by zmughal · Pull Request #23435 · microsoft/vcpkg · GitHub

zeromq build failure because of port file with invalid hash (inconsistent) · Issue #23461 · microsoft/vcpkg · GitHub

run-vcpkg GitHub Action

vcpkgGitURL seems to have no effect · Issue #135 · lukka/run-vcpkg · GitHub

GitHub (the website itself)

Patch URL from pull requests should use full index to maintain stable contents · Discussion #12531 · community · GitHub