libcurl static linking instructions and/or pkg-config file #39
Comments
Reported-by: avih on github Ref: #39
|
@avih: Thanks for your post. I understand your pains with static linking. You summed up very well what issues can be expected and how to fix them. However, besides the informal Two suggestions that may help jumping through this:
I've also added the ngtcp2 crypto lib to the Some points are not specific to these binaries, so may have a better place in the main curl repo. Some of them are already there, e.g. As for what supports UTF-8, this is an open issue on Windows, and "it depends". It's not officially documented in curl. See also: https://github.com/curl/curl/wiki/libcurl-and-expected-string-encodings. OpenSSL recently got support for built-in Windows certs. I haven't looked into it, and it might not made it to a release yet. It will resolve this faster then waiting for Schannel to catch up with OpenSSL features. Having said that, the |
|
Thanks for the reply. While it would be great if all configurations would be supported and documented equally, my focus when filing this issue was the official build of curl and libcurl for windows, which is available via https://curl.se/windows/ . Am I correct to assume that, among others, this repo is used to build the official windows binary at https://curl.se/windows/ ? I presime this zip and the libs/binaries within it are supposed to be used practically and not only as a proof of concept, right? And linking statically with this libcurl is hard to automate in practice. I think that other than the CURL_STATICLIB which is perhaps generic (but still would be useful to mention at the zip), the rest are specific to this shipped configuration:
The latter (https doesn't work) is actually not trivial at all to realize (other than the fact that it just doesn't work), and to me is quite surprising. So I think that a readme specific to the official zip (or to the configuration from which it's built) could be useful. If you agree, I can try to write such readme file with the issues and solutions mentioned in my first post, but a bit nicer ;) Is this the correct repo for such readme? Where should be be placed? |
|
Sure, these builds are the "official" curl builds for Windows. We even link there from here. By all means you're welcome to create such document. My concern is keeping it up to date. It feels like a huge effort, over the already huge effort curl-for-win takes. But, if the document is useful, we can link to it from here, or from other pages. It may even fit curl's INSTALL/FAQ docs, which already mentions static linking. (There is also |
|
Ref: curl/curl#9944 |
Agreed.
The FAQ maybe, but as far as I can tell INSALL explicitly does not refer to using the curl binary distribution, and does refer to building curl from source - same as curl/curl#9944 What I had in mind is exactly these two parts:
|
|
(I've not built libcurl myself) does building libcurl generate a pkg-config file in general? It looks like it does - the main repo has Does it not get generated when building libcurl-for-windows using this repo? If it is generated, why is it not packaged at the zip? |
|
These builds use bare GNU Make and don't generate Alternate CMake and autotools builds do, though |
|
Regarding the CA: Perhaps this information would fit best inside curl somewhere, the FAQ maybe, or libcurl docs? It's a general issue when using llibcurl + any TLS backend that doesn't read the OS CA store. Maintaining a list of these TLS backends / versions would also be useful. |
Probably, but regardless, I do feel strongly that the zip should include some notes specific to this distribution. Like listing the build options, maybe build logs, and the two subjects I mentioned above. I can think of at least one doc which could be updated - https://curl.se/libcurl/c/CURLOPT_CAINFO.html . It wasn't obvious to me that it can be empty by default and so initially I tried placing This: Does not imply that it can be empty... (other than schannel or secure transport where it's unused). |
|
re 871a25b , as far as I can tell Also, why |
|
Because IMO the best indication of the necessary libs is the actual list of Including logs (or even links to logs) break reproducibility, making every build producing a different hash, so that's not workable sadly. So instead we link to the log from the download page. I agree that CA bootstrapping details would be nice to have on the |
|
I see. I think this means that no readme should be created/updated, and that there are two todo items outside the scope of this issue: improve CURLOPT_CAINFO docs (default can be empty, non-absolute paths search dirs, UTF-8), and maybe improve the CA bootstrap implementation/configuration so that https works out of the box for libcurl without additional setup compared to *nix? I this case, I'll just summarize the info here for posterify (and then you can close it and/or help me improve it, or both, or edit it yourself, or whatever). These notes refer to using the mingw libcurl as distributed at https://curl.se/windows/ as of version 7.86. This libcurl supports both OpenSSL and Schannel TLS backends, and defaults to OpenSSL - which needs a CA bundle file. However, a default CA bundle path is not configured at build-time, and so by default https will fail. Possible solutions:
It's recommended to link with libcurl dynamically - with Static linking is also possible, but less simple. Some notes:
Example (posix) LIBS="-lcurl -Wl,--start-group"
for f in ./lib/*.a; do
case $f in *curl*) continue; esac
f=${f##*/lib}; f=${f%.a}
LIBS="$LIBS -l$f"
done
printf %s\\n "$LIBS -Wl,--end-group" |
|
Few more, potentialy interesting points:
Applies to current default official builds. curl-for-win allows to toggle certain features (e.g. UCRT), toolchains and dependencies. UPDATE: Added |
Despite our "best effort" to keep these up-to-date, they often fall out of sync with reality. Also their order was not the one expected by the linker (`ld` specifically), and guaranteeing the correct ones is a hard task. Also, since earlier this year, curl-for-win switched to a unified distro package (a single .zip), making it straightforward to find out all the static lib dependencies simply by looking at the .a files distributed in the package. As for the correct order of them, either use llvm's lld, which isn't sensitive to it like binutils' ld, or list them between the options `-Wl,--start-group` and `-Wl,--end-group`. With this, we're re dropping these manual lists. Ref: #39
Thanks. except as noted below, I'll leave them as additions in your own post, because you know about them (log links, etc) more than me.
I've edited it. better? |
|
@avih: Yes, thank you! |
|
Thanks for helping me understand the constraints better. |
|
@avih: Can you describe how would you use a FWIW they look like this (copied from local BoringSSL + mbedTLS builds): autotools: |
Well, I've not linked with libcurl before so it's not something I do frequently, and my first attempt was while building txiki.js which uses CMake. I'd hope that the CMake recipe would be able to pick the libs automatically, but I did not try it in practice.
Yeah, I've seen the templates. I don't think the |
|
I understand. Well, yeah, these have many problems:
Some of these are fallouts from build tricks I need to use, some others are cmake/autotools "things", and some others are Anyhow, if you manage to settle on a well-working |
Not sure what this means, because I don't think I've seen .pc files used "manually". But I think it does bring up an important question: who is this libcurl for? The But a library, and tools related to it, are designed to be usable in some non-arbitrary environments. Judging by "mingw" at the zip name, is it intended to be used only by mingw compiler? or also by msvc? Either way, the rest below refers to mingw only, but msvc might need some addressing too. I think by far the most common environments for mingw are a posix-ish. That includes cross-compiling on *nix or cygwin, and MSYS2 (either as cross or native-ish via the mingw shortcut). However, there exist also less posix-y envs, like w64devkit. (I'm ignoring msys(1), as it's not used these days I think, and either way it's similar to MSYS2, and also based on (old) cygwin). The .pc file is then used by whatever build system the "client" project uses. I'd think that typically it can be (alphabetically/historically) autotoos, cmake, or meson. So I'd say that as a high level goal, we'd want these to be able to link/build using this .pc file in a posix-y envisonment. It's also important to keep in mind that sometimes these mingw envs already have their own libcurl recipe/binary (e.g. mingw curl as MSYS2 or cygwin or MXE package), however, it's still useful to be able to use the official library as well, but it could be tricky if a libcurl is already installed at this mingw env. I'd think the most important places where it should work would be relatively barebones mingw envs, without their own libcurl recipe, like the debian mingw-w64 package, possibly fedora's (didn't check), and maybe w64devkit (though it's unclear what would use the .pc file other than manual build). |
|
Your summary is on point. The target for these libs is indeed "limited" considering the almost infinite variations of lib formats, ABIs and other variations these can come on Windows. (I find this a quite sad toolchain landscape and not a single bit better than say 15 years ago.) In short, this might be useful for someone using mingw-w64 or llvm-mingw, either gcc or llvm/clang, who might want to link to a libcurl that isn't bound to any specific unixy environment, such as MSYS2, either dynamically, statically or via any other assumptions of these envs. Perfect to ship a standalone Windows .exe, also from a non-Windows env via a cross-build, if that's someone's thing. MSVC never was a goal for these builds. Nor was Cygwin. And MSYS2/friends, as you said, already offer their own curl packages, though those I've found not really suitable for shipping your own app binaries. But, yes, nowadays the main goal is to provide Having said that, the original aim of these builds was in fact to provide the static libs! Now it's the byproduct. [ I still have a somewhat idle open source project that expects these libs. For my own purposes I switched to |
Right, FWIW, the txiki.js project vendors few sub-projects (git submodules) which it builds and links statically with, and expects libcurl to exist someplace. It does not depend on further libraries. I was trying to create a semi reproducible mingw build env (in the sense of getting from zero to a working binary, not necessarily bit-identical), and I used a debian live dvd with the mingw toolchain installed from the distro packages after boot, and then needed only libcurl to complete the build deps, where the one from https://curl.se/windows/ was just what the doctor ordered ;) Even with the shared dll I consider it a good success, and to be honest, I did not expect it to work. I didn't even realize initially that it includes development files, as in the past I only used it for the So it can have real-world use cases, and also serves as a reference to $something. But yeah, OSS/posix build envs for windows are such a pain...
Maybe revive that with a "fat" |
|
I'm glad this could fit into your project, and it's something nice to hear. [ I find "mingw-w64" working rather well, and What's lacking on Windows in general is standards or conventions even. No ABI, filename, dirname, lib format, or anything has any standard. Especially with MSVC which doesn't even have proper C99 support (or by now it has?). Making a cross-vendor DLL is a pain (it used to be with mingw-w64 at least). Thus, everyone keeps guessing and nothing is reusable. Microsoft could change that, but why would they? ] Building for Windows is in effect like building for 4-8-16-64 distinct platforms, depending on settings, compiler vendor, version, make tool and whatnot, where the only thing in common is the Win32 API and the PE .exe format. Though "fat" static lib seems doable technically, it's a step back for reproducibility and transparency IMO. Which is a question of convention, because for DLL and EXE a huge binary is okay, but for static libs, it seems unusual. Maybe an opportunity to create such script and publish it as a Gist for those interested. |
I presume that by "mingw-w64' you refer to the toolchain sources here https://www.mingw-w64.org/ which most mingw setups use these days, and not to a specific set of binaries, right? (agreed) And that by "llvm-mingw" you refer to the project and a specific set of binaries here https://github.com/mstorsjo/llvm-mingw ? how do you use it? from an MSYS2 setup with the path modified to include the bin/ dir, and then cross compile using its various prefixes?
Agreed :) |
|
Yes, that's what I meant by mingw-w64 and I've found If you're interested, the toolchain setup happens in In fact this is the only toolchain that doesn't bump into the static winpthread + UCRT bug, so the only one able to create BoringSSL builds. llvm strip tool has some (niche) bugs, so for proper sripping the binuntils strip is necessary. You can ignore that if reproducibility across platforms isn't your goal. |
Actually, I think this mostly applies to static linking. Judging from my own anecdotical experience, linking with the shared dll passed all txiki.js tests, but linking statically (after working around the roadbumps) failed some tests. I didn't mention it here because I thought I'm still doing something incorrect with the linking. But then I saw https://www.msys2.org/docs/environments/#msvcrt-vs-ucrt which mentions:
I'm not familiar with the libcurl API, but I'm guessing it's not passing |
|
There are a couple of libcurl functions that can exchange Question what else may be affected, because the above quote cites |
Right, then I guess txiki.js either does not use them, or does not use them in tests, or by luck no issues were observed. Generally speaking though, it seems that a UCRT libcurl should not be used with msvcrt application (which is how I compiled txiki.js for now), because even if today it doesn't use such APIs, tomorrow it might.
I'd imagine any libc objects which are created with one crt and are manipulated/used at the other, like FILE* objects, malloc in one and free/realloc at the other, etc. |
|
Yeah, the sure way is not to mix CRTs. One thing for consideration is using UCRT instead of msvcrt. I might have missed stuff here, but to me it seemed that unless you need to target XP and especially older Windows versions, UCRT is fine. As for open-source tooling: mingw-w64 hopefully will make it easier to enable it and remaining bugs will also be ironed out, such as the static winphtread one. |
|
I agree that if one doesn't care about XP then UCRT is probably the way to go. I don't know how common are UCRT mingw setups, and I think at leastr until very recently the common mingw setups were for MSVCRT. So far I was using the debian mingw-w64 package, which as far as I can tell uses MSVCRT. As far as I can tell debian does not have UCRT mingw setup. I did not yet look into setting up other UCRT mingw setup |
|
The default is MSVCRT in the distros I'm aware of. Switching to UCRT isn't much straightforward though: https://stackoverflow.com/questions/57528555/how-do-i-build-against-the-ucrt-with-mingw-w64 The mentioned Presumably in a few years this will become smoother or the default even. |
Hi,
I'm trying to build txiki.js on debian 11 using the mingw distro toolchain, linked with https://curl.se/windows/latest.cgi?p=win64-mingw.zip (currently
curl-7.86.0_2-win64-mingw.zip), and couldn't quite figure out what I should link with.First, is there some README file at the zip or elsewhere which instructs how to use libcurl at the zip? If there is one, hopefully it answers at least some of the subjects below.
I first tried linking with
libcurl.dll.a, which worked, and the resulting binary requires the curl dll (as expected). So that's a success.I then wanted to instead link statically with
libcurl.a, and encountered several missing libcurl symbols. After some searches I realized I neededCURL_STATICLIBdefined, and I then used it successfuly.Then it ended up with many (non-libcurl) missing symbols, but couldn't find which libs it depends on. I tried this list
-lcurl -lz -lcrypt32 -lbcrypt -lwldap32 -lnghttp2 -lssh2 -lgsasl -lssl -lcrypto -lbrotlidec -lbrotlicommon -lzstd -lnghttp3 -lngtcp2which appears at this repo README for the "Default build with OpenSSL and Schannel", but it failed with undefined reference tongtcp2_crypto_openssl_configure_client_contextand few more.I added
-lngtcp2_crypto_opensslwhich resolved this, but now I had a missing reference tozlibVersionandinflateand few more, so I moved-lzat the list to the end:It then failed with undefined references to
__imp___sys_nerrand__imp___sys_errlist, so (after some more googling) I added-lucrtbaseto the list.This seem to resolve all the undefined references, however, it then failed due to
SHA1defined multiple times, first at the txtki.js sources, and then atlibcrypto.a(libcrypto.a(libcrypto-lib-sha1_one.obj):sha1_one.c:(.text+0x70): multiple definition of 'SHA1'; <the original definition location>).I'll need to think what to do about it, maybe rename it at the txiki.js sources.
Seeing that the question of static linking has been asked more than once, and that the generic FAQ doesn't say much about it (docs/FAQ "2.1 configure fails when using static libraries"), I think it could help to have some info specific to this zip distribution about static linking. Specifically:
libcurl.ainside this zip (shared is relatively simple).CFLAGS=-DCURL_STATICLIBor some such which is needed for static linking (not sure if this is windows only?).libcurl.awhich embeds all the other static dependencies (similar to how the dll is linked statically with its deps), which could solve many issues (deps list, symbols which the project might not be expecting, etc).httpsWILL fail because it defaults to OpenSSL but without a default ca bundle (embedded/at some path), unless one of the following happens (I think):curl_easy_setopt(curl_h, CURLOPT_CAINFO, "curl-ca-bundle.crt");and then put this file at the same dir as the binary (seems to work, but I couldn't find docs for the search dirs of CURLOPT_CAINFO if it's not an absolute path, and also not what to do about unicode paths - does it expect UTF-8?).curl-ca-bundle.crtis used bycurl.exeat the zip, so I thought it could be a default name for libcurl at the zip as well), and fallback to schannel if it's missing?The text was updated successfully, but these errors were encountered: