• On Windows, automatically enable ICU if AC_CHECK_HEADER([icu.h]) finds it. To support building in older environments, don't build ICU support if the header is not found. To support running on older Windows versions, dynamically load icu.dll with LoadLibraryW/GetProcAddress. This means no linking flags are needed.

Just to be clear, it sounds from this that the minimum version of Windows required to run the official distributions of Racket available from download.racket-lang.org would not increase? (Currently that minimum version is given as Windows 7, although for 8.8 and 8.9 it was given as Windows 10. There was some discussion in #4542 about what versions of Windows are actually supported.)

• On Windows, automatically enable ICU if AC_CHECK_HEADER([icu.h]) finds it. To support building in older environments, don't build ICU support if the header is not found. To support running on older Windows versions, dynamically load icu.dll with LoadLibraryW/GetProcAddress. This means no linking flags are needed.

Just to be clear, it sounds from this that the minimum version of Windows required to run the official distributions of Racket available from download.racket-lang.org would not increase? (Currently that minimum version is given as Windows 7, although for 8.8 and 8.9 it was given as Windows 10. There was some discussion in #4542 about what versions of Windows are actually supported.)

That's my intention. We should definitely check that it works in practice, though!

If we went this route with configure:

But native Windows builds do not use Autoconf, and I'm not sure how to do the equivalent of AC_CHECK_HEADER([icu.h]). (I cannot overstate the extent to which I am not a Windows person—perhaps ironically, I have only tested this so far on GNU/Linux.) The simplest thing to do would probably be to omit the check and require an explicit --disable-icu (/disableICU?) to build in old Windows environments.

Then building Racket in old environments would require a configuration change, but running it should not.

I don't think we need to support compile-time configuration of ICU for Windows. It should just be always on in DLL mode and used as a fallback when iconv isn't available. (It looks like things are in place where we could avoid #includeing icu.h on Windows even if it's available, but I'm not sure I'm reading things right.) So, --enable-icu-dll could go away and winfig.bat wouldn't need to change. Maybe one day we'll switch the default from iconv to ICU.

An --enable-icu for other platforms makes sense, and the various configure improvements there look good.

I don't think we need to support compile-time configuration of ICU for Windows. … So, --enable-icu-dll could go away and winfig.bat wouldn't need to change. Maybe one day we'll switch the default from iconv to ICU.

That would certainly make things easier.

(It looks like things are in place where we could avoid #includeing icu.h on Windows even if it's available, but I'm not sure I'm reading things right.)

I initially thought this wouldn't work, but then I ended up getting halfway there anyway, and now I think it probably would work, so I will try filling in the rest and see how it goes.

I think the biggest thing that had seemed problematic to me originally were comments pointing to https://unicode-org.atlassian.net/browse/ICU-12420 like the one deprecating U_STANDARD_ERROR_LIMIT because The numeric value may change over time. Now, though, I think that issue is actually specific to constants that were meant to mark the end of open-ended lists. The numeric values of the actual error codes we need to test, including for U_SUCCESS and U_FAILURE, do seem to be covered by ICU's ABI stability.

I had also seen that there is some complicated logic allowing for alternate definitions of UChar. I haven't fully understood it, but it seems like it is for niche situations that wouldn't apply here.

@LiberalArtist Did you decide that this wasn't worthwhile, or just lose interest, or just run out of time?

I remain interested in this improvement for both Racket and Chez Scheme — though admittedly not enough to try anything myself, so far.

I just ran out of time and haven't gotten back to it. I still think it's a viable approach. It is on my to-do stack to get back to, but that won't be for the next month or so, at least. If someone else wants to fill things in before then, so much the better!

I started looking at this again and found I had actually done but not committed much of the work.

Looking at the CI errors https://github.com/racket/racket/actions/runs/16229453902/job/45828973905?pr=4844#step:8:79:

Errors were [4]:
(Section (got expected (call)))
((portlib) ((#"\357\277\275\357\277\275") Error (port->bytes (reencode-input-port (open-input-bytes #"\377\377") "utf-8"))))
((portlib) ((not-an-error) Error (let () (port->bytes (reencode-input-port (open-input-bytes #"\377\377") "utf-8")) (quote not-an-error))))
((portlib) ((#<void>) Error (let ((o (reencode-output-port (open-output-bytes) "utf-8"))) (display #"\377\377" o) (flush-output o))))
((portlib) ((not-an-error) Error (let () (let ((o (reencode-output-port (open-output-bytes) "utf-8"))) (display #"\377\377" o) (flush-output o)) (quote not-an-error))))

Are these cases that were expected to be errors but now work? I'm having trouble locating the tests.

Those tests are here https://github.com/racket/racket/blob/master/pkgs/racket-test-core/tests/racket/portlib.rktl

I found the file, I just haven't found the specific tests. For example, grepping that file for not-an-error doesn't find any results.

That in particular is from the expansion of err/rt-test. My guess is the relevant lines are around here: https://github.com/racket/racket/blob/master/pkgs/racket-test-core/tests/racket/portlib.rktl#L964

Some notes on my investigation so far:

As @samth suggested, the failing tests are specifically:

The relevant parts boil down to this expected behavior, i.e. catching invalid encoding:

> (bytes-convert (bytes-open-converter "UTF-8" "UTF-8") #"\xFF\xFF")
#""
0
'error

(Note that #"\377\377" is the way the printer writes #"\xFF\xFF".)

One surprise is that this goes through the iconv/ICU path at all: (bytes-open-converter "UTF-8" "UTF-8") is one of the guaranteed combinations, even when no iconv/ICU library is available, so there is built-in support for this somewhere. Does the implementation switch to use always use the foreign library if present, even for guaranteed combinations? Something to investigate further.

Nonetheless, I think we do want to understand what's going on in these test failures before merging.

I noticed that:

> (bytes-convert (bytes-open-converter "UTF-8-permissive" "UTF-8") #"\xFF\xFF")
#"\357\277\275\357\277\275"
2
'complete

matching the unexpected result in the first test case.

So it seems to me that we are currently calling ICU in permissive mode, and need to figure out how to not do that.

Apparently ICU's default behavior is to emit a substitution character for an invalid sequence, but setting callbacks can override this. I've pushed a first attempt.

That took care of half of the test failures. What remains:

Errors were [2]:
(Section (got expected (call)))
((portlib) ((#<void>) Error (let ((o (reencode-output-port (open-output-bytes) "utf-8"))) (display #"\377\377" o) (flush-output o))))
((portlib) ((not-an-error) Error (let () (let ((o (reencode-output-port (open-output-bytes) "utf-8"))) (display #"\377\377" o) (flush-output o)) (quote not-an-error))))

Also, we should double-check that I resolved the merge conflicts from the new parallel threads correctly.

One surprise is that this goes through the iconv/ICU path at all: (bytes-open-converter "UTF-8" "UTF-8") is one of the guaranteed combinations, even when no iconv/ICU library is available, so there is built-in support for this somewhere. Does the implementation switch to use always use the foreign library if present, even for guaranteed combinations? Something to investigate further.

Apparently the matching for the guaranteed combinations is case-sensitive, which is presumably why the tests use "utf-8":

While doing a build on Debian for debugging, I found that there are some uses of encodings called UCS-4LE or UCS-4BE. It looks like they are defined in the same place as the implementation of _string/ucs-4, but not actually used by it, at least on CS. The specific places are the ones changed in LiberalArtist@bbd48c7.

At some point in the process of building Racket, something tries to use bytes-open-converter to convert between UTF-8 and UCS-4{L,B}E (the one corresponding to the native endianness). This is a problem, because ICU doesn't recognize either of those as encoding names.

In more detail, these are the related encodings recognized by ICU vs. GNU libiconv built with --enable-extra-encodings. Names listed on the same line are aliases, and matching is case-insensitive (and flexible in other details I don't recall off-hand):

;; ICU
UTF-32 ISO-10646-UCS-4 ibm-1236 ibm-1237 csUCS4 ucs-4 
UTF-32BE UTF32_BigEndian ibm-1232 ibm-1233 ibm-9424 
UTF-32LE UTF32_LittleEndian ibm-1234 ibm-1235 
UTF32_PlatformEndian 
UTF32_OppositeEndian
;; GNU libiconv w/ --enable-extra-encodings
ISO-10646-UCS-4 UCS-4 CSUCS4
UCS-4BE
UCS-4LE
UTF-32
UTF-32BE
UTF-32LE
UCS-4-INTERNAL
UCS-4-SWAPPED

The Unicode Standard, Version 17.0.0, Appendix C§2, “Encoding Forms in ISO/IEC 10646” says:

ISO/IEC 10646:2011 has significantly revised its discussion of encoding forms, compared to earlier editions of that standard. The terminology for encoding forms (and encoding schemes) in 10646 now matches exactly the terminology used in the Unicode Standard. Furthermore, 10646 is now described in terms of a codespace U+0000..U+10FFFF, instead of a 31-bit codespace, as in earlier editions. This convergence in codespace description has eliminated any discrepancies in possible interpretation of the numeric values greater than 0x10FFFF. As a result, this section now merely notes a few items of mostly historic interest regarding encoding forms and terminology.

UCS-4. UCS-4 stands for “Universal Character Set coded in 4 octets.” It is now treated simply as a synonym for UTF-32, and is considered the canonical form for representation of characters in 10646.

Furthermore, neither the IANA Character Sets Registry nor the WHATWG Encoding Standard recognizes endianness-specific variants of UCS-4. (WHATWG doesn't recognize any variant of UCS-4 as a name.) In brief searching, I can see a few, perhaps older, pieces of software that mention the endianness-specific variants, but only a few, and (while I haven't read it) I'm guessing they may not actually have been defined by ISO 10646.

The only reason I can guess for GNU libiconv to treat UTF-32 and UCS-4 as distinct encodings, rather than aliases, is that it might support, or once have supported, older versions of ISO 10646 in which only a subset of UTF-32 was valid UCS-4, and it could have raised errors if anything out of that subset was used.

My conclusion is that we ought to just change to UTF-32LE and UTF-32BE. That's what I've done in LiberalArtist@bbd48c7 (part of this PR, likely to be squashed eventually). I haven't done it yet, but I also think we should update the docs for _string/ucs-4 and _string/utf-16, neither of which say anything about endianness or the lack of a BOM.

The existence of this code reinforces my thinking from LiberalArtist@bbd48c7 that we ought to support all of the Unicode variants without relying on an iconv implementation. But I'm not intending to look at that until after ICU support is done, because I don't want to mask any errors.

These equivalent expressions:

(bytes-convert (bytes-open-converter "utf-8" "UTF-8") #"\xFF\xFF")
(bytes-convert (bytes-open-converter "UTF-8" "utf-8") #"\xFF\xFF")

with ICU both evaluate to:

#""
1
'error

but with iconv evaluate to:

#""
0
'error

I used DrRacket's step debugger in an --enable-icu build on the failing test:

(let ([o (reencode-output-port (open-output-bytes) "utf-8")]) 
  (display #"\xFF\xFF" o)
  (flush-output o))

and found that it got to this code:

got-c => 0
used-c => 1
status => error

so, at line 2022, (zero? used-c) is #f, and the block labeled No progress before an encoding error. never executes.

The test reaches line 2022 twice, then gets to this block of code:

There doesn't seem to be any path in the code that leads to an exception when there persistently is progress before an encoding error.

I'm not sure why the ICU and iconv backends are returning different second results from (bytes-convert (bytes-open-converter "utf-8" "UTF-8") #"\xFF\xFF"). It's very possible that there could be something wrong in my C code, but I can also imagine that the iconv API may not make a portable guarantee about this. I will investigate further, though I'd be glad for any suggestions.

Note that using capitalized "UTF-8" in both cases does in fact get Racket's built-in implementation, so this happens even with --enable-icu:
``

(bytes-convert (bytes-open-converter "UTF-8" "UTF-8") #"\xFF\xFF")
#""
0
'error

After ICU support is in place, this further suggests to me that we should support all of the Unicode variants without relying on a foreign library.

Re parallel threads, as long as this still holds:

The docs for ucnv_convertEx() say that, on output:

*source points to the first unit after the last char read

That is less than clear about whether "the last char read" is a char read from the erroneous character, or if it's the last char successfully read. However, an article by JeanHeyd Meneide (the editor of the ISO C standard) says:

An even better version of this updates the pointer only if the operation was successful, rather than just doing it willy-nilly. This is the way ICU works, and it is incredibly helpful when *pErrorCode is not set to U_SUCCESS. It allows you to insert replacement characters and/or skip over the problematic input if needed, allowing for greater control over how errors are handled.

But my understanding is that the byte #"\xFF" cannot appear anywhere in UTF-8, so I don't see how it can possibly have been successfully read.

For another case, consider (bytes-convert (bytes-open-converter "UTF-8" "utf-8") #"abc\xF0\x9F\xFF"). That byte string comes from:

(bytes-append #"abc"
              (let ([bs (string->bytes/utf-8 "🎜")])
                (subbytes bs 0 (/ (bytes-length bs) 2)))
              #"\xFF")

i.e., chopping off the second half of #\🎜 in UTF-8 and replacing it with the invalid #"\xFF".
The results we get are:

;; ICU
#"abc"
5
'error
;; iconv
#"abc"
3
'error

The iconv backend counts only the three bytes that were successfully converted, whereas the ICU backend counts the 5 bytes that could be part of valid UTF-8, but not, in this case, the bad #"\xFF" (why?).

It looks like the lower-level functions ucnv_fromUnicode() and ucnv_toUnicode() provide enough granular offset information that this should be implementable, but I really thought ucnv_convertEx was supposed to have this capability.

I've put together a little C program to verbosely explore the various input/output pointers and results: https://gist.github.com/LiberalArtist/3f32e8c0b4d9a7db2a23a0cafe444d12

I found some comments about the state during error callbacks and where the various pointers are pointing:
https://github.com/unicode-org/icu/blob/4974e9b262366fe5b16eb10293c35172d55ce2a9/icu4c/source/common/unicode/ucnv_cb.h#L29-L59
I think our problem will be solvable by supplying custom callbacks.

We now have a different test failure:

((portlib) ((EXN #(struct:exn:fail:contract "user port peek: contract violation\n  expected: \n   (or/c exact-nonnegative-integer? eof-object? evt? pipe-input-port? #f procedure?)\n  received: #<void>" #<continuation-mark-set>)) #f (#<procedure:regexp-match?> #px#"temple" #<input-port:string>)))

The test in question is here:

Status Update

I put some time into this last month, inspired by RacketCon, with the thought that 9.0 would have been an auspicious time for the switch, but there is more to do. Aside from tracking down the test failure in #4844 (comment), there is a bigger-picture issue I've been thinking about.

ICU's converters are stateful. In particular, they use a pivot buffer containing an internal representation (usually UTF-16 for historical reasons, but IIUC there are fast paths for UTF-8) when converting from a source encoding to a target encoding. This part of the documentation is relevant:

The converters always consume the source buffer as far as possible, and advance the source pointer.

The converters write to the target all converted output as far as possible, and then write any remaining output to the internal services buffer. When the conversion routines are called again, the internal buffer is flushed out and written to the target buffer before proceeding with any further conversion.

Thus, from ICU's perspective, there may exist bytes that have been consumed from the source but not yet written to the target.

Initially, I thought this would be trivial: after all, in the general case, correctly using the byte converter API requires using bytes-convert-end to potentially get more output after all input has been consumed.

On reflection, though, the qualifier “in the general case” seems significant. The bytes-convert-end docs say:

instead of converting bytes, this procedure generates an ending sequence for the conversion (sometimes called a “shift sequence”), if any. Few encodings use shift sequences, so this function will succeed with no output for most encodings. In any case, successful output of a (possibly empty) shift sequence resets the converter to its initial state.

By implication, a user who knows that the specific encodings in some application do not use shift sequences would seem justified in never calling bytes-convert-end. In other words, bytes-convert-end is not allowed to emit anything other than a (possibly empty) shift sequence.

On further investigation, the requirement seems to follow from the POSIX iconv specification, which says:

For state-dependent encodings, the conversion descriptor cd is placed into its initial shift state by a call for which inbuf is a null pointer, or for which inbuf points to a null pointer. When iconv() is called in this way, and if outbuf is not a null pointer or a pointer to a null pointer, and outbytesleft points to a positive value, iconv() shall place, into the output buffer, the byte sequence to change the output buffer to its initial shift state.

This part of the POSIX spec explicitly requires that input be counted as consumed only when corresponding output has been produced, and it seems to ban any state other than shift sequences in conversion descriptors:

If the output buffer is not large enough to hold the entire converted input, conversion shall stop just prior to the input bytes that would cause the output buffer to overflow. The variable pointed to by inbuf shall be updated to point to the byte following the last byte successfully used in the conversion. The value pointed to by inbytesleft shall be decremented to reflect the number of bytes still not converted in the input buffer. … For state-dependent encodings, the conversion descriptor shall be updated to reflect the shift state in effect at the end of the last successfully converted byte sequence.

Reluctantly, I am coming to the conclusion that we shouldn't use bytes-convert-end to flush an ICU converter's pivot buffer. While hypothetically we could change the specification of the Racket API, and existing uses that correctly handle the general case would continue to be correct, the whole point of this exercise in supporting niche non-Unicode encodings is backward compatibility.

It definitely is still possible to use ICU to implement the iconv API.

In the worst-case scenario, we could use preflighting mode, or possibly character mode.

Ideally, though, we would want to stay within buffered or streamed mode. I'm still trying to figure out how to do that.

This mailing list thread is one part I want to better understand. The documentation for ucnv_fromUCountPending() and ucnv_toUCountPending() says they are “useful for mapping semantics of ICU's converter interface to those of iconv.”

A lingering concern, though, is that some of the suggestions seem to work like this: Imagine an identity conversion, UTF-8 to UTF-8, with abcdefg as the initial input, an empty two-byte target buffer, and an empty two-byte pivot buffer. After the first call, ab will be in the target buffer, cd will be in the pivot buffer, and, from ICU's perspective, efg will not have been consumed. An adapter layer could report instead, per iconv semantics, that only ab were consumed. Then, the client would make a second call with cdefg as the input and, let's say, another empty two-byte target buffer. The adapter layer would offset the input by the content of the pivot buffer, calling ICU with just efg as input and cd in the pivot. The result would have cd in the output buffer, ef in the pivot buffer, and, from ICU's perspective, just g not having been consumed.

The problem I see with that approach is that, in the second call, there is a requirement that the input must start with cd to match the content of the pivot buffer. Most of the time that will obviously be true, especially in reasonable, practical use-cases. But I don't think the iconv semantics require that to be true. Instead, a client could supply vwxyz for the second call and end up with cd wrongfully emitted, vw discarded completely, and xy in the pivot buffer.

Another approach could use ucnv_getMaxCharSize() and ucnv_getMinCharSize() to calculate a safe buffer size for batch processing, perhaps in combination with ucnv_safeClone() to do limited preflighting or character-by-character conversion only when needed. This might have better performance than using preflighting mode or character mode all the time. The downside would be a potentially more complex implementation in rktio.