Support Java 25 runtime compatibility#424
Conversation
|
This sounds promising! On which platform, and with which exact command did you run the regression tests? |
|
I ran this on my Mac laptop. Are there additional tests other than the CI tests Codex ran? Happy to run those if you point me in the right direction...
|
777dcbc to
4e8bac7
Compare
|
@ohumbel I expanded the scope of the PR. I ran the following commands:
Let me know if there is anything else I can do to help. |
@robertpatrick When running The same error both on Mac and Ubuntu, using Java 8: |
4e8bac7 to
a4d3939
Compare
|
@ohumbel Sorry, I don't use Open JDK. PR updated, please try again. |
|
@robertpatrick This all looks relevant and helpful. Thanks for tracking down so many apparent version-dependent changes. The additions to I know you've contributed before, but I can't trace a record of signing the CLA. It may be just that when you did, it wasn't correlated with a GitHub id. Or it may be we weren't all that strict. This is really too many things in one PR. To help us know which changes are related, as reviewers or when someone does the archaeology later to understand a change they find problematic, could we divide this up please? Two of them @wfouche has cooking as PRs for anyway, as you identify, so if we can bank those we need not include them here. |
@robertpatrick If you have a look at ant-regrtest.yml, you can see that the pipelines also use the Eclipse Temurin distributions. That's the reason I test with those. I have a positive and negative feedback for you. I consistently get Thats a good sign for me.
|
Done
Well... It is hard to know if a PR is useful if you cannot actually build and test it with the stated goal.
And how exactly would you suggest we do that? Send a bunch of related PRs, none of which actually achieve the stated goal? How exactly do you vet those PRs individually?
Again, I need Jython support for JDK 25 and so I achieved this and met all of the goals that have been stated. I did give credit to those two PRs but if they were ready, why haven't you merged them already? I seem to recall that you had some feedback on one of them that they did not achieve the objective the way that you wanted. |
a4d3939 to
43bc8e8
Compare
|
@ohumbel Updated the PR to address the issue with the launchertest-built target.
Those two skips are from the existing Ant/regrtest resource configuration, not from this PR. Both They do not enable
I am happy to go do more work to make sure that all of your requirements for OS and JDK versions are met. However, I think you and @jeff5 need to decide what you want to do. I do not want to waste time. We need a working version of Jython that runs properly on JDK25, and we need it soon. I tried to help the process along with this PR but apparently, didn't do it the way @jeff5 wanted. So, what is the plan for getting JDK 25 support added in the near future if it is not this PR? Note that I can continue my work on this PR to verify OS/JDK support. However, if you two decide not to merge it after I satisfy the OS/JDK requirements, we will be forced to fork Jython, apply the PR, and move forward. As one of the largest consumers of Jython, that would be bad for Jython as a whole, and create extra work for us but we have dates to meet. |
|
@robertpatrick Being "only" a contributor like you, I just wanted to help verify the PR locally - time is always short on all sides. |
|
@robertpatrick |
Please, let's remind ourselves that everyone is working on this voluntarily. No need for "threats" like this - I really see no way how Jython 2.7.5 could be released without including this PR the one or other way. So, no worries - it will surely be accepted once remaining issues are solved. That said, if we merge @wfouche 's PRs first, the diff should just adopt and we can merge this PR on top, I suppose. (Perhaps it will require resolving a mild merge conflict if at all.) |
|
@ohumbel I will go download and install these non-standard JDK distributions and try to resolve the issue. |
|
CLA: thank-you.
I think the obvious approach is to address test failures (in |
|
@jeff5 I have no time or interest in dividing this up. Just so we are clear, I am doing this work because my employer needs these changes and we want to be good open-source citizens. I was not trying to make threats earlier, I was stating the facts that we have to ship this and have dates that we need to meet. I will continue to focus on testing and resolving issues in their non-standard JDKs and Linux/Windows. If you want to break it up, into separate PRs, be my guest. For me, this is not about credit, my ego, or any such thing. It is about getting the changes made as soon as possible to meet the dates my employer has. |
|
@ohumbel I cannot reproduce your issue using the same Eclipse Temurin JDK 1.8.0_492 on macOS Tahoe 26.4.1. Same result without the -D overrides for the JDBC jars. |
43bc8e8 to
0ba1cae
Compare
|
Tested on Oracle Linux 9.6. The only update that was required was adding a system property that allowed me to override the test_socket.py file's default, hard-coded unreachable host The test matrix run was:
I will go test on Windows next. |
|
|
||
| if (exception != null) { | ||
| logger.log(Level.WARNING, "Unable to format console messages: {0}", | ||
| logger.log(Level.FINE, "Unable to format console messages: {0}", |
There was a problem hiding this comment.
Is this relevant for a potential doctest failure? Or is it just silencing some noise?
There was a problem hiding this comment.
I think that in normal circumstances, a user or application manager might want to know that they aren't going to get properly formatted log messages. ISTR it emerges as "noise" in a test involvig a security manager, but I've never found it objectionable.
There was a problem hiding this comment.
I've been pondering this noise-silencing decision, too. IMHO noise silencing is not necessarily a bad thing. Ideally, the warning would pop up only in verbose mode. Would Level.FINE achieve this? Since this tiny detail has not the urgency of Java 25 compatibility, I would ideally like to see the discussion decoupled and in a separate issue. However, I'm personally fine with keeping the change for now as-is. We can open a separate issue for this and assess if it can be made somehow verbose-mode only and adjust it in a new PR.
There was a problem hiding this comment.
Supporting Java 25 does not motivate a change from WARNING to FINE. By all means take the discussion elsewhere. I have learned not to change behaviour without a positive reason.
There was a problem hiding this comment.
@jeff5 Well...the fact is that this logger serves two purposes so Codex was simply trying to clean up unnecessary noise in the test. We could either split these two purposes so that each logs a distinct message at the right level or just restore it to WARNING. At the end of the day, I could care less what we do here.
However, let's not pretend that there isn't technical debt in the code...
There was a problem hiding this comment.
This is a good observation on this bit of code, but has nothing to do with Java 25. I am content with this noise.
|
@robertpatrick Then I am sorry to suffer under the "it's not working on my machine" syndrome. The pipelines decide. Therefore I won't bother you with failing Ubuntu and Windows, both behind corporate firewall and proxy. |
No worries, I am also behind a corporate firewall and proxy. I was able to get it to pass in that environment by exporting the JAVA_OPTS environment variable with the necessary proxy -Ds. |
7887892 to
9af02e1
Compare
|
For the record, as always, when binary files occur in a PR I routinely verify their authenticity. I hereby confirm that the provided binary files in this PR @robertpatrick Thanks for addressing all of my remarks thoroughly! |
jeff5
left a comment
There was a problem hiding this comment.
@robertpatrick A lot of relly good stuff here as commented earlier.
The changes made to permit testing in a proxied environment are a worthy objective in their own right, independent of the Java 25 compatibility. They're not things I know how to test but you're obviously a sophisticated user. (We'll credit you not Codex.)
The only blocker I think is that the PR must not modify lib-python. An alternative exists.
Other comments are either appreciative or a question.
|
|
||
| if (exception != null) { | ||
| logger.log(Level.WARNING, "Unable to format console messages: {0}", | ||
| logger.log(Level.FINE, "Unable to format console messages: {0}", |
There was a problem hiding this comment.
I think that in normal circumstances, a user or application manager might want to know that they aren't going to get properly formatted log messages. ISTR it emerges as "noise" in a test involvig a security manager, but I've never found it objectionable.
77419e5 to
6532e8e
Compare
|
@jeff5 @Stewori I just updated the commit with the changes made for Windows build and test (except for Issue #426) and moved changes out of
|
6532e8e to
dd3c22d
Compare
|
I just pushed what I hope to be the last cleanup changes to the PR. One of the things in this commit is trying to clean up race conditions between the socket tests, which were mostly relying on implicit closing of sockets bound to ports that are reused across mutliple tests, primarily in test_socket.py. Note that my Windows VM did not see these errors but my MBP M4 Pro did. I do not believe that this PR introduced these race conditions and I am hopeful we have cleaned at least some of them up. However, race conditions are difficult to detect so I honestly cannot say that the PR fixes every one of them. |
|
I configured a matrix to build with java 8 and then run the ci tests with java versions up to 25. If there are changes to PR #424, I can manually trigger the pipelines again. Update 2026-05-19: The link is now working |
@jeff5 I think the problem is that this line is doing two different jobs at the moment. It reports both:
The first is mostly a formatting degradation and can be noisy under a security manager; the second is more operationally significant because -v logging may not behave as intended. My suggestion is to split those cases instead of using one final message for both. I’d log the formatter-property failure at FINE or CONFIG, but keep a WARNING when we fail to attach/configure the console handler. That preserves the useful signal for users/application managers without making the security-manager test noise drive the level for both cases. |
|
I think that's a good analysis. Also, the change is not of the essence for Java 25. |
There was a typo in the link - sorry for that. |
jeff5
left a comment
There was a problem hiding this comment.
Having studied this carefully, I now understand it in three main parts:
- Bringing JARs up to date.
- Compatibility with Java 25: mainly native access for Java 25, but includes
test_float_jy. - Changes to network support to permit testing behind a proxy.
These are all useful. There are other minor tweaks in the PR too, but I have commented where I think these are misconceived or unnecessary. I'm open to correction on those comments.
I would like to split the PR into those parts so that contributors coming after may more easily associate a change to its intent. They pretty much split by file, although build.xml has elements of them all.
As you've said @robertpatrick that you don't have the time for that reorganisation, I'll do it and aim to credit you in the commits. If some end up with my name on, it's my lack of git foo, not intentional.
|
|
||
| if (exception != null) { | ||
| logger.log(Level.WARNING, "Unable to format console messages: {0}", | ||
| logger.log(Level.FINE, "Unable to format console messages: {0}", |
There was a problem hiding this comment.
This is a good observation on this bit of code, but has nothing to do with Java 25. I am content with this noise.
There was a problem hiding this comment.
Did you encounter a problem with line-endings? This test runs fine for me on Windows. The test material has LF line-endings and we use autocrlf=input.
There was a problem hiding this comment.
@jeff5 things are not as clear cut as relevant to JDK 25 or not. Many tests don’t clean up after themselves and create race conditions. Since the ONLY way to validate changes are the tests, I had no choice but to fix issues causing test failures. Since many of the test changes were addressing race conditions in the tests, by definition, slower machines may not observe these test failures while faster machines will. Do whatever you want but it is naive to think that just because you don’t observe an issue yourself that the issue doesn’t exist.
I spent many hours (and hundreds of millions of tokens) creating and testing the changes in the PR across Mac, Linux, and Windows. How many other contributors do this?
There was a problem hiding this comment.
The particular comment is about whether this test encounters CRLF line endings, which it is hard to see as a concurrency, a processor speed or a clean-up issue. If you see such line endings in the files themselves, they were most likely added by git during checkout.
Taking your comment as general to the review, yes, some tests can be flakey, and none of them are designed to run concurrently: many use the same file name as a temporary for example. I also understand that you would want to make the test suite run in your proxied environment before working on the Java 25 specific fixes.
There was a problem hiding this comment.
The race conditions have nothing to do with running tests concurrently. They are all about proper resource cleanup (e.g., ports, mailboxes, etc.) when one test completes so that the next test being run can reuse the same resource without encountering unexpected conditions (e.g., port already in use, mailbox already exists, etc.).
One common pattern I found was that the tests were relying on JVM garbage collection to clean up these external resources. That approach is not reliable and creates a race condition on whether or not GC clears the resource prior to the next test trying to use it.
Do you have an ETA when the changes will be merged?
There was a problem hiding this comment.
One common pattern I found was that the tests were relying on JVM garbage collection to clean up these external resources. That approach is not reliable and creates a race condition on whether or not GC clears the resource prior to the next test trying to use it.
Yes, we've fixed tests ourselves that (rather) rely on CPython garbage collection, which is immediate, of course. Guidance says you shouldn't rely on that but authors are not reminded by failure.
It's reasonable to think I can get it ready to merge over the weekend.
There was a problem hiding this comment.
This test runs fine for me., but I see now that I have just been lucky with backslashes and Python 2 is tolerant about them.
So it's a bug found, but it has nothing to do with Java 25, so thank you Codex and I shall extract this one to a separate change set. I have a smaller fix anyway.
Is this Codex being an enthusiastic novice? Noticing unrelated bugs and being unable to resist the temptation to fix them.
| self._delete_recursively(foo_path) | ||
| self._delete_recursively(bar_path) |
There was a problem hiding this comment.
Only these two lines are new. They don't exist in the latest upstream test. (Other improvements do.)
Again, this seems to be Codex making improvements unrelated to the objective. The original runs fine for me. Did you really have trouble? If not, I'd revert the shadowing file.
There was a problem hiding this comment.
Yes, more race conditions
| try: | ||
| thelen = str(os.path.getsize(body.name)) | ||
| except (AttributeError, OSError, TypeError): |
There was a problem hiding this comment.
These 3 lines are the reason for creating a shadow copy of the file. But why are they here? CPython 2 does not find it necessary to add this. (Things are changed a lot in Python 3, so I can't tell.)
I would revert if there is no compelling reason.
There was a problem hiding this comment.
Whether or not some random, CPython version from 20 years ago felt the need to add this, I was seeing test failures on at least one platform due to this. Again, like caused by race conditions. If the tests don’t pass on modern machines, how do you plan to validate the changes?
There was a problem hiding this comment.
It turns out the root cause is that os.fstat (line 1099) fails on later Java versions unless --add-opens=java.base/java.io=ALL-UNNAMED is given. But since other parts of the patch add that, I think we're safe. I'll be able to test that.
| def tearDown(self): | ||
| # setUp installs a process-global opener with proxies disabled so these | ||
| # localhost tests ignore any external proxy environment. Restore the | ||
| # default opener afterwards: on Windows matrix runs the next urllib2 | ||
| # network tests rely on the preserved proxy environment to reach | ||
| # external HTTP resources. | ||
| urllib2.install_opener(None) | ||
| super(TestUrlopen, self).tearDown() |
There was a problem hiding this comment.
So the motivation is the proxied environment, even though this test does not use external network resources, because use of install_opener in setUp() messes up subsequent tests that do. And the change entails the shadow copy. :(
I think CPython is treating the same problem here: https://github.com/python/cpython/blob/8d21aa21f2cbc6d50aab3f420bb23be1d081dac4/Lib/test/test_urllib2_localnet.py#L470 . Might this be a better solution?
I think I would like longer term to update the lib-python version from CPython and remove the shadow. However, it has a lot of other differences: useful differences but with unknown impact.
|
Just noting here that asm JARs are already at version 9.10.1 on Maven Central so I'll do an independent PR for that, instead of part 1 of this. |
Fixes #413
Summary
This PR updates Jython for runtime compatibility with JDK 25 while keeping the release built with the lowest supported JDK target. The release artifacts were built with JDK 8 and then tested on JDK 8, 11, 17, 21, and 25 from the same build output. No multi-release JAR was needed.
This builds on the already-merged jffi update from #409 and incorporates/adapts fixes from:
stdout.encoding/sun.stdout.encodinghandling on newer JDKsChanges
--enable-native-access=ALL-UNNAMED--sun-misc-unsafe-memory-access=allowjava.base/java.iojava.base/sun.nio.chEnable-Native-Access: ALL-UNNAMEDAdd-Opens: java.base/java.io java.base/sun.nio.chjython.test.java.optsso CI/test runs can pass Java proxy/system properties without editing build files.jython.test.socket.unreachableHostsystem property to override the test_socket unreachable-host timeout target while preserving the existing192.0.2.42default.test-builtAnt target that runs the tests without calling developer-buildmodjytest failures propagate as process failures.test_mailbox.test_cleanrobust against stale deterministic tmp/foo and tmp/bar files.profile.propertiesintodist/javaliband adjust the bundled JIP profiler properties so the Java 8/11--profilelauncher tests produce output without noisy agent stack traces.Proxy-sensitive network tests
When an external HTTP proxy is configured, this PR skips only the network cases that cannot work through an HTTP proxy:
test_urllib2netexternal FTP test is skipped when an external HTTP proxy is configured because it requires direct FTP access. Neither FTP-over-HTTP-proxy support nor Java FTP proxy property support is implemented in this PR.test_ssl_jyraw external SSL socket tests are skipped because they open raw sockets and cannot use HTTP CONNECT.test_smtpnetraw external SMTP SSL tests are skipped because they require direct SMTP-over-SSL socket access.Higher-level external HTTPS tests were adjusted to use CONNECT where appropriate. Localhost/direct tests continue to run directly.
The relevant generic configuration surfaces for proxied test environments are:
jython.test.java.optssystem property:http.proxyHosthttp.proxyPorthttp.nonProxyHostshttps.proxyHosthttps.proxyPorthttps.nonProxyHosts.Validation
Release artifacts were built with JDK 8 using
ant clean full-build:dist/jython.jardist/jython-standalone.jardist/jython-installer.jarThe resulting artifacts still target JDK 8 bytecode/release metadata.
The regression matrix was run with:
ant testant test-builtant test-builtant test-builtant test-builtResults:
The two top-level skipped tests are existing resource-gated tests:
test_codecmaps_hk, requiresurlfetchtest_curses, requirescursesThey are not JDK 25 failures and are not proxy-related.
Additional focused checks used during development included JDK 25 command-line behavior, float behavior, imports, Java integration, subprocess/threading behavior, HTTP/URL handling, warning suppression,
test_mailboxcleanup, and JDK 21/JDK 25 socket/SSL behavior.