Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds Java version 17 in Github CI build #1609

Merged

Conversation

DarshitChanpura
Copy link
Member

@DarshitChanpura DarshitChanpura commented Feb 11, 2022

Description

Adds support for Java 17 in Github CI build.

Issues Resolved

#1502

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Update:

There is a known Java bug mentioned in #1620 which should be addressed in a separate PR.

@DarshitChanpura DarshitChanpura requested a review from a team February 11, 2022 21:01
davidlago
davidlago previously approved these changes Feb 11, 2022
Copy link

@davidlago davidlago left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pending passing CI

peternied
peternied previously approved these changes Feb 11, 2022
Copy link
Member

@peternied peternied left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Is there any documentation that we need to update around support for java 17?

@codecov-commenter
Copy link

codecov-commenter commented Feb 11, 2022

Codecov Report

Merging #1609 (645be1c) into main (5b994af) will increase coverage by 0.03%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##               main    #1609      +/-   ##
============================================
+ Coverage     64.57%   64.60%   +0.03%     
- Complexity     3215     3216       +1     
============================================
  Files           247      247              
  Lines         17351    17352       +1     
  Branches       3082     3082              
============================================
+ Hits          11205    11211       +6     
+ Misses         4597     4594       -3     
+ Partials       1549     1547       -2     
Impacted Files Coverage Δ
...a/org/opensearch/security/tools/SecurityAdmin.java 47.59% <0.00%> (-0.14%) ⬇️
...ecurity/ssl/rest/SecuritySSLReloadCertsAction.java 85.10% <0.00%> (+0.32%) ⬆️
...earch/security/ssl/util/SSLConnectionTestUtil.java 95.45% <0.00%> (+2.27%) ⬆️
...urity/ssl/transport/SecuritySSLNettyTransport.java 73.40% <0.00%> (+4.25%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b994af...645be1c. Read the comment docs.

@DarshitChanpura
Copy link
Member Author

Thanks! Is there any documentation that we need to update around support for java 17?

We do not mention it specifically anywhere in the plugin code as far as java versions go....as it is inferred from OpenSearch's documentation... But perhaps we should?

@DarshitChanpura DarshitChanpura dismissed stale reviews from peternied and davidlago via 0e8c87b February 11, 2022 23:48
@davidlago
Copy link

Ok, let's keep working throuhg these failures as those are the breaking ones for upgrading to 17 once we support it in 2.0. Next up is https://bugs.openjdk.java.net/browse/JDK-8251547, it seems like some of the elliptic curves we support for JWT authentication are no longer supported in JDK17:

Removal of the following elliptic curves from the SunEC provider: secp112r1, secp112r2, secp128r1, secp128r2, secp160k1, secp160r1, secp160r2, secp192k1, secp192r1, secp224k1, secp224r1, secp256k1, sect113r1, sect113r2, sect131r1, sect131r2, sect163k1, sect163r1, sect163r2, sect193r1, sect193r2, sect233k1, sect233r1, sect239k1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, X9.62 c2tnb191v1, X9.62 c2tnb191v2, X9.62 c2tnb191v3, X9.62 c2tnb239v1, X9.62 c2tnb239v2, X9.62 c2tnb239v3, X9.62 c2tnb359v1, X9.62 c2tnb431r1, X9.62 prime192v2, X9.62 prime192v3, X9.62 prime239v1, X9.62 prime239v2, X9.62 prime239v3, brainpoolP256r1 brainpoolP320r1, brainpoolP384r1, brainpoolP512r1

@dblock is this something we should explicitly document (for example, add to 2.0 list of breaking changes in release notes etc) or is it assumed that if something does not work on JDK17, then it doesn't work with OpenSearch using JDK17? My gut says err on the side of documenting these things.

@dblock
Copy link
Member

dblock commented Feb 14, 2022

Yes, please open documentation issues or make changes as part of the PR.

@dblock
Copy link
Member

dblock commented Feb 14, 2022

Actually it looks like we're not on the same page re:1.3.0. We are trying to replace version 14 with an LTS version 11 for builds, and are testing the complete distribution with the bundled version 17. We are bundling 17 in 1.3.0!

While users can use any JDK from the list, does this introduce a backwards incompatible change to users in the complete distribution or the -min distribution? Will someone doing, for example, a rolling upgrade from 1.2.4, run into a problem?

@DarshitChanpura
Copy link
Member Author

DarshitChanpura commented Feb 14, 2022

Trying to figure out the exception from the CI run for Java 17 build.

Exception call stack from Base64Helper$DescriptorNameSetter
java.lang.NoClassDefFoundError: Could not initialize class org.opensearch.security.support.Base64Helper$DescriptorNameSetter
	at org.opensearch.security.support.Base64Helper$DescriptorReplacer.lambda$replace$1(Base64Helper.java:166)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at org.opensearch.security.support.Base64Helper$DescriptorReplacer.replace(Base64Helper.java:160)
	at org.opensearch.security.support.Base64Helper$SafeObjectOutputStream.writeClassDescriptor(Base64Helper.java:223)
	at java.base/java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1295)
	at java.base/java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1244)
	at java.base/java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1440)
	at java.base/java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1191)
	at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:354)
	at org.opensearch.security.support.Base64Helper.serializeObject(Base64Helper.java:242)
	at org.opensearch.security.transport.SecurityInterceptor.ensureCorrectHeaders(SecurityInterceptor.java:237)
	at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:201)
	at org.opensearch.security.OpenSearchSecurityPlugin$7$2.sendRequest(OpenSearchSecurityPlugin.java:661)
	at org.opensearch.transport.TransportService.sendRequest(TransportService.java:763)
	at org.opensearch.transport.TransportService.sendRequest(TransportService.java:707)
	at org.opensearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:254)
	at org.opensearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:148)
	at org.opensearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:65)
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:194)
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:234)
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154)
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192)
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169)
	at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:95)
	at org.opensearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:91)
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:193)
	at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:336)
	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:153)
	at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:647)
	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:91)
	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:244)
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:127)
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:109)
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:759)
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:170)
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:145)
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:110)
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:94)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:833)

@DarshitChanpura
Copy link
Member Author

Update:
Seems like an issue with initialization using Java 17

Stacktrace from error caused by initialization in Base64Helper$DescriptorNameSetter.getField method
Caused by: java.lang.RuntimeException: java.lang.reflect.InaccessibleObjectException: Unable to make field private java.lang.String java.io.ObjectStreamClass.name accessible: module java.base does not "opens java.io" to unnamed module @574caa3f
	at org.opensearch.security.support.Base64Helper$DescriptorNameSetter.getField(Base64Helper.java:149)
	at org.opensearch.security.support.Base64Helper$DescriptorNameSetter.<clinit>(Base64Helper.java:118)
	... 70 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private java.lang.String java.io.ObjectStreamClass.name accessible: module java.base does not "opens java.io" to unnamed module @574caa3f
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at org.opensearch.security.support.Base64Helper$DescriptorNameSetter.getFieldPrivileged(Base64Helper.java:126)
	at org.opensearch.security.support.Base64Helper$DescriptorNameSetter.lambda$getField$0(Base64Helper.java:143)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
	at org.opensearch.security.support.Base64Helper$DescriptorNameSetter.getField(Base64Helper.java:143)
	... 71 more
java.lang.NoClassDefFoundError: Could not initialize class org.opensearch.security.support.Base64Helper$DescriptorNameSetter
	at org.opensearch.security.support.Base64Helper$DescriptorReplacer.lambda$replace$1(Base64Helper.java:176)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)

@peternied
Copy link
Member

peternied commented Feb 14, 2022

I think to figure out what we need to work around or fix, we should print out the class names and field names that are attempting to be modified, I'm not sure ObjectStreamClass is the actual type with the problem, then we can figure out if there should be an alterative approach or we need to implement a work around for jdk17


More background on JDK visibility changes over time:

If you cannot obtain or deploy newer versions of tools and libraries, then there are two command-line options that enable you to grant access to specific internal APIs for older versions of tools and libraries:

--add-exports: If you have an older tool or library that needs to use an internal API that has been strongly encapsulated, then use the --add-exports runtime option. You can also use --add-exports at compile time to access the internal APIs.
--add-opens: If you have an older tool or library that needs to access non-public fields and methods of java.* APIs by reflection, then use the --add-opens option.

From https://docs.oracle.com/en/java/javase/17/migrate/migrating-jdk-8-later-jdk-releases.html

@@ -42,6 +43,7 @@
import org.opensearch.security.util.FakeRestRequest;
import com.google.common.io.BaseEncoding;

@Ignore
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include a string value to provide context as these tests need to be skipped. This seems like a major functionality regression - why do all the tests need to be blocked instead of some?

Suggested change
@Ignore
@Ignore("reasoning...")

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
… reverts changes made to use junit 5

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
@@ -50,6 +53,15 @@
new SecureRandom().nextBytes(secretKey);
}

/*
This test fails during Java 17 build due to a known bug: https://bugs.openjdk.java.net/browse/JDK-8251547
TODO: This method should be removed once a fix is implemented
Copy link
Member

@peternied peternied Feb 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than create a TODO can you create an issue that captures what is wrong and what should be done (If you know)?

Copy link
Member Author

@DarshitChanpura DarshitChanpura Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I already created an issue and connected it to this PR as well

#1620

@peternied peternied merged commit 77d8861 into opensearch-project:main Feb 17, 2022
@cliu123 cliu123 mentioned this pull request Mar 10, 2022
1 task
wuychn pushed a commit to ochprince/security that referenced this pull request Mar 16, 2023
* Adds Java version 17 in Github CI build
* Disables HTTPJwtAuthenticatorTest for Java 17 build using junit 4 and reverts changes made to use junit 5

Signed-off-by: Darshit Chanpura <dchanp@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants