Netty

Clone Tools
  • last updated a few seconds ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
KQueueEventLoop | EpollEventLoop may incorrectly update registration when FD is reused.

Motivation:

The current KQueueEventLoop implementation does not process concurrent domain socket channel registration/unregistration in the order they actual

happen since unregistration are delated by an event loop task scheduling. When a domain socket is closed, it's file descriptor might be reused

quickly and therefore trigger a new channel registration using the same descriptor.

Consequently the KQueueEventLoop#add(AbstractKQueueChannel) method will overwrite the current inactive channels having the same descriptor

and the delayed KQueueEventLoop#remove(AbstractKQueueChannel) will remove the active channel that replaced the inactive one.

As active channels are registered, events for this file descriptor won't be processed anymore and the channels will never be closed.

The same problem can also happen in EpollEventLoop. Beside this we also may never remove the AbstractEpollChannel from the internal map

when it is unregistered which will prevent it from be GC'ed

Modifications:

- Change logic of native KQueue and Epoll implementations to ensure we correctly handle the case of FD reuse

- Only try to update kevent / epoll if the Channel is still open (as otherwise it will be handled by kqueue / epoll itself)

- Correctly remove AbstractEpollChannel from internal map in all cases

- Make implementation of closeAll() consistent for Epoll and KQueueEventLoop

Result:

KQueue and Epoll native transports correctly handle FD reuse

Co-authored-by: Norman Maurer <norman_maurer@apple.com>

KQueueEventLoop won't unregister active channels reusing a file descriptor

Motivation:

The current KQueueEventLoop implementation does not process concurrent domain socket channel registration/unregistration in the order they actual

happen since unregistration are delated by an event loop task scheduling. When a domain socket is closed, it's file descriptor might be reused

quickly and therefore trigger a new channel registration using the same descriptor.

Consequently the KQueueEventLoop#add(AbstractKQueueChannel) method will overwrite the current inactive channels having the same descriptor

and the delayed KQueueEventLoop#remove(AbstractKQueueChannel) will remove the active channel that replaced the inactive one.

As active channels are registered, events for this file descriptor won't be processed anymore and the channels will never be closed.

Modifications:

Change the logic of KQueueEventLoop#remove(AbstractKQueueChannel) channels so it will check channels equality prior removal.

Result:

KQueueEventLoop won't remove anymore active channels reusing a file descriptor.

Always include classes from all native transports no matter on which platfrom netty-all is build (#9111)

Motivation:

While building netty-all we should always include all classes for native transports no matter if the native part can be build or not. This was it is easier to test locally with a installed snapshot of netty-all when the code that uses it does enable a specific native transport depending on if the native bits can be loaded or not.

Modifications:

Always include classes of native transports no matter on which platfrom we build. When a release is done we ensure we include the native bits by using the uber-staging profile.

Result:

Easier testing with netty-all snapshots.

Always include classes from all native transports no matter on which platfrom netty-all is build (#9111)

Motivation:

While building netty-all we should always include all classes for native transports no matter if the native part can be build or not. This was it is easier to test locally with a installed snapshot of netty-all when the code that uses it does enable a specific native transport depending on if the native bits can be loaded or not.

Modifications:

Always include classes of native transports no matter on which platfrom we build. When a release is done we ensure we include the native bits by using the uber-staging profile.

Result:

Easier testing with netty-all snapshots.

[maven-release-plugin] prepare for next development iteration

  1. … 24 more files in changeset.
[maven-release-plugin] prepare release netty-4.1.36.Final

  1. … 24 more files in changeset.
Always include classes from all native transports no matter on which platfrom netty-all is build

Motivation:

While building netty-all we should always include all classes for native transports no matter if the native part can be build or not. This was it is easier to test locally with a installed snapshot of netty-all when the code that uses it does enable a specific native transport depending on if the native bits can be loaded or not.

Modifications:

Always include classes of native transports no matter on which platfrom we build. When a release is done we ensure we include the native bits by using the uber-staging profile.

Result:

Easier testing with netty-all snapshots.

Implement Http2StreamChannel.bytesBeforeWritable() and bytesBeforeUnwritable()

Motivation:

Our Http2StreamChannel implementation did not implement bytesBeforeWritable() and bytesBeforeUnwritable(). We should implement these to make the channel implementation more feature complete.

Modifications:

- Implement both methods

- Update tests to cover the implementations.

Result:

Fixes https://github.com/netty/netty/issues/8148.

Introduce DynamicAddressConnectHandler which can be used to dynamically change remoteAddress / localAddress when a connect is issued (#8982)

Motivation:

Bootstrap allows you to set a localAddress for outbound TCP connections, either via the Bootstrap.localAddress(localAddress) or Bootstrap.connect(remoteAddress, localAddress) methods. This works well if you want to bind to just one IP address on an interface. Sometimes you want to bind to a specific address based on the resolved remote address which should be possible.

Modifications:

Add DynamicAddressConnectHandler and tests

Result:

Fixes https://github.com/netty/netty/issues/8940.

Introduce DynamicAddressConnectHandler which can be used to dynamically change remoteAddress / localAddress when a connect is issued (#8982)

Motivation:

Bootstrap allows you to set a localAddress for outbound TCP connections, either via the Bootstrap.localAddress(localAddress) or Bootstrap.connect(remoteAddress, localAddress) methods. This works well if you want to bind to just one IP address on an interface. Sometimes you want to bind to a specific address based on the resolved remote address which should be possible.

Modifications:

Add DynamicAddressConnectHandler and tests

Result:

Fixes https://github.com/netty/netty/issues/8940.

Motivation: (#9106)

Subclasses of `OpenSslKeyMaterial` implement `ReferenceCounted`. This means that a new object should have an initial refcount of 1. An `OpenSslPrivateKey.OpenSslPrivateKeyMaterial` object shares its refcount with the enclosing `OpenSslPrivateKey` object. This means the enclosing object's refcount must be incremented by 1 when an instance of `OpenSslPrivateKey.OpenSslPrivateKeyMaterial` is created. Otherwise, when the key material object is `release()`-ed, the refcount on the enclosing object will drop to 0 while it is still in use.

Modification:

- Increment the refcount in the constructor of `OpenSslPrivateKey.OpenSslPrivateKeyMaterial`

- Ensure we also always release the native certificates as well.

Result:

Refcount is now correct.

Invoke channelAcquired callback on first time channel acquire (#9093)

Motivation:

SimpleChannelPool provides ability to provide custom callbacks/handlers

on major events such as "channel acquired", "channel created" and

"channel released". In the current implementation, when a request to

acquire a channel is made for the first time, the internal channel pool

creates the channel lazily. This triggers the "channel created" callback

but does not invoke the "channel acquired" callback. This is contrary to

caller expectations who assumes that "channel acquired" will be invoked

at the end of every successful acquire call. It also leads to an

inconsistent API experience where the acquired callback is sometimes

invoked and sometimes it isn't depending on wheather the internal

mechanism is creating a new channel or re-using an existing one.

Modifications:

Invoke acquired callback consistenly even when creating a new channel

and modify the tests to support this behaviour

Result:

Consistent experience for the caller of acquire API. Every time they

call the API, the acquired callback will be invoked.

Http2MultiplexCodec.DefaultHttp2StreamChannel should handle ChannelConfig.isAutoClose() in a consistent way as AbstractChannel (#9108)

Motivation:

Http2MultiplexCodec.DefaultHttp2StreamChannel currently only act on ClosedChannelException exceptions when checking for isAutoClose(). We should widen the scope here to IOException to be more consistent with AbstractChannel.

Modifications:

Replace instanceof ClosedChannelException with instanceof IOException

Result:

More consistent handling of isAutoClose()

Http2MultiplexCodec.DefaultHttp2StreamChannel should handle ChannelConfig.isAutoClose() in a consistent way as AbstractChannel (#9108)

Motivation:

Http2MultiplexCodec.DefaultHttp2StreamChannel currently only act on ClosedChannelException exceptions when checking for isAutoClose(). We should widen the scope here to IOException to be more consistent with AbstractChannel.

Modifications:

Replace instanceof ClosedChannelException with instanceof IOException

Result:

More consistent handling of isAutoClose()

Adjust pom.xml to be able to build with graalvm (#9107)

Motivation:

When trying to use graalvm and build netty we currently fail because our build configuration is not compatible with it.

Modification:

- Skip plugins that are not supported when graal is used

- Correctly configure surefire plugin for graal so it not produces a NPE

Result:

We can build and test with graalvm.

Adjust pom.xml to be able to build with graalvm (#9107)

Motivation:

When trying to use graalvm and build netty we currently fail because our build configuration is not compatible with it.

Modification:

- Skip plugins that are not supported when graal is used

- Correctly configure surefire plugin for graal so it not produces a NPE

Result:

We can build and test with graalvm.

Split OpenSslPrivateKey and OpenSslPrivateKeyMaterial refcnt to prevent leaking certficate pointer

Split OpenSslPrivateKey and OpenSslPrivateKeyMaterial refcnt to prevent leaking certficate pointer

Add SVM metadata and minimal substitutions to build graalvm native image applications. (#8963)

Motivation:

GraalVM native images are a new way to deliver java applications. Netty is one of the most popular libraries however there are a few limitations that make it impossible to use with native images out of the box. Adding a few metadata (in specific modules will allow the compilation to success and produce working binaries)

Modification:

Added properties files in `META-INF` and substitutions classes (under `internal.svm`) will solve the compilation issues. The substitutions classes are not visible and do not have a public constructor so they are not visible to end users.

Result:

Fixes #8959

This fix is very conservative as it applies the minimum config required to build:

* pure netty servers

* vert.x applications

* grpc applications

The build is having trouble due to checkstyle which does not seem to be able to find the copyright notice on property files.

    • -0
    • +122
    /testsuite-native-image/pom.xml
Add SVM metadata and minimal substitutions to build graalvm native image applications. (#8963)

Motivation:

GraalVM native images are a new way to deliver java applications. Netty is one of the most popular libraries however there are a few limitations that make it impossible to use with native images out of the box. Adding a few metadata (in specific modules will allow the compilation to success and produce working binaries)

Modification:

Added properties files in `META-INF` and substitutions classes (under `internal.svm`) will solve the compilation issues. The substitutions classes are not visible and do not have a public constructor so they are not visible to end users.

Result:

Fixes #8959

This fix is very conservative as it applies the minimum config required to build:

* pure netty servers

* vert.x applications

* grpc applications

The build is having trouble due to checkstyle which does not seem to be able to find the copyright notice on property files.

    • -0
    • +122
    /testsuite-native-image/pom.xml
Add docker-compose file to compile / test with graalvm (#9072)

Motivation:

We should try to compile / test with graalvm as well.

Modifications:

Add docker-compose file for graalvm

Result:

Be able to also compile / test with graalvm

    • -0
    • +22
    /docker/docker-compose.centos-6.graalvm1.yaml
Add docker-compose file to compile / test with graalvm (#9072)

Motivation:

We should try to compile / test with graalvm as well.

Modifications:

Add docker-compose file for graalvm

Result:

Be able to also compile / test with graalvm

    • -0
    • +22
    /docker/docker-compose.centos-6.graalvm1.yaml
Fix flaky GlobalEventExecutorTest.* (#9074)

Motivation:

In GlobalEventExecutorTest we used Thread.sleep(...) which can produce flaky results (as seen on the CI). We should use another alternative during tests.

Modifications:

Replace Thread.sleep(...) with join()

Result:

No more flaky GlobalEventExecutor tests.

Fix flaky GlobalEventExecutorTest.* (#9074)

Motivation:

In GlobalEventExecutorTest we used Thread.sleep(...) which can produce flaky results (as seen on the CI). We should use another alternative during tests.

Modifications:

Replace Thread.sleep(...) with join()

Result:

No more flaky GlobalEventExecutor tests.

Update to latest java releases (#9101)

Motivation:

There were new releases of various Java versions.

Modifications:

Adjust used java versions of the latest releases and so use these on our CI

Result:

Use latest java versions on our CI.

    • -1
    • +1
    /docker/docker-compose.centos-6.111.yaml
    • -1
    • +1
    /docker/docker-compose.centos-6.112.yaml
    • -1
    • +1
    /docker/docker-compose.centos-6.18.yaml
    • -1
    • +1
    /docker/docker-compose.centos-6.openj9111.yaml
    • -1
    • +1
    /docker/docker-compose.centos-7.111.yaml
    • -1
    • +1
    /docker/docker-compose.centos-7.112.yaml
    • -1
    • +1
    /docker/docker-compose.centos-7.18.yaml
    • -1
    • +1
    /docker/docker-sync-compose.centos-6.18.yaml
Update to latest java releases (#9101)

Motivation:

There were new releases of various Java versions.

Modifications:

Adjust used java versions of the latest releases and so use these on our CI

Result:

Use latest java versions on our CI.

    • -1
    • +1
    /docker/docker-compose.centos-6.111.yaml
    • -1
    • +1
    /docker/docker-compose.centos-6.112.yaml
    • -1
    • +1
    /docker/docker-compose.centos-6.18.yaml
    • -1
    • +1
    /docker/docker-compose.centos-6.openj9111.yaml
    • -1
    • +1
    /docker/docker-compose.centos-7.111.yaml
    • -1
    • +1
    /docker/docker-compose.centos-7.112.yaml
    • -1
    • +1
    /docker/docker-compose.centos-7.18.yaml
    • -1
    • +1
    /docker/docker-sync-compose.centos-6.18.yaml
Throw SignatureException if OpenSslPrivateKeyMethod.* return null to prevent segfault (#9100)

Motivation:

While OpenSslPrivateKeyMethod.* should never return null we should still guard against it to prevent any possible segfault.

Modifications:

- Throw SignatureException if null is returned

- Add unit test

Result:

No segfault when user returns null.

Throw SignatureException if OpenSslPrivateKeyMethod.* return null to prevent segfault (#9100)

Motivation:

While OpenSslPrivateKeyMethod.* should never return null we should still guard against it to prevent any possible segfault.

Modifications:

- Throw SignatureException if null is returned

- Add unit test

Result:

No segfault when user returns null.

Http2ConnectionHandler to allow decoupling close(..) from GOAWAY graceful close (#9094)

Motivation:

Http2ConnectionHandler#close(..) always runs the GOAWAY and graceful close

logic. This coupling means that a user would have to override

Http2ConnectionHandler#close(..) to modify the behavior, and the

Http2FrameCodec and Http2MultiplexCodec are not extendable so you cannot

override at this layer. Ideally we can totally decouple the close(..) of the

transport and the GOAWAY graceful closure process completely, but to preserve

backwards compatibility we can add an opt-out option to decouple where the

application is responsible for sending a GOAWAY with error code equal to

NO_ERROR as described in https://tools.ietf.org/html/rfc7540#section-6.8 in

order to initiate graceful close.

Modifications:

- Http2ConnectionHandler supports an additional boolean constructor argument to

opt out of close(..) going through the graceful close path.

- Http2FrameCodecBuilder and Http2MultiplexCodec expose

gracefulShutdownTimeoutMillis but do not hook them up properly. Since these

are already exposed we should hook them up and make sure the timeout is applied

properly.

- Http2ConnectionHandler's goAway(..) method from Http2LifecycleManager should

initiate the graceful closure process after writing a GOAWAY frame if the error

code is NO_ERROR. This means that writing a Http2GoAwayFrame from

Http2FrameCodec will initiate graceful close.

Result:

Http2ConnectionHandler#close(..) can now be decoupled from the graceful close

process, and immediately close the underlying transport if desired.

Http2ConnectionHandler to allow decoupling close(..) from GOAWAY graceful close (#9094)

Motivation:

Http2ConnectionHandler#close(..) always runs the GOAWAY and graceful close

logic. This coupling means that a user would have to override

Http2ConnectionHandler#close(..) to modify the behavior, and the

Http2FrameCodec and Http2MultiplexCodec are not extendable so you cannot

override at this layer. Ideally we can totally decouple the close(..) of the

transport and the GOAWAY graceful closure process completely, but to preserve

backwards compatibility we can add an opt-out option to decouple where the

application is responsible for sending a GOAWAY with error code equal to

NO_ERROR as described in https://tools.ietf.org/html/rfc7540#section-6.8 in

order to initiate graceful close.

Modifications:

- Http2ConnectionHandler supports an additional boolean constructor argument to

opt out of close(..) going through the graceful close path.

- Http2FrameCodecBuilder and Http2MultiplexCodec expose

gracefulShutdownTimeoutMillis but do not hook them up properly. Since these

are already exposed we should hook them up and make sure the timeout is applied

properly.

- Http2ConnectionHandler's goAway(..) method from Http2LifecycleManager should

initiate the graceful closure process after writing a GOAWAY frame if the error

code is NO_ERROR. This means that writing a Http2GoAwayFrame from

Http2FrameCodec will initiate graceful close.

Result:

Http2ConnectionHandler#close(..) can now be decoupled from the graceful close

process, and immediately close the underlying transport if desired.