Clone
 

scott mitchell <smitchel@akamai.com> in Netty

Backport header improvements from 5.0

Motivation:

The header class hierarchy and algorithm was improved on the master branch for versions 5.x. These improvments should be backported to the 4.1 baseline.

Modifications:

- cherry-pick the following commits from the master branch: 2374e17, 36b4157, 222d258

Result:

Header improvements in master branch are available in 4.1 branch.

  1. … 54 more files in changeset.
Backport header improvements from 5.0

Motivation:

The header class hierarchy and algorithm was improved on the master branch for versions 5.x. These improvments should be backported to the 4.1 baseline.

Modifications:

- cherry-pick the following commits from the master branch: 2374e17, 36b4157, 222d258

Result:

Header improvements in master branch are available in 4.1 branch.

  1. … 54 more files in changeset.
Netty Headers Class Restructure and Algorithm Updates

Motivation:

Headers within netty do not cleanly share a common class hierarchy. As a result some header types support some operations

and don't support others. The consolidation of the class hierarchy will allow for maintenance and scalability for new codec.

The existing hierarchy also has a few short comings such as it is not clear when data conversions are happening. This

could result unintentionally getting back a collection or iterator where a conversion on each entry must happen.

The current headers algorithm also prepends all elements which means to find the first element or return a collection

in insertion order often requires a complete traversal followed by a collections.reverse call.

Modifications:

-Provide a generic base class which provides all the implementation for headers in netty

-Provide an extension to this class which allows for name type conversions to happen (to accommodate legacy CharSequence to String conversions)

-Update the headers interface to clarify when conversions will happen.

-Update the headers data structure so that appends are done to avoid unnecessary iteration or collection reversal.

Result:

-More unified class hierarchy for headers in netty

-Improved headers data structure and algorithms

-headers API more clearly identify when conversions are required.

  1. … 91 more files in changeset.
HTTP/2 Read Decompression Flow Control Fix

Motivation:

The current implementation of the HTTP/2 decompression does not integrate with flow control properly.

The decompression code is giving the post-decompression size to the flow control algorithm which

results in flow control errors at incorrect times.

Modifications:

-DecompressorHttp2FrameReader.java will need to change where it hooks into the HTTP/2 codec

-Enhance unit tests to test this condition

Result:

No more flow control errors because of decompression design flaw

HTTP/2 Unit Tests EventLoopGroup cleanup

Motivation:

The HTTP/2 unit tests are suffering from OOME on the master branch.

These unit tests allocating a large number of threads (~706 peak live) which may

be related to this memory pressure.

Modifications:

Each EventLoopGroup shutdown operation will have a `sync()` call.

Result:

Lower peek live thread count and less associated memory pressure.

HTTP/2 Unit Test Leak Fixes

Motivation:

The HTTP/2 tests do not always clean up ByteBuf resources reliably. There are issues with the refCnt, over allocating buffers, and potentially not waiting long enough to reclaim resources for stress tests.

Modifications:

Scrub the HTTP/2 unit tests for ByteBuf leaks.

Result:

Less leaks (hopefully none) in the HTTP/2 unit tests. No OOME from HTTP/2 unit tests.

Fix Native EPOLL Build Failure

Motiviation:

If sendmmsg is already defined then the native epoll module failed to build because of conflicting definitions.

The mmsghdr type was also redefined on systems that already supported this structure.

Modifications:

Provide a way so that systems which already define sendmmsg and mmsghdr can build

Provide a way so that systems which don't define sendmmsg and mmsghdr can build

Result:

The native EPOLL module can build in more environments

    • -6
    • +8
    /transport-native-epoll/src/main/c/io_netty_channel_epoll_Native.c
Fix Native EPOLL Build Failure

Motiviation:

If sendmmsg is already defined then the native epoll module failed to build because of conflicting definitions.

The mmsghdr type was also redefined on systems that already supported this structure.

Modifications:

Provide a way so that systems which already define sendmmsg and mmsghdr can build

Provide a way so that systems which don't define sendmmsg and mmsghdr can build

Result:

The native EPOLL module can build in more environments

    • -6
    • +8
    /transport-native-epoll/src/main/c/io_netty_channel_epoll_Native.c
Fix Native EPOLL Build Failure

Motiviation:

If sendmmsg is already defined then the native epoll module failed to build because of conflicting definitions.

The mmsghdr type was also redefined on systems that already supported this structure.

Modifications:

Provide a way so that systems which already define sendmmsg and mmsghdr can build

Provide a way so that systems which don't define sendmmsg and mmsghdr can build

Result:

The native EPOLL module can build in more environments

    • -6
    • +8
    /transport-native-epoll/src/main/c/io_netty_channel_epoll_Native.c
Correcting HttpDecoder SuppressWarnings parameter

Motiviation:

The HttpContentDecoder.getTargetContentEncoding has a SuppressWarnings(unused) on its parameter.

This should be SuppressWarnings(UnusedParameters).

Modifications:

SuppressWarnings(unused) -> SuppressWarnings(UnusedParameters)

Result:

Correctly suppressing warnings due to HttpContentDecoder.getTargetContentEncoding

HTTP/2 Cipher Suite Support

Motivation:

The HTTP/2 specification places restrictions on the cipher suites that can be used. There is no central place to pull the ciphers that are allowed by the specification, supported by different java versions, and recommended by the community.

Modifications:

-HTTP/2 will have a security utility class to define supported ciphers

-netty-handler will be modified to support filtering the supplied list of ciphers to the supported ciphers for the current SSLEngine

Result:

-Netty provides unified support for HTTP/2 cipher lists and ciphers can be pruned by currently supported ciphers

ALPN java implementation Motivation:

Netty only supports a java NPN implementation provided by npn-api and npn-boot.

There is no java implementation for ALPN.

ALPN is needed to be compliant with the HTTP/2 spec.

Modifications:

-SslContext and JdkSslContext to support ALPN

-JettyNpn* class restructure for NPN and ALPN common aspects

-Pull in alpn-api and alpn-boot optional dependencies for ALPN java implementation

Result:

-Netty provides access to a java implementation of APLN

  1. … 11 more files in changeset.
Exclude bin directory from git Motivation:

The 'bin/' directory is currently not in the .gitignore file.

Eclipse creates this directory to support IDE operations.

These directory is generated and not related to source code and therefore does not belong in git.

Modifications:

-Add 'bin/' to .gitignore

Result:

-No more 'bin/' directory confusion for eclipse users.

HTTP/2 draft 14 HTTP message flow Motivation:

HTTP/2 draft 14 came out a couple of weeks ago and we need to keep up

with the spec.

Modifications:

-Revert back to dispatching FullHttpMessage objects instead of individual HttpObjects

-Corrections to HttpObject comparitors to support test cases

-New test cases to support sending headers immediatley

-Bug fixes cleaned up to ensure the message flow is terminated properly

Result:

Netty HTTP/2 to HTTP/1.x translation layer will support the HTTP/2 draft message flow.

  1. … 8 more files in changeset.
HTTP/2 Priority Tree Restructure Motivation:

The current interface and implementation for HTTP/2 priority tree events does not notify listeners of all parent change events. As a result operations which depend upon knowing about parent change events may be missing events and result in stale data. This interface also allows new listeners to easily consume priority tree change events.

Modifications:

-Http2Connection.Listener interface will change to support notifications on every node after the priority has changed and tree events have settled

-This will affect the outbound flow controller, DefaultHttp2Connection, and other listeners using the old interface

Result:

A modified (hopefully simplified and correct) Listener interface to get priority tree change event notification

Add HTTP/2 codec to netty-all maven dependency Motivation:

The HTTP/2 codec is missing from the netty-all maven dependency.

This means third party libraries have to pull in additional dependencies

above netty-all to use this codec.

Modifications:

Add HTTP/2 codec to netty-all

Result:

Third party libraries don't need to pull in additional dependencies

to use HTTP/2 codec.

Introduced HTTP/2 frame to HTTP/1.x translation layer.

Motivation:

The HTTP/2 codec currently provides direct callbacks to access stream events/data. The HTTP/2 codec provides the protocol support for HTTP/2 but it does not pass messages up the context pipeline. It would be nice to have a decoder which could collect the data framed by HTTP/2 and translate this into traditional HTTP type objects. This would allow the traditional Netty context pipeline to be used to separate processing concerns (i.e. HttpContentDecompressor). It would also be good to have a layer which can translate FullHttp[Request|Response] objects into HTTP/2 frame outbound events.

Modifications:

Introduce a new InboundHttp2ToHttpAdapter and supporting classes which will translate HTTP/2 stream events/data into HttpObject objects. Introduce a new DelegatingHttp2HttpConnectionHandler which will translate FullHttp[Request|Response] objects to HTTP/2 frame events.

Result:

Introduced HTTP/2 frame events to HttpObject layer.

Introduced FullHttp[Request|Response] to HTTP/2 frame events.

Introduced new unit tests to support new code.

Updated HTTP/2 client example to use new code.

Miscelaneous updates and bug fixes made to support new code.

  1. … 8 more files in changeset.
The temporary variable is no longer used in an attempt to accumulate data over onDataRead method class