Netty: Memory Exhaustion via HTTP/3 Reserved Frame Types

QFinch assessment

Netty's Http3FrameCodec buffers incoming data for HTTP/3 reserved frame types up to the specified payload length without any limits. The payload length is read directly from the wire and trusted without validation. A bad actor can send a reserved frame with a payload length of up to Integer.MAXVALUE, causing the server to buffer the data in memory. This leads to an OOM and a gradual Denial of Service due to memory exhaustion as multiple streams are opened.

Potential impact

Denial of Service due to gradual memory exhaustion. Any application using Netty's HTTP/3 codec is impacted.

Technical context

io.netty.handler.codec.http3.Http3FrameCodecdecodeFrame handles reserved frame types as follows: The payLoadLength is read directly from the wire and trusted implicitly. Since payLoadLength can be up to Integer.MAXVALUE and there is no maximum payload length enforcement for reserved frames, the decoder will accumulate bytes in memory until the wire-provided length is reached. This allows a bad actor to exhaust server memory by opening multiple QUIC streams and sending reserved frames with large payload lengths, followed by a small amount of data (e.g., up to the defined limit) on each stream.