Scylla Documentation Logo Documentation
  • Server
    • Scylla Open Source
    • Scylla Enterprise
    • Scylla Alternator
  • Cloud
    • Scylla Cloud
    • Scylla Cloud Docs
  • Tools
    • Scylla Manager
    • Scylla Monitoring Stack
    • Scylla Operator
  • Drivers
    • CQL Drivers
    • DynamoDB Drivers
Download
Menu
Scylla Java Driver Manual Developer docs Netty pipeline

Netty pipeline¶

With the protocol layer in place, the next step is to build the logic for a single server connection.

We use Netty for network I/O (to learn more about Netty, this book is an excellent resource).

   +----------------+
   | ChannelFactory |
   +----------------+
   | connect()      |
   +-------+--------+
           |                              Application
           |creates     +----------------------------------------------+
           V            | Outgoing                                     |
   +-------+--------+   |  |         +---------------------+        ^  |
   | DriverChannel  |   |  |         | ProtocolInitHandler |        |  |
   +-------+--------+   |  |         +---------------------+        |  |
           |            |  |                                        |  |
   +-------+--------+   |  |         +---------------------+        |  |
   |    Channel     |   |  |         |   InFlightHandler   |        |  |
   |    (Netty)     |   |  |         +---------------------+        |  |
   +-------+--------+   |  |                                        |  |
           |            |  |         +---------------------+        |  |
   +-------+--------+   |  |         |   Heartbeathandler  |        |  |
   |ChannelPipeline +---+  |         +---------------------+        |  |
   |    (Netty)     |   |  |                                        |  |
   +----------------+   |  |  +--------------+   +--------------+   |  |
                        |  |  | FrameEncoder |   | FrameDecoder |   |  |
                        |  |  +--------------+   +--------------+   |  |
                        |  |                                        |  |
                        |  |         +---------------------+        |  |
                        |  |         |      SslHandler     |        |  |
                        |  |         |       (Netty)       |        |  |
                        |  V         +---------------------+        |  |
                        |                                     Incoming |
                        +----------------------------------------------+
                                             Network                   

Each Cassandra connection is based on a Netty Channel. We wrap it into our own DriverChannel, that exposes higher-level operations. ChannelFactory is the entry point for other driver components; it handles protocol negotiation for the first channel.

A Netty channel has a pipeline, that contains a sequence of handlers. As a request is sent, it goes through the pipeline top to bottom; each successive handler processes the input, and passes the result to the next handler. Incoming responses go the other way.

Our pipeline is configured with the following handlers:

SslHandler¶

The implementation is provided by Netty (all the others handlers are custom implementations).

Internally, handler instances are provided by SslHandlerFactory. At the user-facing level, this is abstracted behind SslEngineFactory, based on Java’s default SSL implementation.

See also the Extension points section below.

FrameEncoder and FrameDecoder¶

This is where we integrate the protocol layer, as explained here.

Unlike the other pipeline stages, we use separate handlers for incoming and outgoing messages.

HeartbeatHandler¶

The heartbeat is a background request sent on inactive connections (no reads since x seconds), to make sure that they are still alive, and prevent them from being dropped by a firewall. This is similar to TCP_KeepAlive, but we provide an application-side alternative because users don’t always have full control over their network configuration.

HeartbeatHandler is based on Netty’s built-in IdleStateHandler, so there’s not much in there apart from the details of the control request.

InFlightHandler¶

This handler is where most of the connection logic resides. It is responsible for:

  • writing regular requests:

    • find an available stream id;

    • store the ResponseCallback provided by the client under that id;

    • when the response comes in, retrieve the callback and complete it;

  • cancelling a request;

  • switching the connection to a new keyspace (if a USE statement was executed through the session);

  • handling shutdown: gracefully (allow all request to complete), or forcefully (error out all requests).

The two most important methods are:

  • write(ChannelHandlerContext, Object, ChannelPromise): processes outgoing messages. We accept different types of messages, because cancellation and shutdown also use that path. See DriverChannel, which abstracts those details.

  • channelRead: processes incoming responses.

Netty handlers are confined to the channel’s event loop (a.k.a I/O thread). Therefore the code doesn’t have to be concurrent, fields can be non-volatile and methods are guaranteed not to race with each other.

In particular, a big difference from driver 3 is that stream ids are assigned within the event loop, instead of from client code before writing to the channel (see also connection pooling). StreamIdGenerator is not thread-safe.

All communication between the handler and the outside world must be done through messages or channel events. There are 3 exceptions to this rule: getAvailableIds, getInflight and getOrphanIds, which are based on volatile fields. They are all used for metrics, and getAvailableIds is also used to balance the load over connections to the same node (see ChannelSet).

ProtocolInitHandler¶

This handler manages the protocol initialization sequence on a newly established connection (see the STARTUP message in the protocol specification).

Most of the logic resides in InitRequest.onResponse, which acts as a simple state machine based on the last request sent.

There is also a bit of custom code to ensure that the channel is not made available to clients before the protocol is ready. This is abstracted in the parent class ConnectInitHandler.

Once the initialization is complete, ProtocolInitHandler removes itself from the pipeline.

Extension points¶

NettyOptions¶

The advanced.netty section in the configuration exposes a few high-level options.

For more elaborate customizations, you can extend the context to plug in a custom NettyOptions implementation. This allows you to do things such as:

  • reusing existing event loops;

  • using Netty’s native Epoll transport;

  • adding custom handlers to the pipeline.

SslHandlerFactory¶

The user-facing API (advanced.ssl-engine-factory in the configuration, or SessionBuilder.withSslContext / SessionBuilder.withSslEngineFactory) only supports Java’s default SSL implementation.

The driver can also work with Netty’s native integration with OpenSSL or boringssl. This requires a bit of custom development against the internal API:

  • add a dependency to one of the netty-tcnative artifacts, following these instructions;

  • implement SslHandlerFactory. Typically:

    • the constructor will create a Netty SslContext with SslContextBuilder.forClient, and store it in a field;

    • newSslHandler will delegate to one of the SslContext.newHandler methods;

  • extend the context and override buildSslHandlerFactory to plug your custom implementation.

PREVIOUS
Native protocol layer
NEXT
Request execution
  • 4.13.0.x
    • 4.13.0.x
    • 4.12.0.x
    • 4.11.1.x
    • 4.10.0.x
    • 4.7.2.x
    • 3.11.2.x
    • 3.11.0.x
    • 3.10.2.x
    • 3.7.2.x
  • Java Driver for Scylla and Apache Cassandra®
  • API Documentation
  • Manual
    • API conventions
    • Case sensitivity
    • Core driver
      • Address resolution
      • Asynchronous programming
      • Authentication
      • Bill of Materials (BOM)
      • Compression
      • Configuration
        • Reference configuration
      • Control connection
      • Custom codecs
      • Detachable types
      • Using the driver in GraalVM native images
      • Query idempotence
      • Integration
      • Load balancing
      • Logging
      • Metadata
        • Node metadata
        • Schema metadata
        • Token metadata
      • Metrics
      • Native protocol
      • Non-blocking programming
      • Paging
      • Performance
      • Connection pooling
      • Query timestamps
      • Reactive Style Programming
      • Reconnection
      • Request tracker
      • Retries
      • Using the shaded JAR
      • Speculative query execution
      • SSL
      • Statements
        • Batch statements
        • Per-query keyspace
        • Prepared statements
        • Simple statements
      • Temporal types
      • Request throttling
      • Query tracing
      • Tuples
      • User-defined types
    • Developer docs
      • Administrative tasks
      • Common infrastructure
        • Concurrency
        • Driver context
        • Event bus
      • Native protocol layer
      • Netty pipeline
      • Request execution
    • Mapper
      • Integration
        • Kotlin
        • Lombok
        • Java 14 Records
        • Scala
      • DAOs
        • Custom result types
        • Delete methods
        • GetEntity methods
        • Increment methods
        • Insert methods
        • Null saving strategy
        • Query methods
        • Query provider methods
        • Select methods
        • SetEntity methods
        • Statement attributes
        • Update methods
      • Entities
      • Mapper interface
    • OSGi
    • Query builder
      • Conditions
      • DELETE
      • Idempotence in the query builder
      • INSERT
      • Relations
      • Schema builder
        • Aggregate
        • Function
        • Index
        • Keyspace
        • Materialized View
        • Table
        • Type
      • SELECT
      • Terms
      • TRUNCATE
      • UPDATE
  • Upgrade guide
  • Frequently asked questions
  • Changelog
  • Create an issue
  • Edit this page

On this page

  • Netty pipeline
    • SslHandler
    • FrameEncoder and FrameDecoder
    • HeartbeatHandler
    • InFlightHandler
    • ProtocolInitHandler
    • Extension points
      • NettyOptions
      • SslHandlerFactory
Logo
Docs Contact Us About Us
Mail List Icon Slack Icon
© 2022, ScyllaDB. All rights reserved.
Last updated on 25 May 2022.
Powered by Sphinx 4.3.2 & ScyllaDB Theme 1.2.2