Request throttling¶

Quick overview¶

Limit session throughput.

• advanced.throttler in the configuration; defaults to pass-through (no throttling), also available: concurrency-based (max simultaneous requests), rate-based (max requests per time unit), or write your own.

• metrics: throttling.delay, throttling.queue-size, throttling.errors.

Throttling allows you to limit how many requests a session can execute concurrently. This is useful if you have multiple applications connecting to the same Cassandra cluster, and want to enforce some kind of SLA to ensure fair resource allocation.

The request throttler tracks the level of utilization of the session, and lets requests proceed as long as it is under a predefined threshold. When that threshold is exceeded, requests are enqueued and will be allowed to proceed when utilization goes back to normal.

From a user’s perspective, this process is mostly transparent: any time spent in the queue is included in the session.execute() or session.executeAsync() call. Similarly, the request timeout encompasses throttling: it starts ticking before the request is passed to the throttler; in other words, a request may time out while it is still in the throttler’s queue, before the driver has even tried to send it to a node.

The only visible effect is that a request may fail with a RequestThrottlingException, if the throttler has determined that it can neither allow the request to proceed now, nor enqueue it; this indicates that your session is overloaded. How you react to that is specific to your application; typically, you could display an error asking the end user to retry later.

Note that the following requests are also affected by throttling:

• preparing a statement (either directly, or indirectly when the driver reprepares on other nodes, or when a node comes back up – see how the driver prepares);

• fetching the next page of a result set (which happens in the background when you iterate the synchronous variant ResultSet).

• fetching a query trace.

Configuration¶

Request throttling is parameterized in the configuration under advanced.throttler. There are various implementations, detailed in the following sections:

datastax-java-driver {
  advanced.throttler {
    class = PassThroughRequestThrottler
  }
}

datastax-java-driver {
  advanced.throttler {
    class = ConcurrencyLimitingRequestThrottler
    
    # Note: the values below are for illustration purposes only, not prescriptive
    max-concurrent-requests = 10000
    max-queue-size = 100000
  }
}

datastax-java-driver {
  advanced.throttler {
    class = RateLimitingRequestThrottler
    
    # Note: the values below are for illustration purposes only, not prescriptive
    max-requests-per-second = 5000
    max-queue-size = 50000
    drain-interval = 1 millisecond
  }
}

Monitoring¶

Enable the following metrics to monitor how the throttler is performing:

datastax-java-driver {
  advanced.metrics.session.enabled = [
    # How long requests are being throttled (exposed as a Timer).
    #
    # This is the time between the start of the session.execute() call, and the moment when the
    # throttler allows the request to proceed.
    throttling.delay,
    
    # The size of the throttling queue (exposed as a Gauge<Integer>).
    #
    # This is the number of requests that the throttler is currently delaying in order to
    # preserve its SLA. This metric only works with the built-in concurrency- and rate-based
    # throttlers; in other cases, it will always be 0.
    throttling.queue-size,
    
    # The number of times a request was rejected with a RequestThrottlingException (exposed as a
    # Counter)
    throttling.errors,
  ]
}

If you enable throttling.delay, make sure to also check the associated extra options to correctly size the underlying histograms (metrics.session.throttling.delay.*).