Caution
You're viewing documentation for a previous version of Scylla Java Driver. Switch to the latest stable version.
Limit session throughput.
advanced.throttler in the configuration; defaults to pass-through (no throttling), also
available: concurrency-based (max simultaneous requests), rate-based (max requests per time unit),
or write your own.
metrics: throttling.delay, throttling.queue-size, throttling.errors.
Throttling allows you to limit how many requests a session can execute concurrently. This is useful if you have multiple applications connecting to the same Cassandra cluster, and want to enforce some kind of SLA to ensure fair resource allocation.
The request throttler tracks the level of utilization of the session, and lets requests proceed as long as it is under a predefined threshold. When that threshold is exceeded, requests are enqueued and will be allowed to proceed when utilization goes back to normal.
From a user’s perspective, this process is mostly transparent: any time spent in the queue is
included in the session.execute() or session.executeAsync() call. Similarly, the request timeout
encompasses throttling: it starts ticking before the request is passed to the throttler; in other
words, a request may time out while it is still in the throttler’s queue, before the driver has even
tried to send it to a node.
The only visible effect is that a request may fail with a RequestThrottlingException, if the throttler has determined that it can neither allow the request to proceed now, nor enqueue it; this indicates that your session is overloaded. How you react to that is specific to your application; typically, you could display an error asking the end user to retry later.
Note that the following requests are also affected by throttling:
preparing a statement (either directly, or indirectly when the driver reprepares on other nodes, or when a node comes back up – see how the driver prepares);
fetching the next page of a result set (which happens in the background when you iterate the
synchronous variant ResultSet).
fetching a query trace.
Request throttling is parameterized in the configuration under
advanced.throttler. There are various implementations, detailed in the following sections:
datastax-java-driver {
advanced.throttler {
class = PassThroughRequestThrottler
}
}
This is a no-op implementation: requests are simply allowed to proceed all the time, never enqueued.
Note that you will still hit a limit if all your connections run out of stream ids. In that case,
requests will fail with an AllNodesFailedException, with the getErrors() method returning a
BusyConnectionException for each node. See the connection pooling page.
datastax-java-driver {
advanced.throttler {
class = ConcurrencyLimitingRequestThrottler
# Note: the values below are for illustration purposes only, not prescriptive
max-concurrent-requests = 10000
max-queue-size = 100000
}
}
This implementation limits the number of requests that are allowed to execute simultaneously. Additional requests get enqueued up to the configured limit. Every time an active request completes (either by succeeding, failing or timing out), the oldest enqueued request is allowed to proceed.
Make sure you pick a threshold that is consistent with your pooling settings; the driver should
never run out of stream ids before reaching the maximum concurrency, otherwise requests will fail
with BusyConnectionException instead of being throttled. The total number of stream ids is a
function of the number of connected nodes and the connection.pool.*.size and
connection.max-requests-per-connection configuration options. Keep in mind that aggressive
speculative executions and timeout options can inflate stream id consumption, so keep a safety
margin. One good way to get this right is to track the pool.available-streams metric
on every node, and make sure it never reaches 0. See the connection pooling page.
datastax-java-driver {
advanced.throttler {
class = RateLimitingRequestThrottler
# Note: the values below are for illustration purposes only, not prescriptive
max-requests-per-second = 5000
max-queue-size = 50000
drain-interval = 1 millisecond
}
}
This implementation tracks the rate at which requests start, and enqueues when it exceeds the configured threshold.
With this approach, we can’t dequeue when requests complete, because having less active requests
does not necessarily mean that the rate is back to normal. So instead the throttler re-checks the
rate periodically and dequeues when possible, this is controlled by the drain-interval option.
Picking the right interval is a matter of balance: too low might consume too many resources and only
dequeue a few requests at a time, but too high will delay your requests too much; start with a few
milliseconds and use the cql-requests metric to check the impact on your latencies.
Like with the concurrency-based throttler, you should make sure that your target rate is in line with the pooling options; see the recommendations in the previous section.
Enable the following metrics to monitor how the throttler is performing:
datastax-java-driver {
advanced.metrics.session.enabled = [
# How long requests are being throttled (exposed as a Timer).
#
# This is the time between the start of the session.execute() call, and the moment when the
# throttler allows the request to proceed.
throttling.delay,
# The size of the throttling queue (exposed as a Gauge<Integer>).
#
# This is the number of requests that the throttler is currently delaying in order to
# preserve its SLA. This metric only works with the built-in concurrency- and rate-based
# throttlers; in other cases, it will always be 0.
throttling.queue-size,
# The number of times a request was rejected with a RequestThrottlingException (exposed as a
# Counter)
throttling.errors,
]
}
If you enable throttling.delay, make sure to also check the associated extra options to correctly
size the underlying histograms (metrics.session.throttling.delay.*).