When a query fails, it sometimes makes sense to retry it: the error might be temporary, or the query might work on a different host, or with different options.
The driver uses a configurable set of rules to determine when and how to retry.
When the driver executes a statement, it first obtains a query plan (a list of hosts) from the load balancing policy. Then it picks the first host and sends it the request; this host acts as the coordinator for the query, it will communicate with the rest of the cluster and reply to the client.
If the coordinator can’t be reached or replies with an error, there are various things that the driver can do; they are expressed as RetryDecision objects:
retry(): retry the query on the same host. It’s possible to retry with a different consistency level than the one that was originally requested;
tryNextHost(): retry on the next host in the query plan. Again, it can be with a different CL;
rethrow(): rethrow the exception to the user code. This means it will be thrown from the
session.executecall (or returned as a failed future if
ignore(): mark the request as successful, and return an empty result set.
If the driver retries on every host and reaches the end of the query plan, a NoHostAvailableException is thrown to the user code. You can use its getErrors() method to find out what went wrong on each host.
RetryPolicy is a pluggable component that determines the retry decisions for various types of errors. It is configured when initializing the cluster:
Cluster cluster = Cluster.builder() .addContactPoint("127.0.0.1") .withRetryPolicy(new MyCustomPolicy()) .build();
Once the cluster has been built, you can’t change the policy, but you may inspect it at runtime:
RetryPolicy policy = cluster.getConfiguration().getPolicies().getRetryPolicy();
If you don’t explicitly configure it, you get a DefaultRetryPolicy.
The policy’s methods cover different types of errors:
A read request reached the coordinator, which initially believed that there were enough live replicas to process it.
But, for some reason, one or several replicas were too slow to answer within the predefined timeout
cassandra.yaml), and the coordinator replied to the client with a
This could be due to a temporary overloading of these replicas, or even
that they just failed or were turned off. During reads, Cassandra doesn’t request data from every replica to minimize
internal network traffic; instead, some replicas are only asked for a checksum of the data. A read timeout may occur
even if enough replicas responded to fulfill the consistency level, but only checksum responses were received (the
dataRetrieved parameter allow you to check if you’re in that situation).
If the policy rethrows the error, the user code will get a ReadTimeoutException.
Note: do not confuse this error with a driver read timeout, which happens when the coordinator didn’t reply at all to the client.
This is similar to
onReadTimeout, but for write operations. The reason reads and writes are handled separately is
because a read is obviously a non mutating operation, whereas a write is likely to be. If a write times out at the
coordinator level, there is no way to know whether the mutation was applied or not on the non-answering replica.
If the policy rethrows the error, the user code will get a WriteTimeoutException.
This gets called for any other error occurring after the request was sent.
The method receives the exception as a parameter, so that implementations can refine their decision based on what happened. The possible exceptions are:
ServerError: thrown by the coordinator when an unexpected error occurs. This is generally a Cassandra bug;
ConnectionException: thrown by the client for any network issue while or after the request was written;
OverloadedException: thrown by the coordinator when replicas are down and the number of hinted handoffs gets too high; the coordinator temporarily refuses writes for these replicas (see hinted handoffs in the Cassandra documentation).
There are a few cases where retrying is always the right thing to do. These are not covered by
hard-coded in the driver:
any error before a network write was attempted: to send a query, the driver selects a host, borrows a connection from the host’s connection pool, and then writes the message to the connection. Errors can occur before the write was even attempted, for example if the connection pool is saturated, or if the host went down right after we borrowed. In those cases, it is always safe to retry since the request wasn’t sent, so the driver will transparently move to the next host in the query plan.
re-preparing a statement: when the driver executes a prepared statement, it may find out that the coordinator doesn’t know about it, and need to re-prepare it on the fly (this is described in detail here). The query is then retried on the same host.
trying to communicate with a host that is bootstrapping: this is a rare edge case, as in practice the driver should never try to communicate with a bootstrapping host (the only way is if it was specified as a contact point). Anyway, it is again safe to assume that the query was not executed at all, so the driver moves to the next host.
Similarly, some errors have no chance of being solved by a retry. They will always be rethrown directly to the user. These include:
Retries and idempotence¶
If a query is not idempotent, the driver will not retry it if that could produce inconsistent results:
onReadTimeoutis always safe, since by definition this error indicates that the query was a read, which didn’t mutate any data;
onUnavailableis safe: the coordinator is telling us that it didn’t find enough replicas, so we know that it didn’t try to apply the query.
onWriteTimeoutis not safe: some replicas failed to reply to the coordinator in time, but they might still have applied the mutation;
onRequestErroris not safe either: the query might have been applied before the error occurred. In particular, an
OperationTimedOutExceptioncould be caused by a network issue that prevented a successful response to come back to the client.
Therefore, the driver does not retry after a write timeout or request error if the statement is not idempotent. This is handled internally, the retry policy methods are not even invoked in those cases.
Note that this behavior was introduced in version 3.1.0 of the driver. In previous versions, it was up to retry policy
implementations to handle idempotence (the new behavior is equivalent to what you achieved with