My Adventures in Coding

February 22, 2019

Cassandra – FSReadError on Startup

Filed under: Cassandra — Brian @ 5:43 pm
Tags: , , , ,

Recently we encountered an error with one node in a Cassandra cluster where the Cassandra service said it was running but we would get a failure when we tried to connect:

# cqlsh
Connection error: ('Unable to connect to any servers', 
{'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. 
Last error: Connection refused")})

So we decided to tail the journal to see if we could find a useful error message:

journalctl -f -u cassandra

While monitoring the journal output we saw the following exception recurring roughly every minute:

Nov 19 04:17:35 pcu077al188 cassandra[17259]: Exception (org.apache.cassandra.io.FSReadError) encountered during startup: java.io.EOFException
Nov 19 04:17:35 pcu077al188 cassandra[17259]: FSReadError in /var/lib/cassandra/hints/b22dfb1b-6a6e-44ce-9c7c-fda1e75293af-1542627895660-1.hints
Nov 19 04:17:35 pcu077al188 cassandra[17259]: at org.apache.cassandra.hints.HintsDescriptor.readFromFile(HintsDescriptor.java:235)

The file that Cassandra was having problems with was a 0 byte hint file.

The following Stack Overflow post suggested that to resolve the problem you just needed to remove this file. We tried this solution and it worked.

Steps to Fix FSReadError Startup Problem

Stop Cassandra

systemctl stop cassandra

Move the suspect hint file into a temporary folder (just to be safe)

mv /var/lib/cassandra/hints/b22dfb1b-6a6e-44ce-9c7c-fda1e75293af-1542627895660-1.hints /tmp

Start Cassandra

systemctl start cassandra

Verify the error has stopped

journalctl -f -u cassandra

Now verify you can connect using CQLSH

# cqlsh
Connected to PointServiceClusterV3 at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh>

Note: In our case this happened on a Cassandra instance in a test environment that had not been shutdown cleanly so there was no worry about data integrity. However, if this had happened on a node in a production environment I would recommend running nodetool repair on the node.

nodetool repair

I hope that helps!

January 29, 2019

Cassandra – Switching from SimpleStrategy to NetworkTopologyStrategy

When we started using Cassandra I setup clusters for our test, staging, and production environments. Then we created the initial keyspace for our application, added tables, started using them, and everything worked fine.

Later we decided that for our up-time requirements we wanted to have a second cluster in another data center to act as a hot fail-over on production. No problem, Cassandra has us covered. However, when we had originally created our application’s keyspace, it was created with the default replication strategy SimpleStrategy. For a fail-over cluster we need Cassandra to be configured with the NetworkTopologyStrategy. No big deal right, should be an easy fix?

After reading through the documentation on Changing keyspace replication strategy, I was left with one question:

“What do I use for the data center name?”.

With SimpleStrategy you specify the number of nodes to which you want each item replicated by specifying the parameter “replication_factor”, for example (‘replication_factor’ : 3) . However, when using NetworkTopologyStrategy you use the data center name to specify how many nodes you want to have copies of the data, for example (‘mydatacentername’, 3). I was worried that if I altered the strategy on one node then the cluster thought the node was not part of the same data center, this would cause some serious problems.

Fortunately, it turns out that Cassandra has a default data center name which you can use when making this switch, kudos to the person who replied to this StackOverflow post.

Of course I was not going to try this switch out on any of our clusters until I was confident it would work. I setup a three node cluster with SimpleStrategy with replication factor set to 2, added data to the cluster, ran a nodetool repair, then I altered the strategy for the keyspace, verifyied nothing had changed as expected, then I ran nodetool repair again, and once again verified all my data was intact. So it worked as promised.

Switch Replication Strategy

Note: In this example, the keyspace we are switching the replication strategy on is called “examplekeyspace”.

Open a cqlsh prompt on any node in the cluster

Check the current replication strategy

SELECT * FROM system_schema.keyspaces;

show_keyspaces_before

Verify the default data center name

SELECT data_center FROM system.local;

show_data_center_name

Alter the existing Keyspace

Alter the keyspace using the data center name (make sure you copy it exactly!) as the replication factor and set the number of nodes to replicate to to be the same as before.

ALTER KEYSPACE ExampleKeyspace WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter1': '3'};

alter_keyspace_network_topology

Now if you check the keyspace on each node in the cluster you will see that the replication strategy is now NetworkTopologyStrategy.

Nodetool Repair

Switching the replication strategy does not cause any data to be moved between nodes, you would need to run nodetool repair to do that. However, if all you are doing is switching an existing cluster with a single rack and datacenter from SimpleStrategy to NetworkTopologyStrategy it should not require any data be moved. But if you would like to be thorough it does not hurt to run a nodetool repair.

Run nodetool repair

nodetool repair -pr examplekeyspace

Using the option “pr – primary range only” means repair will only repair the keys that are known to the current node where repair is being run, and on other nodes where those keys are replicated. Make sure to run repair on each node, but only do ONE node at a time.

Conclusion

When I started using Cassandra I did not realize that for data replication how much of a limitation SimpleStrategy imposes. So if all you want is a single rack in a single datacenter, SimpleStrategy works, however if there is even the slightest possibility you might one day add a failover cluster in another data center or nodes in one data center but on multiple racks, use NetworkTopologyStrategy. Personally, for anything other than a local test cluster, I would always go with NetworkTopologyStrategy.

That is all!

January 27, 2018

Cassandra – Getting Started with Java

Filed under: Cassandra — Brian @ 9:15 pm
Tags: , ,

Cassandra is a great tool for storing time series data and I happen to be using it on my current project for that exact purpose.

There are several ways to use Cassandra from Java and many ways to improve performance, but here I just want to provide a simple “Getting Started” example. So here it is!

First, download the current version of Cassandra V3 from here.

Extract the tar.gz file:

 tar -zxvf apache-cassandra-3.11.1-bin.tar.gz
 

Change directory into the bin folder:

 cd apache-cassandra-3.11.1/bin
 

Start Cassandra:

 ./cassandra -f
 

Create a Java project, if using Maven, you can add the following dependencies to your pom.xml file:

<dependency>
    <groupId>com.datastax.cassandra</groupId>
    <artifactId>cassandra-driver-core</artifactId>
    <version>3.3.0</version>
</dependency>

Here is a simple Java example showing how to connect to Cassandra, create a keyspace, create a table, insert a row, and select a row:

import com.datastax.driver.core.*;

import java.time.Instant;
import java.time.ZoneId;
import java.util.Date;
import java.util.UUID;

public class CassandraV3Tutorial {

    private final static String KEYSPACE_NAME = "example_keyspace";
    private final static String REPLICATION_STRATEGY = "SimpleStrategy";
    private final static int REPLICATION_FACTOR = 1;
    private final static String TABLE_NAME = "example_table";

    public static void main(String[] args) {

        // Setup a cluster to your local instance of Cassandra
        Cluster cluster = Cluster.builder()
                .addContactPoint("localhost")
                .withPort(9042)
                .build();

        // Create a session to communicate with Cassandra
        Session session = cluster.connect();

        // Create a new Keyspace (database) in Cassandra
        String createKeyspace = String.format(
                "CREATE KEYSPACE IF NOT EXISTS %s WITH replication = " +
                        "{'class':'%s','replication_factor':%s};",
                KEYSPACE_NAME,
                REPLICATION_STRATEGY,
                REPLICATION_FACTOR
        );
        session.execute(createKeyspace);

        // Create a new table in our Keyspace
        String createTable = String.format(
                "CREATE TABLE IF NOT EXISTS %s.%s " + "" +
                        "(id uuid, timestamp timestamp, value double, " +
                        "PRIMARY KEY (id, timestamp)) " +
                        "WITH CLUSTERING ORDER BY (timestamp ASC);",
                KEYSPACE_NAME,
                TABLE_NAME
        );
        session.execute(createTable);

        // Create an insert statement to add a new item to our table
        PreparedStatement insertPrepared = session.prepare(String.format(
                "INSERT INTO %s.%s (id, timestamp, value) values (?, ?, ?)",
                KEYSPACE_NAME,
                TABLE_NAME
        ));

        // Some example data to insert
        UUID id = UUID.fromString("1e4d26ed-922a-4bd2-85cb-6357b202eda8");
        Date timestamp = Date.from(Instant.parse("2018-01-01T01:01:01.000Z"));
        double value = 123.45;

        // Bind the data to the insert statement and execute it
        BoundStatement insertBound = insertPrepared.bind(id, timestamp, value);
        session.execute(insertBound);

        // Create a select statement to retrieve the item we just inserted
        PreparedStatement selectPrepared = session.prepare(String.format(
                "SELECT id, timestamp, value FROM %s.%s WHERE id = ?",
                KEYSPACE_NAME,
                TABLE_NAME));

        // Bind the id to the select statement and execute it
        BoundStatement selectBound = selectPrepared.bind(id);
        ResultSet resultSet = session.execute(selectBound);

        // Print the retrieved data
        resultSet.forEach(row -> System.out.println(
                String.format("Id: %s, Timestamp: %s, Value: %s",
                row.getUUID("id"),
                row.getTimestamp("timestamp").toInstant().atZone(ZoneId.of("UTC")),
                row.getDouble("value"))));

        // Close session and disconnect from cluster
        session.close();
        cluster.close();
    }
}

If you would like to look at the data in your local Cassandra database, you can use the CQLSH command line tool.

So from the bin folder type:

 ./cqlsh

This will take you to a “cqlsh>” prompt.

To view all available Keyspaces:

 DESCRIBE KEYSPACES;

You will now see our “example_keyspace” database:

 system_schema system system_traces
 system_auth system_distributed example_keyspace

To switch to that Keyspace:

 USE example_keyspace;

To show all tables in the keyspace:

 DESCRIBE TABLES;

You will be shown all tables which includes “example_table”.

Now from the command line you can view the data in the table by using a select statement:

 select * from example_table;

Which will show the following information:

 id | timestamp | value
 --------------------------------------+---------------------------------+-------
 1e4d26ed-922a-4bd2-85cb-6357b202eda8 | 2018-01-01 01:01:01.000000+0000 | 123.45

I hope that helps!

Note: The documentation on the DataStax website is very good.

Blog at WordPress.com.