My Adventures in Coding

April 27, 2018

Redis – Setup a Test Cluster on Ubuntu

Filed under: Linux,Redis,Ubuntu — Brian @ 4:49 pm
Tags: , , , ,

I have been working with Redis lately and in my local development environment I was just testing against a single node setup of Redis. However, this did not give me the functionality I wanted to test against since our production environment uses a Redis cluster of six nodes.

Connecting to a Redis cluster requires a different configuration from a single Redis instance and is also functionally different. For example, a standard six node cluster of Redis is actually three pairs of Master-Replica where each pair stores a third of the data. So for those reasons I really wanted to test my application setup against a Redis cluster in my local development environment, which was Ubuntu.

At the the time of this post I was using Ubuntu 16.04 and Redis 4.0.9.

So here is how to spin up a six node test cluster of Redis on a single Ubuntu instance.

Download and Build Redis

wget http://download.redis.io/releases/redis-4.0.9.tar.gz
tar -zxvf redis-4.0.9.tar.gz
cd redis-4.0.9
make

Create Six Redis Nodes

First, edit the create-cluster startup file:

cd utils/create-cluster
vi create-cluster

In my setup, I changed the following:

  • Port – set to 7000 (This will be the starting port)
  • Timeout – set to 10000 (I found this handy in my dev enivornment)
  • Nodes – set to 6 (We use a 6 node cluster in production, so I wanted the same)
  • Replicas – set to 1 (This means each master with have one replica, 6 nodes = 3 Masters and 3 Replicas)

Edit_Create_Cluster_Script

Now we can start all six nodes:

./create-cluster start

You will see the nodes have been started:
Redis-Start-Cluster

If you do an “ls” in the folder you will notice that some files have been created:

  • Each node has a log file (For example: 7001.log)
  • Each node has a cluster configuration file (For example: nodes-7001.conf)

Cluster_config_files

You can also confirm the nodes are running by connecting to a node using the “redis-cli” command line tool by specifying the port of a node:

cd redis-4.0.9/src
./redis-cli -p 7001

Redis-Cli-Cluster-Key-Added

Join the Nodes as a Cluster

Even though at this stage we have six Redis nodes, they know nothing about each other. So the next step is to join these nodes as a cluster.

Prerequisites: Ruby and Redis Gem

The script that connects the nodes as a cluster is written in Ruby and requires the Redis Gem, which requires Ruby 2.2.2 or higher.

Update Ruby

sudo apt-get install ruby-full

Install Redis Gem

sudo gem install redis

To create a cluster where you only need to connect to it from the local machine you can use:

cd redis-4.0.9/src
./redis-trib.rb create --replicas 1 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006

NOTE: If you plan to connect to your Redis cluster remotely, you will need to use the public IP of the machine instead of 127.0.0.1.

When you run the join cluster script it will show you the generated configuration and then ask you to confirm the setup:
Redis-Join-Nodes-To-Cluster_001

After you answer “yes” and press enter, the nodes will be joined in a cluster. If you see the following the cluster creation was successful:
Redis-Join-Nodes-Success

That is all! You now have a functional Redis Cluster that you can connect to from your application!

For more information on Redis Clusters I recommend reading the Redis Cluster Tutorial.

Advertisements

April 1, 2018

Java – Using SQLServerBulkCopy in Java with an InputStream

Filed under: Java,SQL Server — Brian @ 9:59 pm
Tags: , , ,

Up until recently, the only constructor available in SQLServerBulkCopy took in a file path. This was kind of awkward since if you wanted to use bulk copy with data you had in memory, you either had to write it to a file only to read it back, or extend the class and write a constructor that could take in an InputStream. Luckily in version (I think 6.3) Microsoft finally added a new constructor with support for InputStream.

The following is a simple example in Java showing how to use SQLServerBulkCopy to write to a table in SQL Server.

Create a table in your SQL Server database:

CREATE TABLE [dbo].[SqlBulkInsertExample](
	[Id] [int] NOT NULL,
	[Name] [varchar](100) NOT NULL 
) ON [PRIMARY]
GO

Create a Java project. If using Maven, you can add the following dependency to your pom.xml file:

<dependency>
    <groupId>com.microsoft.sqlserver</groupId>
    <artifactId>mssql-jdbc</artifactId>
    <version>6.4.0.jre8</version>
</dependency>

Here is a simple Java example application showing how to take some data, convert it to a CSV, and save it to the database using bulk insert:

import com.microsoft.sqlserver.jdbc.SQLServerBulkCSVFileRecord;
import com.microsoft.sqlserver.jdbc.SQLServerBulkCopy;
import com.microsoft.sqlserver.jdbc.SQLServerBulkCopyOptions;

import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.Types;
import java.util.HashMap;
import java.util.Map;

public class SqlBulkCopyExample {

  private static String JDBC_URL = "jdbc:sqlserver://localhost;DatabaseName=Playground;InstanceName=SQL2016";
  private static String USERNAME = "client";
  private static String PASSWORD = "client";
  private static String TABLE_NAME = "dbo.SqlBulkInsertExample";

  public static void main(String[] args) {
    try {
      // Create some data to insert into our database table
      Map data = new HashMap();
      data.put(1, "John Smith");
      data.put(2, "Steve Smith");
      data.put(3, "Molly Smith");

      // We are going to build a CSV document to use for the bulk insert
      StringBuilder stringBuilder = new StringBuilder();

      // Add table column names to CSV
      stringBuilder.append("id, name\n");

      // Copy data from map and append to CSV
      for (Map.Entry entry : data.entrySet()) {
        stringBuilder.append(
                String.format("%s,%s\n", entry.getKey(), entry.getValue()));
      }

      byte[] bytes = stringBuilder.toString().getBytes(StandardCharsets.UTF_8);
      try (InputStream inputStream = new ByteArrayInputStream(bytes)) {

        // Pass in input stream and set column information
        SQLServerBulkCSVFileRecord fileRecord = new SQLServerBulkCSVFileRecord(
                inputStream, StandardCharsets.UTF_8.name(), ",", true);

        fileRecord.addColumnMetadata(1, "id", Types.INTEGER, 0, 0);
        fileRecord.addColumnMetadata(2, "name", Types.VARCHAR, 0, 0);

        try (Connection connection = DriverManager.getConnection(
                JDBC_URL, USERNAME, PASSWORD)) {

          // Set bulk insert options, for example here I am setting a batch size
          SQLServerBulkCopyOptions copyOptions = new SQLServerBulkCopyOptions();
          copyOptions.setBatchSize(10000);

          // Write the CSV document to the database table
          try (SQLServerBulkCopy bulkCopy = new SQLServerBulkCopy(connection)) {
            bulkCopy.setBulkCopyOptions(copyOptions);
            bulkCopy.setDestinationTableName(TABLE_NAME);
            bulkCopy.writeToServer(fileRecord);
          }
        }
      }
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

Now if you login to your database and run the following select:

SELECT * FROM dbo.SqlBulkInsertExample

You should see that you have data!
Query_Results

I hope that helps!

Create a free website or blog at WordPress.com.