My Adventures in Coding

January 2, 2016

Java – Simple GZIP Utility to Compress and Decompress a String

Filed under: Java — Brian @ 10:15 pm
Tags: , , ,

I wanted to have a simple utility class in our app so we could easily compress a String to a byte[] when our REST API received a GET request with the header “Accept-Encoding: gzip” and also be able to decompress a byte[] to a String when we received a PUT with the header “Content-Encoding: gzip”. So I wrote a simple utility class to GZIP a String to a byte[] and to unzip a GZIP byte[] to a String.

So here is a simple GzipUtil class:

import java.io.*;
import java.nio.charset.StandardCharsets;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

public class GzipUtil {

    public static byte[] zip(final String str) {
        if ((str == null) || (str.length() == 0)) {
            throw new IllegalArgumentException("Cannot zip null or empty string");
        }

        try (ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream()) {
            try (GZIPOutputStream gzipOutputStream = new GZIPOutputStream(byteArrayOutputStream)) {
                gzipOutputStream.write(str.getBytes(StandardCharsets.UTF_8));
            }
            return byteArrayOutputStream.toByteArray();
        } catch(IOException e) {
            throw new RuntimeException("Failed to zip content", e);
        }
    }

    public static String unzip(final byte[] compressed) {
        if ((compressed == null) || (compressed.length == 0)) {
            throw new IllegalArgumentException("Cannot unzip null or empty bytes");
        }
        if (!isZipped(compressed)) {
            return new String(compressed);
        }

        try (ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(compressed)) {
            try (GZIPInputStream gzipInputStream = new GZIPInputStream(byteArrayInputStream)) {
                try (InputStreamReader inputStreamReader = new InputStreamReader(gzipInputStream, StandardCharsets.UTF_8)) {
                    try (BufferedReader bufferedReader = new BufferedReader(inputStreamReader)) {
                        StringBuilder output = new StringBuilder();
                        String line;
                        while((line = bufferedReader.readLine()) != null){
                            output.append(line);
                        }
                        return output.toString();
                    }
                }
            }
        } catch(IOException e) {
            throw new RuntimeException("Failed to unzip content", e);
        }
    }

    public static boolean isZipped(final byte[] compressed) {
        return (compressed[0] == (byte) (GZIPInputStream.GZIP_MAGIC)) && (compressed[1] == (byte) (GZIPInputStream.GZIP_MAGIC >> 8));
    }
}

Also here are a set of JUnit tests for this utility:

import org.junit.Test;
import static org.junit.Assert.*;

public class GzipUtilTest {

    @Test(expected = IllegalArgumentException.class)
    public void zip_shouldThrowIllegalArgumentException_whenStringToCompressIsNull() {
        GzipUtil.zip(null);
    }

    @Test(expected = IllegalArgumentException.class)
    public void zip_shouldThrowIllegalArgumentException_whenStringToCompressIsEmpty() {
        GzipUtil.zip("");
    }

    @Test
    public void zip_shouldGzipString_whenStringIsNotEmpty() {
        String xml = "<Hello>World</Hello>";

        byte[] actual = GzipUtil.zip(xml);

        assertTrue(GzipUtil.isZipped(actual));
    }

    @Test(expected = IllegalArgumentException.class)
    public void unzip_shouldThrowIllegalArgumentException_whenByteArrayToDecompressIsNull() {
        GzipUtil.unzip(null);
    }

    @Test(expected = IllegalArgumentException.class)
    public void unzip_shouldThrowIllegalArgumentException_whenByteArrayToDecompressIsEmpty() {
        GzipUtil.unzip(new byte[0]);
    }

    @Test
    public void unzip_shouldReturnInputByteArrayAsString_whenByteArrayContentIsNotGzipped() {
        String xml = "<Hello>World</Hello>";
        byte[] bytes = xml.getBytes();

        String actual = GzipUtil.unzip(bytes);

        assertEquals(xml, actual);
    }

    @Test
    public void unzip_shouldDecompressByteArrayGzippedContent() throws Exception {
        String xml = "<Hello>World</Hello>";
        byte[] compressed = GzipUtil.zip(xml);

        String actual = GzipUtil.unzip(compressed);

        assertEquals(xml, actual);
    }

    @Test
    public void isZipped_shouldReturnFalse_whenContentIsNotGzipped() {
        byte[] bytes = new byte[] {1,2,3};

        boolean actual = GzipUtil.isZipped(bytes);

        assertFalse(actual);
    }

    @Test
    public void isZipped_shouldReturnTrue_whenContentIsGzipped() {
        byte[] bytes = GzipUtil.zip("1,2,3");

        boolean actual = GzipUtil.isZipped(bytes);

        assertTrue(actual);
    }
}

I hope you find this useful!

August 14, 2015

Java – Automate database schema updates with Flyway

Filed under: Java — Brian @ 1:44 pm
Tags: , , , , , ,

I am currently working on a Java 8 project which is a REST API deployed as a WAR file to Jetty. Our deploy process is very simple, our deploy pipeline just copies the WAR file into the Jetty directory in each environment then verifies the app is up and running with the correct version and runs some integration tests.

We wanted to be able to apply database migration scripts automatically in each environment (Dev, Test, QA, Staging, Prod) as we did our deploy, so we would no longer have to worry about manually applying scripts. In the past for Java, Scala, and .NET projects I have used several different tools, but for this project we decided to use Flyway which is very flexible and simple to setup.

The documentation for Flyway is excellent, however I decided to just post what we did in our app in case it might help someone else out. Here is our “Quick Start” setup.

1. Add the Flyway dependency

Flyway can be setup using Maven, Gradle, SBT, Ant, etc. In our project we used Maven, so all we did was add the following to our pom.xml file:

<dependency>
    <groupId>org.flywaydb</groupId>
    <artifactId>flyway-core</artifactId>
    <version>3.2.1</version>
</dependency>

2. Code

Now, to get Flyway to check for any database changes to apply, whenever the application is started, put the following code somewhere in your applications startup sequence.

Flyway flyway = new Flyway();
//Create the dbo.schema_version table if it does not already exist
flyway.setBaselineOnMigrate(true);
//Where the method "getDataSource()" provides your DataSource 
//object that has the jdbc url, username, and password.
flyway.setDataSource(getDataSource()); 
flyway.migrate();

3. SQL Script Location

All migrations scripts by default must go in the following folder in your app:

/resources/db/migration

4. SQL Script Naming convention

Scripts are run in version number order based on their names, the default naming convention is:

Version__description.sql
For example: “V1_0_1__create_tables.sql”

All scripts must start with the letter “V”, followed by major/minor version numbers, with two underscores “__” separating the version from the description.

5. Schema Version Table

Flywaydb will automatically create a table in each database called “dbo.schema_version” which stores a log of all migration scripts that have been applied.

The table looks like:

version_rank installed_rank version description type script checksum installed_by installed_on execution_time success
1 1 1 Flyway Baseline BASELINE Flyway Baseline NULL JohnSmith 2015-07-30 15:55:49.337 0 1
1 2 1.0.1 create tables SQL V1_0_1__create_tables.sql -440358290 JohnSmith 2015-07-30 15:55:49.337 109 1

6. Handling Failure

If a script fails, the app will fail to start and the failure information is written to our log files as well as a monitor is tripped. Flywaydb applies each script as a transaction so all changes in the script will be rolled back if any part of the script fails. This is very handy because if you commit a script with some invalid SQL syntax, all you have to do is update the script with the corrected syntax, commit it again, and let the build pipeline apply the changes from the fixed script. No messy cleanup or reset to worry about.

So that is it, you should have all you need to get your database schema changes easily synced up with the deploy of your app!

January 11, 2015

Java to Sql Server – Cannot create PoolableConnectionFactory

Filed under: Java,SQL Server,Uncategorized — Brian @ 11:14 pm
Tags: , ,

If you are connecting a Java application to a SQL Server database, which is up and running, but your connection fails with the following error:

Could not acquire a connection from DataSource – Cannot create PoolableConnectionFactory

The error is most likely that the windows service “SQL Server Browser” is disabled.

To fix the problem:

  • Start -> Control Panel -> Systems & Security -> Administrative tools -> Services
  • SQL Server Browser -> right click -> properties

image2014-10-30 19-17-34

  • Set Startup type to “Automatic”
  • Apply -> Start

image2014-10-30 19-19-16

Now you should be able to connect to SQL Server from your Java application!

July 30, 2014

Java – Creating a simple retry command with function passing in Java 8

Filed under: Java — Brian @ 8:45 am
Tags: , , ,

Recently we have been working on an application that imports data from a number of different sources, where the network connection between us and each of these sources is not very reliable. So in our Gateway that makes these REST calls I wanted to be able to write a reusable piece of code that we could use in different calls, that in the event of a failure, would retry the command a few more times before finally giving up.

Java 7
I wanted to be able to write this retry logic and error handling code in one place and use it for a number of different method calls. There are several ways to do this, but previously in Java 7 I would have just written an abstract class with a single abstract method such as:

public abstract class RetryCommand<T> {
    private int maxRetries;

    public RetryCommand(int maxRetries) {
        this.maxRetries = maxRetries;
    }

    // This abstract command is the method that will be implemented 
    public abstract T command();

    public final T run() throws RuntimeException {
        try {
            return command();
        } catch (Exception e) {
            return retry();
        }
    }

    private final T retry() throws RuntimeException {
        System.out.println("FAILED - Command failed, will be retried " + maxRetries + " times.");
        int retryCounter = 0;
        while (retryCounter < maxRetries) {
            try {
                return command();
            } catch (Exception e) {
                retryCounter++;
                System.out.println("FAILED - Command failed on retry " + retryCounter + " of " + maxRetries + " error: " + ex );
                if (retryCounter >= maxRetries) {
                    System.out.println("Max retries exceeded.");
                    break;
                }
            }
        }
        throw new RuntimeException("Command failed on all of " + maxRetries + " retries");
    }
}

Then in my Gateway code, for each method that I want to wrap with my retry logic I would just do the following:

public class MyGateway {
    private RetryCommand<String> retryCommand;
    public MyGateway(int maxRetries) {
        retryCommand = new RetryCommand<>(maxRetries);
    }

    // Inline create an instance of  the abstract class RetryCommand
    // Define the body of the "command" method
    // Execute the "run" method and return the result
    public String getThing(final String id) {
        return new RetryCommand<String>() {
            public String command() {
                return client.getThatThing(id);
            }
        }.run();
    }
}

The reason for this layout was I could not pass a function as a parameter to a method, like I have done in Scala, Python, and C#. However, now that we have Java 8, we can finally pass functions as parameters using the handy features in java.util.function package!

Java 8
Java 8 uses Functional Interfaces, which are interfaces with a single abstract method. The package java.util.function defines a number of standard functional interfaces, so most of the time you will be able to use one of these. Some example functional interfaces are Function (function with return value and input param), Supplier (function with return value but no input param), and Consumer (function with input param but no return value). However, if one of these standard functional interfaces does not meet your needs you can always define your own. In the following example I used Supplier.

So now in Java 8 I would create a new RetryCommand class that has a “run” method which takes in a function:

import java.util.function.Supplier;

public class RetryCommandJava8<T> {
    private int retryCounter;
    private int maxRetries;

    public RetryCommandJava8(int maxRetries)
    {
        this.maxRetries = maxRetries;
    }

    // Takes a function and executes it, if fails, passes the function to the retry command
    public T run(Supplier<T> function) {
        try {
            return function.get();
        } catch (Exception e) {
            return retry(function);
        }
    }

    public int getRetryCounter() { return retryCounter; }

    private T retry(Supplier<T> function) throws RuntimeException {
        System.out.println("FAILED - Command failed, will be retried " + maxRetries + " times.");
        retryCounter = 0;
        while (retryCounter < maxRetries) {
            try {
                return function.get();
            } catch (Exception ex) {
                retryCounter++;
                System.out.println("FAILED - Command failed on retry " + retryCounter + " of " + maxRetries + " error: " + ex );
                if (retryCounter >= maxRetries) {
                    System.out.println("Max retries exceeded.");
                    break;
                }
            }
        }
        throw new RuntimeException("Command failed on all of " + maxRetries + " retries");
    }
}

So now in my gateway code, I would create my fancy new retry command executer:

public class MyGateway {
    private RetryCommandJava8<String> retryCommandJava8;
    public MyGateway(int maxRetries) {
        retryCommandJava8 = new RetryCommandJava8<>(maxRetries);
    }

    // Passing function using a Lamba expression 
    public String getThing(final String id) {
        return retryCommandJava8.run(() -> client.getThatThing(id));
    }
}

Also, this setup is fairly easy to unit test, here are some example tests:

import junit.framework.TestCase;

public class RetryCommandJava8Test extends TestCase {

    public String SUCCESS = "success";
    public int MAXRETRIES = 3;
    public int SECONDSTOWAIT = 0;
    RetryCommandJava8<String> retryCommandJava8;

    public void testRetryCommandShouldNotRetryCommandWhenSuccessful() {
        retryCommandJava8 = new RetryCommandJava8<>(MAXRETRIES, SECONDSTOWAIT);

        String result = retryCommandJava8.run(() -> SUCCESS);

        assertEquals(SUCCESS, result);
        assertEquals(0, retryCommand.getRetryCounter());
    }

    public void testRetryCommandShouldRetryOnceThenSucceedWhenFailsOnFirstCallButSucceedsOnFirstRetry() {
        retryCommand = new RetryCommandJava8<>(MAXRETRIES, SECONDSTOWAIT);

        String result = retryCommandJava8.run(() -> {
            if (retryCommand.getRetryCounter() == 0) throw new RuntimeException("Command Failed");
            else return SUCCESS;
        });

        assertEquals(SUCCESS, result);
        assertEquals(1, retryCommand.getRetryCounter());
    }

    public void testRetryCommandShouldThrowExceptionWhenMaxRetriesIsReached() {
        retryCommandJava8 = new RetryCommandJava8<>(MAXRETRIES, SECONDSTOWAIT);

        try {
            retryCommand.run(() -> {throw new RuntimeException("Failed");});
            fail("Should throw exception when max retries is reached");
        } catch (Exception e) { }
    }
}

Of course this example is a stripped down version of what we use, which does waits between retries, back off retries, and proper logging of errors, etc. I just wanted to use a retry command as my example code for trying out function passing in Java 8. However, I hope maybe you will find this useful if you are trying to get a working example going for your first use of function passing in Java 8.

If you are new to Java 8 (just like I am) I recommend reading Everything About Java 8.

March 7, 2010

MongoDB queries in Java using Conditional Operators

Filed under: Java,NOSQL — Brian @ 4:29 pm
Tags: , , , ,

When I first started using MongoDB, I used the interactive shell to learn the query syntax. The syntax is simple and straightforward. However, when I started using the Java driver I was not sure how to translate some of my command line queries into Java code, for example some of the conditional operators like “$in”. The Java documentation on the MongoDB website only showed how to use some basic conditionals like greater than and less than, but not for using other conditionals that use lists like the $in option.

A simple query for all cars with the make “Ford” that match any of several models listed:

SQL:

SELECT * FROM dbo.Cars 
WHERE make="Ford" 
AND model IN ("Galaxy","Mustang","Meteor")

MongoDB interactive shell:

db.cars.find( { "make":"Ford", "model":{ $in: ["Galaxy","Mustang","Meteor"] } } )

MongoDB Java driver:

BasicDBObject query = new BasicDBObject();
query.put("make", "Ford");
String models[] = new String[]{"Galaxy", "Mustang", "Meteor"};
query.put("model", new BasicDBObject("$in", models));
DBCursor resultsCursor = carsCollection.find(query);

July 5, 2009

Java – REST APIs and Thread Safety

Filed under: Java,REST — Brian @ 7:26 pm
Tags:

In my current job, a few months ago we ran into some strange issues on production. Customers were complaining that sometimes they would receive incorrect data when accessing our REST APIs. We were never able to reproduce the issue and the strange results customers were receiving seemed entirely random. Until one day while a member of our team was accessing a REST API at the same time as a customer. As a result the customer received the employee’s information! After this discovery we found that when accessing a REST call in our system concurrently with different parameters we were consistently able to reproduce the problem the customer was complaining about. Yep, we had classes that were not thread safe and needed to be fixed!

The Bug

In our case the thread safety issue was related to the use of local variables inside a class declared as a singleton in our Spring configuration. The issue was a race condition between when the lookup parameter (e.g., customerId) was set and when it was used to lookup a requested value (e.g., email). Take a look at this example class:

public class CustomerInfo {
    int customerId;

    public void setCustomerId(int customerId) {
        this.customerId = customerId;
    }

    public String getEmail() {
        return customerRepository.getEmail(customerId);
    }
}

Let’s say we have two customers placing an order:
Customer 1: has an email address of jsmith@nowhere.com
Customer 2: has an email address of jdoe@nowhere.com

Now we have two callers accessing our REST API at the same time so two threads are using our CustomerInfo class:

  • Thread A: setCustomerId(1) – customerId=1
  • Thread B: setCustomerId(2) – customerId=2
  • Thread A: getEmail() – customerId=2 so email jdoe@nowhere.com is returned
  • Thread B: getEmail() – customerId=2 so email jdoe@nowhere.com is returned

The Fix

So even though the code calling the getEmail() method was calling it immediately after setting the customerId with the setCustomerId() method, if two threads were running at the same time the chances of the customerId being updated by one thread and affecting the results of another were very likely. There was no reason why we needed to set a customerId value in the class, this was just extra code that was not needed. So removing unnecessary code and fixing our thread safety issue ended up being connected:

public class CustomerInfo{
    public String getEmail(int customerId) {
        return customerRepository.getEmail(customerId);
    }
}

The way we were able to find and track down this concurrency bug was by writing a concurrency test first.

A Simple Concurrency Test

A concurrency test for a REST call does not have to be anything complicated. Just take a REST call you want to test and write a test following this basic outline:

  • One REST call to test
  • A list of valid parameters to use with that REST call and an expected value for each parameter.
  • This expected value is used to check that the request was successful (e.g., parameter CustomerID=2938 returns Email=johnsmith@gmail.com).
  • Pool of threads, each calling the REST call for each of the parameters in the list.
  • Verify every REST call in each thread returns the expected value.

Now run the test and if any of the REST calls return data that does not match the expected value, you know you have some code that is not thread safe. Now that you have this failing test in place you can proceed with finding and fixing your thread safety issues!

Create a free website or blog at WordPress.com.