My Adventures in Coding

January 2, 2016

Java – Simple GZIP Utility to Compress and Decompress a String

Filed under: Java — Brian @ 10:15 pm
Tags: , , ,

I wanted to have a simple utility class in our app so we could easily compress a String to a byte[] when our REST API received a GET request with the header “Accept-Encoding: gzip” and also be able to decompress a byte[] to a String when we received a PUT with the header “Content-Encoding: gzip”. So I wrote a simple utility class to GZIP a String to a byte[] and to unzip a GZIP byte[] to a String.

So here is a simple GzipUtil class:

import java.io.*;
import java.nio.charset.StandardCharsets;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

public class GzipUtil {

    public static byte[] zip(final String str) {
        if ((str == null) || (str.length() == 0)) {
            throw new IllegalArgumentException("Cannot zip null or empty string");
        }

        try (ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream()) {
            try (GZIPOutputStream gzipOutputStream = new GZIPOutputStream(byteArrayOutputStream)) {
                gzipOutputStream.write(str.getBytes(StandardCharsets.UTF_8));
            }
            return byteArrayOutputStream.toByteArray();
        } catch(IOException e) {
            throw new RuntimeException("Failed to zip content", e);
        }
    }

    public static String unzip(final byte[] compressed) {
        if ((compressed == null) || (compressed.length == 0)) {
            throw new IllegalArgumentException("Cannot unzip null or empty bytes");
        }
        if (!isZipped(compressed)) {
            return new String(compressed);
        }

        try (ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(compressed)) {
            try (GZIPInputStream gzipInputStream = new GZIPInputStream(byteArrayInputStream)) {
                try (InputStreamReader inputStreamReader = new InputStreamReader(gzipInputStream, StandardCharsets.UTF_8)) {
                    try (BufferedReader bufferedReader = new BufferedReader(inputStreamReader)) {
                        StringBuilder output = new StringBuilder();
                        String line;
                        while((line = bufferedReader.readLine()) != null){
                            output.append(line);
                        }
                        return output.toString();
                    }
                }
            }
        } catch(IOException e) {
            throw new RuntimeException("Failed to unzip content", e);
        }
    }

    public static boolean isZipped(final byte[] compressed) {
        return (compressed[0] == (byte) (GZIPInputStream.GZIP_MAGIC)) && (compressed[1] == (byte) (GZIPInputStream.GZIP_MAGIC >> 8));
    }
}

Also here are a set of JUnit tests for this utility:

import org.junit.Test;
import static org.junit.Assert.*;

public class GzipUtilTest {

    @Test(expected = IllegalArgumentException.class)
    public void zip_shouldThrowIllegalArgumentException_whenStringToCompressIsNull() {
        GzipUtil.zip(null);
    }

    @Test(expected = IllegalArgumentException.class)
    public void zip_shouldThrowIllegalArgumentException_whenStringToCompressIsEmpty() {
        GzipUtil.zip("");
    }

    @Test
    public void zip_shouldGzipString_whenStringIsNotEmpty() {
        String xml = "<Hello>World</Hello>";

        byte[] actual = GzipUtil.zip(xml);

        assertTrue(GzipUtil.isZipped(actual));
    }

    @Test(expected = IllegalArgumentException.class)
    public void unzip_shouldThrowIllegalArgumentException_whenByteArrayToDecompressIsNull() {
        GzipUtil.unzip(null);
    }

    @Test(expected = IllegalArgumentException.class)
    public void unzip_shouldThrowIllegalArgumentException_whenByteArrayToDecompressIsEmpty() {
        GzipUtil.unzip(new byte[0]);
    }

    @Test
    public void unzip_shouldReturnInputByteArrayAsString_whenByteArrayContentIsNotGzipped() {
        String xml = "<Hello>World</Hello>";
        byte[] bytes = xml.getBytes();

        String actual = GzipUtil.unzip(bytes);

        assertEquals(xml, actual);
    }

    @Test
    public void unzip_shouldDecompressByteArrayGzippedContent() throws Exception {
        String xml = "<Hello>World</Hello>";
        byte[] compressed = GzipUtil.zip(xml);

        String actual = GzipUtil.unzip(compressed);

        assertEquals(xml, actual);
    }

    @Test
    public void isZipped_shouldReturnFalse_whenContentIsNotGzipped() {
        byte[] bytes = new byte[] {1,2,3};

        boolean actual = GzipUtil.isZipped(bytes);

        assertFalse(actual);
    }

    @Test
    public void isZipped_shouldReturnTrue_whenContentIsGzipped() {
        byte[] bytes = GzipUtil.zip("1,2,3");

        boolean actual = GzipUtil.isZipped(bytes);

        assertTrue(actual);
    }
}

I hope you find this useful!

August 14, 2015

Java – Automate database schema updates with Flyway

Filed under: Java — Brian @ 1:44 pm
Tags: , , , , , ,

I am currently working on a Java 8 project which is a REST API deployed as a WAR file to Jetty. Our deploy process is very simple, our deploy pipeline just copies the WAR file into the Jetty directory in each environment then verifies the app is up and running with the correct version and runs some integration tests.

We wanted to be able to apply database migration scripts automatically in each environment (Dev, Test, QA, Staging, Prod) as we did our deploy, so we would no longer have to worry about manually applying scripts. In the past for Java, Scala, and .NET projects I have used several different tools, but for this project we decided to use Flyway which is very flexible and simple to setup.

The documentation for Flyway is excellent, however I decided to just post what we did in our app in case it might help someone else out. Here is our “Quick Start” setup.

1. Add the Flyway dependency

Flyway can be setup using Maven, Gradle, SBT, Ant, etc. In our project we used Maven, so all we did was add the following to our pom.xml file:

<dependency>
    <groupId>org.flywaydb</groupId>
    <artifactId>flyway-core</artifactId>
    <version>3.2.1</version>
</dependency>

2. Code

Now, to get Flyway to check for any database changes to apply, whenever the application is started, put the following code somewhere in your applications startup sequence.

Flyway flyway = new Flyway();
//Create the dbo.schema_version table if it does not already exist
flyway.setBaselineOnMigrate(true);
//Where the method "getDataSource()" provides your DataSource 
//object that has the jdbc url, username, and password.
flyway.setDataSource(getDataSource()); 
flyway.migrate();

3. SQL Script Location

All migrations scripts by default must go in the following folder in your app:

/resources/db/migration

4. SQL Script Naming convention

Scripts are run in version number order based on their names, the default naming convention is:

Version__description.sql
For example: “V1_0_1__create_tables.sql”

All scripts must start with the letter “V”, followed by major/minor version numbers, with two underscores “__” separating the version from the description.

5. Schema Version Table

Flywaydb will automatically create a table in each database called “dbo.schema_version” which stores a log of all migration scripts that have been applied.

The table looks like:

version_rank installed_rank version description type script checksum installed_by installed_on execution_time success
1 1 1 Flyway Baseline BASELINE Flyway Baseline NULL JohnSmith 2015-07-30 15:55:49.337 0 1
1 2 1.0.1 create tables SQL V1_0_1__create_tables.sql -440358290 JohnSmith 2015-07-30 15:55:49.337 109 1

6. Handling Failure

If a script fails, the app will fail to start and the failure information is written to our log files as well as a monitor is tripped. Flywaydb applies each script as a transaction so all changes in the script will be rolled back if any part of the script fails. This is very handy because if you commit a script with some invalid SQL syntax, all you have to do is update the script with the corrected syntax, commit it again, and let the build pipeline apply the changes from the fixed script. No messy cleanup or reset to worry about.

So that is it, you should have all you need to get your database schema changes easily synced up with the deploy of your app!

January 11, 2015

Java to Sql Server – Cannot create PoolableConnectionFactory

Filed under: Java,SQL Server,Uncategorized — Brian @ 11:14 pm
Tags: , ,

If you are connecting a Java application to a SQL Server database, which is up and running, but your connection fails with the following error:

Could not acquire a connection from DataSource – Cannot create PoolableConnectionFactory

The error is most likely that the windows service “SQL Server Browser” is disabled.

To fix the problem:

  • Start -> Control Panel -> Systems & Security -> Administrative tools -> Services
  • SQL Server Browser -> right click -> properties

image2014-10-30 19-17-34

  • Set Startup type to “Automatic”
  • Apply -> Start

image2014-10-30 19-19-16

Now you should be able to connect to SQL Server from your Java application!

July 30, 2014

Java – Creating a simple retry command with function passing in Java 8

Filed under: Java — Brian @ 8:45 am
Tags: , , ,

Recently we have been working on an application that imports data from a number of different sources, where the network connection between us and each of these sources is not very reliable. So in our Gateway that makes these REST calls I wanted to be able to write a reusable piece of code that we could use in different calls, that in the event of a failure, would retry the command a few more times before finally giving up.

Java 7
I wanted to be able to write this retry logic and error handling code in one place and use it for a number of different method calls. There are several ways to do this, but previously in Java 7 I would have just written an abstract class with a single abstract method such as:

public abstract class RetryCommand<T> {
    private int maxRetries;

    public RetryCommand(int maxRetries) {
        this.maxRetries = maxRetries;
    }

    // This abstract command is the method that will be implemented 
    public abstract T command();

    public final T run() throws RuntimeException {
        try {
            return command();
        } catch (Exception e) {
            return retry();
        }
    }

    private final T retry() throws RuntimeException {
        System.out.println("FAILED - Command failed, will be retried " + maxRetries + " times.");
        int retryCounter = 0;
        while (retryCounter < maxRetries) {
            try {
                return command();
            } catch (Exception e) {
                retryCounter++;
                System.out.println("FAILED - Command failed on retry " + retryCounter + " of " + maxRetries + " error: " + ex );
                if (retryCounter >= maxRetries) {
                    System.out.println("Max retries exceeded.");
                    break;
                }
            }
        }
        throw new RuntimeException("Command failed on all of " + maxRetries + " retries");
    }
}

Then in my Gateway code, for each method that I want to wrap with my retry logic I would just do the following:

public class MyGateway {
    private RetryCommand<String> retryCommand;
    public MyGateway(int maxRetries) {
        retryCommand = new RetryCommand<>(maxRetries);
    }

    // Inline create an instance of  the abstract class RetryCommand
    // Define the body of the "command" method
    // Execute the "run" method and return the result
    public String getThing(final String id) {
        return new RetryCommand<String>() {
            public String command() {
                return client.getThatThing(id);
            }
        }.run();
    }
}

The reason for this layout was I could not pass a function as a parameter to a method, like I have done in Scala, Python, and C#. However, now that we have Java 8, we can finally pass functions as parameters using the handy features in java.util.function package!

Java 8
Java 8 uses Functional Interfaces, which are interfaces with a single abstract method. The package java.util.function defines a number of standard functional interfaces, so most of the time you will be able to use one of these. Some example functional interfaces are Function (function with return value and input param), Supplier (function with return value but no input param), and Consumer (function with input param but no return value). However, if one of these standard functional interfaces does not meet your needs you can always define your own. In the following example I used Supplier.

So now in Java 8 I would create a new RetryCommand class that has a “run” method which takes in a function:

import java.util.function.Supplier;

public class RetryCommandJava8<T> {
    private int retryCounter;
    private int maxRetries;

    public RetryCommandJava8(int maxRetries)
    {
        this.maxRetries = maxRetries;
    }

    // Takes a function and executes it, if fails, passes the function to the retry command
    public T run(Supplier<T> function) {
        try {
            return function.get();
        } catch (Exception e) {
            return retry(function);
        }
    }

    public int getRetryCounter() { return retryCounter; }

    private T retry(Supplier<T> function) throws RuntimeException {
        System.out.println("FAILED - Command failed, will be retried " + maxRetries + " times.");
        retryCounter = 0;
        while (retryCounter < maxRetries) {
            try {
                return function.get();
            } catch (Exception ex) {
                retryCounter++;
                System.out.println("FAILED - Command failed on retry " + retryCounter + " of " + maxRetries + " error: " + ex );
                if (retryCounter >= maxRetries) {
                    System.out.println("Max retries exceeded.");
                    break;
                }
            }
        }
        throw new RuntimeException("Command failed on all of " + maxRetries + " retries");
    }
}

So now in my gateway code, I would create my fancy new retry command executer:

public class MyGateway {
    private RetryCommandJava8<String> retryCommandJava8;
    public MyGateway(int maxRetries) {
        retryCommandJava8 = new RetryCommandJava8<>(maxRetries);
    }

    // Passing function using a Lamba expression 
    public String getThing(final String id) {
        return retryCommandJava8.run(() -> client.getThatThing(id));
    }
}

Also, this setup is fairly easy to unit test, here are some example tests:

import junit.framework.TestCase;

public class RetryCommandJava8Test extends TestCase {

    public String SUCCESS = "success";
    public int MAXRETRIES = 3;
    public int SECONDSTOWAIT = 0;
    RetryCommandJava8<String> retryCommandJava8;

    public void testRetryCommandShouldNotRetryCommandWhenSuccessful() {
        retryCommandJava8 = new RetryCommandJava8<>(MAXRETRIES, SECONDSTOWAIT);

        String result = retryCommandJava8.run(() -> SUCCESS);

        assertEquals(SUCCESS, result);
        assertEquals(0, retryCommand.getRetryCounter());
    }

    public void testRetryCommandShouldRetryOnceThenSucceedWhenFailsOnFirstCallButSucceedsOnFirstRetry() {
        retryCommand = new RetryCommandJava8<>(MAXRETRIES, SECONDSTOWAIT);

        String result = retryCommandJava8.run(() -> {
            if (retryCommand.getRetryCounter() == 0) throw new RuntimeException("Command Failed");
            else return SUCCESS;
        });

        assertEquals(SUCCESS, result);
        assertEquals(1, retryCommand.getRetryCounter());
    }

    public void testRetryCommandShouldThrowExceptionWhenMaxRetriesIsReached() {
        retryCommandJava8 = new RetryCommandJava8<>(MAXRETRIES, SECONDSTOWAIT);

        try {
            retryCommand.run(() -> {throw new RuntimeException("Failed");});
            fail("Should throw exception when max retries is reached");
        } catch (Exception e) { }
    }
}

Of course this example is a stripped down version of what we use, which does waits between retries, back off retries, and proper logging of errors, etc. I just wanted to use a retry command as my example code for trying out function passing in Java 8. However, I hope maybe you will find this useful if you are trying to get a working example going for your first use of function passing in Java 8.

If you are new to Java 8 (just like I am) I recommend reading Everything About Java 8.

March 7, 2010

MongoDB queries in Java using Conditional Operators

Filed under: Java,NOSQL — Brian @ 4:29 pm
Tags: , , , ,

When I first started using MongoDB, I used the interactive shell to learn the query syntax. The syntax is simple and straightforward. However, when I started using the Java driver I was not sure how to translate some of my command line queries into Java code, for example some of the conditional operators like “$in”. The Java documentation on the MongoDB website only showed how to use some basic conditionals like greater than and less than, but not for using other conditionals that use lists like the $in option.

A simple query for all cars with the make “Ford” that match any of several models listed:

SQL:

SELECT * FROM dbo.Cars 
WHERE make="Ford" 
AND model IN ("Galaxy","Mustang","Meteor")

MongoDB interactive shell:

db.cars.find( { "make":"Ford", "model":{ $in: ["Galaxy","Mustang","Meteor"] } } )

MongoDB Java driver:

BasicDBObject query = new BasicDBObject();
query.put("make", "Ford");
String models[] = new String[]{"Galaxy", "Mustang", "Meteor"};
query.put("model", new BasicDBObject("$in", models));
DBCursor resultsCursor = carsCollection.find(query);

October 11, 2009

Second Level Caching for Hibernate with Terracotta

Filed under: Caching,Hibernate — Brian @ 4:54 pm
Tags: , , ,

Traffic on our system has been increasing, putting more pressure on our database causing the response time of our application to degrade. Since we are currently using Hibernate in our project for a Java app that surfaces a REST API, we decided to look into caching strategies available. After looking around and trying a few technologies we decided to tested out the open source version of Terracotta.

To start let’s just explain the difference between First Level Caching and Second Level Caching in Hibernate.

First-level Cache

First level caching is associated with the Session object and is caching on a per transaction basis. An example would be an object where you make several updates to it as part of a single transaction. Hibernate, rather than submitting several update statements to the database, will pool the group of updates to the object into a single update statement.

Second-level Cache

Second level caching is associated with the SessionFactory object and maintains a cache of previously loaded objects accross transactional boundaries. These cached objects are available to any query for the same object, not just a single transaction. So in this case the entire application has access to retrieve any cached object. For example, if my app performs a lookup for a Product object describing a product, if the application later makes a request for the same product, this object can be retrieved from the Second-level Cache rather than making an additional trip to the database.

Why Terracotta?

Handles asynchronus queueing of writes. This feature offers some failover capabilities for when our database is not available. Originally we attacked the problem separately figuring we needed a caching strategy for reads and a fault tollerant queuing strategy for writes to deal with our current single point of failure (our database). Fortunately Terracotta has functionality that allows us to cover both of these cases.

The Developer Console. This tool allows you to monitor the traffic between your app, the Terracotta server, and your database. With the Developer Console we were able to test out different caching strategies to see which ones gave us the highest percentage of cache hits. For us developers, this was a major plus, having such a useful tool to aid us in tuning our cache settings. It made Terracotta feel much less like a black box.

Simple to install and configure. Since Terracotta just plugs in to Hibernate, there is no real development work required to get it up and running. Setting up a Terracotta server is easy, adding it to the application just meant minor additions to our pom.xml file and our hibernate settings file, and tuning our cache settings was relatively simple using the provided Developer Console.

Install and Start the Terracotta Server

  • Download Terracotta
  • Install by running the command: java -jar terracotta-3.1.0-installer.jar
  • Start the Terracotta server: ~/terracotta/terracotta-3.1.0/bin/start-tc-server.sh
  • Start the developer console: ~/terracotta/terracotta-3.1.0/bin/dev-console.sh

Add Terracotta support to your Maven/Java/Hibernate/Spring Application

  • Add the Terracotta dependencies to your Maven pom.xml file
  • <dependency>
        <artifactId>terracotta-hibernate-agent</artifactId>
        <version>1.0.0</version>
        <groupId>org.terracotta.hibernate</groupId>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <artifactId>terracotta-hibernate-cache</artifactId>
        <version>1.0.0</version>
        <groupId>org.terracotta.hibernate</groupId>
    </dependency>
    
  • Add the following properties to the hibernate properties section of the sessionFactory bean in your Spring database settings file (e.g., app/src/main/resouces/sping/database/database.xml)
  • <bean id="sessionFactory">
        <property name="dataSource" ref="dataSource"/>
        <property name="hibernateProperties">
            <props>
                ...
                <prop key="hibernate.cache.use_second_level_cache">true</prop>
                <prop key="hibernate.cache.provider_class">
                     org.terracotta.hibernate.TerracottaHibernateCacheProvider
                </prop>
                ...
            </props>
        </property>
    ...
    </bean>
    
  • Add a cache annotation to any Hibernate Entity in your application that you wish to have cached: @Cache(usage=CacheConcurrencyStrategy.READ_WRITE)
  • Create a Terracotta cache settings file. This is important because Terracotta by default NEVER evicts an object from cache. Once the memory allocated to Terracotta has been filled, Terracotta will start to write to disk unless eviction settings have been put in place. In the “Resources” folder of your application, create a file called “tc-hibernate-cache.xml”. To create the content for this file you can either look at the Terracotta Hibernate Configuration Reference Guide or use the Developer Console to generate a cache settings file using the GUI tool.

Running your project with Jetty

In our local development environments we use Jetty as our web server (Tomcat on production). To run your app from Maven using the Jetty plugin make sure you first do the following:

  • Set the MAVEN_OPTS environment variable with the “javaagent” parameter
  • export MAVEN_OPTS="-javaagent:/path/to/terracotta-hibernate-agent-1.0.0.jar -cp /path/to/terracotta-hibernate-cache-1.0.0.jar -Xmx512m"
    
  • NOTE: We appended to the MAVEN_OPTS environment variable the value “-Xmx512m” to tell Jetty to use 512mbs (the default is 128). We have found that with an application using Hibernate, 128 makes performance of the application sluggish.
  • Now start the app with the command: mvn jetty:run

Conclusion

Overall we have found Terracotta to be a useful tool. It requires very little effort to update an existing project using Spring/Hibernate to use it. Terracotta offers more than just Second Level Caching, but also handles queuing of writes and ensuring data is written to the SOR (System or Record) in the event the database is not available for a brief period. Also, stale cache entries are not an issue as well since Terracotta updates the cache immediately on any write made to the SOR. As a final note, the Developer Console is a really nice addition!

I hope this post helps!

Blog at WordPress.com.