In Flux, we have made improvements in recent versions to handle database or network failures and recover from such failures gracefully without needing to restart Flux. Sometimes, it would be nice to notify administrators about these failures and they can act upon them in case of an unscheduled outage. Flux is dependent on the database for maintaining job states and schedules, so fixing this problem is critical.

BoneCP, a popular JDBC Connection Pool has an interesting and useful feature which allows the connection pool to automatically recover from such DB outages and it replays the transactions when a healthy connection becomes available. This feature became available since its 0.6.5 release and I had tested this before and found this to be very useful when configured with Flux.

In BoneCP, we can implement a connection hook which gets triggered when database failures occur and it is easy to configure in Flux to use BoneCP as the datasource pool provider. In case of any database failure, you may want to send an email notification or initiate a SNMP trap so other downstream systems can be handled accordingly.

Here is a basic DatabaseShutdownHook would look like:

import com.jolbox.bonecp.ConnectionHandle;
import com.jolbox.bonecp.hooks.AbstractConnectionHook;
import com.jolbox.bonecp.hooks.AcquireFailConfig;

import java.util.Date;

public class DatabaseShutdownHook extends AbstractConnectionHook {
    @Override
    public boolean onConnectionException(ConnectionHandle connection, String state, Throwable t) {
        // handle notifications here: SNMP or SMTP
        System.out.println("Database down at " + new Date());
        return super.onConnectionException(connection, state, t);
    }

    @Override
    public boolean onAcquireFail(Throwable t, AcquireFailConfig acquireConfig) {
        // handle notifications here: SNMP or SMTP
        System.out.println("Failure to acquire connection at " + new Date() + ". Retry attempts remaining : " + acquireConfig.getAcquireRetryAttempts());
        return super.onAcquireFail(t, acquireConfig);
    }

}

Let us now see how to configure BoneCP as a Data source in Flux.

import com.jolbox.bonecp.BoneCPDataSource;
import flux.Configuration;
import flux.DatabaseType;
import flux.Engine;
import flux.Factory;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.sql.Connection;
import java.sql.SQLException;

public class FluxEngine {

    private static Context initialContext;

    static {
        try {
            System.setProperty(Context.INITIAL_CONTEXT_FACTORY, "org.apache.naming.java.javaURLContextFactory");
            System.setProperty(Context.URL_PKG_PREFIXES, "org.apache.naming");
            initialContext = new InitialContext();
            initializeDataSource();
        } catch (NamingException e) {
            // log exception
        } catch (SQLException e) {
            // log exception
        }
    }

    public static void initializeDataSource()  throws NamingException, SQLException {
        BoneCPDataSource ds = new BoneCPDataSource();
        ds.setJdbcUrl("jdbc:mysql://localhost:3306/flux710?relaxAutoCommit=true");
        ds.setUsername("flux");
        ds.setPassword("secret");
        ds.setMinConnectionsPerPartition(10);
        ds.setMaxConnectionsPerPartition(50);
        ds.setPartitionCount(1);
        ds.setConnectionHook(new DatabaseShutdownHook());// Required only if you need notifications.
        ds.setTransactionRecoveryEnabled(true);// Important: This should be enabled
        ds.setAcquireRetryAttempts(10);//default is 5
        ds.setAcquireRetryDelay(10000);// default is 7 secs
        ds.setReleaseHelperThreads(5);

        Connection con = ds.getConnection();
        if (con != null) {
            initialContext.rebind("FluxDataSource", ds);
            con.close();
        }
        System.out.println("DataSource configured.");
    }

    public static void main(String[] args) throws Exception {
        Factory f = Factory.makeInstance();
        Configuration c = f.makeConfiguration();
        c.setDatabaseType(DatabaseType.MYSQL);
        c.setDataSource("FluxDataSource");
        Engine engine = f.makeEngine(c);
        engine.start();
        System.out.println("Engine started.");
    }
}

Data sources should be made available via JNDI for Flux to use them. So, in this example I used Tomcat JNDI support to expose the BoneCP datasource via JNDI. There are also other ways to expose this via JNDI when running Flux as a standalone server. Tomcat JNDI was easy to configure though as you can see from the code above. The BoneCP data source should be configured for transaction recovery and you can set the number of recovery attempts and retry delay.

I have created a simple flow chart which has a Timer Trigger followed by a Java Action. The timer is configured to fire every 15 seconds for 5 times as shown below.

process_data.png

Here is a sample output of running this job in Flux configured with BoneCP and MySQL database. I shutdown MySQL server when this job was running and you can see the BoneCP recovery attempts and after a while I brought back the MySQL server and BoneCP successfully recovered. Now, you can see Flux execute the last occurrence of this job successfully.

DataSource configured.
Engine started.
Done processing data

Done processing data

Done processing data

Done processing data

Database down at Sun Oct 10 12:18:12 MDT 2010
Oct 10, 2010 12:18:12 PM com.jolbox.bonecp.ConnectionHandle markPossiblyBroken
SEVERE: Database access problem. Killing off all remaining connections in the connection pool. SQL State = 08007
Oct 10, 2010 12:18:12 PM com.jolbox.bonecp.MemorizeTransactionProxy invoke
SEVERE: Connection failed. Attempting to recover transaction on Thread #70
Oct 10, 2010 12:18:14 PM com.jolbox.bonecp.hooks.AbstractConnectionHook onAcquireFail
SEVERE: Failed to acquire connection Sleeping for 10000ms and trying again. Attempts left: 10. Exception: java.net.ConnectException: Connection refused: connect
Failure to acquire connection at Sun Oct 10 12:18:14 MDT 2010. Retry attempts remaining : 10
Oct 10, 2010 12:18:26 PM com.jolbox.bonecp.hooks.AbstractConnectionHook onAcquireFail
SEVERE: Failed to acquire connection Sleeping for 10000ms and trying again. Attempts left: 9. Exception: java.net.ConnectException: Connection refused: connect
Failure to acquire connection at Sun Oct 10 12:18:26 MDT 2010. Retry attempts remaining : 9
Oct 10, 2010 12:18:38 PM com.jolbox.bonecp.hooks.AbstractConnectionHook onAcquireFail
SEVERE: Failed to acquire connection Sleeping for 10000ms and trying again. Attempts left: 8. Exception: java.net.ConnectException: Connection refused: connect
Failure to acquire connection at Sun Oct 10 12:18:38 MDT 2010. Retry attempts remaining : 8
Oct 10, 2010 12:18:48 PM com.jolbox.bonecp.MemorizeTransactionProxy invoke
SEVERE: Recovery succeeded on Thread #70
Done processing data

The following BoneCP dependencies are required in the classpath:
bonecp-0.7.0.jar
guava-r07.jar
slf4j-api-1.6.1.jar
slf4j-jdk14-1.6.1.jar (Note: You can use any logger bindings supported by slf4j. Using JDK logger for simplicity and one less jar.)

To configure Tomcat JNDI, you need to have these jars in classpath:
catalina.jar
tomcat-juli.jar

Let me know if you have any trouble setting up this in Flux.

(Update: 10/13): Updated to reference the latest stable 0.7.0 release which now uses Google guava instead of the retired Google collections library and fixed slf4j dependency requirement.

Possibly Related Posts:


Try-with-resources (originally known as Automatic Resource Management) is one of the Project Coin proposals that made its way into recent JDK 7 builds. Traditionally, developers had to manually terminate any resources (Files, Database connections, etc) they use in their applications and sometimes it was painful to keep track of those things and failing to do so may have serious problems such as resource leaks which could potentially lead to application failures that are hard to debug and triage. The burden of managing resources is no more a developer’s headache as this will be natively supported in Java 7.

C# has “using” blocks and C++ destructors served a similar purpose. This is a more sought after language feature for years and Joshua Bloch brings this to real for Java developers with some syntax sugaring. While this construct looks alien to Java, I believe this is something we have to accept and move forward as this solves common programming errors. IDEs can fill this space to support intelligent code completion for ARM constructs and make it easy for developer adoption. The other disadvantage outlined in the proposal is increased coupling between the language specification and libraries. To support this feature, a class must implement a designated interface AutoCloseable to make it eligible for automatic resource management.

Let us look at the example used in the ARM proposal, copying a file involving two resources.

    public void copy(String src, String dest) throws IOException {
        InputStream in = new FileInputStream(src);
        try {
            OutputStream out = new FileOutputStream(dest);
            try {
                byte[] buf = new byte[8192];
                int n;
                while ((n = in.read(buf)) >= 0)
                    out.write(buf, 0, n);
            } finally {
                out.close();
            }
        } finally {
            in.close();
        }
    }

With the new try-with-resources construct, we should be able to rewrite the above code with much simplicity:

    public void copy(String src, String dest) throws IOException {
        try(InputStream in = new FileInputStream(src);
            OutputStream out = new FileOutputStream(dest)){
            byte[] buf = new byte[8192];
            int n;
            while ((n = in.read(buf)) >= 0)
                out.write(buf, 0, n);
        }
    }

The bloated code we use these days will become more crisp with fewer lines of code, while continuing to maintain readability.

Java APIs that involve dealing with resources are mostly retrofitted to support this new AutoCloseable interface. This feature brings heap of benefits to the JDBC world where this may prove very useful when managing database connections and its subsidiaries.

Here is the traditional way of programming in JDBC where the user has to manage JDBC resources such as ResultSet, Statement and Connection.

        Connection conn = null;
        Statement stmt = null;
        ResultSet rs = null;
        try {
            conn = DriverManager.getConnection("jdbc:h2:mem:test", "sa", "");
            stmt = conn.createStatement();
            rs = stmt.executeQuery("SELECT * FROM INFORMATION_SCHEMA.USERS");
            while (rs.next()) {
                System.out.println(rs.getString(1));
            }
        } catch (SQLException e) {
            // handle exception
        } finally {
            if (rs != null) {
                try {
                    rs.close();
                } catch (SQLException e) {
                }
                rs = null;
            }
            if (stmt != null) {
                try {
                    stmt.close();
                } catch (SQLException e) {
                }
                stmt = null;
            }
            if (conn != null) {
                try {
                    conn.close();
                } catch (SQLException e) {
                }
                conn = null;
            }

        }

You guessed it right, there is so much boilerplate code that is involved here. Here is the same code with our try-with-resources construct.

        try(Connection conn = DriverManager.getConnection("jdbc:h2:mem:test", "sa", "");
            Statement stmt = conn.createStatement();
            ResultSet rs = stmt.executeQuery("SELECT * FROM INFORMATION_SCHEMA.USERS")) {// no trailing semicolon in the last line of the block containing automatic resource definition
            while (rs.next()) {
                System.out.println(rs.getString(1));
            }
        } catch (SQLException e) {
            // handle exception
        }

Wow, it is compact and powerful. JDBC support for the new try-with-resources construct will be part of JDBC 4.1 specification and the implementation is available in the recent JDK 7 build 112.

In these try-with-resources examples, an exception thrown during the lifetime of a try-with-resources statement receives priority over an exception thrown by an automatically generated close invocation. The proposal addresses automatic retention of those suppressed exceptions which are associated with the primary exception thrown during the try-with-resources statement.

The following methods are added to java.lang.Throwable to support extraction of these suppressed exceptions.

    public synchronized void addSuppressedException(Throwable exception);

    public synchronized Throwable[] getSuppressedExceptions();

You may want to go through the recent JavaOne presentation which details Project Coin features that are included in Java 7, slated for release in mid 2011.

Do you think Project Coin will become a crown jewel of Java 7, since Lambda expressions and Project Jigsaw are no longer the main contenders? In the coming days, I will be exploring many other language features that are included in Java 7 and help you decide the crown jewel of Java 7.

Keep watching this space for more Java 7 delicacies 🙂

Possibly Related Posts: