Comet technics (ie. polling, long-polling, streaming) and more recently, the WebSocket protocol, have made possible various webpush applications.

Today, when you want to enable realtime push on your own Java-based webapp, you have several solutions:

• Use an externalized system that will handle that for you (like Pusher), avoiding you the providing of any additional server
• Use a commercial solution (like Kaazing or Diffusion), generally implementing a Message Broker underneath, with both embedded and standalone server
• Use a container-specific api (like Jetty Continuation, Tomcat’s CometProcessor, Grizzly CometHandler or Netty WebSocket), but your application will be strongly coupled with it
• Or use one of the few Open Source frameworks available (like Atmosphere, cometD or JWebSocket)

In this article, I will focus on Atmosphere. After a brief presentation of the framework, I will demonstrate how easy it is to make push-capable applications, whatever the container you use and the nature of your clients.

Before we start, you can take a look at the following video, that demonstrates what I achieved thanks to the Atmosphere framework (more details below, of course):

http://youtu.be/1Abv88t5igc

What is Atmosphere ?

Atmosphere is a Java OpenSource framework led by Jean-Francois Arcand. Started around 2009, it evolved significantly and powers today websites like smartmoney.com. The project is very active (daily commits, lively mailing list, 300 Github followers). Currently in version 0.9, Atmosphere is close to the major 1.0 release planned for May.

One of the strength of the framework is that it is container agnostic: you can deploy an Atmosphere application in Jetty, Tomcat, JBoss, Grizzly,… Indeed, Atmosphere will seamlessly use specific compatibility modules and enable WebSocket depending on the detected container. The user does not need to learn container-specific Comet or WebSocket implementations anymore, only the way Atmosphere runs.

Atmosphere takes advantage of asynchronous I/O assuming the container supports it (e.g. Tomcat APR Connector). Indeed, as realtime connections are all maintained server side (either with WebSocket or Comet), clients are permanently connected. If you don’t use advanced mechanisms, available threads in connection pool can run out very quickly. Here comes async I/O, that breaks the 1 client / 1 thread paradigm and allows one server to handle numerous connected clients asynchronously. Of course, if your container can’t run async I/O, Atmosphere will fallback with common blocking thread per request.

Atmosphere can run in two modes: embedded or standalone. The first one is the classic approach and the one you generally choose when you start from scratch. The second one, called Nettosphere, relies on the combination of the Netty container and the Atmosphere framework, allowing you to extend your existing Java webapp with realtime push capabilities in a non-intrusive way. Moreover, Nettosphere fits well when you need to do integration test on your realtime resources as you can instantiate it programmatically on demand, similarly to the Jersey-test-framework.

Usually, when you want to achieve standard comet (long-polling, streaming) communication, you don’t have to worry about the underneath container: they are compatible with Atmosphere. About WebSocket nonetheless, the compatibility will depend on the version of the WebSocket specification implemented by the container running Atmosphere. To this day, Jetty, Grizzly, Netty and even Tomcat are some WebSocket-capable containers.

A variety of modules and plugins are bundled with the framework, letting the user benefit from Atmosphere in various way: you are able to write Atmosphere applications in Java, Scala, JRuby and Groovy, and use other frameworks such as Jersey, GWT, Spring or JSF.

Another interesting thing: Atmosphere was built to scale thanks to clustering plugins. These plugins actually wrap a clustering layer implementation among JMS, Redis, XMPP or Hazelcast to name a few…  So if one server is not enough, you have tools to scale-out as needed.

Among all Atmosphere’s components, we find the jQuery plugin: this Javascript API connects your web pages to your server and offers methods to exchange realtime data. It can detect client’s capabilities and switch between protocols as necessary via a fallback mechanism. Therefore, the same user experience can be offered from IE6 to Chrome18.

From now on, we will focus on the Jersey’s Atmosphere extension, very convenient as it largely reduces lines to code thanks to JAX-RS implementation (annotations, json mapping, etc…).

Grab your keyboard, launch your Eclipse!

To demonstrate the use of Atmosphere, I chose to build an application that draws realtime events on a map: user can generate events by clicking on the map, and server can generate random events. A generated event (by either client or server) is delivered to each connected clients.

Thus, this application – called MapPush (I was not able to find anything more explicit!) – is composed of:

• The client, a simple HTML page that uses Javascript/jQuery to process logic
• The server, a JEE webapp backed by Atmosphere and Jersey frameworks

Note: the application is hosted on Github at the address http://github.com/ncolomer/MapPush. All the snippets below are extracted from this project. Feel free to browse the source code or clone the project!

Project bootstrap

In Eclipse, create a Maven project with a “webapp” archetype. As Atmosphere is available on Sonatype repositories (snapshots, releases), you can simply add the following to your pom.xml :

<!-- Sonatype repositories -->
<repositories>
	<repository>
		<id>Sonatype snapshots</id>
		<url>https://oss.sonatype.org/content/repositories/snapshots</url>
	</repository>
	<repository>
		<id>Sonatype releases</id>
		<url>https://oss.sonatype.org/content/repositories/releases</url>
	</repository>
</repositories>

<!-- Dependencies -->
<dependencies>
	<!-- Atmosphere -->
	<dependency>
		<!-- Atmosphere's Comet Portable Runtime (CPR) -->
		<groupId>org.atmosphere</groupId>
		<artifactId>atmosphere-runtime</artifactId>
		<version>0.9.7</version>
	</dependency>
	<dependency>
		<!-- Atmosphere's Jersey module -->
		<!-- Transitivity will pull all necessary dependencies -->
		<!-- ie. Jersey 1.10 core, server, etc... -->
		<groupId>org.atmosphere</groupId>
		<artifactId>atmosphere-jersey</artifactId>
		<version>0.9.7</version>
	</dependency>
	<dependency>
		<!-- Atmosphere's jQuery plugin -->
		<groupId>org.atmosphere</groupId>
		<artifactId>atmosphere-jquery</artifactId>
		<version>0.9.7</version>
		<type>war</type>
	</dependency>
	<!-- Jersey's JSON mapper -->
	<dependency>
		<groupId>com.sun.jersey</groupId>
		<artifactId>jersey-json</artifactId>
		<version>1.10</version>
	</dependency>
</dependencies>

To start the framework we need a servlet, and not any… an AtmosphereServlet! In case the Atmosphere’s Jersey module is detected at load time, the framework will wrap around the Jersey Servlet (ContainerServlet) to load our resources and extend it with Atmosphere capabilities (IoC, annotations, etc…). Therefore, Jersey’s init-params are still available.

Open the web.xml file of your project and paste the following snippet:

<!-- Atmosphere -->
<servlet>
    <description>AtmosphereServlet</description>
    <servlet-name>AtmosphereServlet</servlet-name>
    <servlet-class>org.atmosphere.cpr.AtmosphereServlet</servlet-class>
    <init-param>
        <!-- Jersey base package of your resources -->
        <param-name>com.sun.jersey.config.property.packages</param-name>
        <param-value>org.mappush.resource</param-value>
    </init-param>
    <init-param>
        <!-- Enable Jersey's JSON mapping feature -->
        <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
        <param-value>true</param-value>
    </init-param>
    <load-on-startup>0</load-on-startup>
</servlet>
<servlet-mapping>
    <servlet-name>AtmosphereServlet</servlet-name>
    <url-pattern>/api/*</url-pattern>
</servlet-mapping>

We are now ready to code both client and server application.

The server

If you have already used the official JAX-RS implementation, Jersey, you won’t be disapointed. For others, you’ll find out that programming Atmosphere push APIs is really… dead simple!

Among the essential pieces of Atmosphere, one is called Broadcaster. This is the object that manages connected clients and delivers broadcasted messages to them. A connected client is seen as an AtmosphereResource by the framework.

In the push communication lifecycle, we can define three steps:

• the client sends a request, that is suspended (in case of comet) or upgraded (in case of websocket) by the server.
• then, server and client can exchange data : server generally broadcasts/pushes data to clients, and in case of a duplex protocol like WebSocket, client can send data back to it.
• finally, either one closes the connection.

Atmosphere offers two convenient ways to handle this lifecycle: you can choose between annotation or programmatic API (or even a combination of both), the last one allowing more customization from my point of view.

• To suspend a request, use the @Suspend annotation, or simply return a SuspendResponse
• To broadcast data, use the  @Broadcast annotation and simply return a Broadcastable. For a more specific use of broadcast mechanism, you are able to manually deal with the Broadcaster and its broadcast(…) methods to push messages to all, a subset or a particular AtmosphereResource.

Here is an exemple of Atmosphere resource using the main concepts we have described above:

@Path("/")
@Singleton
public class EventResource {

	private final Logger logger = LoggerFactory.getLogger(EventResource.class);

	private EventListener listener;
	private EventGenerator generator;

	private @Context BroadcasterFactory bf;

	/**
	 * Programmatic way to get a Broadcaster instance
	 * @return the MapPush Broadcaster
	 */
	private Broadcaster getBroadcaster() {
		return bf.lookup(DefaultBroadcaster.class, "MapPush", true);
    }

	/**
	 * The @PostConstruct annotation makes this method executed by the
	 * container after this resource is instanciated. It is one way
	 * to initialize the Broadcaster (e.g. by adding some Filters)
	 */
	@PostConstruct
	public void init() {
		logger.info("Initializing EventResource");
		BroadcasterConfig config = getBroadcaster().getBroadcasterConfig();
		config.addFilter(new BoundsFilter());
		listener = new EventListener();
		generator = new EventGenerator(getBroadcaster(), 100);
	}

	/**
	 * When the client connects to this URI, the response is suspended or
	 * upgraded if both client and server arc WebSocket capable. A Broadcaster
	 * is affected to deliver future messages and manage the
	 * communication lifecycle.
	 * @param res the AtmosphereResource (injected by the container)
	 * @param bounds the bounds (extracted from header and deserialized)
	 * @return a SuspendResponse
	 */
	@GET
	@Produces(MediaType.APPLICATION_JSON)
	public SuspendResponse<String> connect(@Context AtmosphereResource res,
			@HeaderParam("X-Map-Bounds") Bounds bounds) {
		if (bounds != null) res.getRequest().setAttribute("bounds", bounds);
		return new SuspendResponse.SuspendResponseBuilder<String>()
				.broadcaster(getBroadcaster())
				.outputComments(true)
				.addListener(listener)
				.build();
	}

	/**
	 * This URI allows a client to send a new Event that will be broadcaster
	 * to all other connected clients.
	 * @param event the Event (deserialized from JSON by Jersey)
	 * @return a Response
	 */
	@POST
	@Path("event")
	@Consumes(MediaType.APPLICATION_JSON)
	public Response broadcastEvent(Event event) {
		logger.info("New event: {}", event);
		getBroadcaster().broadcast(event);
		return Response.ok().build();
	}

	// ...

}

Listen for client messages

In case of a duplex protocol, we said a client can send data back to the server using the connection. But how do we handle such messages? Atmosphere provides a listener mechanism, implementable via interfaces such as the AtmosphereResourceEventListener or its WebSocket specialization, the WebSocketEventListener. The WebSocket interface exposes useful methods like onConnect, onHandshake, onDisconnect, or onMessage. Listeners are declared when the client connects (ie. when suspending a response).

public class EventListener extends WebSocketEventListenerAdapter {

	private final Logger logger = LoggerFactory.getLogger(EventListener.class);

	@Override
	public void onMessage(WebSocketEvent event) {
		Bounds bounds = JsonUtils.fromJson(event.message(), Bounds.class);
		if (bounds == null) return;
		logger.info("New bounds {} for resource {}",
				event.message(), event.webSocket().resource().hashCode());
		AtmosphereRequest req = event.webSocket().resource().getRequest();
		req.setAttribute("bounds", bounds);
	}

}

Filter your broadcasted messages

You may also have observed the init() method… but what is done inside exactly ?

A possibility offered by the framework is the ability to include logic before delivering message to each connected client. We can achieve that thanks to the BroadcastFilter interface and its specialization, the PerRequestBroadcastFilter interface. The first interface allows to execute logic once (common to all clients) whereas the second one can apply to each client according to their associated context (session data, first connection headers, etc…).

Each BroadcastFilter can CONTINUE or ABORT a broadcast. While the broadcast is not aborted, the filter chain is executed in order (BroadcastFilters first, PerRequestBroadcastFilters then) and finally delivered (or not) to each client. The following snippet is an example of PerRequestBroadcastFilter implementation:

public class BoundsFilter implements PerRequestBroadcastFilter {

    private final Logger logger = LoggerFactory.getLogger(BoundsFilter.class);

    @Override
    public BroadcastAction filter(Object originalMessage, Object message) {
        return new BroadcastAction(ACTION.CONTINUE, originalMessage);
    }

    @Override
    public BroadcastAction filter(AtmosphereResource res,
            Object originalMessage, Object message) {
        logger.info("BoundsFilter triggered for AtmosphereResource {} "+
				"with message {}", res.hashCode(), message);
        Event event = (Event) message;
        try {
            Bounds bounds = (Bounds) res.getRequest().getAttribute("bounds");
            if (bounds == null) throw new NoBoundsException("no bounds");
            if (bounds.contains(event)) {
                String json = JsonUtils.toJson(event); // Manual serialization
                return new BroadcastAction(ACTION.CONTINUE, json);
            } else {
                return new BroadcastAction(ACTION.ABORT, message);
            }
        } catch (NoBoundsException e) {
            logger.info("Applying default action CONTINUE, cause: {}",
					e.getMessage());
            String json = JsonUtils.toJson(event); // Manual serialization
            return new BroadcastAction(ACTION.CONTINUE, json);
        } catch (Exception e) {
            logger.info("Filter BoundsFilter aborted, cause: {}",
					e.getMessage());
            return new BroadcastAction(ACTION.ABORT, message);
        }
    }

}

The client

Now that all is ready server side, we are able to connect our clients to the realtime endpoint. If you plan to use WebSocket, several clients are compatible with Atmosphere. Let’s focus on Java and Javascript ones:

• In java, you find several projects like Java-WebSocket or async-http-client. Some webapp containers also provide a WebSocket client implementation (e.g. Jetty)
• In Javascript, most of recent browsers implement the WebSocket interface. But, in case the browser is not compatible with WebSocket, this API can’t fallback into another protocol. Here comes the jQuery Atmosphere Plugin and its fallback mechanism.

In our case, you probably guessed, we’ll use the Atmosphere jQuery plugin. Please note that due to the jQuery dependency, we have to import the jQuery library in addition to the Atmosphere plugin.

Once done, connect to the server is no more complicated than the following connect() javascript routine:

var endpoint;
function connect() {

    var callback = function callback(response) {
        // Websocket events.
        if (response.state == "opening") {
            console.log("Connected to realtime endpoint");
        } else if (response.state == "closed") {
            console.log("Disconnected from realtime endpoint");
        } else if (response.transport != 'polling' &&
                response.state == 'messageReceived') {
            if (response.status == 200) {
                var data = response.responseBody;
                if (data.length > 0) {
                    statsAgent.notify();
                    console.log("Message Received: " + data +
                            " & detected transport is " + response.transport);
                    var json = JSON.parse(data);
                    mapsAgent.drawEvent(json);
                }
            }
        }
    };

    var bounds = mapsAgent.getBounds();
    var header = bounds.southLat + "," + bounds.northLat +
            "," + bounds.westLng + "," + bounds.eastLng;
    endpoint = $.atmosphere.subscribe(url, callback, {
        transport: 'websocket',
		/* available transports: websocket, jsonp, long-polling,
			polling, streaming */
        attachHeadersAsQueryString: true,
        headers: {"X-Map-Bounds": header}
    });
}

You can see that we attach headers when connecting. They are used here to transmit some client context when connecting (actually, the current bounds of the map). As the WebSocket handshake doesn’t allow the client to pass any headers, they have to be serialized in the query string, justifying the attachHeadersAsQueryString: true entry. On the server side, the query string will be translated to headers so application wise, you don’t have to care about the difference.

To process message pushed from the server, we can add a callback. It will be triggered on each received message, passing a response (Javascript object) containing values as status, state, transport etc…

The endpoint variable – that stores the connection – was made global to be used when you want to disconnect the client from the realtime endpoint or push data to the server. The following snippet shows you the two corresponding javascript routines.

function disconnect() {
	$.atmosphere.unsubscribe();
	endpoint = null;
}

function update(bounds) {
	console.log("### Map bounds changed:", JSON.stringify(bounds));
	if (!endpoint) return;
	endpoint.push(JSON.stringify(bounds));
}

Testing your WebSocket resources

In addition to regular browser testing, you can easily try your realtime URIs with a shell and cURL:

curl -v -N -XGET http://localhost:8080/MapPush/api

• -v/--verbose: Make the operation more talkative
• -N/--no-buffer: Disable buffering of the output stream
• -X/--request <command>: Specify request command to use

With the second option, you will be able to observe all data sent by the server. Nonetheless, note that you’ll not be able to send data back to it.

cURL also allows you to send headers with the request:

boundsHeader='48.0,49.0,2.0,3.0'
curl -v -N -XGET http://localhost:8080/MapPush/api -H "X-MAP-BOUNDS: $boundsHeader"

• -H/--header <line>: Custom header to pass to server

You can also go deeper and analyse WebSocket frames with tools such as ngrep or Wireshark: these are a bit more complex tools but it may become very useful in some situations!

Conclusion

Atmophere provides a powerful ecosystem that simplifies the creation of push applications and makes easy full-duplex communication between a server and any kind of client. Its intensive use of async I/O and its ready-to-use clustering plugins give it both performance and scalability. Moreover, the project is under Open source Apache license, the community is growing quickly and the last 0.9 version is stabilizing fast.

In short, the Atmosphere framework has a bright future :)

Additional Resources

• Atmosphere project is hosted on GitHub:
  • Download the Atmosphere’s whitepaper
  • Take a look at the samples
  • Browse the Javadoc
• The community is reachable via Atmosphere’s Google Group
• You can also follow Atmosphere via Jean-Francois Arcand’s blog and Twitter

My first approach was to put a custom index.php file on the old domain that permanently redirects all root requests to my new domain:

<?php
// Permanent redirection
header("HTTP/1.1 301 Moved Permanently");
header("Location: http://www.ncolomer.net/");
exit();
?>

But what about the old URLs that map my blog articles, the one that was indexed by Google? In a such situation, the best way to do the least possible harm (ie. do not lost SEO robots) is to redirect all old URLs to my new domain. This can be done using a simple .htaccess containing the following:

RewriteEngine on
RedirectMatch 301 ^(.*)$ http://www.ncolomer.net$1

Thus, all requests on colomern.net/2011/03/bonjour-tout-le-monde for instance will instantly be redirected (301 Moved Permanently) to ncolomer.net/2011/03/bonjour-tout-le-monde.

The wiki page is available here.

Edit: I added some tips to optimize Ubuntu when running on a SSD

In my precedent blog post, we introduced the way to install and start playing with Redis. Now, I propose you to go further, with a more advanced use case. In my personnal project development (f.ollow.us), I faced a new problem:

How can we manage efficiently a fixed number of “slots” that have a TTL (i.e. Time To Live) and can be held (i.e. be available or not) in a high concurrency context? Don’t look further, Redis is the solution!

The problem

To be more explicit about what we try to solve here, i’ll give you another example. This one involves phone operators. You are aware that they dispose of a (large) range of phone number that can be used or vacant. Sometimes, customers break their contract and their phone number becomes available after a while. Sometimes, new customers subscribe a phone plan and they are given a random phone number. It’s exactly the same problem here.

Actually, we have to answer at least four constraints:

• Know if a phone number is available or not
• Retrieve an available phone number (preferably, the oldest one)
• Ask for them transactionally (race condition support)
• Expire phone numbers with time (they become available)

What about the Redis TTL feature you’ll say, does it solve the problem? I think the response is no. Ho yeah, we can probably imagine a solution, in which you select all used slots (not expired, existing in the db), store them in a set, and finally make a difference with another set that contains all known slots… but these operations take time, CPU and memory.

Why bother when the solution is so close? The Redis “sorted set” data type is our solution.

The solution

What is a Sorted Set?

Accordingly to the official redis.io website, sorted sets are “non repeating collections of Strings”. In reality, this simple definition hides the most advanced Redis data type. Sorted sets are advanced Redis sets, that associate each element with a score. This score permits elements to be taken permanently in order (and not ordered afterwards), resulting great performances when accessing to the set entries.

Redis offers useful commands to access such data sets by score (ZSCORE) or by rank (ZRANK), or select only a range of them (once again) by score (ZRANGEBYSCORE) or by rank (ZRANGE). For the three last commands, time complexity is equivalent to O(log(N)), with N the number of elements in the sorted set: no longer doubt about performances! To get a full list of Sorted sets dedicated commands, you can have a see to this web page.

We don’t need more to achieve our task!

Code it with Java

The code you’ll find below is a Java implementation of what we need to solve the “slot management” problem. It boils down to a simple class named PhoneNumberManager, which contains some static methods.

To converse with a Redis instance, I used the Jedis API (the Java Redis client). Jedis instances are retrieved from the RedisManager described in my precedent post about Redis.

First, we have to populate the sorted set with all phone numbers that we wish to use. These numbers are stored as simple strings. Each element is associated with the default 0 score. This initialization is done by using the init() method.

Once initialized, we can retrieve available phone numbers with the method getAvailableSlot(). A retrieved phone number have to be held or released via respective methods hold() and release().

There are also three more useful methods:

• getSize() returns the current number of elements in the sorted set
• getExpirationTimestamp() returns the expiration timestamp for a given phone number
• isAvailable() checks the availability of a phone number

Each of these methods are atomic (unique read/write atomic access to Redis) except the getAvailableSlot() method, which use transaction mechanism (WATCH, MULTI, EXEC) to support race condition on giving available phone number.

I think it’s time to paste some Java code…

import java.util.Date;
import java.util.Set;

import redis.clients.jedis.Jedis;
import redis.clients.jedis.Response;
import redis.clients.jedis.Transaction;

/**
 * This class manage all available phone numbers via static methods.
 * It uses Jedis instances from a JedisPool.
 */
public class PhoneNumberManager {

	/**
	 * This method populates the "numbers" sorted set.
	 * @param number The number of slots to allocate
	 */
	public static void init(int number) {
		Jedis jedis = RedisManager.getJedis();
		// Delete the key if exists
		jedis.del("numbers");
		// Populate the sorted set
		int number = Integer.parseInt("600000000");
		for (int i = 0; i < 10; i++) {
			jedis.zadd("numbers", i, "+33" + Integer.toString(number++));
		}
		RedisManager.returnJedis(jedis);
	}

	/**
	 * Returns the number of phone number.
	 * @return the number of phone number
	 */
	public static long getSize() {
		Jedis jedis = RedisManager.getJedis();
		Long card = jedis.zcard("numbers");
		RedisManager.returnJedis(jedis);
		return card;
	}

	/**
	 * This methods searches for an available phone number.
	 * The operation is atomic: The phone number (if found) is owned
	 * only by the claimant.
	 * You can release the phone number by using {@link #release()}.
	 * You can hold the phone number by using {@link #hold()}.
	 * @return A free phone number
	 * @throws Exception If no one is available
	 */
	public static String getAvailableSlot() throws Exception {
		Jedis jedis = RedisManager.getJedis();
		// Start watching the sorted set (preventing any modification)
		jedis.watch("numbers");
		try {
			String number = null;
			// Try to get an available phone number
			Set<String> availableNumber = jedis.zrangeByScore("numbers",
					Double.NEGATIVE_INFINITY, new Date().getTime(), 0, 1);
			if (!availableNumber.isEmpty()) {
				number = availableNumber.iterator().next();
				// Start transaction to remove the phone number from list
				Transaction t = jedis.multi();
				Response<Long> poped = t.zrem("numbers", number);
				t.exec();
				if (poped.get() != 1)
					throw new Exception("Problem getting available" +
						" phone number, please retry");
				else return number;
			} else {
				throw new Exception("No phone number available");
			}
		} catch (Exception e) {
			throw e;
		} finally {
			jedis.unwatch();
			RedisManager.returnJedis(jedis);
		}
	}

	/**
	 * This method holds a phone number for a given time.
	 * @param number The phone number to hold
	 */
	public static void hold(String number) {
		long timestamp = (long)(new Date().getTime()*1E-3 + duration);
		Jedis jedis = RedisManager.getJedis();
		jedis.zadd("numbers", timestamp, number);
		RedisManager.returnJedis(jedis);
	}

	/**
	 * This method release a given phone number.
	 * @param number The phone number to release
	 */
	public static void release(String number) {
		Jedis jedis = RedisManager.getJedis();
		jedis.zadd("numbers", 0, number);
		RedisManager.returnJedis(jedis);
	}

	/**
	 * This method returns the expiration timestamp of a phone number
	 * @param number The phone number
	 * @return The phone number expiration timestamp, null if not found
	 */
	public static Long getExpirationTimestamp(String number) {
		Jedis jedis = RedisManager.getJedis();
		Double score = jedis.zscore("numbers", number);
		RedisManager.returnJedis(jedis);
		return score.longValue();
	}

	/**
	 * This method checks the availability of a phone number.
	 * @param number The phone number
	 * @return false if slot does not exist or already expired,
	 * true otherwise
	 */
	public static boolean isAvailable(String number) {
		Long timestamp = getExpirationTimestamp(number);
		if (timestamp == null) return false;
		if (timestamp <= (long)(new Date().getTime()*1E-3)) return false;
		return true;
	}

}

The community around Redis is very active, resulting a lot of client for many languages: Java, Node, Ruby, PHP, etc. On this post, after a brief “How to set up Redis”, I will focus on one of the most active Java API, Jedis.

First steps with Redis

To familiarize with, as first approach, you can try the excellent online interactive tutorial.

Once you are ready to go further, simply run these commands on a shell to install Redis:

wget http://redis.googlecode.com/files/redis-2.2.11.tar.gz
tar xzf redis-2.2.11.tar.gz
cd redis-2.2.11
make

Then, you can execute the Redis server with:

src/redis-server

And finally, interact with it by running the Redis client:

src/redis-cli

Throughout your tests, you will probably generate a lot of keys. Here is a useful Redis command to delete all keys, once.

flushdb

Jedis, the Java Redis API

The Jedis API is an open project hosted on Github. It supports most of Redis features, including connection handling & pooling, transactions, pipelining or publish/subscribe. Please refer to the Related links part below to get some pointers about Jedis.

Jedis contributors done things well. They created a class that enable the API to work in a multithreaded environment. Here is a quotation from their wiki page.

A single Jedis instance is not threadsafe! To avoid these problems, you should use JedisPool, which is a threadsafe pool of network connections. You can use the pool to reliably create several Jedis instances, given you return the Jedis instance to the pool when done.

You can store the pool somewhere statically, it is thread-safe.

No sooner said than done! We can write a simple RedisManager to handle our pool of jedis instances. Here is a way to do so:

import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisPoolConfig;

public class RedisManager {
	private static final RedisManager instance = new RedisManager();
	private static JedisPool pool;
	private RedisManager() {}
    public final static RedisManager getInstance() {
        return instance;
    }
	public void connect() {
		// Create and set a JedisPoolConfig
		JedisPoolConfig poolConfig = new JedisPoolConfig();
		// Maximum active connections to Redis instance
		poolConfig.setMaxActive(10);
		// Tests whether connection is dead when connection
		// retrieval method is called
		poolConfig.setTestOnBorrow(true);
		/* Some extra configuration */
		// Tests whether connection is dead when returning a
		// connection to the pool
		poolConfig.setTestOnReturn(true);
		// Number of connections to Redis that just sit there
		// and do nothing
		poolConfig.setMaxIdle(5);
		// Minimum number of idle connections to Redis
		// These can be seen as always open and ready to serve
		poolConfig.setMinIdle(1);
		// Tests whether connections are dead during idle periods
		poolConfig.setTestWhileIdle(true);
		// Maximum number of connections to test in each idle check
		poolConfig.setNumTestsPerEvictionRun(10);
		// Idle connection checking period
		poolConfig.setTimeBetweenEvictionRunsMillis(60000);
		// Create the jedisPool
		pool = new JedisPool(poolConfig, "localhost", 6379);
	}
	public void release() {
		pool.destroy();
	}
	public Jedis getJedis() {
		return pool.getResource();
	}
	public void returnJedis(Jedis jedis) {
		pool.returnResource(jedis);
	}
}

Note that JedisPool class depends on the Apache commons-pool library. You can download this lib from the main page of the project.

The token problem

This problem is about transaction. Redis offers some native atomic functions like INCR or DECR, but also a transactional mechanism, also called “check-and-set”. This is what we will use below.

The problem I had is quite simple: I have to set keys that expire (TTL). Some Threads scan asynchronously for expired keys. If a thread detects that a key is absent, it executes some http requests which is time expensive (and a blocking io). We have to avoid that two or more threads executes the same request!

This problem can be solved by using a token. This is how I made it work:

• The thread test the presence of a key. If the key exists, it ignores it and search for another key. This test has to be atomic.
• If the key does not exist, the thread set a provisional value for this key that does not expire: It the the token. Other threads that will ask for this key in the future will show that it is already set (or in progress).
• After the execution, the worker thread, that get responses to requests (or not), set the final value, that expire after a fixed time.

Here is the code:

	// Get Jedis instance from pool
	Jedis jedis = redisManager.getJedis();
	jedis.watch("key:"+myKey);
	String value = jedis.get("key:"+myKey);
	if (type == null) {
		// No value for the key, we set the token atomically
		Transaction t = jedis.multi();
		Response<String> response = t.set("key:"+myKey, "Searching");
		t.exec();
		if (response != null) {
			// The key was correctly set
			// You can do stuff here...
			// And finally update the key
			jedis.setex("key:"+myKey, TIMEOUT, myValue);
		}
	}
	// Return the jedis instance to the pool
	jedis.unwatch();
	redisManager.returnJedis(jedis);

Related links

• To learn more on how Redis handles transactions, you can read the official related topic
• The page presenting all available client APIs
• Some extra links about Jedis:
  • The download page, where you can get latest Jars
  • The wiki page, where concept and usage are explained
  • Some extra documentation

For that matter, prefer a silent and energy efficient HDD (e.g. “green” products from Seagate, Samsung or Western Digital). These HDD rotate more often at 5400 rpm. That’s enough to do daily backups (not so IO intensive).

Here is a step by step procedure to change your Time Capsule disk in no more than 15 minutes!

Changing the disk

Please note that all data on your old drive will be lost. You’ll have to reconfigure your Mac’s Time Machine once you finished, and start a full backup.

Here is the underside of my Time Capsule

Remove the rubber. Don’t be afraid, it is just stuck, you’ll be able to stuck it again at the end. Once done, you should see something like that.

Unscrew all the visible screws (ten that fix the hull and four that fix the HDD) and lift the metal part. Be careful, a fan is attached to the part you are removing. If you don’t want to unplug the fan, place the metal part as in the following image.

Unstuck the little temperature sensor from the disk surface (the solid foam square). Unplug the disk SATA and power connectors.

You are now up to extract the disk carefully.

Finally, plug and place the new disk in the same way than the old one. Stuck the temperature sensor, replace the metal part and screw it. Power your Time Capsule. The status light should blink in orange and, after a while, be fixed green.

Use the Apple Airport utility to configure your Time Capsule if necessary (disk tab once connected).

About disk partitioning

It may be useful when you want to share data on your network (public file depot) and use Time Machine backups. Partioning disk seems to be feasible, although more binding. I have not experienced myself (the adapter is missing), but some people already achieved this. You need:

• An USB to SATA adapter, recycled from an USB SATA disk drive or easily foundable on eBay.
• A Mac to format your disk with Mac OS partition table and type. I tried with Ubuntu, without success.

1. Connect your disk to your adapter and plug the USB cable on your Mac.
2. Open the MacOS Disk Utility or spotlight it. You should see your USB hard drive on the left panel. Select it.
3. Click to the Partition tab, and create as many partition as your need in your Time Capsule.
   Note that you have to choose Apple partition table to be recognized by your Time Capsule. You can use MacOS Extended (Journalized) as partition format, even though Time Capsule automatically format your partitions if it was not done or in wrong format. You can also name your partitions, they will be displayed by the Time Capsule.
4. Unmount your USB disk from your Mac, unplug your hard drive from the adapter, and plug it into the Time Capsule.
5. Once powered and connected to your network, display the Time Capsule settings via Airport Utility and go to the disk tab. You should see your partitions. You can access them with the finder by connecting to your Time Capsule.

Set the backup frequency

By default, the Time Machine backup frequency is set to 3600 seconds, i.e. one hour. You can set this value by opening a terminal and execute the following bash command:

sudo defaults write /System/Library/LaunchDaemons/com.apple.backupd-auto StartInterval -int 86400

Here, the backup frequency is set to 86400 seconds, i.e. one day.

You have to restart for this change to take effect.

Related links

• (FR) Allain Le Gallou partitioning Time Capsule page
• (FR) MacBidouille change Time Capsule disk page

The main feature of OpenStreetMap is to provide a map, as Google do. You can access it on there main site or embed it in your own website. But you can also access OSM data in other ways than viewing them in a browser…

OpenStreetMap available API

Indeed OpenStreetMap provides several API to create, modify and read geographic content. In addition, some API provide advanced querying features (e.g. geocoding or reverse geocoding features).

All the available API are listed below:

• API: OpenStreetMap editing API for fetching and saving raw geodata from/to the OpenStreetMap database.
• Xapi: OpenStreetMap extended read-only API, based on a modified version of the standard API, that provides enhanced querying capabilities, as bouding-box or X-path.
• Nominatim: OpenStreetMap search engine, for searching OSM data by name and address and generating synthetic addresses of returned OSM points (i.e. reverse geocoding).

Xapi and Nominatim are what will interest us today. These API are used as one of the sources for the Search box on the OpenStreetMap home page, but also powers other websites like MapQuest.

Because of official OpenStreetMap servers are mostly overloaded and out of date, their use are not recommended. Fortunately, other providers are available (you can see the additional link section for more information). We will focus on one of the most relevant providers: the MapQuest Open Initiative.

MapQuest is a mapping company based in the United States and wholly-owned by AOL. In 2010, The company announced their support for OpenStreetMap, making MapQuest the first large online mapping service to embrace OSM. They since earmarked $1 million in resources to help improve the OSM data in the United States, with the stated intention of possibly using OSM data for their maps of the U.S. in the future.

The MapQuest Open Initiative is one of the results of their support efforts. In addition to localized websites, it provides an API to read and query OSM data. These data are frequently updated from OSM (every 15 minutes for map data, every 5 minutes for search data and daily for routing data). So there is no problem to get fresh data quickly!

Using OpenStreetMap API

The data model

In order to use API via MapQuest, we first have to understand the data model used by OpenStreetMap. OSM data is stored as XML. Basically, it is a list of instances of 3 data primitives:

• Node, a single geospatial point.
• Way, an ordered interconnection of at least 2 nodes that describe a linear feature (e.g. street, footpath, railway line, river, …).
• Relation, used to group way or node objects that are geographically related (i.e. connected or adjacent to one another).

To summarize, querying the API should return all or part of the following XML schema:

<?xml version='1.0' standalone='no'?>
<osm version='0.6' generator='xapi: OSM Extended API'
 xmlns:xapi='http://www.informationfreeway.org/xapi/0.6'
 xapi:uri='/api/0.6/*[amenity=hotel]'
 xapi:planetDate='200803150826'
 xapi:copyright='2008 OpenStreetMap contributors'
 xapi:instance='zappy2'>
  <node id='218963' lat='52.5611324692581' lon='-1.79024812573334' timestamp='2006-03-22T16:47:48+00:00' version='1' changeset='2211'>
  </node>
  <node id='331193' lat='53.7091237972264' lon='-1.50282510180841' timestamp='2007-03-31T00:09:22+01:00' version='1' changeset='2211'>
    <tag k='amenity' v='hotel'/>
  </node>
  ...
  <way id='4958218' timestamp='2007-07-25T01:55:35+01:00' version='1' changeset='2211'>
    <nd ref='218963'/>
    <nd ref='331193'/>
    ...
    <tag k='amenity' v='hotel'/>
    <tag k='building' v='hotel'/>
  </way>
  <relation id='123456' timestamp='2007-10-25T03:05:34Z' version='32' changeset='2211'>
    <member type='node' ref='331193' role=''/>
    <member type='node' ref='331194' role=''/>
    ...
    <tag k='amenity' v='hotel'/>
    <tag k='operator' v='Premier Inns'/>
    <tag k='type' v='operators'/>
  </relation>
</osm>

Querying the API

The previous XML result can be obtained by querying Xapi. The querying engine allows the use of filters also called predicates. There are 3 types of predicates:

• Tag predicates, to match any node that has a key with the wanted value. For instance, use the link api/0.6/node[amenity=hospital] to get all nodes in which tag amenity exists and equals hospital
• BBox (i.e. bounding box) predicates, which limit the result to the selected bounding box region. For instance, the link api/0.6/node[bbox=-6,50,2,61] will return all objects localized in the region limited by the [left=-6°, bottom=50°, right=2°, top=61°] box.
• Child element predicates, to select objects matching X-path like conditions. For instance, the link api/0.6/way/14310041 will return the way object matching 14310041 id.

However, there is limitations on the use of predicates. Currently each request is limited to one tag predicate and one bbox predicate. Child element predicates are used to query a specific OSM object with a known ID.

A simple example

To illustrate the use of our API, imagine that we want to know the highway type (e.g. motorway, trunk, primary,…) of a road for a given position. To do so, we have to execute 2 requests:

• The first one is a reverse geocoding request, which will give us the street address of a location and the OSM ID of the associated way.
• The second is a Xapi request, which will give us highway information for a give highway id, like its type.

Assume that the location correspond to the following geo coordinates (48.888320; 2.249920).

The first step can be accomplished by sending the following GET request (follow the link to see the result):

http://open.mapquestapi.com/nominatim/v1/reverse?lat=48.888320&lon=2.249920

In the result XML, we can see the ...osm_type="way" osm_id="14310041"... string. We can conclude that the returned object type is way, and its OSM id is 14310041.

These data will allow us to write and execute the second request (follow the link to see the result):

http://open.mapquestapi.com/xapi/api/0.6/way/14310041

Among the result’s tags, we can find the <tag k="highway" v="motorway"/> key/value tag. We finally can assert that the highway type is motorway!

Additional links

• To learn more about OpenStreetMap, you can have a see on their FAQ.
• The community greatly highlights the ability for common user to contribute to the project (create and update geographic content). To learn more on how to contribute, you can read their beginners’ guide.
• To learn more on the MapQuest API, see their developer’s guide.
• To learn more on using Xapi, you can read the dedicated wiki page and test it on MapQuest Xapi page.
• To learn more on using Nominatim, you can read the dedicated wiki page and test it on MapQuest Nominatim page.
• Other geocoding web services using OpenStreetMap exist on the web. You can have a look at Worldwide Geocoding or GeoNames.

Here is some procedures and advices to understand how to handle post multipart form with JEE.

The API we will use is called FileUpload from Apache commons API.

As usual, some web resources about this API:

• Apache FileUpload main page
• … Where you can find the user guide
• … And some JavaDoc

You can download the latest version of this FileUpload API (1.2.1 – 18 January 2008) using this link. Simply add the .jar library to your Eclipse Project to use it.

The first thing is to create our JSP file, which will contain our form:

 ...
 <%= request.getAttribute("message") %>
 <form action="." enctype="multipart/form-data" method="post">
 <input accept="image/*" name="file" type="file" />
 <input type="submit" value="Send" />
 </form>
 ...

Then, write the web.xml deployment descriptor as following in order to map correctly our servlet.

<?xml version="1.0" encoding="UTF-8"?>
<web-app id="WebApp_ID" version="2.4"
	xmlns="http://java.sun.com/xml/ns/j2ee"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">
	<!-- WebApp Name -->
	<display-name>MyPostServlet</display-name>
	<!-- Servlet declaration -->
	<servlet>
		<servlet-name>post</servlet-name>
		<servlet-class>foo.post.Servlet</servlet-class>
	</servlet>
	<!-- Servlet url mapping -->
	<servlet-mapping>
		<servlet-name>post</servlet-name>
		<url-pattern>/post</url-pattern>
	</servlet-mapping>
	<!-- Default welcome page -->
	<welcome-file-list>
		<welcome-file>post</welcome-file>
	</welcome-file-list>
</web-app>

We can now create our main servlet which will handle GET and POST requests:

• The doGet method simply dispatch to our form.jsp,
• The doPost method parse the multipart request and then… call the doGet method.

Before process the post as a multipart one, you have to test it (see line 21).

When you parse the multipart post request with the ServletFileUpload.parseRequest(req) method, you get a List as return (see line 37).

Finally, loop with an Iterator to get all the FileItem of the request (see line 43).

public class Servlet extends HttpServlet {

	private static final String CACHE_PATH = "./temp/";
	private static final int CACHE_SIZE = 100*(int)Math.pow(10,6);
	private static final int MAX_REQUEST_SIZE = 10*(int)Math.pow(10,6);
	private static final int MAX_FILE_SIZE = 1*(int)Math.pow(10,6);

	@Override
	protected void doGet(HttpServletRequest req, HttpServletResponse resp)
			throws ServletException, IOException {

		// Redirecting to the test form
		req.getRequestDispatcher("form.jsp").forward(req, resp);

	}

	@Override
	public void doPost(HttpServletRequest req, HttpServletResponse resp)
			throws ServletException, IOException {

		if (ServletFileUpload.isMultipartContent(req))
		{
			// Create a factory for disk-based file items
			DiskFileItemFactory factory = new DiskFileItemFactory();
			// Set factory constraints
			factory.setRepository(new File(CACHE_PATH));
			factory.setSizeThreshold(CACHE_SIZE);

			// Create a new file upload handler
			ServletFileUpload upload = new ServletFileUpload(factory);
			// Set overall request size constraint
			upload.setSizeMax(MAX_REQUEST_SIZE);
			upload.setFileSizeMax(MAX_FILE_SIZE);

			// Parse the request
			@SuppressWarnings("unchecked")
			List<FileItem> items = upload.parseRequest(req);

			// Process the uploaded items
			Iterator iter = items.iterator();
			while (iter.hasNext())
			{
				FileItem item = (FileItem) iter.next();

				if (item.isFormField())
				{
					// The FileItem is a (text) form field
					// You can save it to a database
				}
				else
				{
					// The FileItem is a file
					// You can write it on the disk
				}
			}
		}
		// Add a validation message to the request
		req.setAttribute("message","The file was uploaded");
		// Redirect as a GET request to print the form again
		doGet(req,resp);

	}

}

The procedure splits in three steps:

• Obtain the debug certificate (official procedure here)
• Sign up for a Maps API (official procedure here)
• Start playing with MapView !

Step one: obtain the debug certificate

Each installed Android SDK have a unique debug certificate. You’ll have to get the MD5 Fingerprint of this certificate in order to pretend a Maps API key.

First, we have to locate the debug.keystore file on the computer and note down its path. This file is generally located at these following paths:

# Linux / MacOS
~/.android/debug.keystore

# Windows
C:\Documents and Settings\<user>\.android\debug.keystore

Please note that .android is a hidden directory in both OS.

Then, we must generate the MD5 Fingerprint of our debug keystore. To do so, we will use keytool, a tool bundled with any java JDK. You can find it in the following paths (or similar).

# Linux
$JAVA_HOME/bin/keytool
# MacOS
JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
$JAVA_HOME/bin/keytool

# Windows
C:\Program Files\Java\jdk1.6.0_23\bin\keytool.exe

Open a terminal or a dos box and change the working directory to the java bin/ path seen above (the keytool directory). You can now launch the following command:

keytool -list -alias androiddebugkey -keystore "path_to_your_debug.keystore" \
-storepass android -keypass android

Note down the generated MD5 Fingerprint present in the result. It should look like:

01:E9:C8:C0:BF:E3:4C:5F:A3:EE:28:33:BD:10:D3:19

Step two: sign up for a Maps API key

We have now to register the debug certificate with the Android Maps API service.

Connect to the Maps API Signup page and enter your MD5 Fingerprint. (Read and) Accept terms of use and validate the form.

You just obtain the key you need to load map tiles in your Android application. It should look like:

16ahjArI_Z393VYwzeGYJFT62pmG3LuKgZ0jy9w

Step three: start playing with MapView !

To draw a MapView in your application, simply paste the following code snippet in a XML layout. Don’t forget to write your own API key in the android:apiKey property.

<com.google.android.maps.MapView
 android:id="@+id/MyMapView"
 android:layout_width="match_parent"
 android:layout_height="match_parent"
 android:clickable="true"
 android:apiKey="16ahjArI_Z393VYwzeGYJFT62pmG3LuKgZ0jy9w"
 />

You’ll probably need to change the API key for a production one. Instead of putting the API key into the XML layout, prefer define a string value by editing the res/values/strings.xml as following:

<?xml version="1.0" encoding="utf-8"?>
<resources>
	<string name="apikey">16ahjArI_Z393VYwzeGYJFT62pmG3LuKgZ0jy9w</string>
</resources>

And replace the android:apiKey value, in a XML layout, by:

android:apiKey="@string/maps_apikey"

Related links

• More information about Google APIs add-on here (and the associated javadoc API Reference)