CouchDb on Android

14 Nov 2011 CouchDb on Android

Posted by

Synchronising advanced data structures with a backend sucks. Especially when the sync is both ways; user updates to server, server updates to user. Couchbase for Mobile attempts to solve this issue by bringing CouchDb and it’s spiffy replication to Android.

What? Why?

Why would you ever want CouchDb on your phone, you ask. Well, along with giving you some fancy pants replication, which I’ll rant about soon, some of the benefits over Android’s built in SQLite support are:

  • Unstructured data storage in JSON documents
  • Map/reduce for querying
  • Versioning of documents
  • And more..

In a bit more detail ..

JSON documents with whatever you want in them

The data in CouchDb is document oriented, in that it stores and returns your content in JSON documents. These documents can have whatever fields and whatever JSON acceptable values you wish without any table changes. This gives you the power to support dynamically added fields. If you wanted to create a form that accepts additional user defined fields, CouchDb or something like it is what you want.

A CouchDb document, requested with a GET to /database/boo, is read and written something like

{
  "_id": "booboo",
  "_rev": "gibberish",
  "favouriteFoods": [ "Bangers", "Mash" ],
  "bodyWeight": 300
}

While a horribly non standard representation of a SQL table structure, that allows dynamically added fields, would be something like

Form#1()
FormValue#1(
  formId: 1,
  fieldName: "favouriteFoods",
  fieldValue: "Bangers",
  fieldType: "String"
)
FormValue#2(
  formId: 1,
  fieldName: "favouriteFoods",
  fieldValue: "Mash",
  fieldType: "String"
)
FormValue#3(
  formId: 1,
  fieldName: "bodyWeight",
  fieldValue: "300",
  fieldType: "Number"
)

Retrieved with queries like

SELECT * FROM Form
SELECT * FROM FormValue WHERE formId = 1

As you can see the CouchDb approach is much nicer, and in SQLite’s case the structure of Form and FormValue need to be defined when the table is created. Meaning updates to the structure of these tables involve fancy scripts or complicated SQLiteOpenHelper.onUpgrade functions.

Map/reduce functions

CouchDb has a different method of querying the data compared to normal SQL databases, depending on your usage this could be a deal maker or a deal breaker.

In CouchDB you create views of your data, written in Javascript, emitting a key and a value. You then look up the data you want using the key you specified. You can also reduce the values for summing or other fun things. For example:

"byName": {
  "map": function (doc) {
    if (doc.name) {
      emit(doc.name, 1);
    }
  },
  "reduce": function (key, values, rereduce) {
    return sum(values);
  }
}

When this has been indexed you will have an index like:

"Jim", 1
"Jim", 1
"Jack", 1

Calling it with reduce will give you:

"Jim", 2
"Jack", 1

Versioning of documents

CouchDb versions documents. Every time you update a document the previous revision will still exist and be accessible (until compaction which I’ll ignore), which may be useful for basic undo or rollback functionality.

Everything CouchDb gives you


And all of the other things CouchDb provides, like hugs, kittens and obese indexes (don’t quote me).

Replication

10 points if you get the reference
The main benefit you’ll see with using CouchDb for your application (presumably) is data synchronisation via replication.

CouchDb supports really easy start and stop replication, with it’s own conflict resolution. You could have multiple databases that haven’t seen each other in months, tell them about each other and each of them will eventually catch up with the other as far as documents go. This is awesome for “when I want” syncing, and offline editing of data that syncs when a connection is established again. Some use cases:

Taking notes

This would be great for a note taking application that supports mobile as well as web edits. The user could add notes on both sides and sync to make both sides consistent when they want / when a wireless connection is available.

Real time when available

You could have replication going any time the application is in the foreground. This would allow for applications that work in real time but don’t explode when the network is lost.

And I’m sure other more exciting things my boring brain chose to shield me from.

I’m sold. How do I make the magic happen?

First up, make an Android 2.x project in your favourite Java IDE (Eclipse), which you have obviously already set up for Android development.

Next you need to get the latest version of CouchDb for Android (still in Developer preview), here. This website also covers the instructions for getting your project CouchDb friendly, basically it’s “run an Ant script and hope it’s not doing anything too horrible”. Without going too into it, it’s configuring your project for CouchDb, Erlang (which CouchDb uses behind the scenes) and some funky native libraries.

You can also add Ektorp, a Java API for CouchDb with Android support. This wraps the CouchDb calls so you don’t have to generate your own HTTP requests. That’s really up to you, but the code examples below use this, and Couchbase recommends it as far as Android CouchDb libraries go.

I’m listening..

If you’re anything like me, you’ve probably skipped to this section or closed the page when you didn’t instantly see code, but here goes.

Starting CouchDb

First we’ll set up a CouchbaseDelegate in our main Activity. You use this to capture startup errors and kick off initialisation steps / replication when the database has started.

ICouchbaseDelegate couchDelegate = new ICouchbaseDelegate() {
  @Override
  public void exit(String error) {
    // Show the error to the user
  }

  @Override
  public void couchbaseStarted(String host, int port) {
    // CouchDb has started, set up your connector
    // in our case initialise Ektorp
    startEktorp(host, port);
  }
};

Now we’ll actually request for CouchDb to be started, referencing the delegate we created above. In my case I’m calling this when the activity starts.

private void startCouch() {
  CouchbaseMobile couch = new CouchbaseMobile(
    getBaseContext(), couchDelegate
  );
  couchConnection = couch.startCouchbase();
}

CouchbaseMobile will do its magic and eventually call your delegate.

Connecting to the database

Now that CouchDb has started, we’ll connect using the host and port the delegate gave us.

clientBuilder = new AndroidHttpClient.Builder();
httpClient = (AndroidHttpClient)clientBuilder
  .host(host)
  .port(port)
  .build();
dbInstance = new StdCouchDbInstance(httpClient);
// The last parameter true will create the database
// if it doesn't exist.
dbConnector = dbInstance.createConnector(DATABASE_NAME, true);

The dbConnector is now ready for normal database use. Let the CRUD begin.

Creating the initial design documents

CouchDb is pretty useless without a view over your data (unless you know ids or want to just work with all documents). So after CouchDb has started you should create (or update) the design documents specifying your views.

designDocId = "_design/slyncy";
byNameView = "byName";
byNameMap = "function(doc) {if (doc.name) emit(doc.name, doc);}";
DesignDocument dDoc = new DesignDocument(designDocId);
dDoc.addView(byNameView, new DesignDocument.View(
  byNameMap
));
dbConnector.create(dDoc);

Putting a list adapter on top of the view

Assuming you’re displaying the data in a list, you’d want to wrap a view in a custom CouchbaseViewListAdapter. There’s a horrible implementation in the example code showing basic view mapping.

ViewQuery query = new ViewQuery()
  .designDocId(designDocId)
  .viewName(byNameViewName)
  .descending(true);

// The third parameter specifies that you should listen to
// the _changes feed from CouchDb enabling this means the
// list will automatically update for you when data changes
adapter = new CouchbaseViewListAdapterImpl(
  dbConnector, query, true
);

The final parameter (_changes: true) is pretty powerful. It works like Android content provider data change notifications, allowing the list to update when the data changes. Delete a document and it will be removed from the list.

Something Ektorp is missing with their base CouchbaseViewListAdapter class is support for include_docs when querying the view. This means your document will have to be emitted as an indexed value, which could lead to obese indexes if your data set is big enough.

Starting replication

Starting replication is pretty simple. You build a ReplicationCommand, specifying the source, target and continuous (once until complete (false), or continually run until shut down (true)).

pushReplicationCommand = new ReplicationCommand.Builder()
  .source(DATABASE_NAME)
  .target(syncUrl)
  .continuous(true)
  .build();

EktorpAsyncTask pushReplication = new EktorpAsyncTask() {
  @Override
  protected void doInBackground() {
    dbInstance.replicate(pushReplicationCommand);
  }
};
pushReplication.execute();

Switch the source and target around for pull replication. Depending on how your system is you would probably want both (user updates -> server, server updates -> user).

CRUD

After the connection is set up, CRUD is simple. You just call create/update on a document (where document is a Map in my case), or delete with a given id and revision.

dbConnector.create(document);
dbConnector.get(Map.class, id);
dbConnector.update(document);
dbConnector.delete(id, revision);

Summing up

So, as you can see, connecting to CouchDb on Android is fairly simple, if a little verbose, but that’s the Java way.

You can find a working (you know, sometimes) example application here, based on AndroidGrocerySync.

Concerns

Data Security

Sure, replication sounds dandy, but how do I stop the user replicating changes of documents they don’t own?

CouchDb has a method for this called Validation Functions. Basically you can define a Javascript function that will look at an update and allow you to reject the change. The last parameter of these functions is a user context. So using this you can work out who’s writing and if their write is valid. For example:

function (newDoc, savedDoc, userCtx) {
  if (savedDoc) {
    // This is an update, make sure the user
    // isn't watching the world burn.
    if (newDoc.userName !== userCtx.name) {
      throw({ unauthorised: 'You shall not pass ' + userCtx.name });
    }
  }
}

Internal phone security

A completely uninformed concern (I’m sure Couchbase has thought of it) is security from other applications running on the phone. What if you have private data you don’t want Mr Naughty’s application to push to his cloud of deceit?

Completely based on a Stack Overflow thread here, it seems that connecting to Couch and access the database itself is secure enough, but the data is written to external storage, which isn’t secure at all.

Conclusion

I’ll sum it up in pros and cons form.

Pros:

  • CouchDb on Android makes replication easy. It’s basically allout of the box, just point and say go
  • The JSON strucure of CouchDb documents gives your application a lot of flexibility and a nice way to store data.
  • The actual code behind it isn’t too painful or that far away from ‘typical Android’ code.

Cons:

  • Starting up CouchDb is slow, from what I’ve heard mostly due to loading an Erlang interpeter. You’ll see the lag between starting the application and initialising CouchDb if you run the example application
  • Map/Reduce while providing some friendliness also removes the possibility of on the fly SQL style queries
  • Packaging CouchDb with your application adds about 10mb of bloat to your installation.

All up I’ll give it 3.33 out of 5 overly intelligent mice, ready to take over the world.

Links / More interesting reads:

Note: Images randomly pilfered from Google Image Search.

No Comments

Leave a Reply