Processing Huge JSON Files with Jackson
[caption id="attachment_1628" align="alignnone" width="300" caption="Weird Al's not the only one who knows how to go large with Jackson"]
[/caption]
A couple of months ago I was asked to build a processor that would take a JSON file; perform a few elementary checks and transformations and upload the resulting records into a Couch DB. I hadn't done any JSON, or Couch and, naturally, the whole thing had to be done yesterday - but at first glance it didn't look like much of a challenge.
However, looking a little deeper, my processor was going to be part of a replication process across two databases housed in separate enterprises. The replication was going to be based on a daily snapshot - each JSON file would be a copy of the entire database - and a few back-of-the-envelope calculations suggested that the files may become rather large (40+GB) over time.
23 December, 2011
/ 1 Comment