[caption id="attachment_1628" align="alignnone" width="300" caption="Weird Al's not the only one who knows how to go large with Jackson"][/caption]
A couple of months ago I was asked to build a processor that would take a JSON file; perform a few elementary checks and transformations and upload the resulting records into a Couch DB. I hadn't done any JSON, or Couch and, naturally, the whole thing had to be done yesterday - but at first glance it didn't look like much of a challenge.
However, looking a little deeper, my processor was going to be part of a replication process across two databases housed in separate enterprises. The replication was going to be based on a daily snapshot - each JSON file would be a copy of the entire database - and a few back-of-the-envelope calculations suggested that the files may become rather large (40+GB) over time.
“Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis mollis auctor libero quis cursus. Mauris ultricies lacus est. Praesent interdum mauris sed turpis imperdiet, et laoreet dolor tincidunt. Donec porttitor neque a lacus condimentum ullamcorper.
Aliquam aliquet mauris a ante mattis luctus. Cras tellus ante, placerat sed ex vel, tincidunt lobortis velit.”