Will Swift be the next king of server side development?

Swift throne

In June 2015, Apple announced at WWDC that they were open-sourcing the Swift language and its runtime libraries. On December 3rd that year they made good on their promise. In this post I’d like to talk about why this is significant, particularly for server-side developers.

Messages in the sky

contrailscience.com_skitch_skitched_20130315_131709

One of the projects that I’m currently working on is developing a solution whereby millions of rows per hour are streamed real-time into Google BigQuery. This data is then available for immediate analysis by the business. The business likes this. It’s an extremely interesting, yet challenging project. And we are always looking for ways of improving our streaming infrastructure.

As I explained in a previous blog post, the data/rows that we stream to BigQuery are ad-impressions, which are generated by an ad-server (Google DFP). This was a great accomplishment in its own right, especially after optimising our architecture and adding Redis into the mix. Using Redis added robustness, and stability to our infrastructure.  But – there is always a but – we still need to denormalise the data before analysing it.

In this blog post I’ll talk about how you can use Google Cloud Pub/Sub to denormalize your data in real-time before performing analysis on it.

YOW! Connected 2015: Conference Report

yowconnected

Last week I had the privilege of attending the YOW! Connected conference in Melbourne, Australia. YOW! Connected offers a look at all the interesting things that are happening in the mobile and IoT (Internet of Things) space, from the perspective of both software developers and UX designers.

On the mobile front it included a variety of talks relating to both the iOS and Android platforms and yes, even a little bit of Windows.

In general all the talks that I went to were pretty good,  but here I will write about a few that particularly interested me.

Biometric authentication with Touch ID

Touch-ID-640x384

Passwords have become such a pain in the neck. Switching between different apps and websites you will usually find different password security policies. One site will inform you no more than 10 characters, other a 6 digits pin. While some other sites will insist in making you change your password every 15 days, and so on.

Why does this have to be so complicated, and hard to manage? Wouldn’t it be so much easier to use something that you have as a key instead of having to remember so many different passwords across a plethora of different sites!?

Well, there is something that most of us have. Something we were born with. Something we use every single day. That is our fingerprint. Each fingerprint has a unique pattern that can be used as key. In this blog post, I’ll talk about my experience of playing around with Apple’s Touch ID.

Securing your Spring App using 2FA

2a_02 Not so long ago, a good old username and password were considered more than enough to secure access to our applications and favourite web sites. But back then, nobody could have imagined the countless ways in which a hacker can now get a hold of our precious login credentials. From software exploits to social engineering, security has been drawn into the spotlight like never before, and software developers must really think hard about security when building any type of software solution. In this blog post, I’ll explain how you can secure your Spring applications using 2FA (Two Factor Authentication).

Running Geospatial Queries with GeoTools

4331986510_bb69fd7a3c

Geospatial information analysis normally requires pretty complex calculations and transformations between different representation types.  The Google Maps APIs are a great tool because they hide all the complexity of these operations. However when the geospatial information that you need to analyse is not from Google Maps, things get more complicated. Operations like finding polygons representing geographical places, or finding polygons that intercept other polygons, require a lot of time and intricate code to have any acceptable  solution. At Shine we are working on a solution for a big telecommunication customer that requires being able to query large geospatial databases without degrading performance.

Using ‘HugeCollections’ To Manage Big Data

OLYMPUS DIGITAL CAMERA

Introduction

When writing complex software things don’t always go as planned. You implement a new feature that works perfectly well locally and in a test environment, but when your code hits the real world everything falls apart. Sadly, that’s one of the things we all have to deal with as software developers. On a recent project for a major telecommunications client we needed to be able to process more than 20 million records every night. That equated to 5GB of storage and unfortunately the environment where our process was running had up to 4GB of memory.

Processing such a vast amount of data brings a lot of challenges along with it, especially when you also need to combine it with a few more million records that are located in a database. Adding code to retrieve associated information and transform raw data might take an extra few milliseconds per each record. However, when you repeat that operation 20 million times those few milliseconds can easily turn into hours.

So there we were asking ourselves why it was taking so long. Is it an index we forgot to add to the database? A network latency problem? These things can be very hard to pin down.

We needed to think outside the box to get around this one.

Put On Your Streaming Shoes

Saturday-Night-Fever-2

The Kick-Off Meeting

It went something along the lines of:

  • Client: “We have a new requirement for you..”
  • Shiners: “Shoot..”
  • Client: “We’d like you to come up a solution that can insert 2 million rows per hour into a database and be able to deliver real-time analytics and some have animated charts visualising it. And, it should go without saying, that it needs to be scalable so we can ramp up to 100 million per hour.”
  • Shiners: [inaudible]
  • Client: “Sorry, what was that?”
  • Shiners: [inaudible]
  • Client: “You’ll have to speak up guys..”
  • Shiners: “Give us 6 weeks”

We delivered it less than 4.