It’s a simple question, often asked by project managers, data scientists, and quality engineers on every data engineering project when that first data source is ingested. How do we know the data that has been ingested into a data lake is accurate and error-free?

Cloudflare Dev Workshop 2020 In mid-February, I had the privilege to attend the first Melbourne Cloudflare dev event. This was just one of a series of sessions they ran across the country to reach out to developers and help educate people around their thinking and the...

Out at Shine's various client sites, our teams often meet to discuss the pros and cons of various technical solutions. And in the past, there was one particular Shine manager who, if he was in attendance, would regularly pipe up and ask the question: what's the problem we're actually trying to solve?

The year was 1997. The Red Hot Chili Peppers were musing on love and the motions of amusement park rides, Pathfinder landed on Mars and Leonardo DiCaprio drew Kate Winslet as per one of his French associates.  It was around this time I had heard about a thing called “Java”, a fancy new language everyone was talking about. The word on IRC was that it was based on work Sun Microsystems had originally done for embedded software on set-top boxes and other smart appliances.

When I started out as a developer the internet was made of wood and owl feathers, held together with spit, pluck, gumption and whatever else it is that you kids today no longer seem to have (job security? the possibility of owning your own home? a habitable climate?). We had to chisel our code out of the rocks 26 hours a day, 10 days a week, wait two years for it to compile, and the only way of knowing if it worked was if the old wise woman of the company divined the error messages in the entrails of a junior developer. I am now that wise old woman, and so I must pass on the things I have learned.