Scheduling BigQuery jobs: this time using Cloud Storage & Cloud Functions

Intro

Post update: My good friend Lak over at Google has come up with a fifth option! He suggests using Cloud Dataprep to achieve the same. You can read his blog post about that over here. I had thought about using Dataprep, but because it actually spins up a Dataflow job under-the-hood, I decided to omit it from my list. That’s because it will take a lot longer to run (the cluster needs to spin up and it issues export and import commands to BigQuery), rather than issuing a query job directly to the BigQuery API. Also, there are extra costs involved with this approach (the query itself, the Dataflow job, and a Dataprep surcharge – ouch!). But, as Lak pointed out, this would be a good solution if you want to transform your data, instead of issuing a pure SQL request. However, I’d argue that can be done directly in SQL too ūüėČ

Not so long ago, I wrote a blog post about how you can use Google Apps Script to schedule BigQuery jobs. You can find that post right here. Go have a read of it now. I promise you’ll enjoy it. The post got quite a bit of attention, and I was actually surprised that people actually take the time out to read my drivel.

It’s clear that BigQuery’s popularity is growing fast. I’m seeing more content popping up in my feeds than ever before (mostly from me because that’s all I really blog about). However, as awesome as BigQuery is, one glaring gap in its arsenal of weapons is the lack of a built-in job scheduler, or an easy way to do it outside of BigQuery.

That said however, I’m pretty sure that the boffins over in Googley-woogley-world are currently working on remedying that – by either adding schedulers to Cloud Functions, or by baking something directly into the BigQuery API itself. Or maybe both? Who knows!

Putting together the pieces: Server-side rendering with React Router v4 and Redux

Server-side rendering a React app is a miracle on-par with childbirth and modern air travel.

OK, that opening sentence was a little over-the-top. I apologise to birth mothers and those in the aviation industry.

Let me start again: server-side rendering a React app is…kind of cool.

That said, it can be a little tricky to get started, especially if you’re trying to do it with an existing app.

In this post I‚Äôll explain one way you can implement server-side rendering (SSR) for an app that’s using ¬†React Router v4 and Redux Thunks.

Along the way we’ll discuss the fundamental difference between JavaScript clients and servers, how it forces us to change the way we do routing, and the small “missing-link” that enables us to bridge React Router v4 with Redux thunks.

We’ll build up a simple example to demonstrate. I’m going to assume you’ve got some knowledge of:

  • React
  • Redux
  • React Router v4

However, you are not required to have knowledge of:

  • Childbirth
  • Aeronautics

Let’s do this.

Scheduling BigQuery jobs using Google Apps Script

Do you recoil in horror at the thought of running yet another mundane SQL script just so a table is automatically rebuilt for you each day in BigQuery? Can you barely remember your name first thing in the morning, let alone remember to click “Run Query” so that your boss gets the latest data refreshed in his fancy Data Studio charts, and then takes all the credit for your hard work?

Well, fear not my fellow BigQuery’ians. There’s a solution to this madness.

It’s simple.

It’s quick.

Yes, it’s Google Apps Script to the rescue.

Disclaimer: all credit for this goes to the one and only Felipe Hoffa. He ‘da man!

Extending React’s container/presentation pattern to share business logic between apps

One of the hottest JavaScript libraries out there right now is React. Whilst there are many good reasons for React being so popular, the main reason my team picked React for our new project is because of the commonality between React and React Native. The concepts and the language are the same, so if you can write React, then you can write React Native. There is also the benefit of being able put both in the same project and then share business logic between them, which is precisely what we need to be able to do.

There are a number of ambitious projects out there that aim to eliminate duplication between platforms by using a common set of React primitives. react-native-web, ReactXP and react-primitives spring to mind. However, these rely on not only a common set of components, but also a common approach to things like styling and animations.

Whilst we wait for the dust to settle on which is the best generic solution to use, my team has come up a simple approach that extends upon a pattern that is already quite well known in the React community. In this post I’ll outline this approach.

Ampersandjs at the beach

Ampersand.js

Ampersand.js was created by the good people at &yet. It is based on Backbone.js, and refers to itself as a ‚Äėnon-frameworky framework‚Äô. In a world where Javascript frontend frameworks abound, it can be difficult to choose ‚Äėthe best‚Äô one for your project. Most of the time you would go with that which was used before, which is the most popular, or which is most familiar. In 2015 I found myself in a position where a series of small, discrete, single-purpose, front-end ‚Äėwidgets‚Äô were proposed. The small development team was able to choose what we wanted, and Ampersand.js was selected. In this post¬†I’ll talk¬†about why I like using it for our projects, and what I think makes a good framework in general.

Triggering Dataflow Pipelines With Cloud Functions

Do you have an unreasonable fear of cronjobs? Find spinning up VMs to be a colossal waste of your towering intellect? Does the thought of checking a folder regularly for updates fill you with an apoplectic rage? If so, you should probably get some help. Maybe find another line of work.

In the meantime, here’s one way to ease your regular file processing anxieties. With just one application of Google Cloud Functions, eased gently up your Dataflow Pipeline, you can find lasting relief from troublesome cronjobs.

TypeScript, Flow and the importance of toolchains over tools

EDIT: The initial version of this post stated that the TypeScript compiler won’t emit code if it finds errors in the source. This is not correct. By default, the compiler will emit code even if it detects errors, unless the –noEmitOnError flag has been set. This post has been updated accordingly.

I’ve recently been working on a project that uses TypeScript. I also have been playing with Flow on a personal project. In this post I want to talk about why I think these tools are important, some of the fundamental differences between them, and why the choice of which one to use might best be determined by the broader toolchain that you are working within, rather than just the particular technical merits of one or the other.

YOW! Connected 2016: The typed-functional future of UI platforms

I just spent a couple of days at the YOW! Connected conference and had a great time, despite nursing a bit of a cold. There were a tonne of great talks at the conference covering a wide range of topics, but in this post I’m going to briefly reflect on¬†one specific trend that interested me at the event: the way in which UI platforms are advancing to adopt modern languages, and are even influencing each other in the process. The end-result: they’re all moving towards languages that are¬†both functional and statically typed.

Full disclosure:¬†This year I was a member of the programme committee for the conference. So¬†in writing this post, there’s a bit of a risk that I’m¬†creating an echo chamber for myself. All I can really say in my defence is that I hadn’t consciously¬†made¬†these connections in advance – it was only afterwards that I¬†saw¬†a trend!

Creating a serverless ETL nirvana using Google BigQuery

Quite a while back, Google released two new features in BigQuery. One was federated sources. A federated source allows you to query external sources, like files in Google Cloud Storage (GCS), directly using SQL. They also gave us user defined functions (UDF) in that release too. Essentially, a UDF allows you to ram JavaScript right into your SQL to help you perform the map phase of your query. Sweet!

In this blog post, I’ll go¬†step-by-step through how I combined BigQuery’s federated sources and¬†UDFs to create a scalable, totally serverless, and cost-effective ETL pipeline¬†in BigQuery.

Style Guides and AEM: Fitting a Square Peg in a Round Hole

Adobe-Marketing-Cloud

There is a push in the industry to code against an external style guide to maintain consistent styling, have a reusable set of components to build applications with and provide a shared vocabulary for teams to communicate. The goal is that any web application built in an organisation can make use of this style guide to re-use existing CSS rules and/or Javascript functions. For example, one of the more well-known and recently-published style guides is the U.S. Web Design Standards, which will enable U.S. government agencies to create a unified user experience throughout their web applications.

When dealing with modern web applications, integrating a style guide is a relatively straightforward process. It often involves leveraging the existing Javascript build tooling to pull assets down via npm, and then having those assets processed as part of the build pipeline along with your application’s styles/scripts. Alternately, you can simply copy-paste a version of your style guide’s artifacts into your application Рthe quick and dirty way.

However, things are not always this easy. In my experience working with AEM/CQ, integrating a style guide into a project has consistently been a challenge.

To be fair, style guides are not to be blamed for this. AEM’s strict file structure, meta-data (.content.xml files and the like) and reliance on Java technologies can make it challenging to integrate with Javascript-based tooling. Each project generally ends up with a set of custom scripts to achieve this integration, resulting in a solution that is simply not maintainable. If you’ll indulge me for a moment, I’d like to take this opportunity to run through a few solutions I’ve come across to integrate style guides with AEM.