Node.js

AppSync integrates seamlessly with a DynamoDB database. And as demonstrated in my previous article, AWS Amplify CLI can create the DynamoDB tables and generate the API CloudFormation infrastructure from an annotated GraphQL schema. However, using a relational data source with AppSync is more complex as...

10 January, 2019 /

Learn more >

Our use case

It has been close to 3 years since Drupal 8 was released. With lots of amazing new features released in every 6 months we are getting more and more excited to see how Drupal has evolved and improved. With the release of Drupal 8.4, it is starting to use ES6 in core. However, there is not yet a standard practice to write ES6 in custom modules and themes. The ideal solution is for compiled JS to still stay in the original module or theme and make our output JS still be compatible with Drupal 8. Drupal is heavily modular, so Javascript files written for a module should ideally be stored with it. This atomic design pattern allows a module to easily be enabled or uninstalled, and makes our code clean. JS files could be also added in a theme level if they are more related to that theme. In this blog we will share our story on how to configure the project with webpack and other tools to facilitate ES6 in the whole Drupal 8 project including both custom modules and at the theme level.
05 November, 2018 /

Learn more >

Some background

When we started using Google BigQuery - almost five years ago now - it didn't have any partitioning functionality built into it.  Heck, queries cost $20 p/TB back then too for goodness' sake!  To compensate for this lack of functionality and to save costs, we had to manually shard our tables using the well known _YYYYMMDD suffix pattern just like everyone else.  This works fine, but it's quite cumbersome, has some hard limits, and your SQL can quickly becomes unruly. Then about a year ago, the BigQuery team released ingestion time partitioning.  This allowed users to partition tables based on the load/arrival time of the data, or by explicitly stating the partition to load the data into (using the $ syntax).  By using the _PARTITIONTIME pseudo-column, users were more easily able to craft their SQL, and save costs by only addressing the necessary partition(s).  It was a major milestone for the BigQuery engineering team, and we were quick to adopt it into our data pipelines.  We rejoiced and gave each other a lot of high-fives.
09 April, 2018 /

Learn more >

Intro

Post update: My good friend Lak over at Google has come up with a fifth option! He suggests using Cloud Dataprep to achieve the same. You can read his blog post about that over here. I had thought about using Dataprep, but because it actually spins up a Dataflow job under-the-hood, I decided to omit it from my list. That's because it will take a lot longer to run (the cluster needs to spin up and it issues export and import commands to BigQuery), rather than issuing a query job directly to the BigQuery API. Also, there are extra costs involved with this approach (the query itself, the Dataflow job, and a Dataprep surcharge - ouch!). But, as Lak pointed out, this would be a good solution if you want to transform your data, instead of issuing a pure SQL request. However, I'd argue that can be done directly in SQL too ;-) Not so long ago, I wrote a blog post about how you can use Google Apps Script to schedule BigQuery jobs. You can find that post right here. Go have a read of it now. I promise you'll enjoy it. The post got quite a bit of attention, and I was actually surprised that people actually take the time out to read my drivel. It's clear that BigQuery's popularity is growing fast. I'm seeing more content popping up in my feeds than ever before (mostly from me because that's all I really blog about). However, as awesome as BigQuery is, one glaring gap in its arsenal of weapons is the lack of a built-in job scheduler, or an easy way to do it outside of BigQuery. That said however, I'm pretty sure that the boffins over in Googley-woogley-world are currently working on remedying that - by either adding schedulers to Cloud Functions, or by baking something directly into the BigQuery API itself. Or maybe both? Who knows!
21 January, 2018 /

Learn more >

Do you recoil in horror at the thought of running yet another mundane SQL script just so a table is automatically rebuilt for you each day in BigQuery? Can you barely remember your name first thing in the morning, let alone remember to click "Run Query" so that your boss gets the latest data refreshed in his fancy Data Studio charts, and then takes all the credit for your hard work? Well, fear not my fellow BigQuery'ians. There's a solution to this madness. It's simple. It's quick. Yes, it's Google Apps Script to the rescue. Disclaimer: all credit for this goes to the one and only Felipe Hoffa. He 'da man!
01 November, 2017 /

Learn more >

Do you have an unreasonable fear of cronjobs? Find spinning up VMs to be a colossal waste of your towering intellect? Does the thought of checking a folder regularly for updates fill you with an apoplectic rage? If so, you should probably get some help. Maybe find another line of work. In the meantime, here's one way to ease your regular file processing anxieties. With just one application of Google Cloud Functions, eased gently up your Dataflow Pipeline, you can find lasting relief from troublesome cronjobs.
23 March, 2017 /

Learn more >

Creating rich email templates can be a real pain, not only having to discard many best practices when it comes to writing HTML and CSS, but also needing to ensure your email renders correctly in all the commonly used email clients, many of which will throw away or ignore large portions of your carefully crafted styling.
07 November, 2016 /

Learn more >

Show more
Loading...