Running a Web Crawler in a Docker Container


A website may have hundreds, thousands, or even millions, of public facing pages. When we are responsible for maintaining such a website, it’s impractical to traverse it manually looking for broken links. We need an automated testing tool: one which can scan the whole website and log any broken links, so we can get them fixed sooner rather than later.

In this blog, I am going to describe a web crawler project which can easily and efficiently achieve the goal. The primary technologies used in this project are Scrapy and Docker.

Universal Links – A Few Things to be Prepared for

I recently had the opportunity to work for a client who wanted to develop what they termed “app indexing”. What they meant by this was that they wanted their users to be directed into a specific screen of their iPhone app when they tapped on a particular Google search result. Put differently, they wanted the user to feel as if Google had returned search results specifically for their iPhone app.

They also wanted to be able to send out links via email, SMS or other marketing channels. If the app was installed, opening such a link on their phone would result in the user being taken to the relevant points in the iPhone app. If the app wasn’t installed then they would just be taken to the mobile website.

The way this is achieved is through what Apple refer to as “Universal Links”. In this post I’m going to discuss how we implemented Universal Links at a client of ours, some of the obstacles we faced, and how we overcame those obstacles.

Test Driving Google Cloud Dataflow


Back in June 2014, at the annual Google IO in San Francisco, Google unveiled their newest, and much hyped cloud product, Cloud Dataflow. The demo they did that day, using a live twitter feed to analyze supporter sentiment during the 2014 world cup, got my mouth watering at the prospect of working with it. It looked downright freaking awesome, and I just couldn’t wait to get my hands on it to take it for a spin.

Concordion Integration With Jenkins


We recently introduced acceptance test driven development (ATDD) at a client. The idea was for the product owners, developers, and testers to work as a team to come up with the acceptance criteria for user stories before development begins. We adopted this approach as an attempt to increase a shared understanding of user stories, as well as a shared agreement on the definition of ‘done’.

As we introduced more functionality to the application, it became evident that more and more effort had to be put into regression testing prior to a software release. The application has to be supported on mobile (iOS and Android) and desktop (Safari, Chrome, Firefox, IE8 and above). Our team of 2 to 3 testers would spend up to 3 days testing for regressions! In an attempt to reduce the need for talented testers to carry out monkey work, the team began to push for a focus on automated browser-level testing.

In this post I will demonstrate the acceptance testing framework we used and describe how the test suite can be tied to a Jenkins build (with a beautiful results page!).

Unit Testing PHP and Silex using PHPUnit


As a newcomer to PHP I was puzzled by how to unit test controllers and services when using Silex (if you’re wondering, Silex is based on the Symfony 2 framework and draws a lot of modules from it).

In this post I’ll talk about my experiences getting tests going, and how you can setup your own unit tests using PHPUnit and WebTestCase. I’ll start with my initial attempts to test directly against the DB, and the approach that I ultimately found most useful and practical.

I’ve also created an example Silex project that can be used to try out the various features of PHPUnit. The only prerequisite is PHP 5.4, which comes with most modern OS’s. You don’t even need Apache installed to try it out!

Respect The Javascript

Like two-time WWE Champion Daniel Bryan, Javascript demands respect
Like two-time WWE Champion Daniel Bryan, Javascript demands respect

Yes, it’s true. Javascript is so popular it can now be a full-time job. Many people used to coast through their daily jobs as a PHP, Rails, Java developer with a few $(..) statements, maybe a $('#message').hide() (or a $('#message').slideUp(); if you were hip).

But these days, Javascript is coming of age. It’s grown from being a scrappy little toy we have loved and abused with code like onclick="alert('...')", to be something that has grown up with us, and now appreciate. It is the only runtime language I can think of that will run on any modern processing device. It’s ubiquitous and the ecosystem has evolved beyond what anyone could have imagined, with many implementations (Rhino, V8, now Nashorn) and it’s very own matching server-side equal, node.js

Testing Backbone Views with QUnit and Sinon

Clock parts

Unit-testing Backbone Views is hard. You need to cover enough for the test to be meaningful (for example DOM updates and server calls), without getting too tangled up in gory details. In this post I’ll talk about how we use QUnit and Sinon.JS to unit-test our Backbone views.

There’s already material out there on using these frameworks together, so I’m not going to get into setting things up. However, having done it for a year now I feel I’ve learnt enough additional tips, tricks and traps to warrant sharing them with the world.

This post will cover the three things you have to think about when testing Backbone Views, then talk about three strategies for simplifying your tests. I’ll also share some more general thoughts on Backbone’s suitableness for building large apps.

Testing for Android with Robotium

Android’s built-in testing framework is lacking on a number of levels. Enter Robotium, an open-source project. Robotium connects the dots by extending Android’s testing framework and providing convenience methods to help developers rapidly create tests.

In this entry I will show how Robotium can be used for data input, finding and testing views and buttons, testing between multiple activities, and testing activities with extras. I’ll also show how to work around issues with race-conditions in UI tests. I have created a GitHub project with a sample application and test project.

Unit Testing with Groovy

This post is about unit testing with Groovy. Groovy was the first language I used after Java. It lets me do a lot of the things I do with Java, but more quickly. Like many developers who are using another language after Java for the first time – whether it be Ruby, Scala, Groovy or something else – I’ve quickly come to love it.

In this post I’m going to show you how Groovy can make your tests more succinct. We’ll focus on syntax sugar for creating and consuming Maps and Lists, which is something you do a lot of when preparing data for a unit test. Than I’ll give a quick introduction to closures, which will lead us into some of the techniques we can use for mocking in Groovy.

How to Stop Cucumber Becoming Technology Roadkill

I’ve worked on a number of projects with Cucumber now, and I think it is a terrific tool. However, I’ve been seeing it used in ways that it was not originally intended for, the consequence being acceptance-test suites that grow increasingly slow, brittle and difficult to maintain.

These problems occur when developers start using Cucumber to drive overly-detailed tests. The problem of excessive UI testing with Cucumber has been blogged about in the past by Aslak Hellesøy – the creator of Cucumber – but I thought I’d cover some of my own experiences and opinions in this post.