Running a Web Crawler in a Docker Container

September 13, 2018September 13, 2018 Jiaqi Yin Leave a comment

Introduction

A website may have hundreds, thousands, or even millions, of public facing pages. When we are responsible for maintaining such a website, it’s impractical to traverse it manually looking for broken links. We need an automated testing tool: one which can scan the whole website and log any broken links, so we can get them fixed sooner rather than later.

In this blog, I am going to describe a web crawler project which can easily and efficiently achieve the goal. The primary technologies used in this project are Scrapy and Docker.

Continuous Integration/iPhone/Mac/Mobile/Testing/Web

Universal Links – A Few Things to be Prepared for

June 15, 2017June 16, 2017 Chaise Hocking 2 Comments

I recently had the opportunity to work for a client who wanted to develop what they termed “app indexing”. What they meant by this was that they wanted their users to be directed into a specific screen of their iPhone app when they tapped on a particular Google search result. Put differently, they wanted the user to feel as if Google had returned search results specifically for their iPhone app.

They also wanted to be able to send out links via email, SMS or other marketing channels. If the app was installed, opening such a link on their phone would result in the user being taken to the relevant points in the iPhone app. If the app wasn’t installed then they would just be taken to the mobile website.

The way this is achieved is through what Apple refer to as “Universal Links”. In this post I’m going to discuss how we implemented Universal Links at a client of ours, some of the obstacles we faced, and how we overcame those obstacles.

Cloud/Google Cloud/Java/Testing

Test Driving Google Cloud Dataflow

February 25, 2015October 15, 2018 Shine Solutions Group 4 Comments

Back in June 2014, at the annual Google IO in San Francisco, Google unveiled their newest, and much hyped cloud product, Cloud Dataflow. The demo they did that day, using a live twitter feed to analyze supporter sentiment during the 2014 world cup, got my mouth watering at the prospect of working with it. It looked downright freaking awesome, and I just couldn’t wait to get my hands on it to take it for a spin.

Continuous Integration/Testing/Tools

Concordion Integration With Jenkins

May 9, 2014 Jack Low Leave a comment

Monkey_Office_1A

We recently introduced acceptance test driven development (ATDD) at a client. The idea was for the product owners, developers, and testers to work as a team to come up with the acceptance criteria for user stories before development begins. We adopted this approach as an attempt to increase a shared understanding of user stories, as well as a shared agreement on the definition of ‘done’.

As we introduced more functionality to the application, it became evident that more and more effort had to be put into regression testing prior to a software release. The application has to be supported on mobile (iOS and Android) and desktop (Safari, Chrome, Firefox, IE8 and above). Our team of 2 to 3 testers would spend up to 3 days testing for regressions! In an attempt to reduce the need for talented testers to carry out monkey work, the team began to push for a focus on automated browser-level testing.

In this post I will demonstrate the acceptance testing framework we used and describe how the test suite can be tied to a Jenkins build (with a beautiful results page!).

Testing/Tools/Uncategorized

Unit Testing PHP and Silex using PHPUnit

February 6, 2014February 6, 2014 Colin Nevin Leave a comment

As a newcomer to PHP I was puzzled by how to unit test controllers and services when using Silex (if you’re wondering, Silex is based on the Symfony 2 framework and draws a lot of modules from it).

In this post I’ll talk about my experiences getting tests going, and how you can setup your own unit tests using PHPUnit and WebTestCase. I’ll start with my initial attempts to test directly against the DB, and the approach that I ultimately found most useful and practical.

I’ve also created an example Silex project that can be used to try out the various features of PHPUnit. The only prerequisite is PHP 5.4, which comes with most modern OS’s. You don’t even need Apache installed to try it out!

Agile/Javascript/Testing

Respect The Javascript

October 29, 2013October 30, 2013 Rob Pocklington 10 Comments

Like two-time WWE Champion Daniel Bryan, Javascript demands respect

Yes, it’s true. Javascript is so popular it can now be a full-time job. Many people used to coast through their daily jobs as a PHP, Rails, Java developer with a few $(..) statements, maybe a $('#message').hide() (or a $('#message').slideUp(); if you were hip).

But these days, Javascript is coming of age. It’s grown from being a scrappy little toy we have loved and abused with code like onclick="alert('...')", to be something that has grown up with us, and now appreciate. It is the only runtime language I can think of that will run on any modern processing device. It’s ubiquitous and the ecosystem has evolved beyond what anyone could have imagined, with many implementations (Rhino, V8, now Nashorn) and it’s very own matching server-side equal, node.js

AJAX/Javascript/Testing/UI

Testing Backbone Views with QUnit and Sinon

June 27, 2013September 6, 2013 Ben Teese Leave a comment

Unit-testing Backbone Views is hard. You need to cover enough for the test to be meaningful (for example DOM updates and server calls), without getting too tangled up in gory details. In this post I’ll talk about how we use QUnit and Sinon.JS to unit-test our Backbone views.

There’s already material out there on using these frameworks together, so I’m not going to get into setting things up. However, having done it for a year now I feel I’ve learnt enough additional tips, tricks and traps to warrant sharing them with the world.

This post will cover the three things you have to think about when testing Backbone Views, then talk about three strategies for simplifying your tests. I’ll also share some more general thoughts on Backbone’s suitableness for building large apps.

Android/Java/Testing/Tools/UI

Testing for Android with Robotium

September 5, 2012 Tim Osmond 5 Comments

Android’s built-in testing framework is lacking on a number of levels. Enter Robotium, an open-source project. Robotium connects the dots by extending Android’s testing framework and providing convenience methods to help developers rapidly create tests.

In this entry I will show how Robotium can be used for data input, finding and testing views and buttons, testing between multiple activities, and testing activities with extras. I’ll also show how to work around issues with race-conditions in UI tests. I have created a GitHub project with a sample application and test project.

Java/Testing

Unit Testing with Groovy

June 23, 2012June 28, 2012 Kon Soulianidis 3 Comments

This post is about unit testing with Groovy. Groovy was the first language I used after Java. It lets me do a lot of the things I do with Java, but more quickly. Like many developers who are using another language after Java for the first time – whether it be Ruby, Scala, Groovy or something else – I’ve quickly come to love it.

In this post I’m going to show you how Groovy can make your tests more succinct. We’ll focus on syntax sugar for creating and consuming Maps and Lists, which is something you do a lot of when preparing data for a unit test. Than I’ll give a quick introduction to closures, which will lead us into some of the techniques we can use for mocking in Groovy.

Testing/Tools

How to Stop Cucumber Becoming Technology Roadkill

April 15, 2012April 18, 2012 Ben Teese 1 Comment

I’ve worked on a number of projects with Cucumber now, and I think it is a terrific tool. However, I’ve been seeing it used in ways that it was not originally intended for, the consequence being acceptance-test suites that grow increasingly slow, brittle and difficult to maintain.

These problems occur when developers start using Cucumber to drive overly-detailed tests. The problem of excessive UI testing with Cucumber has been blogged about in the past by Aslak Hellesøy – the creator of Cucumber – but I thought I’d cover some of my own experiences and opinions in this post.

Category: Testing

Introduction