Faking Interactive Video on Mobile

12 Dec 2013 Faking Interactive Video on Mobile

Problem solving is a major aspect of software development, there are often many different solutions to a problem and a good developer will strive for the most simple without compromising maintainability. However, there are certain times when there’s simply no elegant way to solve a problem so you end up writing what’s commonly referred to in the industry as a hack. You probably won’t be proud of it and you might even have committed the code under a pseudonym so no one can git blame you, but however ugly, the hack still solves a problem and the next thing you know, you’re being asked to write a blog detailing its every hideous crevice.

Let’s take a step back in time for a moment, it’s a typical work day, you’re just getting settled into some always enjoyable defect fixing when you get called over by upper management. You’re at a client site and there’s a problem with an online campaign built by an external marketing agency. The campaign’s landing page embeds a video from YouTube, it’s an interactive game that includes buttons that will load different videos when clicked.

Unfortunately, the interactive elements don’t work on mobile devices, exacerbated by the fact that 40% of their traffic is mobile and that they’ve already launched TV commercials, website banners and mail inserts pointing customers to this page. It seems like it shouldn’t be too difficult to fix, you probably have a whole bunch of ideas surfacing already…

The YouTube-sanctioned solution

Looking closer, the buttons inside the video use the now common technique of including annotations to create a clickable hotspot within a YouTube video. Your first move is to hit up the YouTube documentation; there are many different options when embedding a video on a webpage, so the careless marketing agency probably just forgot to tick the box that enables annotations.

After a little digging you surface a page with a footnote indicating that annotations aren’t supported on mobile players. Management aren’t happy with this result so they call Google directly, their tone firm with a hint of dismay, more so since Google aren’t bearing any kind of miracle solution.

“Annotations aren’t supported on mobile and they won’t be coming anytime soon. You shouldn’t be using them.”

Although disappointing, it becomes painfully obvious why Google has taken this stance, but it’s a realisation that doesn’t hit you until you’re fully immersed in the labyrinth of mobile video manipulation.

HTML mock annotations

So Google won’t help, it’s no big deal, when was the last time you needed them anyway? Oh. The buttons already exist visually within the video so your task is to find a way to detect the clicks (or mobile touches) and perform the action of the invisible annotations to recreate the effect of an interactive video.

YouTube actually has a fairly nice JavaScript API so loading a new video into the same embedded player isn’t an issue after the click has been detected, but the fact that the player is running inside an iFrame is a little bit of a problem. An iFrame is the web equivalent of a black hole — the user can click all they like but you won’t know where the cursor was or whether the click even took place.

You might have more luck if you control the page inside the iFrame but the player and its content come directly from YouTube so cross-site scripting measures will shut you down before you can even think about writing that function call.

//Are you kidding? This isn't going to work!
$("iframe").click(function () {
  player.loadVideoById(video);
  player.playVideo();
});

However, you do control the webpage the video is embedded on, so the more effective work-around is to absolutely position some transparent div elements over the video in the same visual space that the buttons appear and detect the clicks on the div instead. There’s some manual work in positioning and timing when the div should overlay but the hack seems to work just fine on an Android device … but not on an iPad.

What’s with that iPad?

The iOS devices are a completely different breed of pain and suffering. Whilst you can display an HTML element over the top of the iPad video player just fine, you can’t detect any clicks on it. Internally, the iOS web engine cuts an invisible hole in the area occupied by the video player so it can use a hardware renderer for that portion of the page. Although clicks still get detected in this area, they’re sent directly to the video element and the click event will pass straight through overlapping elements as if they were merely manifestations of the mind.

After giving it some thought, you come up with a solution which is much uglier and has a slight impact on the user experience. For this particular campaign, the call to actions in the video appear near the end and although the video continues to play, it’s just a background idle animation with the main focus being on the buttons to encourage user interaction.

Using the same timing detection from the Android fix, you write a script to hide the video element and replace it with a static screenshot positioned in exactly the same location. It gives the illusion of the video pausing when the buttons appear and as an added bonus, it also provides a mechanism for detecting clicks and restoring the functionality of the interactive video.

Breaking the iPhone player

Unfortunately when it comes to playing video, the iPhone takes all the good that’s come of consistent web standards and swiftly throws them out the window. When you play a video using the iPhone browser, it will instead launch the native iOS video player in full screen which takes you away from the web page. It’s roughly at this point that the control over the user experience is ripped from your clutches and you’re left cursing the overbearing Apple gods.

In a brilliant circle of poetic justice, Apple encountered this very issue themselves when building the marketing page for the iPhone 5. The page at the time included a very short inline video of an iPhone unlocking once you scrolled down. That’s right, their iPhone marketing page worked on every device except the iPhone. Given the circumstances that placed them in this situation, their solution (which involves re-implementing video encoding and playback using JPG images and background HTTP requests) can only be described as ludicrous.

However, this technique won’t work for your video campaign since the video length makes the solution infeasible. Instead, through a lot of trial and error, you discover that you’re able to use JavaScript to remove the entire iFrame element during playback which is enough to disrupt the iOS native video player and cause it to revert back to the webpage. It also means you’ll need to re-initialise the YouTube Player API in preparation for playing the next video.

$("#video").html("<div id='video-player'></div>");

var player = new YT.Player("video-player", {
  height: "505",
  width: "100%",
  videoId: nextVideoId,
  playerVars: { "autoplay": 0 },
});

Once back in the context of the webpage, you now have the ability to implement the static screenshot fix used for iPad, resulting in a less than ideal user experience but one that allows the user to click the button and navigate to the next video.

Something feels dirty

As you start reaping in the kudos from these glorious fixes, you can’t shake the disgusting feeling in the back of your mind you that there must be a better way. It occurs to you that if you could somehow control the content within the iFrame, you wouldn’t have the cross-site security issues that were preventing more elegant and transparent solutions.

You whip up a quick backend service that curls the content from YouTube’s iFrame and sends it back as a standard HTML response. It’s a type of proxy that makes it appear like the video player content is coming from the same domain as the host page and it allows you to inject a script that can detect the user’s actions within the iFrame (removing the need for mock divs or static screenshots).

The response from YouTube needs some tweaking, many of the links are relative so you need to programmatically replace them with absolute paths pointing to YouTube. After quite a bit of manipulation it starts to work, so you go home for the day delighted that you’ve transformed your ugly hack into a much more impressive dodgy fix.

The following morning you arrive at your desk, ready to spend the next hour revelling in your improved solution but are quickly cut down when the brittleness of last night’s approach is highlighted through an overnight change in YouTube’s video player response. Not just a minor change, they’ve completely restructured the HTML loaded into the iFrame which renders your fix completely useless. Without knowing how frequently this may occur, you decide it’s better to discard the proxy solution and revert to the less impressive manual hack.

Awaiting a more simple solution

That side experiment wasn’t without its insights; it proved that there’s a lot that can be done to enhance the interactive experience on mobile if you can control the video player’s HTML. At one point YouTube had partial support for annotations but presumably this was dropped when they were unable to provide a consistent experience across all mobile devices due to iOS constraints.

However, they’re already providing an inconsistent experience between the desktop and mobile platforms so why draw the line there? Understandably, Google’s top engineers are probably too busy inventing self-driving cars and building computer brains to look into this, but surely the intern keeping an eye on the world’s number one video sharing website has a spare hour or two to improve the current situation.

You’d also be kind to excuse the marketing agency that both conceived and originally developed the interactive video concept. Evidently no testing was performed on mobile devices, whether the importance of mobile was never conveyed or if it was simply overlooked, they were quick to dust their hands of any responsibility. Their proposed solution? Cut all videos just before the clickable buttons appear and redirect the user’s attention to links displayed outside the video element. In a day and age where creative agencies are twisting technology to achieve marketing brilliance, these half-baked solutions aren’t going to cut it.

“Mobile makes up almost 40% of YouTube’s global watch time”

According to YouTube’s own statistics and assuming infinite reach, that’s potentially hundreds of millions of users per month who won’t be able to fully immerse themselves in your interactive video campaign. Which makes you wonder why Google doesn’t address this issue for 75% of mobile users by adding support on a platform they control. In the meantime, you’ll be left winning admirers by hacking together four different solutions to solve a problem that never really should’ve existed in the first place.