Universal Links – A Few Things to be Prepared for

15 Jun 2017 Universal Links – A Few Things to be Prepared for

I recently had the opportunity to work for a client who wanted to develop what they termed “app indexing”. What they meant by this was that they wanted their users to be directed into a specific screen of their iPhone app when they tapped on a particular Google search result. Put differently, they wanted the user to feel as if Google had returned search results specifically for their iPhone app.

They also wanted to be able to send out links via email, SMS or other marketing channels. If the app was installed, opening such a link on their phone would result in the user being taken to the relevant points in the iPhone app. If the app wasn’t installed then they would just be taken to the mobile website.

The way this is achieved is through what Apple refer to as “Universal Links”. In this post I’m going to discuss how we implemented Universal Links at a client of ours, some of the obstacles we faced, and how we overcame those obstacles.

The Basics

Apple’s documentation provides some instruction on setting up Universal Linking between your app and your website, so I’m not going to repeat that information here. However, if you’re not already familiar with the process, here are the essential steps that we took initially to set up Universal Linking:

1. We created an apple-app-site-association JSON file, which described the paths we wished to link, and the app ids the links were for
2. We hosted a file in the root of each of our domains that the links are for (ie, we had our web-servers serve up each file)
3. We added an entry for each domain under the associated domains key in the entitlements file of our iPhone app

Easy, huh? Well, you may not be surprised to hear that it didn’t work straight away. We were going to have to troubleshoot this thing.

One weird trick for testing on the simulator

Before we started troubleshooting, we had to first ask ourselves: what’s the easiest way for us to quickly check our changes?

Unlike most of the discussion you’ll see on the web that suggests otherwise, it turns out you can actually test out your Universal Links on the iPhone simulator. It’s kind of a hack, but at least it makes it possible. So without further ado, here’s our five-step approach to using Universal Links in an iOS simulator:

1. Start the iOS 10 simulator and run your app
2. Switch out of the app using multitasking
3. Open up the Messages app in the simulator
4. Create a message with the link you want to test and send it
5. Click on the link in the message you just sent

The simulator will subject the link to the same checks that an actual iOS device would. This means that, if everything is setup correctly, the link will be recognised as a Universal Link, and your app will open correctly.

The First Hurdle

So on to our first actual issue. This concerned allowing devices to hit the apple-app-site-association file served from our website. We had built our file and placed a single path in it that we wished to support. Following the instruction from Apple to a tee, we then configured Nginx to serve up the file by adding the following to our nginx.conf file:

location = /apple-app-site-association {
    default_type application/json;
}

Unfortunately, even though we had the app installed, when we tapped on a link we were still just taken to the mobile website. After much investigation, we found that the robots.txt file on our server was blocking access to the apple-app-site-association file.

Fortunately, the answer turned out to be simple. We just had to permit access to that route in our Nginx config:

Allow: /apple-app-site-association

And the problem was solved: our first Universal Link was successful!

SSL Issues

As described in Apple’s documentation, you must serve your apple-app-site-association from your HTTPS web server. Now if you have a setup anything like ours, then the mobile web developers on your team won’t have their own SSL certificates for local development. What this means is you can’t setup Universal Linking between local instances of your mobile web app and the iPhone app. This is because when the iPhone app is installed it tries to fetch the apple-app-site-association file from the SSL web server defined in the associated domains file of the app. However, if the server does not have an SSL certificate, then the file will not be fetched and the Universal Links will fail.

The end result of this is that any changes made to the apple-app-site-association file need to be deployed into a test environment that supports SSL before you’ll know if the changes worked. In our case, this meant that they needed to be committed to our version control system and then deployed via our CI pipeline. This was a very arduous process with a very slow feedback loop and we really wanted to improve it.

At first we tried to use Charles to simply proxy the iPhone’s network requests through our local Macs and map the request made for the apple-app-site-association file to a different file containing the changes. However, whilst this was a good idea in theory, another obstacle immediately presented itself. When you install the Charles root certificate, it is trusted by Safari and other apps, but requests for the apple-app-site-association file are not actually coming from an app, they are made by the operating system itself. This meant that even though we could see the request for the association file and we could map it to something else, the process of SSL proxying this request was actually failing.

As a last-ditch attempt, I also tried setting up our mobile web environment with a self-signed SSL certificate to serve over HTTPS, allowing me to provide the association file locally. This meant changing the associated domains entry of the iPhone app to point at my Mac, so that it now treated my Mac as the source of the website. However the problem arose once again when the phone attempted to download the apple-app-site-association file. Because the SSL certificate was not from a trusted source, it refused to follow the route and the file could not be retrieved.

So unfortunately, we weren’t able to do any better than deploying changes directly to our test environments through our CI, and were left unable to test them locally first.

Association File Intricacies

Something that’s not well explained in Apple’s documentation is the parsing process used when processing the apple-app-site-association file. It explains that an * indicates any substring and a ? is used to match any single character. However it was unclear how to allow links for paths like:

/laptops/abc

but deny paths that are an extension of this, for example:

/laptops/abc/def

The documentation specifies that you could write:

"NOT /laptops/*/*",
"/laptops/*"

This should deny anything that has more than one component after “laptops”, and allow a link that has exactly one component like our /laptops/abc. However, it doesn’t seem to work. What happens is NOT /laptops/*/* actually matches /laptops/abc. What appears to be happening is the path is actually url-compared, and trailing asterisks and slashes don’t affect a url. Put differently, as an * can represent nothing, /laptops/abc/ is considered equal to /laptops/abc, so the url matches.

Interestingly enough, if an * is in the middle of a url, it does not allow this to be nothing. For example, if a path was declared as:

"/tablets/*/2016"

and a link /tablets/2016 was clicked, this would not match, and the link would fail.

So back to the original problem. We wanted to deny paths longer than the one we were allowing for our link. The solution we found was to use the ? character before the *. As described earlier, * characters on the end of paths are treated as [0 – many], but question marks are always [1]. So we changed our rules to:

"NOT /laptops/*/?*",
"/laptops/?*"

Now this meant that /laptops/abc would not be denied by the first rule, as it doesn’t have a character on the end while the ? is expecting one, and will therefore be allowed by the second rule. Awesome!

Watch out for the User-Agent

Out next issue concerned the user-agent used when the association file is downloaded by the iPhone.

As described earlier, when the iPhone app is downloaded from the store, the phone makes a request for the apple-app-site-association file from the web server. We have a rule in our pre-production environment load balancer that directs request to either our desktop site or our mobile website based on the user-agent. The rule isn’t completely straight forward, but basically if the user-agent contains WebKit then the user will be directed to the mobile site.

When our testing reached pre-production where we serve our mobile and desktop sites from different web-servers, we couldn’t work out why when we tapped on a link it failed and kept linking us directly to the mobile website. After some investigation and monitoring our logs on our server, we discovered that the apple-app-site-association file was never reaching the phone when the app was installed.

It turns out that when the phone was making requests to retrieve the apple-app-site-association file, it was actually being directed to the desktop website, rather than the mobile website where the association file lived.

By inspecting the logs, we could see the user-agent of the request was actually:

swcd (unknown version) CFNetwork/758.2.8 Darwin/15.0.0

Because WebKit was absent from this string, the load balancer was receiving the request and as expected, sending it straight to the desktop site, where no apple-app-site-association lived.

There is no WebKit in this string because it is not the browser making the request for the file, but iOS itself. So we tweaked the load balancer rule to redirect requests with Darwin in the user-agent to continue through to the mobile website too. This change fixed the issue and the links began working in the pre-production and subsequently production environments.

Difference in iOS Versions

In the Apple documentation they specify that you can host the apple-app-site-association file in either the root of your web server, or in the /.well-known subdirectory. We chose to host it in the root of our web server and everything was working fine… until we tested on iOS 10.

Pre iOS 10, iOS would first check for the file https://<your-domain>/apple-app-site-association, then if that didn’t exist, it would go on to check https://<your-domain>/.well-known/apple-app-site-association. As of iOS 10 however, for some unexplained reason the order has been reversed.

This subtle, yet significant change really caught us off guard. On our website, we had a mechanism in place whereby any requests made to an unknown path would return a 200 status code, and show the user the home page. This meant that https://<your-domain>/.well-known/apple-app-site-association actually returned a 200. However, even though the content was HTML rather than JSON, iOS would not go on to check https://<your-domain>/apple-app-site-association to see if a valid file was there.

This resulted in all our Universal Links failing, as the paths for the actual file were never retrieved, causing them all to fail. Fortunately, the solution to this problem was quite simple: add an Apache rewrite rule to prevent requests to/.well-known from returning 200s.

In Summary

The documentation provided by Apple on Universal Linking is quite limited and does not provide much detail of how the Universal Linking system really works. If you too are to implement your own Universal Linking solution, it’s likely that you may face some similar complications to those I’ve described. I hope that this brief summary of tips, tricks and traps I’ve encountered will leave you slightly better placed to face some of the curve balls that Universal Linking might throw at you.