Deep inside the insurance industry, insurance agency software, claims management and actuarial science.

There are a number of reasons that tracking referrals to your website from third-party websites can be problematic if you attempt to do so with server-side code.

This is a technical article intended for insurance software developers but the problem described affects all of us who work with affiliates and resellers and attempt to track referrals server-side.

Let me break it down so you see the process, and the flaws.

Before I start, let me just make it clear: REFERRER and REFERER are alternative spellings of the same word, I think probably UK and US respectively. Since I’m in the UK right now I’ll be sticking to ‘referrer’ most of the time but forgive me if I lapse and switch

The tracking process we usually use to capture referrer is as follows:

1. The affiliate (e.g. an aggregator like CompareTheMarket or MoneySupermarket, or an independent broker or authorised representative, or even a shopping affiliate like Froogle, Kelkoo, SiteRunner, Ciao) publishes a link with their ID and our ID on their website.
2. A visitor clicks on the link with the intention to buy from your site.
3. The link takes the visitor back to the site of the affiliate that logs the visit, referrer, etc. and sets a cookie in the visitor’s browser.
4. The visitor is redirected to your site, where you log the new visitor as coming from the affiliate either by reading an ID in the querystring or by reading the *referrer value* (HTTP_REFERER) server side.
5. The visitor proceeds to shop while we track them around the site with an id binding them to the original affiliate.
6. The visitor either leaves or places an order, in which case we record the affiliate details in the order/quote/policy record.

How affiliates usually track us:

Since affiliates don’t trust their merchants to report this information back to them, there are two very similar methods they ask us to use to help them to capture more information:

1. They give us a snippet of code to be embedded in the source code of the ‘Thank you’ page that connects to the affiliate site notifying them the details of the successful transaction.
2. They give us an IMG tag to embed instead, linking to a server-side script on their site.

Note that this is not reliable, since we could easily randomize or fake the reporting of click-through. They are very much at our mercy, just as – in the other direction – we are at theirs, as you will see…

The industry-wide problem with all of this is:

There is absolutely no way to guarantee the referral data, for a number of reasons. In some respects HTTP itself is a flawed protocol in this context: namely the HTTP_REFERER server variable that we all use is not reliable – often empty or containing fraudulent data – and this server variable is the only way to capture the origin of a click event required to credit an affiliate referrer.

It can be empty because:

1. Some affiliates use a meta-refresh to redirect people to the merchant site (a-la Kelkoo), allowing them to deliver an extra banner ad, but often ditching the all important referrer info.
2. Some browsers don’t pass this variable at all.
3. Some browsers discard this variable if the site is opened in a new window, or if you drag a link and drop it on another window, or use a desktop/start menu shortcut, bookmark, etc.
4. Some people disable it manually, for privacy (usually outside of a single domain).
5. Some firewalls block it (e.g. Norton, usually outside of a single domain).
6. Many e-mail programs like Outlook filter it.
7. Most download managers don’t send it.

Furthermore, in order to track the referrer as people move around the site it has to be stored in the session (usually the only solution for storing information about someone who hasn’t yet registered or logged-in, but a very volatile mechanism which can easily expire), or a cookie (which people often disable, firewalls block etc.).

Also, if people enter the domain name of the site manually or set a bookmark once they found us on the affiliate site and then return later, there is nothing to track or detect.

Are there any workarounds?

Unfortunately the entire system is based on trust between affiliate and merchant, and worse still, trust of the browser/software/network/pc of the visitor, all of which is unreliable, and data – however stored – can be faked, since there is not actual authentication between the affiliate and us.

However, there is a dim light at the end of the tunnel, if you squint and look hard enough…

• Javascript, client based tracking like Google Analytics, Mint or BLVDStatus do a good job of telling you where visitors are coming from.  They’re not so good at showing clearly when visitors who originally came from a referrer return at a later date, because of the ‘last click wins behaviour‘.  To get around this when using Google Analytics, you need to override the last click wins behaviour.  You may also like to track all of your outgoing links.
• Alternatively or in addition, affiliates could provide authentication by SSL or a similar mechanism to prove to us that they are who they say they are and visa-versa, making the whole process of delivering customers a secure transaction.  I’ve not seen anything like this in practice but I’m sure it goes on.  If anyone has any examples, let me know!
• Often the only practical short-term solution is to create a different URL for each affiliate, which forces them to visit a different link to their counterparts, (e.g. http://www.yoursite.com/froogle ), but this can lead to resubmissions by fraudulent affiliates, since there is no way for us to stop it.

In my experience, use every method you can and compare the data coming from each approach which will help you highlight any grey areas or holes, and pinpoint fraudulent activity by any web partners you are working with.

As ever, your comments are very welcome!

As always, based on any comments received I will update this article so it remains authoritative.

{ 0 comments }