We’ve now set our crawler live across all publisher sites, so it’s time for a formal introduction.

What is VANCrawler?

VANCrawler is a web crawler which downloads a copy of all pages showing our ad units onto our own servers, where we apply our own natural language processing for advanced contextual targeting, scan pages for potential issues, and run countless other processing to help us target ads better.

What Do I need to do?

We have spotted several sites which have issues with the text encoding used on their pages. We suggest that all publishers double-check that they are declaring the correct encoding (“charset”) for their site.

Without this information, crawlers don’t know how to treat words containing accented letters such as “publicité”, or non-latin letters such as “Διαδίκτυο” or “網站”. If you’re not using these letters you may still be affected, as without the correct character set we may believe you are.

This is a problem for crawlers which index pages for search engines, and for users around the world as well as our own crawler, so fixing this issue on sites where it occurs may increase your search engine traffic as well.

You can specify the character encoding for your website in the Content-Type HTTP header, in the XML declaration for XHTML, or using a special tag for HTML. Please ensure that this matches the character encoding which your documents use when they are served.

If you are uncertain which encoding you are using, you can generally find the “charset” specified somewhere in your web browser (e.g. under “View>Character Encoding” in Firefox. For European users using windows, this will probably be “Windows-1252″. For Users on any other operating system (or for advanced websites on Windows), it will probably be “UTF-8″.

We have also noticed several sites serving pages which are not encoded using a valid encoding at all.

This may occur from using a server-side language such as PHP to concatenate strings which use different encodings. For example a string stored in a database may be encoded using UCS-4 (where each letter is represented as 4-bytes), and may then be concatenated with an ascii string (where each letter is one byte). This situation is even more difficult for crawlers to deal with, and so we suggest that publishers who program their own server-side pages check that they are not mixing different character sets when generating their pages.

Can I prevent VANCrawler from accessing my site?

VANCrawler follows the robots.txt standard, and will follow rules for User-Agent: VANCrawler..

If you want to limit how often we crawl pages on your site, you can use the crawl-delay robots.txt option to specify the minimum number of seconds we should wait between crawls.

As VANCrawler is being used to target ads to increase your revenue, we recommend you do not disallow access to pages on your site, however you may enter urls to exclude in the robots.txt format.

You can find more information on our ad crawler here: http://www.viraladnetwork.net/crawler-info

The expanding lightbox will be displayed as a site overlay and will grab the viewer’s attention, whilst keeping them on your site (therefore no site traffic will be lost). This also means that we can host flash games or more complex video campaigns via the Fun Links on your site.

Here’s an example…

Expanded ‘Lightbox’ Unit

You can also customise your Fun Links to suit your site – another cool feature! Here are some examples of customised Leaderboard ad units…

(note – if you are comfortable editing html then it’s still advised you manually add the script tags to your template, as this will enable more advanced ad formats)

Step 1:

Log into Blogger and click on “Design” under your blog

Step 2:

Click “Add a Gadget” where you would like to insert the ad.

Make sure you have chosen a space on your page that will be large enough to hold the ad. Most of our ads are 300px wide and 250 px high.

Step 3:

Select “Add your own” from the left hand column in the gadget directory window.

Step 4:

Enter the following URL into the text box:

http://van-static.com/gadget-resources/blogger/bloggergadget.xml

And then click “Add By URL”.

Step 5:

Look at the embed code for your ad unit from your dashboard, and find the line starting with “var vaunit_id=”. Copy the number (example below). If you haven’t yet signed up to the viral ad network then apply now.

Now paste that number into the “Unit ID” field. Make sure that you leave the height large enough for your ad unit (normally 250 px). Click “Save”.

Done

Your ad should now have been inserted into your blog – keep on blogging, and we’ll keep picking the best ads to show to your users.

Another feature that we pushed in the same update was increased loading speed for pages showing our ads on safari and Chrome.

By loading the ad asynchronously on these browsers, we have measured a time saving of up to 150ms per ad displayed on the page. (that’s time saved before your page’s onload event).

We will continue to look at other possibilities for improving the loading times of our ads across other platforms and web browsers. For more information on what we do to speed up our loading times, take a look at our previous blog post on file size.

These new ads allow the user to view the ad in a larger format, by clicking on the ad to expand it to fill the screen in a lightbox style.

We have recently pushed this new feature out to a wider range of web browsers – including Internet Explorer, and the iPad. If you are one of our publishers, it is likely you’ll start to see some ads like this appearing over the coming weeks.

Here is an example of an ad that makes use of this new feature on browsers that support it:

If you are an advertiser interested in finding out how we can use these features to enhance your campaign, contact our team.

Over the past few years, the size of websites has increased dramatically – driven partly by advertiser demand for shinier creatives, partly by a new generation of web designers who have never focused on file size, and partly by increased broadband penetration.

File size is very important.There used to be a rule of thumb that if your website didn’t load within three seconds then the user would probably leave your website – and I believe that rule still holds. For publishers, that means you are losing out on ad revenue (and visitors) by serving large file sizes.

Even if a viewer does stay on the website, a slow page can irritate the user – which can impact negatively on the brand who’s adverts are being shown.

Not only that, but many people pay for data by the Mb / Gb (especially mobile users) – sending your users larger data than they need costs them real money, and puts them off websites and advertisers.

File size is being overlooked though. If you’re from an ad agency or  a publisher, you are probably willing (and able) to pay for high-speed internet access. Website viewers, the people being advertised to, aren’t always in the same situation though.

As an example, I just opened up the website of a popular web publisher, and noticed the page (including ads) was over 8mb in size.

Here’s how long that website would take to load on a user’s computer, depending on what internet connection the user is on:

1. 8 seconds (10Mbps broadband – you’re probably on this)
2. 45 seconds (2Mbps broadband – most of your users are probably on approximately this speed)
3. 3 minutes 7 seconds (500Kbps broadband – quite a few of your users are probably on this)
4. 40 minutes (56Kbps modem – you’d be surprised how many are still on this)

Of course, people don’t just browse from their homes and offices – users browse from mobile phones and from mobile broadband.

Although mobile broadband can have very fast speeds (8Mbps), this speed goes down very fast if you’re on the move. Travelling on the train this weekend, I recorded my average broadband speed at about 120Kbps. For the page above, it takes about 20 minutes to load the page.

Here in the UK, up to 50% of users have mobile broadband – which means that 50% of viewers to this site  may be have to wait 20 minutes for a page before your ads load.

The situation gets worse in other countries. For example, in Australia the average speed is far lower. If you’re serving ads world-wide, you should be aware that they’re probably taking far longer to load in Australia than they are to users in the USA or UK.

So that’s the bad news about the industry in general, but here’s what we at the Viral Ad Network are doing about the issue:

How the Viral Ad Network reduces Ad size:

For all ads, we are able to do the following

1. We serve the majority of data through compressed protocols (See RFC 2616 – section 3.5)
2. All of our standard javascripts are made as compact as possible, using the most advanced javascript analysis tools available.
3. We only load our standard code when it’s required.
4. We split our ad serving into two parts – the part that’s the same for every ad, and the part that varies depending on which ad is being served. We tell browsers and proxies cache the former as much as possible (including across different websites), and only load the latter each time an ad is displayed.
5. Most Ethernet implementations have a MTU of approximately 1492 bytes. We endeavour to fit our initial ad code into a single TCP packet (we currently serve roughly 870 bytes of initial javascript).

In addition, for ad creatives which we have control over, we do the following:

1. For data feeds loaded from servers, we provide server-side rewriting that compacts the data as much as possible (and compresses it).
2. For images, we re-compress the images to reduce file size (Note re-compression is also a necessary precaution for security)
3. For video players we produce ourselves, we sometimes in-line any images into the swf. This reduces the number of requests that have to be made for the data.
4. We try to post-load data from the server whenever possible. This means the code required by the ad often doesn’t load until after the website and ads have loaded, and the user starts interacting with an ad.

1. More compact, faster, ads (file size for all our ads has been reduced noticeably, which should make them display faster on publisher sites)
2. Language specific targeting. Previously, users have seen ads based purely on their location. Now, ads may be targeted based on the user’s preferred language. For example, a French language ad may now be targeted at French speakers in Canada, in addition to users located in France.
3. Extra precision for targeting options – from now, certain ads may only target individual Operating Systems and devices, while this feature is in test, this may include users across 20 different operating systems.

You may be wondering what the direct effects of that are for you as a website publisher; Well I’m going to let you in on the secret to making money from advertising that magazine publishers have known about for years, so read on…

Remember that the amount you get paid as a publisher every month is based on three things: the amount of viewers you get, the probability of them interacting with your ads, and the average amount you get paid per viewer. To calculate how much you earn, simply multiply these together:

£ [total money made] = [number of viewers] * [rate of interaction] * [average price per viewer]

While you’re working on your website it’s easy to forget about the last two, and simply focus on increasing the number of viewers you get per month – but take some time to think about these points:

• A well-positioned ad will easily get twice as many people interacting with it as a badly-positioned ad
• The price per viewer can change by a factor of 20 or more for different kinds of viewers.

Now think about how much time you spend deciding what kind of viewers you’re trying to attract, or deciding where to place an ad – and compare that to how much time you spend trying to increase the number of viewers.

Where do those magazine publishers come in to this? Well take a look across the magazine shelves next time you’re in a newsagents. There’s a large chance that you’ll see a lot of magazines along the lines of  “which X should I buy” (substitute “car”, “computer”, “morgage” or any other expensive good for “X”).

Most people seldom buy one of these expensive items, so the publishers aren’t expecting to sell a large number of subscriptions – or even (necessarily) sell a huge number of magazines. The publishers have recognised the importance of the other two factors in their income – how likely the readers are to respond to the ads, and how much value the readers bring to the advertisers (and the successful magazine publishers  have got this down to a fine art!)

With this new-found knowledge, what can you do to increase your income? That will depend on your site, but focusing on content that’s likely to appeal to high-value users in countries where there’s a high demand for advertising could make a significant difference to the average value of your website viewers, and hence your monthly income.

There was, however, some very flawed reasoning behind the assumption that that was a reasonable thing to pay by – for an advertiser, the concept of display CPM assumes that every thousand displays of an ad will bring you the same income.

Of course this is not true – showing an ad to 1,000 teenagers will get you a different response from showing your ad to 1,000 parents for example (which of those is better depends on what you are selling).

Less obvious differences exist as well – people are more likely to splurge on large items just after pay-day than half way through the month, and they are more likely to take anti-drinking advertisements seriously on a sunday morning.

Looking beyond the users that view the ads, the internet has changed a lot since the ninties. XHR became widespread at the change of the millenium, allowing “AJAX” websites where the browser doesn’t have to load a new page every time the user did something. Suddenly an entire visit to a website could be done on a single page, and the meaning of “display” got more blurred.

The Viral Ad Network works on a different level – publishers are paid (and advertisers pay) for some kind of interaction with the ad – which may be a video starting playing, a video ending, a clickthrough to a different site, a player reaching the second level in a game, or just about anything. This metric is more closely linked to the value a publisher brings to the advertiser – but there are still variations in how much each viewer is worth to the advertiser.

These differences are carried through in the price that’s actually paid for a viewer – and these prices vary with supply and demand.

Essentially, the more demand advertisers have for viewers on specific sites / from specific countries, the higher the price per viewer will be. Likewise, the more supply there is of similar sites, the lower the price per viewer will be.

The job of our syndication system is to create a market where these prices change efficiently with supply and demand – so publishers get paid as much as they should be, and so advertisers aren’t paying more than they should. We spend a lot of time doing mathematical modeling of how our software works to ensure it fills this role correctly, and we’re proud of the level to which it works.

Ad Targeting

After the success of our previous updates to our ad targeting,  we’re now analysing an extra layer of data in the same manner to target ads better, and increase click-through rates.

Ads targeting specific countries or niches will now be compared with the performance of other ads in those specific target audiences, helping to make the most out of every viewer our publishers get.

Publisher Dashboards

By default, your dashboards will now show you your total earnings in the past 14 days. This makes it easier to glance at your dashboard and immediately estimate your current daily earnings from your site.

To show all earnings since your last payment (the previous setting)  simply select “Since Last Payment” in the Date Range dropdown.

Publisher Offers

We’ve streamlined our system for offering specific ads to publishers. This means that in some situations you may now start to receive automated emails letting you know when there is a new offer – and letting you know when the offer ends.

In addition – some publishers will now be given the option of running offers alongside other ads, all without having to modify the embed code on your site. This means that you can make the most of the efficiency of our automated targeting system while running offers outside of our standard syndicated units.

Tim Wintle