Viral Ad Network

If you check your server logs, you may have noticed that we have been running a new experimental web crawler – “VANCrawler”.

We’ve now set our crawler live across all publisher sites, so it’s time for a formal introduction.

What is VANCrawler?

VANCrawler is a web crawler which downloads a copy of all pages showing our ad units onto our own servers, where we apply our own natural language processing for advanced contextual targeting, scan pages for potential issues, and run countless other processing to help us target ads better.

What Do I need to do?

We have spotted several sites which have issues with the text encoding used on their pages. We suggest that all publishers double-check that they are declaring the correct encoding (“charset”) for their site.

Without this information, crawlers don’t know how to treat words containing accented letters such as “publicité”, or non-latin letters such as “Διαδίκτυο” or “網站”. If you’re not using these letters you may still be affected, as without the correct character set we may believe you are.

This is a problem for crawlers which index pages for search engines, and for users around the world as well as our own crawler, so fixing this issue on sites where it occurs may increase your search engine traffic as well.

You can specify the character encoding for your website in the Content-Type HTTP header, in the XML declaration for XHTML, or using a special tag for HTML. Please ensure that this matches the character encoding which your documents use when they are served.

If you are uncertain which encoding you are using, you can generally find the “charset” specified somewhere in your web browser (e.g. under “View>Character Encoding” in Firefox. For European users using windows, this will probably be “Windows-1252″. For Users on any other operating system (or for advanced websites on Windows), it will probably be “UTF-8″.

We have also noticed several sites serving pages which are not encoded using a valid encoding at all.

This may occur from using a server-side language such as PHP to concatenate strings which use different encodings. For example a string stored in a database may be encoded using UCS-4 (where each letter is represented as 4-bytes), and may then be concatenated with an ascii string (where each letter is one byte). This situation is even more difficult for crawlers to deal with, and so we suggest that publishers who program their own server-side pages check that they are not mixing different character sets when generating their pages.

Can I prevent VANCrawler from accessing my site?

VANCrawler follows the robots.txt standard, and will follow rules for User-Agent: VANCrawler..

If you want to limit how often we crawl pages on your site, you can use the crawl-delay robots.txt option to specify the minimum number of seconds we should wait between crawls.

As VANCrawler is being used to target ads to increase your revenue, we recommend you do not disallow access to pages on your site, however you may enter urls to exclude in the robots.txt format.

You can find more information on our ad crawler here: http://www.viraladnetwork.net/crawler-info

As of next week VAN’s publishers will be able to use a new ad format on their websites/blogs! These new ads will be displayed through the existing Fun Links (Text/Image links) and will feature an expanding ‘lightbox’ when viewers click on the link.

The expanding lightbox will be displayed as a site overlay and will grab the viewer’s attention, whilst keeping them on your site (therefore no site traffic will be lost). This also means that we can host flash games or more complex video campaigns via the Fun Links on your site.

Here’s an example…

Expanded ‘Lightbox’ Unit

You can also customise your Fun Links to suit your site – another cool feature! Here are some examples of customised Leaderboard ad units…