Critical Section


the referer mess

Sunday,  01/20/08  10:33 AM

I started blogging a little over five years ago (!), and at that time referer logs were a really cool thing.  (BTW, yes, "referer" is misspelled; it always is, going back to the early days of the 'net when a bad speller defined the HTTP header syntax.)  It was fun to see who was linking to you, and to find interesting new sites in that way.

{
Quick digression if you don't know what this is about:  Whenever you click a link on a web page, the website which serves the new page gets passed the old "referring" page's address.  By looking at server activity logs, a website owner can find all the websites which link to his.  In my case, I can find other blogs and sites which have linked to my blog, which is fun.
}

So now, in 2008, referer logs are basically a cesspool of misinformation.  If you are ever able to find anything useful of interesting in there, good luck.  First you have all the spiders; programs which access pages on your site for indexing purposes, their referers are often faked or meaningless.  Next you have access via RSS feeds, feed readers typically don't even pass a referer.  And then you have referer spam, which you might remember from my referer spam wars, where various websites send you fake referrals in the home that you'll click through to them.  (This is a weird and despicable practice, first, the volume from such traffic has to be uselessly low, because only webmasters see the spam, and second, no webmaster is going to care about a slimy site which is advertised in this way.  But I digress.)  And finally you have just plain old garbage; various bots which don't access anything useful and which pass junk as a referer.

This is a classic tragedy of the commons, we see this all over the 'net.  In the bad old days it worked great, but people have subverted the mechanism for other purposes to the point where it barely works at all.  The legitimate stuff is till in there but the signal to noise is decreasing to the point of uselessness.  Kind of like email :(

Home
Archive
'11   '10   '09
'08   '07   '06
'05   '04   '03
flight
Re:Cycling
Re:The Book
Re:Software
Re:Philosophy
About Me
W=UH
Email

RSS   OPML

Greatest Hits
Correlation vs. Causality
The Tyranny of Email
Unnatural Selection
Lying
Aperio's Mission = Automating Pathology
On Blame
Try, or Try Not
Books and Wine
Google and Blogs
Emergent Properties
God and Beauty
Moving Mount Fuji The Nest Rock 'n Roll
IQ and Populations
Are You a Bright?
Adding Value
Confidence
The Joy of Craftsmanship
The Emperor's New Code
Toy Story
The Return of the King
Religion vs IQ
Most Spectacular Photos of 2003
In the Wet
the big day
solving bongard problems
visiting Titan
unintelligent design
Shorthorn
the nuclear option
second gear
On the Persistence of Bad Design...
Texas chili cookoff
the inflection point
almost famous design and stochastic debugging
may I take your order?
paper art
triple double
China's olympic gardens
New Yorker covers
Death Rider! (da da dum)
how did I get here (Mt.Whitney)?
the Law of Significance
Holiday Inn
Daniel Jacoby's photographs
room with a view
weird disaster update
in praise of paddle shifting
the first bird
Gödel Escher Bach: Birthday Cantatatata
shining a light
Father's Day (in pictures)
Tour de France 2009
Tour de France 2010
Jobsnotes of note
Tour de France 2011