There has been a lot of talk today about Murdoch wanting to have Google stop indexing his news sites. I’m not going to go into the piss poor business decision this is, but I do need to highlight this:
Mr Murdoch also indicated that he would use legal methods to prevent Google and other search engine “news aggregators” from taking his newspapers’ material.
Really – going to sue Google over this? Let’s take a look at how much Fox News tries to keep Google out.
When a search engine goes to a website it reads a file called robots.txt. This is like an instruction manual for search engines on what to search and not to search. You can view my robots.txt file here.
So what does the robots.txt file on foxnews.com say?
User-agent: * Disallow: /printer_friendly_story Disallow: /projects/livestream # User-agent: gsa-crawler Allow: /printer_friendly_story Allow: /google_search_index.xml Allow: /google_news_index.xml Allow: /*.xml.gz # Sitemap: http://www.foxnews.com/google_search_index.xml Sitemap: http://www.foxnews.com/google_news_index.xml
Well look at that. Not only is Fox allowing Google, but they are giving specific directions to Google to read files and index those items.
The best way to explain this I can think of is the example of a robber. Let’s say you open the door and ask the robber to come on in. You then walk him around your house, pointing out all your valuables for him to take. Now after he takes them and leaves you decide to go after the robber legally. Of course a robber would be a person engaged in illegal activity generally, something Google isn’t.
But the tech behind robots. txt isn’t the only way to stop Google from indexing your site. Google offers numerous ways to have your site removed from their search engine, and have already said they will gladly pull his sites out of there, if he so wishes.
There does seem to be something more sinister to Murdoch’s master plan than all this:
Asked how he reacted to the challenge of Google and others for newspapers such as his to remove their newspapers from search results, Mr Murdoch said that once they had in place the means to charge for news, “I think we will”.
He also challenged the idea that Google and others could take just the headlines and opening lines from his papers’ stories, indicating that he would not tolerate even that.
“[They use] a doctrine called fair use, which we believe can be challenged in the courts and will bar it altogether,” he said.
So Murdoch is planning on using Google until he decides it’s time to make people pay, after that it is all him. So how are you going to know what his new “pay sites” offer if you can’t even find them, let alone have an idea of what is published? One would think that a well written headline would be a sure seller for his new product, but apparently old Rupert doesn’t even want that. Instead it appears as though he is going after a “fork over the money then see what you get” model of business. Oh yeah – that will be a big success there.
I can’t wait for all this to happen. It’s a good thing I am not a big decision maker at Google because I would go ahead and save Rupert the very minimal headache of having to have his sites removed and just do it now. That includes all News Corp sites, including things like Fox Movies. Just imagine when someone wants to check out a trailer for some big new movie and decides to Google it? And let’s not forget about all the Fox affiliates out there. Hey they have a right to republish Fox items, so Google shouldn’t take a chance.
I also hope that Rupert does decide to file legal action against Google. Can you imagine the trial if it ever went there? The defense would just have to say “here are copies of documents placed on Fox servers by Fox people asking our software to index their site and place the information in our search results”. I think the only lawyer Rupert would be able to find to take this case if Orly Taitz.