Google is creating duplicate content on websites. I started noticing this back in 2008, when Webmaster Tools identified some pages that were “broken” on the website. The curious thing about these pages was that they were not actually accessible via the website. But they were in the source code as a part of a Javascript function and Google appeared to have kludged together some URLs based on the root domain name plus the file being called in the Javascript function.
For example, in the ASP.NET environment, pages and code behind are often called via Javascript. Those pages carry variables to the server where they are often rewritten to be “search friendly”, before being returned to the browser. In other words, “products.aspx?id=12345″ gets rewritten on the server to be “super-dooper-blue-products”. The actual “page” being called in Javascript never makes it to the browser.
In looking at GWT back in early 2008, I discovered there were hundreds of URLs causing “404″ errors. All of these URLs followed the pattern of http://root-domain/products.aspx?id=xyxyz. And when I clicked them, yep, they were broken. The reason these URLs were broken was because of the way Google created, guessed at, the proper URL construction from the information they discovered in the Javascript function. Google took the page name and parameter from the function and appended it to the root domain where the function was discovered.
However, the “actual” page that was being executed was not http://rootdomain.com/products.aspx? The real URL structure was more like, http://rootdomain.com/directory1/directory2/products.aspx? When Google executed the erroneous URL that it had created based on its assumptions, the pages were broken because Google logic did not fully understand that the products.aspx page resided in a relative path rather than an absolute path.
Functional duplicate URLs occur when pages such as products.aspx are called in Javascript and also reside in the absolute path. In cases like this, Google pulls page names and parameters from Javascript functions and appends them to the root URL to actually create a functional (although non-intentional) page. Now Google’s assumed URL construction does render an actual page that functions correctly. But that functional page is a duplicate page because, at the same time, a rewritten URL exists that renders the same content.
A few months after making this unfortunate discovery, Google informed us to stop the practice of rewriting URLs. What? In the sense that Google is now collecting “raw” data pages before a rewrite, and that a rewrite can cause duplicate content, Google says that it prefers the raw version of the URLs, the dynamic URLs, rather than rewritten ones.
That would be fine and nice if so many websites weren’t already using URL rewrites. Secondly, even though Google prefers the dynamic URLs now, Bing certainly does not. It would make better sense if Webmasters could instead include a tag in their pages such as “meta name =’discovery’ rel=’noJS’” whereby Google would not try to execute Javascript to “discover” pages that would result in the creation of duplicate content.
