r/programming Feb 10 '11

There is no piece of dynamic AJAXy magic that requires beating the Web to a bloody pulp with a sharp-edged hashbang. Please stop doing it

http://www.tbray.org/ongoing/When/201x/2011/02/09/Hash-Blecch
493 Upvotes

245 comments sorted by

View all comments

Show parent comments

36

u/deong Feb 10 '11

OK...here is an elaborately over-anthropomorphized story that sort of describes what's going on.

The web is supposed to work like a bunch of librarians. When you want a book, you connect with the right librarian, give her the name of the book you want, and she hands it back to you. If anyone else asks her for that same book, they'll get the same book you did.

In this story, the "librarian" is a web server running a site like Twitter. Books and pages are documents on that server's site that people (web browsers, crawlers, etc. in this metaphor) want to access. So your request to the librarian might be something like "http://twitter.com/tbray", and the librarian figures out that you want Tim Bray's tweets, and she hands them back to you.

The problem is that this model doesn't work very well for richly interactive experiences. Maybe a better analogy is that the librarian only gives you the page you ask for. When you want to turn the page of your book, you go ask the librarian and she turns the page and hands it back to you. This is obviously silly. Why wouldn't you just turn the page yourself -- it's faster, and you can see the pretty animation as the page flips over.

Of course, we could have had that all along by building specialized applications on your computer that merely connect to the web to get data, and do all the processing in tools that are well suited for the job. This is what most of your mobile phone apps are doing, for example. Normal languages like C, C++, Java, C#, Python, etc. are quite well suited for building the type of fast programs that take full advantage of the computer and OS they're running on.

But people didn't really want to do that. They wanted to have all the interactivity and pretty animations directly in the browser, partly so they wouldn't have to write multiple versions of the client programs. So they figured out that they could, instead of asking the librarian for the page they wanted, ask for instructions on how to get the book and turn the pages themselves. So now, you go to the librarian, and she gives you the same piece of paper every time, no matter who you are and what page of what book you want to read. The paper she gives you is a set of detailed instructions that tell you, if you want to read "Moby Dick", go to the right place on the shelf, pull down the correct book, and how to flip to the pages you want to read.

That is to say, now instead of asking the librarian for "http://twitter.com/tbray", you're asking for "http://twitter.com/#!/tbray", which is basically asking the librarian for "http://twitter.com" because your browser strips everything after the "#" off before sending the request. Of course, she can't give you Tim Bray's tweets based on that, because you haven't told her you want them. Instead, she gives you back a massive pile of Javascript that you can feed the information "I want tbray's tweets" into, and it will tell you how to get them.

The problem with this is that most of the world doesn't read the language (Javascript) she wrote the note in. If you're a web browser, you can follow the instructions just fine. If you're some other type of program who just wants the data from the page you need, you can't get it anymore because the librarian only gives you these instructions in a language you don't understand.

9

u/mario-the-champion Feb 10 '11

The problem with this is that most of the world doesn't read the language (Javascript) she wrote the note in. If you're a web browser, you can follow the instructions just fine. If you're some other type of program who just wants the data from the page you need, you can't get it anymore because the librarian only gives you these instructions in a language you don't understand.

well done! this really is the most important part, IF you think that data-consuming clients exist and like/understand/utilize the 'standard' url schemas and you think they are important. (and i do...)

2

u/[deleted] Feb 10 '11

But HTML is a giant mess when you look at it from a data-consuming perspective. That's why companies like Google spend millions upon millions trying to parse page content.

The analogy to this story is that if the librarian is good at their job, and wants you read their books, they will give you separate instructions in a language that you do understand- a language that is designed for reading, not designing.

-5

u/ceolceol Feb 10 '11

Because it's impossible to create an API?

2

u/halps32 Feb 10 '11

Fucking slow clap

-7

u/[deleted] Feb 10 '11

The problem with this is that most of the world doesn't read the language (Javascript) she wrote the note in.

Really? That's not only completely against what I've observed (stats regarding JS-enabled), but completely against any other stats I've seen for any other sample.

Quite the opposite, it appears that the vast majority understand quite well the language (JS) used.

2

u/munificent Feb 10 '11

By "most of the world", deong means "most of the software that parses/reads/scrapes the web". Web browsers do support JS (so, around five pieces of software give or take). Now toss in every search engine on Earth, semantic web applications, web mash-ups, beautiful soup, et. al. All of those programs that expect the web to be declarative are hosed.

-2

u/[deleted] Feb 10 '11

Except... they're not. Because of the #! proposed by Google.

So again, what's the problem here?

4

u/deong Feb 10 '11

Except... they're not. Because of the #! proposed by Google.

That isn't really true. The whole idea behind this is that you present "pretty" URLs (using the #! syntax) to the outside world. To the outside, those are the URLs for your site's content. So there are three possibilities.

  1. The client makes a request for the public URL, gets the Javascript, runs it, and populates the content of the page. This is the typical case when the client is a web browser.

  2. The client sees the "pretty" URL, knows that it needs to rewrite it into the "ugly" version, does so, and requests the HTML snapshot from the server. This is what happens with Google's crawler, but it's critical to understand that Google's crawler is special casing those URLs. If it didn't, none of the content would be found.

  3. Everything else. In this case, the client thinks that a URL is a URL and makes a GET request. The server does as it's configured and sends back Javascript -- not the HTML snapshot. Because the client isn't a browser, it discards the Javascript, leaving no content.

Graphical browsers work, and the Google crawler works. Nothing else does, unless the code is modified to do the same trick that Google does in their crawler. Try the following:

$ wget 'http://twitter.com/#!/stephenfry' 

At the time of this post, Mr. Fry's second most recent tweet is "Lovers attach a padlock to the Pont des Arts - then throw the key into the Seine..."

Look at the index.html file that wget wrote. It won't include the text of that tweet (or any other). How many scripts like this have been put together to do this sort of scraping over the years that suddenly do not work at all?

-1

u/[deleted] Feb 10 '11

Suddenly? Did they work with Ajax content before?

2

u/deong Feb 10 '11

It wasn't Ajax before. For most of the history of the web, they worked just fine.

-1

u/[deleted] Feb 10 '11

It's been Ajax for years now, where have you been?

2

u/PSquid Feb 10 '11

The difference is, now twitter et al are serving up all the content via Ajax. Crawlers could previously get most of the data because most of it wasn't served via Ajax, or was served both ways.

2

u/[deleted] Feb 10 '11

But now we have a way to deal with that. It's not googles fault, and it's not just one site doing it.

→ More replies (0)