IE9 the best browser? Not so fast

It began innocently enough--a promotional blog post about Microsoft's newest version of Internet Explorer 9. But in less than a week, it became an illustration of just how rapidly misinformation moves through a hot and increasingly important corner of the software market. The browser market was already competitive a year ago, but the arrival of IE9, currently in beta testing, has added even more energy to the competition. That's because Microsoft has placed support for a host of modern Web technologies front and center, transforming IE from a drag on the Internet into an ally in developers' efforts to bring everyone a more powerful Web.

IE6, nearly a decade old but still widely used, has saddled Microsoft with a reputation for browser neglect. So what could be more eye-catching than the news that IE9 not only is back in the game, but that it leapfrogged the competitors?

Alas, as with many good stories, the truth turns out to be less dramatic and but harder to find. The IE9-is-better idea triggered teeth-gnashing on a Web standard mailing list and a hastily published disclaimer by the World Wide Web Consortium that's developing many of those Web standards. The episode shows the difficulties of keeping track of a browser market in the throes of breakneck change. Even those creating the yardsticks to measure the progress struggle to keep up.

IE9 and standards support

What happened with IE9? Last week, Microsoft released the sixth "platform preview" version of IE9. At the same time, browser makers had begun submitting test results to a new W3C suite of HTML5 standards compliance tests. When Microsoft discussed the new IE9 version on its IEBlog, the company included a link to the tests, calling them "an early version of the W3C's Official HTML5 Test Suite Conformance Results."

Shortly after came the headlines: "W3C Says IE9 Is Currently the Most HTML5-Compatible Browser" from Slashdot. "IE9 Outperforms Other Browsers for HTML5 Compliance" from ReadWriteWeb. "IE 9 Beats Chrome, Firefox, and Safari on Official HTML 5 Test" from Windows IT Pro.

Then came the alarmed reaction by those creating the standards. The W3C page, it turned out, was preliminary at best. "This test suite is vastly incomplete. Publishing unverified results of a vastly incomplete test suite without a big fat warning is extremely silly. Why was this done?" said Anne van Kesteren, an Opera developer who works on standards issues, in a mailing list posting Tuesday.

Added Ian Hickson, the Google employee who's editing the HTML5 specification, "I agree with Anne that it's rather pointless to be publishing results for this test suite. Realistically speaking the test suite isn't even 0.1 percent complete yet." Shortly after, the disclaimer arrived on the Web page with the results: "The HTML5 test suite is still being developed. The number of tests and the results on these tests will change. The results in this document may be updated, replaced, or obsoleted by others documents at any time. It is inappropriate to cite those results as other than work in progress and unstable."

In a presentation this week, leaders of the HTML5 effort said there are 215 tests included in the suite right now, with more than 600 on the way, and that the group needs "a lot more tests!" So declaring IE9 the winner on the test at this stage is something like saying England is better than the United States because it's got lusher lawns. The full range of tests aren't yet available.

Even worse, HTML5 is only part of the full range of Web technologies that are arriving. Others include Scalable Vector Graphics and Canvas for 2D graphics, WebGL for 3D graphics, Web Sockets for better live communications between servers and browsers, Web Workers for multithreaded JavaScript programs, and Cascading Style Sheets for increasingly advanced formatting.

Benchmark brouhaha
All these technologies are instrumental to transforming browsers from passive receptacles for server content into active engines for running Web-based applications. Accompanying the new standards is ever-faster JavaScript, the language for Web-based programs. Browser makers are scrambling to attract and retain users. As new features emerge, a golden age of Web demos has begun with helicopter games, ruffling curtains, exploding videos, computer aquariums, virtual reality, jiggling balls, and blooming flowers.

But what if you want something more authoritative? A natural way to get a handle on the chaos, of course, is to measure whose browser can run Web-based programs the fastest, or build Web pages to tally who supports what new features. Such seemingly neutral areas are where the problems are cropping up.

Back in the days of yore--say, early 2010--there were two prevailing methods that were useful if not perfect. For measuring JavaScript program speed, one could run the SunSpider benchmark. For checking compliance with new-era standards, one could run the Acid3 test.

Now, though, the browser world has begun moving to new tests. The only problem with calling winners in the browser race, though, is that the new tests really aren't settled down yet. The W3C tests are just one illustration. SunSpider stemmed from the WebKit project behind Apple's Safari browser and, more recently, Google's Chrome and any number of mobile browsers. Dissatisfaction is brewing with what's now a relatively elderly test, however.

"Because of...all the progress each browser vendor has made over the last several years, SunSpider is no longer particularly useful as a JS benchmark," Asa Dotzler in a blog post last week. "This is kind of obvious when you see that all of the top scores are pretty much tied. One one-hundredth of a second (across 26 tests) separates the slow from the fast, and that's just not particularly meaningful." Google found SunSpider wanting, too, and introduced its own V8 JavaScript benchmark suite. The first version arrived with Chrome, more than two years ago, but it's now up to its sixth revision.

And in September, Mozilla released Kraken, a new JavaScript speed test. It was developed to try to be closer to real-world computing challenges. Benchmarking is ever an imperfect science, to be sure. It's hard to measure the full breadth of computing chores, hard to weight tests toward the important components, hard to account for different hardware and network constraints, hard to factor in the latest technology. To draw a parallel from the auto industry, miles-per-gallon ratings for cars are reasonably useful--until electric vehicles come along.

And just as cataloging HTML5 feature support is a narrower task than the full panoply of Web standards, clocking JavaScript is only one aspect of browser performance. There's also the time to show a Web page, network performance, scrolling fluidity, and any number of other matters. Microsoft has been releasing a series of tests that exercise several parts of the browser--naturally the IE9 versions shine when it comes to the matter of hardware acceleration. All these tests are useful--as is Find Me By IP's browser support test and the HTML5 test site.

Unfortunately, it'll take time for the testing community to converge on new tests, even assuming the standards settle down enough to create meaningfully complete tests in the first place. So for the time being, we'll all have to live with inconvenient ambiguities in browser testing. Sounds like good practice for the real world.

Source: http://news.cnet.com/8301-30685_3-20021606-264.html#ixzz14MvV3X65

Related articles

• IE9 and Chrome 7 Top Early HTML5 Tests (pcworld.com)
• W3C seeks help and patience with HTML5 tests (news.cnet.com)
• W3C tests HTML5 browser compatibility, crowns IE9 the champ (engadget.com)
• IE9 Leads Pack in HTML5 Support? Not Exactly (wired.com)
• IE9 Scrubs up well against the HTML5 Competition (ghacks.net)