Think Your Code Runs the Same On All Platforms? Think Again!
Two weeks ago the game company Valve launched Steam, their online game distribution platform, for the Macintosh. Mac gamers rejoiced at the availability of a wide library of video games, and then they rejoiced even more when Valve announced that Portal, their award-winning game and cult classic, would also be available for free on the Mac and PC.
Of course a game like Portal hit uTest hard, sweeping its way through the offices like a bad cold in winter. One by one we each became addicts. But as games go Portal is short, and we all quickly returned to our software testing ways not long after. So besides us all letting off some steam (no pun intended), what does this have to do with software testing? Well, it turns out that the fine folks at Valve have been watching Mac vs. PC performance using monitoring tools built-in to Steam, and their results are quite surprising.
Steam has all kinds of tracking mechanisms to monitor game performance, hardware utilization, and even crash rates. Valve was very curious how the port of their own game engine, Source, was performing on the Mac. First, they learned that the Mac port of Portal is slow compared to the PC. That’s not too surprising since they’ve had years to perfect their game engine with Microsoft’s Direct X libraries. Meanwhile, they’ve only just started to develop for the OpenGL libraries on the Mac.
But their other results were also interesting. From their recent blog post:
Portal (with the same code base across platforms) is one fifth as likely to crash on a Mac than on Windows.
It’s not unreasonable to think that a port to a new platform would be slow, but it was entirely unexpected that it would be more stable. Valve currently attributes that to the Mac having less hardware diversity as a platform, but they’re still trying to figure it out themselves.
As testers, you can’t take anything for granted – especially when comparing products across different platforms. An application could be almost exactly the same on multiple platforms, with only the platform specific components processed through some low level abstractions. Valve’s Source Engine works exactly like that, actually. And yet Source still behaves very differently on Macs.
Do you have any good experiences to share about testing the same application across multiple platforms? Tell us about them below. There will be cake for all who reply.
