So to hear stories like these, we must look back in time at the great development projects of the past.  Today the Computer History Museum announced that Apple has donated the source code for the original MacPaint application so that it can be downloaded by anyone.  MacPaint was a drawing application included with the first Macintosh that by today’s standards seems very simple, but in 1984 was completely revolutionary.  Many of the graphic design tools we take for granted, like the paint bucket and lasso select, were invented in MacPaint.

For developers and testers alike, there’s a lot to learn from the development of MacPaint.  Here are a few good stories:

(From Folklore.org)

The last part of finishing MacPaint involved dealing with out of memory problems, since it was really pushing the limits of the 128K Macintosh by using the three, large offscreen buffers. At the worst case, there was only about 100 bytes free in MacPaint’s heap. Most of the bugs we encountered when running MacPaint turned out to be bugs in the underlying system, which were exposed by running so close to the edge of available memory.

(From The Computer History Museum)

In writing MacPaint, [Bill Atkinson] was as concerned with whether human readers would understand the code as he was with what the computer would do with it. He later said about software in general, “It’s an art form, like any other art form… I would spend time rewriting whole sections of code to make them more cleanly organized, more clear. I’m a firm believer that the best way to prevent bugs is to make it so that you can read through the code and understand exactly what it’s doing…

(From The Computer History Museum)

When the Lisa team was pushing to finalize their software in 1982, project managers started requiring programmers to submit weekly forms reporting on the number of lines of code they had written. Bill Atkinson thought that was silly. For the week in which he had rewritten QuickDraw’s region calculation routines to be six times faster and 2000 lines shorter, he put “-2000″ on the form. After a few more weeks the managers stopped asking him to fill out the form, and he gladly complied.

But my favorite story is about The Monkey.  MacPaint kept triggering bugs in the Macintosh’s hardware, but reproducing those errors was often impossible.  So the engineers borrowed the screen recording tools used to build the Mac’s guided tutorial program and then combined them with a little device that simulated hitting the keyboard and moving the mouse around randomly.  It looked like a monkey was playing with the computer.  The end result was a cheap and easy automated testing solution that discovered a number of additional Mac bugs.

But The Monkey had one problem: it would sometimes randomly quit the application, cutting testing short.  So the developers created a system routine called “MonkeyLives” that would disable calls to quit if The Monkey was in use.  For years after that, a mysterious flag existed in the Mac’s system memory called “MonkeyLives” – all in the service of early automated testing.

We recently learned about Apple’s “Top Secret” design and testing lab thanks to MG Seigler of TechCrunch, who was given access to the state-of-the-art facilities just days before he mysteriously disappeared (kidding).

For some, the futuristic lab has conjured up images from the movie Star Gate, although I think it looks more like the Senate floor from Star Wars (episodes I through III). Here’s Seigler with a more technical description, as well as some insight into how Apple actually uses it:

Inside Apple’s headquarters in Cupertino, CA, there are a collection of rooms that house 17 giant anechoic chambers. Basically, they’re rooms where no waves (sound or electromagnetic) can reflect off of anything, so there is absolutely no interference when it comes to wireless testing. Apple places their devices from iPhones to iPads in these chambers to ensure the performance is up to their standards.

So how do they test it? There are four stages. The first is a passive test to study the form factor of the device they want to create. The second stage is what Caballero calls the “junk in the trunk” stage. Apple puts the wireless components inside of the form factor and puts them in these chambers. The third part involves studying the device in one of these chambers but with human or dummy subjects. And the fourth part is a field test, done in vans that drive around various cities monitoring the device’s signal the entire time (both with real people and with dummies).

So where did Apple go wrong? And what can this controversy teach us about the difference between in-the-lab-testing vs. in-the-wild testing? Below the jump are four critical lessons that companies ignore at their own peril:

1. An insanely expensive and elaborate lab is still a lab
Notice how the lab contained rooms where there is “absolutely no interference when it comes to wireless testing.” Does this sound like an environment that exists in the real world? Yes, it’s absolutely helpful for testing some aspects of such a solution. But at some point, you actually want the wireless interference (and all the other stuff that real users must deal with). No matter how much a company spends to replicate reality, it will never become reality. A lab is still a lab.

2. You ALWAYS need a fresh set of eyes (plus ears and hands)
Apparently, testing was completed by those who were extremely close to the actual product – those who knew the device inside and out, top to bottom, front to back. As a result, many usability issues (like the death grip, for one) were never factored into the testing equation. Only through ‘in-the-wild’ testing (i.e. real devices in the hands of real users) can a company expect to uncover these type of unique issues.

3. Non-standard use cases are vital to acing the user test
Apple’s testing team should be applauded for driving around in a van, testing the wireless connection. But why stop there? Why not think of 50 more unusual scenarios – aka “non-standard use cases”? Of course, relying entirely on lab testing to accomplish these tasks would be extremely expensive, even for Apple. But by using elements of crowdsourcing, they would be able to extend their testing coverage relatively cheap.

4.Global users? Global testing!
As the iPhone 4 makes its way into other markets, it will be interesting to see whether users encounter issues specific to their geo-location. Judging from the centralized conditions of the Apple lab, it should not surprise any readers of this blog if such complications appear. To sum it all up: your product needs to work where your users are. For 99.997% of companies and apps, this is NOT in a lab.

Are there other lessons to be learned from Apple’s testing methods? Have you seen other companies pay the price for relying on lab testing, while discounting “in-the-wild” testing? Drop us a comment and tell us all about it.

Here’s a quick video “unveiling” of the Apple’s impressive test lab:

As part of our newly-launched “What Do uThink?” series (more on this shortly), we decided to ask our community which mobile OS they considered to be the best. Here are the results:

1. Android – 38%
2. RIM Blackberry – 28%
3. Apple – 16%
4. Symbian – 12%
5. Windows Mobile – 6%

“What do uThink?” is a weekly poll, where we’ll be asking the uTest community their preferences and feedback on various apps, operating systems and other technologies. To encourage voting, we’ll be awarding monthly and quarterly prizes to randomly selected participants. This quarter, for instance, we’re giving away an iPod Touch. The weekly polls open every Tuesday afternoon and voting takes place in the uTest Forums available to registered testers) as well as on our Facebook page. Got it?

Good. Now back to the mobile OS results…

As you can see from the fractured vote count, there’s a strong case to be made for each mobile operating system. Below are some updated excerpts from the Forums thread where this question first appeared.

The case for Android, posted by “pedro gonzalez”:

“As a Google OS, Android knows better than any other OS how to manage Google maps, Gmail, Geolocators, browsers, and a lot of other Google applications. I’m a Gmail user, so the Gmail sync is one of the most valuable things for me.

There are a lot of tools for developers/testers within SDK, in addition the testing part is much better in Android, since the OS is much flexible than others (less limitations allow the programmer-tester to be creative).

Android is not as mature yet as other OS (Blackberry for example was released 10 years ago), although Android is getting better and better each release. For example, Adobe’s Flash player is supported in mobile devices for the first time ever from the last Android release.

There are a lot of cons too regarding the OS maturity (like the total 254Mb limitation), but it seems that the Android guys are working on this.”

The case for Apple, posted by “madhukarjain”:

I also voted for Apple….

Once you use it you will know why Apple can become a part of you’re life, so much that you can’t think of switching to another phone.

From GPS naviation to gaming to stock trading to radio…..you name it and Apple has an App for it and mostly for free, even if not free then most apps are 99 cents which is worth the money for its use.

Many people might disagree and will consider Blackberry or Android as better, but once you use its full capacity then you realize whats the Magic of Apple.

The case for RIM, posted by “jayadinu”:

1.Obviously its performance
2.GSM for its roaming ability
3.Camera and video capability
4.Full QWERTY keyboard
4.Push Email services and many other factors like “A very good deal that I could afford “

Care to weigh in on this debate? The comment box is all yours. Want to vote in the next weekly poll? Visit the uTest Forums or check out our Facebook page.

I mean, the World Cup is over and nobody really cares about baseball until September, so perhaps this could help fill the competitive void in the meantime…

Here are the current “standings”:

1. Apple
2. Oracle
3. Microsoft
4. HP
5. Adobe Systems
6. IBM
7. VMware
8. Cisco
9. Google
10. Mozilla Organization

As noted earlier, this is really more of a marathon than a sprint, so it would be useful if we went back a little longer than six months to crown a winner. Thankfully, Secunia did just that as part of their key findings:

• Since 2005, no significant up-, or downward trend in the total number of vulnerabilities in the more than 29,000 products covered by Secunia Vulnerability Intelligence was observed.
• A group of ten vendors, including Microsoft, Apple, Oracle, IBM, Adobe, and Cisco, account on average for 38 percent of all vulnerabilities disclosed per year.
• In the two years from 2007 to 2009, the number of vulnerabilities affecting a typical end-user PC almost doubled from 220 to 420, and based on the data of the first six months of 2010, the number is expected to almost double again in 2010 to 760.
• During the first six months of 2010, 380 vulnerabilities or 89% of the figures for all of 2009 has already been reached.

ZDNet’s Adrian Kingsley-Hughes made note of the products that are causing these vendors to rise or fall in the standings. The culprits, he says, are as follows:

• Apple - (iTunes, Quicktime)
• Microsoft – (Windows, Internet Explorer)
• Sun Microsystems – (Java, now part of Oracle)
• Adobe - (Acrobat Reader, Flash)

So testers, who do you see as the Bug Marathon winner (i.e. loser) at the end of the year? Not that we’re keeping score or anything.

After tracking down Alex Brie, a developer who first discovered the issues, PC World reported:

“After Brie’s calculations, Nguyen would have needed at least 3,000 hacked iTunes accounts to reach the ranking he had on Sunday in the App Store…[and] Brie speculates that to achieve such high ratings for his apps, Nguyen had to hack into Apple’s iTunes servers and skip the normal security steps, or run an automated scripted program.”

According to Engadget, Apple responded last night:

The developer Thuat Nguyen and his apps were removed from the App Store for violating the developer Program License Agreement, including fraudulent purchase patterns…

I was under the impression that the App Store approval process was brutal. So, how did this rogue developer get through? What additional security measures and tests need to be put into place to prevent account fraud?

Case in point: The majority of mobile app developers remain overwhelmingly reliant on internal beta testing.

Here with proof is VisionMobile, who recently published a fascinating report on the growing mobile app ecosystem – a must-read for anyone involved in the space (developers, marketers, users, etc). From a QA point of view, the report further establishes that although testing innovations will ALWAYS trail those of development, the gap need not be so wide.

Here’s an excerpt that sums the whole thing up:

Internal beta testing is the most popular technique used by the vast majority (nearly 70 percent) of respondents, with beta testing with users and peer reviewing the next most popular techniques. Only 20 percent of respondents use focus groups or research of their own. Overall, North American developers are somewhat more sophisticated in their application planning, with 97 percent using beta testing as a standard part of application development and with broader use of a portfolio of planning techniques as well.

Yet, small development firms have limited means today to beta test and peer review their applications with a crosssection of representative users. Given the hundreds of thousands of mobile apps, we believe that efficient (crowd-sourced) testing of apps in a global market of users is considerably under-utilized. This presents an opportunity for the few solution providers in this segment – Mob4Hire and uTest.com, for example – but also for network operators, who can generate a channel for testing applications with end users, and provide an open feedback support system back to developers.

Other notable findings included:

Android as mindshare leader. Android stands out as the platform most popular with mobile developers. Our survey results suggest nearly 60 percent of all mobile developers recently developed on Android, assuming an equal number of respondents with experience across each of eight major platforms. iOS (iPhone) is second in terms of developer mindshare, outranking Symbian and Java ME, which were in pole position in 2008.

Debugging. In terms of debugging, our benchmarking shows that Android has the fastest debugging process, compared with iPhone, Symbian and Java ME. Debugging in Symbian takes up more than twice the time it takes on Android.

Open source. On average, 86 percent of respondents who use open source at work use it within development tools such as Eclipse. Android and iPhone developers are three times more likely to lead open source communities, compared to Symbian, revealing the contrasting pedigree of the developer communities. The single key drawback to open source reported by 60 percent of respondents was the confusion created by open source licenses; we believe education on open source realities can be used as a competitive advantage for developer programs launched by operators and OEMs.

The Industrial Revolution era (2010-2014). The next five years will completely remap the mobile industry landscape. RIM and Apple, two verticalised companies, move into the top five, displacing the incumbents, leaving one Finnish and two Korean companies in pole position. The operating system landscape will consolidate into two tiers; the top-end open to iconic products dominated by Apple and followed by the iPhone clones powered by Android; and the feature-phone market where licensable operating systems (Android and BREW) will finally allow handset OEMs to move away from legacy RTOS platforms. Google’s Android will also power a diverse range of new form factors, from picture frames to car dashboards, offering for the first time a simplified platform from which to achieve convergent interconnected services. In this age of Industrial Revolution, mobile developers will be responsible for most of the innovation on mobile devices, and can act independently from the mobile industry powers-that-be – OEMs or network operators – to get their applications to market. In this age, developers have both power and choice.

We believe that Android’s lead in developer mindshare ahead of Apple’s iOS is down to two factors: first the $99 fee developers have to pay in order to deploy their applications, an entry barrier which reduces the innovation from developing countries. Secondly, the very effective use of open source licensing as a marketing technique to attract developers to Google’s Android.

So forget everything you know (or thought you knew) about the current mobile app landscape. Like the folks at VisionMobile said, it will be totally unrecognizable in five years. Except, of course, for the internal beta testing.

Testers, do you see the beta testing trend reversing any time soon? Will Android soon be the king of all things mobile? Will crowdsourced testing still be under-utilized? Let us know what you think.

The hackers claim they reported the flaw to AT&T before sending their discovery to the fine folks at Gawker. AT&T, on the other hand, was not pleased to see their security problems appear in a popular tech blog at all, and had this to say in an email to their iPad customers:

On June 7 we learned that unauthorized computer “hackers” maliciously exploited a function designed to make your iPad log-in process faster by pre-populating an AT&T authentication page with the email address you used to register your iPad for 3G service.

…

The hackers deliberately went to great efforts with a random program to extract possible ICC-IDs and capture customer email addresses.

So who’s to blame for a problem like this? Is it AT&T, or do the hackers themselves deserve some of the blame for the public way they handled their disclosure? Give us your thoughts below.

Yes, the storm clouds are gathering.  Off in the distance, we can see Safari 5, IE9, Chrome 5 and Firefox 4 in various stages of envisioning, development or launch.  And just like the good ole days, the combatants aren’t wasting any time in taking aim at the competition.  MG Siegler over at TechCrunch outlines the initial skirmish in what figures to be a protracted battle among 800 lb. heavyweights.

For those who haven’t yet waded in and taken a side in this looming battle, here are a few product reviews (or previews) from some well-respected sources:

• Safari 5 review from ars technica
• Chrome 5 review from ComputerWorld
• IE9 preview from CNET
• Firefox 4 preview from Wired’s WebMonkey

We’re just beginning to experiment with the betas here in the uTest offices, but I’m curious to hear if any testers or devs have started using these new versions yet.  If so, drop us a comment and share your thoughts. What’s clear is that the latest round of browser wars will be fought along the lines of speed, tab management & placement, extension management and HTML5 support.

New Display – a Treat for Sore Eyes
Before today, I could not have told you the maximum amount of detail my retina could process in a given inch of space.  Now, thanks to Steve Jobs, I know: 300 dots per inch. That’s important because the iPhone 4 features an astonishing 960×640 display that resolves to 326 pixels per inch. That’s apparently more than what your retina can physically see, a fact that Apple took to heart when they decided to name this feature “Retina Display“.

More incredible is how this compares to the iPad. The iPad is a significantly larger device, but with a resolution of 1024×768 it only has 28% more pixels.

Of course, this also means that Apple developers now have three resolutions to worry about when they’re developing apps: iPhone 4, iPad, and iPhone/iPod 2G/3G/3GS.

A4 – A Super Fast Processor with a Super Short Name
With the launch of the iPad, Apple introduced the A4 processor – a custom system-on-a-chip CPU developed by secret engineers Apple has locked away somewhere. The A4 has been important for the iPad for two reasons: it’s incredibly fast and it uses very little power. That gives the iPad a longer battery life while making it insanely speedy.

Now the iPhone 4 will have an A4 processor as well. That means it will feature a huge performance boost over the 3GS while actually using less power. Everybody wins! You should thank those smart Apple engineers, assuming you can actually find them.

Gyroscopes – No Longer Just for Museum Gift Shops
Apple has also augmented the iPhone’s built-in accelerometer with a gyroscope.  With this feature, developers will be able to measure the tilt of an iPhone as it moves.  That will make it easier to develop games and other apps that can use the angle of the phone as an input. Imagine tilting your phone to steer your race car or rotate a game.

Front Facing Camera + Video Phone = Answer Phone Fully Dressed
If you thought Chatroulette was fun, wait until all iPhone users have the ability to make and receive video phone calls. Apple calls the feature “FaceTime” and it will either change your world or make you incredibly paranoid.  Either way, get ready for an awkward learning period where the world figures out video phone etiquette. Rule #1, video phones and driving probably don’t go well together.

iPhone OS 4 – Now iOS 4
Apple has put the letter ‘i’ in front of nearly every word in the computer dictionary, and quite honestly we thought they had run out by now. But they surprised us by rebranding the iPhone OS as simply iOS. The new version, which we discussed previously, will be known as iOS 4.

Other Notes
Two other takeaways from today’s show. First, always have a backup plan. Midway through Steve Jobs’ presentation, the WiFi in the auditorium briefly stopped working. Even though Steve couldn’t finish presenting a feature online, he was able to switch over to showing the feature using photos instead. You can watch the issue unfold here.

Second, Steve also revealed the three most common reasons an app is rejected from the App Store:

1. The app doesn’t function as advertised by the developer
2. The app uses private APIs
3. The app crashes

As usual, software testers still have plenty of work to do.

Through all of this bitterness, many have argued that Flash is ready for the deadpool – some even cheering its demise.  I disagree.  Actually, I believe just the opposite is true.  Here are 5 reasons why Flash won’t be going away anytime soon.

1. HTML5 is still very immature.
HTML5 is everyone’s favorite choice as a Flash replacement. Read the comments sections on just about any blog or article about this topic, and HTML5 is often hailed as the greatest thing to happen to computing since Apple “invented” the mouse (with Xerox’s help).  The problem with HTML5 is that it’s still an immature and unfinished platform.  While it’s supported by the very latest versions of Firefox, Safari, and Chrome, it’s not yet fully supported in Internet Explorer (although IE9 will bring support eventually). If most of the browsers on the web don’t yet support HTML5, it’s not a fully supported standard.

Then consider that even the editor for the HTML5 standard says that it won’t be finalized until 2022, and that it won’t even be in a final draft stage until 2012:

It is estimated by the editor that HTML5 will reach the W3C Candidate Recommendation stage during 2012. That doesn’t mean you can’t start using it yet, though. Different parts of the specification are at different maturity levels. Some sections are already relatively stable and there are implementations that are already quite close to completion, and those features can be used today (e.g. <canvas>). But other sections are still being actively worked on and changed regularly, or not even written yet.

The canvas tag is the feature that most people associate with HTML5, but for browsers to simply support that one tag is a far cry from a Flash replacement.  Still lacking are any tools for syncing audio and video, interactive vector graphics, and any kind of development environment.  HTML5 canvas also has no performance optimization.  A recent study showed that unless you’re using Safari on the Mac, Flash performance beats HTML5 every time.

This isn’t to say that HTML5 is a bad standard, it’s just not yet ready and won’t be for several years.  Even when it is ready, it’s doubtful that it will completely replace Flash.  In the mean time, developers can’t stop and wait for standards to settle before creating new rich applications and animations.

2. HTML5 video is an unresolved issue.
Let’s say that you want to replicate in HTML5 the most famous Flash site on the Internet: YouTube. Sorry, you can’t do that – at least not consistently on every browser. That’s because nobody can agree on whether the HTML5 video standard will be H.264 or Ogg Theora.  And unless something seriously changes (probably involving billions of dollars somewhere), this debate isn’t going anywhere.

The problem here is that H.264 is a patented and proprietary algorithm owned by the MPEG group. The royalty structure is currently vague, and there are no guarantees that users won’t be forced to pay licensing fees in the future.  However, Apple, Google, and probably Microsoft have licensed H.264 and adopted it for their browsers.

Which leaves out Firefox.  Firefox’s open source standards prohibit it from incorporating patented standards that are not specifically royalty free.  For that reason, Firefox only supports Ogg Theora for HTML5 video.  Theora was developed as an open video standard that specifically avoided patents.  The trouble with Theora is that guaranteeing it doesn’t infringe on someone’s patent is impossible, and for that reason no one wants to touch it for risk of becoming a litigation target in the future.

The only ways out of this problem would be for MPEG to freely license H.264 in perpetuity or for someone with deep pockets to really dig into Theora to make sure it’s patent free.  Even if Theora were patent free, it may not be possible to improve on it without infringing on patents.

That means that video publishers will either continue to distribute their videos in Flash (which is universally supported) or distribute their videos using HTML5 in both H.264 and Theora.  Since Theora is a second encoding, then every video would need to be encoded twice.  That means twice the computing costs and twice the storage costs. YouTube has conspicuously left Firefox off of their HTML5 beta for this very reason.

Flash suddenly seems a lot cheaper and easier for video.

3. Flash 10.1 changes everything.
Flash is frequently described as buggy and slow.  Actually, it’s buggy everywhere but only slow on the Mac.  Adobe took that criticism to heart and has made a huge number of fixes and changes to Flash 10.1.  It should be faster and more reliable, especially for Mac users.  Anandtech reviewed the beta release and liked what they saw.  You can try it too by downloading the Flash 10.1 beta for yourself from Adobe Labs.

Which brings me to another issue entirely.  Adobe may have taken forever to put together Flash 10.1 (too long in the opinions of a lot of Mac users), but they still put it together in less time than HTML5 has been in development.  What’s more, as user tastes and requirements change, Adobe can launch new versions of Flash that take advantage of all kinds of exciting new multimedia and design requirements.  HTML5, on the other hand, is a very robust but slow moving standard that won’t be finished until 2022.

What happens when the world wants 3D video, virtual reality goggles, or brainwave YouTube?  HTML5 will undoubtedly include them at some point, but Flash will get there first.

4. Apple’s ecosystem is growing more rigid and more fragile.
As Apple tries to either control or monetize more and more of their ecosystem, they also make it more and more rigid and inflexible.  There are two ways this can turn out: Microsoft in the 1990s or Apple in the 1980s.  TechCrunch considered this point with their article Is Steve Jobs Ignoring History, Or Trying To Rewrite It? Microsoft built a dominant position in desktop computing in the 1990s which produced enormous value for them.  Apple, on the other hand, lost this battle in the 1980s.

Building a tightly controlled ecosystem is a huge challenge over the long-term.  It’s great when customers want your product because you can easily set terms for everyone who develops around it.  If developers want a piece of your big pie, they need to play by your rules.  On the other hand, when market tastes change, a controlled ecosystem makes it much harder for you to adapt. Rigid ecosystems are fragile ecosystems.

It’s very likely that Apple will over time come to view Flash as something that strengthens their platform. Supporting Flash will be an easy way to port applications to the iPhone / iPad when (if) consumers eventually change their tastes.

5. The web is mature, mobile isn’t.
Mobile computing today is a lot like desktop computing was in 1985.  At that time the players were all there and the winner looked like Apple.  Then they made a series of decisions that looked good at the time but were disastrous in hindsight.  That’s the trouble with being the innovator in a new space – you have to make every decision correctly to stay at the front.  The market is maturing so rapidly that if you fail, someone else will quickly take your spot.  Microsoft took Apple’s desktop leadership spot first with Windows 3.1 and then more firmly with Windows 95.

The web, on the other hand, is more mature.  It’s been around for over 20 years, and Flash has been around for 14 of them.  That means that standards slip in and out of place much more slowly.  Think about how entrenched IE6 is and how we’re still testing for it.

Flash is ubiquitous on the web which makes it easy to target for development.  Among web users, Flash is installed on over 90% of computers, and the only reason it’s not more common on mobile is because Adobe hasn’t perfected a low impact version.  I’m betting they’ll figure that out sometime in the next year, and then we’ll find out whether cutting ties with Adobe was a good move or a bad move for Apple.

What do you think?  Sound off below.