Last month, the uTest Community Management team kicked off the uPanel Webinars!

Rather just listening to one person talk in a straight presentation, the uPanel Webinars are very open and involved discussions on helpful topics for the uTest tester community. During each webinar, the panelists respond to questions from both the moderator and the audience.

In one of our most recent (and very popular) webinars, we explored the topic of video documentation for mobile bugs, something that developers are increasingly requiring from their testers. Video documentation helps to clarify the actions leading up to the discovered issue, and especially valuable with in-the-wild testing. Many testers struggle with determining the best method for capturing a bug in video on particular devices in a way that will give the TTL and the customer the most detail possible.

The video capture uPanel Webinar explored these challenges, and provided best practices for capturing video documentation of bugs in mobile test cycles. Kayla and Tara made some great recommendations for tools and tips to follow when trying to grab the right visual info for developers and customers. Check out the webinar below: Read more …

Just how fragmented is the world of real-life, still-in-use software? More fragmented than you’d like to think.

Not only are you contending with frequently released operating systems and versions, the back log of versions you need to support isn’t shrinking as quickly as most developers would probably like. Just how old is some of the software people are still clinging to? Let’s look at some numbers.

The old standby browser, Internet Explorer, is famous for having users who aren’t the quickest to update. Despite repeated rumors of its demise, IE6 is still kicking ever so slightly (with less than 1% of IE market share), according to stats from W3Schools. Internet Explorer’s market share is mostly spread across versions 8-10, though IE 10 hasn’t quite caught up to the others. So, if you want to cover the entire fragmentation of Internet Explorer, you need to test on five different versions.

In the mobile world things aren’t any better. We’re up to the seventh major version of the Android operating systems (with several sub-versions mixed in). If developers want to support all versions with a sizable market share, they need to account for Gingerbread, Ice Cream Sandwich and Jelly Bean – all of which have more than 25% of the Android distribution. While it pales in comparison to the big players, Froyo still accounts for nearly 4% and even Eclair has almost 2%. Total: 3 major versions that definitely need support and 2 older versions that should possibly be considered (location and popular device manufacturer of your target market will be a major deciding factor).

iOS is less forth coming with their distribution stats but David Smith took a look at the users of his Audiobook app to give people a general idea of what overall version distribution might look like.

On all mobile platforms (iPhone, iPad and iPod Touch) users are overwhelmingly using OS versions 6+ (more than 80% of users). Step down a major version and you reach another 10%. Anything lower than OS 5 is pretty nominal and developers like David have stopped supporting these versions (which each grab 1-2%).

Audiobooks no longer supports iOS versions prior to 4.3.0. The population below these limits was below 4% when support was dropped. Residual usage will occur at these OS versions but no new customers are being added.

Now for the kicker. Windows XP was released 12 years ago and is still in use … by a lot of people. From CNET:

Microsoft’s XP operating system is still used on more than a third of the installations out there, according to figures from Net Applications. Is it really still that popular? …

On Friday, figures from Net Applications showed XP with a robust 37.74 percent of all Windows and Mac OS installations worldwide, down only slightly from 38.31 percent in April.

It’d be difficult to support all past versions of software and the hope is that if enough developers stop supporting it eventually these holdouts will update. In the meantime it’s important to pay attention to the version distributions in your field (be it mobile, desktop or web) and support as many versions that have significant users.

As with all data, don’t take these stats blindly. Do a little digging to figure out which versions are most important to your user base and be sure to develop and test for those. If you’re having trouble achieving test coverage in this fragmented world, crowdsourced testing can give you access to a lot more hardware/software combinations than you could ever house yourself. If you’re going to support multiple versions, do it right, don’t skip testing.

Although we don’t play favorites when it comes to dev methodologies, you can’t deny the benefits of Agile development.

The ever-popular approach has helped countless companies increase flexibility within their organization, and thus develop better, more intuitive software.

However, there are ways to do an Agile project all wrong. David Taber, of CIO, recently published a list of the top ways to blow up an Agile project, which he says can make even the best Agile team melt down.

So what are the ‘worst practices’ for Agile development? Here’s a look:

1)      Use The Element of Surprise

“The agile effect comes from trust (through a team that’s deputized and empowered to make key decisions on its own), as well as close communication and relentless focus on the highest-priority goals. Surprise the team with new mandates, unstated top priorities, an uncertain budget, political cross-fire or just a simple lack of trust and it’s hard to imagine anything more damaging to both morale and productivity. Surprise your team and watch it melt.”

2)      Don’t Put Your Best Players on the Team

“One thing’s for sure: The folks who conspired to write The Agile Manifesto are smarter than the average bear. Way smarter. Agile tends to work best when you have users who are articulate and know their business processes cold. Effective agile developers are usually wicked smart.

t’s not just soloist brain-power, though. The team members must respect each other. If they don’t, communication will not be quick and smooth. In my experience, location matters, too. It’s best if team members are within shouting distance, if not in the same room, while it’s almost impossible to have great productivity if the team is spread across multiple time zones.”

3)      Use Politics for Leverage

“You know the drill. Find and emphasize Catch-22 situations, exploit optics and rhetoric, use budgetary threats, shuffle the team, promote and demote people randomly, install a Czar—preferably someone who knows nothing about software but is an I’m-in-charge-here, get-it-done, take-no-prisoners kind of guy—then lather, rinse and repeat. These steps work wonders to show your power, and to break down internal trust and external communication.”

4)      Don’t Give Team Access to Test Resources Until Beta

“What could be duller than testing? Seriously, the continuous building and changing of agile software makes testing an endless chore for the users and a bore for the compliance guys. Besides, the test labs can’t afford to be available all the time.

You’ll hear these kinds of arguments. You have to overrule them. You don’t have to be a card-carrying member of the test-first crowd to know the economics of software defects: Catch them early and fix them cheaply. Here’s the agile answer: If you test throughout the project, you won’t need a beta at all. The software will just migrate into production when it’s ready.

5)      Accel erator-Brake, Accelerator-Brake

“Agile lets developers and users innovate fast and become high-functioning teams. This depends upon continuity of effort and objectives. Sure, detailed priorities shift continuously—but if the global purpose of a project jumps around, the good effects of agile will never take hold.

The same is true if there are interruptions of more than a couple of weeks, as the flywheel never really gets going. Remember: Continuity matters. Due to a number of human factors, restarting an Agile project that’s been stalled for a month is almost worse than starting from scratch.”

Read the rest here>>

As Taber highlights in worst practice #4, beta testing isn’t the right time for testing… and overlooking QA is a sure way to demolish your Agile project. In fact, if testing is done correctly (beginning early on, and done continuously) a beta test won’t be needed at all.

In an Agile setting testing needs to be incorporated into each step of the process, not just at the end of each major release. Real world QA done throughout each step of development, paired with the lab testing that Taber mentions, can help dev teams avoid costly bugs and mistakes. It also keeps the bugs out of the hands of the application’s users, unlike a beta test.  After all, you don’t want to expose your user base to a buggy product. But as you can see from Taber’s mistakes to avoid, the challenges don’t end with testing. Planning for Agile, documenting and organizing Scrum meetings quickly while maintaining an Agile environment requires a unique approach where all the key players are involved and on-board.

Agile development has great potential to help teams quickly and efficiently produce higher quality software – but it has to be done right.

For more best practices on adopting Agile, download this free whitepaper: Ten Tips for Understanding Agile Development and Making it Work for You.

What do you think is the biggest challenges in adopting an Agile development approach? Let us know in the comments section.

Now before you go and actually spend 10,000 testing to answer that, let us first put the question into context.

There’s an idea floating around that in order to become an expert in a given field, one must spend roughly 10,000 hours honing those particular skills. Apparently, no one has taken credit for the 10k hour rule, but it’s believed to have been popularized by author Malcolm Gladwell in his book Outliers. So if you spend 10,000 hours on something and don’t become an expert, blame him. Or at least get your money back.

Anyway, for some professions, the 10k rule would seem to make perfect sense. But we’re not talking about some professions here, we’re talking about testing. And in my view, there’s a lot more that goes into becoming a testing expert than just time. There’s passion, intelligence  curiosity, work ethic and many other “nurture” traits that can never be substituted for time. Time certainly helps – don’t get me wrong – but it’s not everything.

To illustrate my point, take a look at this quote from a CNN.com article on the Secrets of Greatness:

For example: Simply hitting a bucket of balls is not deliberate practice, which is why most golfers don’t get better. Hitting an eight-iron 300 times with a goal of leaving the ball within 20 feet of the pin 80 percent of the time, continually observing results and making appropriate adjustments, and doing that for hours every day – that’s deliberate practice.

In other words, it’s not the amount of time you spend testing that will make you an expert, it’s how you spend that time. If you plan on spending 10,000 hours testing, make sure you spend it wisely! As James Bach once said in on our blog:

Read more …

Not just anyone can claim to be an exceptional software tester. Software testing requires a unique skill set, and the best software testers are the ones that continuously strive to expand their knowledge and hone their testing strategies. In short, good testers always want to get better.

So how can you become a better tester?

We recently posed this question to our community of 80,000 QA professionals in the uTest Forums. Here’s a look at the top 10 responses we got from the uTest community, and why these tips are so important:

1. Test for quality over quantity: “Here’s 10,000 bugs… good luck!” Testers, please don’t ever shoot for quantity. Identifying the most important bugs and glitches, and helping the company or developer make sense of the bugs is ten times more helpful then testing for mere volume.
2. Learn to prioritize: In line with “quality over quantity”, prioritizing what you test is extremely important. Testing the mission critical parts of an application before the minute details of an app will help you to identify the most valuable bugs first. This will also allow the development team to fix the most imperative parts of their application as quickly as possible.
3. Practice and improve your written communication skills: Everyone can right, write? (Ha!). Good testers must have excellent written communication skills in order to write good test cases, bug reports and so on. These testing artifacts are an essential part of QA and must be detailed and easy to consume.
   
4. Learn from your own mistakes – and from others too:  Everyone makes mistakes, but learning from others and from your own will make you better tester. How can you improve your bug report next time? How can you prioritize better during the next test cycle? How can you communicate better with the development team? These are questions you should constantly be asking yourself, and your fellow testers.
   
5. Be objective and professional: Every time you test, go at it with a fresh perspective. Look at the software you are testing without bias or past experience, and test with an open mind. Testers who think “Oh, I know this software” or “I’ve used this before” are at risk of overlooking important bugs. Objectivity is key.
   
6. Don’t be humble with software… think out of the box: Explore the software, ‘test to break’ and be willing to suggest improvements; these are all attributes that make up the attitude of a good software tester.
7. Question. Everything: Does this work as intended? Does it work on all devices? Does it work under every possible use-case, every time? Question. Everything.  
   
8. Think like the user: Remember; your job is to find bugs before the software reaches the hands of users. Pair your technical skills with an end-user’s mindset and you will find the best, most valuable bugs possible.
   
9. Increase the effectiveness of bug reports: Attaching screen shots and providing detailed bug reports will give the developer the information he or she needs to understand the bug and fix it. Where did it occur, when, how many times, on what devices, running what operating system and under what circumstances? Without the right details a bug is useless to a development team.
   
10. Be Passionate! To excel in any field, you need to be passionate for what you do. Read, seek new training opportunities, engage with your fellow testers, attend testing conference, classes – immerse yourself in all things QA.

What do you think it takes to become a better tester? Share your thoughts with us in the comments section, or join in on the discussion in the uTest Community Forums!

Focusing on your staff and production methods can help you meet the challenges of releasing a product quickly and on time. I’m not talking “we’re going to use the Agile method because it’s ‘better’ and faster.” I’m talking about looking at your teams – both as individual players and cohesive units – and understanding how they can work more efficiently.

Expert after expert has said that a methodology is only good if it works for you – you shouldn’t adhere strictly to a method simply for namesake. That same sentiment can be applied to maximizing the efficiency of your teams. Don’t throw work at a team simply so they’re always busy. Pay attention to the time they need to complete a task well and set the flow that way. Overloading a team will bog them down and force them to rush through work to keep up. Rushed work is poorly done work and can cost you down the line when you need to release a patch for an issue that made it through to production.

Derek Huether, of Leading Agile, says that like the original Ford Motor plants, it’s about maximizing the flow of work, not the amount of work.

Henry Ford did not have everyone working at 100% utilization.  If everyone worked at 100%, the result would have been congestion — bottlenecks within his (assembly) system and the production of excessive parts inventory.  Instead, one of the many things he did was focus on limiting lead times.  That’s the time something waits before an activity happens.  By understanding his system, he was able to have the right amount of people, working at the right pace, in the right sequence, in order to maximize flow (delivery through the system).

He recommends understanding four key factors to help maximize the flow:

• Understand Current and Potential Capability and Capacity
• Understand the Delivery System and Establish Goals
• Balance Capacity and Capability with delivery throughput
• Monitor Performance

You can read more about each point at Leading Agile.

The key here is figuring out what works for your teams. Don’t flood them for the sake of keeping them busy, but don’t let them have so much down time that they get bored and lose their focus.

Creating a truly efficient environment needs to spread beyond the QA department. Waiting until the end of the SDLC to address quality and testing is only going to slow down the process. Simple problems need to be addressed or avoided earlier in the process. Have QA professionals review the product’s purpose, features, early designs and any potential road maps – they may be able to spot problems before anything is coded.

Read more …

There’s a lot to like about the technology space. Mainly, the innovative products and services that seem impossible just a short time ago. There’s also a lot to dislike about the technology space. Mainly, anyone that disagrees with you.

I’m talking of course about the tech rivalries that dominate the headlines, comments sections, and now, even television ads. But which rivalry is the most heated? I felt the only reasonably way to decide this was to put it up for a vote:

 Loading ...

Great conversations, insightful panels and keynotes, rows of innovative mobile brands… and J.Lo.

That’s pretty much what CTIA’s 2013 show entailed this week in Las Vegas. Here’s a look at a few things we loved about CTIA 2013:

• Sessions: It was easy to love the sessions this year. Day 1 started off with the “Apps Summit” featuring the latest trends in mobile and a lot of talk around development and design. All the sessions were great, especially “Making Commerce Mobile: Strategy and Trends of Mobile Commerce” led by Sharon Wienbar, member of the uTest board and partner at Scale Venture Partners. The panel was also led by Jake Ward, co-founder and executive director at Application Developer Alliance, and others. The session looked at the latest trends in mobile technology, as well as the challenges brought on by device fragmentation and the ever changing mobile landscape. Another favorite was on day two, and was led by Michael Facemire, Forrester Analyst. The session titled “Technology Solutions for Today’s Mobile App Development Challenges” addressed the variety in mobile technology, time-to-market demands, UX requirements and how to overcome all of these challenges.
• E-Tech Awards: A big highlight of CTIA was the 2013 E-Tech Awards. Attendees voted and an online panel of highly-respected industry experts, reporters and analysts ultimately judged the submissions on “innovation, functionality, technological importance, implementation and overall ‘wow’ factor,” according to the CTIA blog. We’re thrilled that uTest’s Labs’ app analytics tool, Applause, took home 3rd place in the Enterprise Solution – General Business category. Congratulations to Ericsson’s Connected Vehicle Cloud and GPS Dashboard LLC’s “Check in” App for Salesforce.com, which took first and second place – and to all the nominees!
• Cool Brands: The Sands Expo Center was packed this week, with never-ending rows of new and upcoming mobile apps, development tools and countless other types of brands in the mobile space. We had a great time chatting with folks near our booth in the Mobile Apps Pavilion such as Kodak Developer, CallSnap and Speakerfy, as well as all the hundreds of attendees who stopped by our booth to talk about the importance of in-the-wild testing.

So what was the absolute best part about CTIA 2013?

Maybe you won big on the black jack table, or took a gondola ride through the Venetian – but for us it was the continued power of mobile. Every developer, QA Manager, or business exec we talked to, and every keynote or session we sat in on, was further validation of how mobile continues to be the driving force behind brands of all industries and sizes. Businesses know that their app is their brand and source of revenue – and they also know just how important app quality is. We love that.

So there you have it. A big thanks to CTIA for putting on a great show. See you next time, Las Vegas!

In Part I of this month’s Testing the Limits series, Jonathan Kohl talked the life of the Agile methodology, his relationship with and advice about mobile app testing, and how new and season testers alike can advance their skills and feed their passion.

Today, he’ll talk about the biggest hang-ups teams have when it comes to testing, how to overcome those hurdles, how he became a fan of “gamification” in testing and a few other words of wisdom.

********

uTest: When speaking or consulting on mobile application testing, what’s the most common question you encounter and what’s your answer?

JK: This is the question that comes up the most after people have heard me talk about testing on mobile projects:

“Is it true that we need to test on real devices, incorporate movement, and test out in the real world, outside of the office?”

My answer: “Yes.”

Mobile technology has been developed to support movement, and it has a lot of dependencies. For example: networking (while moving you change networks, technology, and encounter lots of errors or weak signal conditions), location services (all about movement and spending time in different locations), weather and environmental conditions (temperature and light have an enormous effect on how some apps work), and movement (the devices have movement sensors, and are interacted with using touch screens and voice control. Combining inputs and sensors getting triggered from movement can be tough for apps to handle.). I know of no tool that helps us sort that out without getting physical with the devices. Hopefully we get better ones soon that focus on mobile technology, instead of treating it like a small PC/web browser. 

uTest: You work a lot with startups, when there’s a shoe string budget how do you communicate the importance of testing and QA?

JK: A successful business requires execution on several dimensions, for example, understanding your customers and their needs, having a great product or service offering that solves real problems for people, having enough investment to enable you to deliver, and a strong grasp of your financials. These require different skills and focus, but there is an over-arching need to create value for your customers, and part of that is great customer service. Technology fits directly into that, especially as we view organizations through the window (screen) of a smartphone or tablet. A poor technology experience is another type of poor customer service, so there is a direct line there. If you let down your customer or provide them with a poor first experience (which we are finding is increasingly on mobile devices), you can lose them forever. So not only do we need to have great people skills and a good strategy for satisfying and impressing our customers in our human interactions, but in our technology interactions as well.

A lot of people don’t realize that we are evaluated on our ability to deliver on the technology front, and that technology is a major part of customer service. It fits into the customer experience, but also in our offering of a product or service. Does the technology help, or hinder? Do we have an awkward experience, or is it seamless. It also has an effect on investment and financials, even if it is indirect. A quality product and technology stack is vital to the success of a business over the long term, so we need to invest in ways to make sure that we meet our customer’s needs there as well. Good quality processes, development and testing are part of that, and we ignore them at our peril.

uTest: Part of your consulting work is to help teams adjust to methodology changes. What’s the biggest hurdle these teams need to overcome and how do you help them do that?

JK: Dealing with people issues – different individual people with different agendas, motivations and fears. On one hand, you have people who have bought the marketing of a methodology and think that it will cure all ills. On the other, you have people who are deeply threatened by the change the methodology represents. It is a difficult balance to reign in the exuberance and unrealistic expectations of one group, while encouraging the laggards to get on board and actually try it out. I spend a lot of time myth busting on both groups. The methodology will not make everything perfect (and it won’t work forever), but it will also not cause you to lose your job, or cause bad things to happen to you. I work with teams by doing – getting in there and demonstrating, and helping address their fears. If I can do it, anyone can do it.

It’s also important to help them be able to measure the success of their methodology adoption in real terms: better product, better customer service, saved costs, etc. Just measuring success on adopting a new process misses the point, and will lead to some difficult situations down the road, particularly if that new process isn’t helping you create value for you, for your team, for the company, and for your customers.

People issues are hard to cope with. They aren’t fun, and it means we have to confront people on difficult topics. However, healthy disagreement and healthy confrontation are good, healthy things. If we suppress conflict, it just goes underground. We reach peaceful solutions through negotiation and co-operation, not by suppressing them in a need to “be positive.” There is also nothing quite like trying something out and seeing whether it works or not. Evidence, generated by a team that is committed to getting results, not just in implementing a methodology change, is powerful, and helps build teamwork and consensus towards reaching goals. Talking is one thing, doing and getting feedback on how we are doing helps enormously.

Read more …

Our Testing the Limits guest this month is Jonathan Kohl, a consultant and technical leader who writes and speaks on a wide range of software development and testing topics.

In this interview we talk to Jonathan about his passion for the field, what’s changed over time, his take on mobile app testing and his advice for new and seasoned testers. To learn more about Jonathan visit his website or follow him on Twitter @jonathan_kohl.

********

uTest: I think it’s safe to say that you’ve jumped into the world of software development and testing whole-heartedly. What drew you to this field and how do you stay passionate about it?

JK: The bottom line: I love to build things, and creating great software with a talented team is incredibly rewarding. Knowing that we have created something that helps make people’s lives easier is gratifying. I have found an ideal mix of problem solving, technology, creativity and satisfying and impressing customers in the software development industry.

I have great friends in the industry, and we keep each other sharp, and work on side projects together. We can talk about the latest technology, tease each other about programming tools we like and dislike, or branch off into just about any topic when we’re together. We also serve as a great support network for each other when someone is struggling, either technically, or personally. When we are working together to solve a difficult problem, nothing describes that energy of collaborating and working past your weaknesses, and that triumph of shipping working software at the end.

I ended up in this field by accident – I got lost on my way to law school. In the mid 90’s, I had three university professors who were incredibly influential: Dr. Michael Kubara (Philosophy), Dr. John Rutland (Management) and Dr. David Cowan (Mathematics). I was taking logic and philosophy classes from Dr. Kubara, and he told me to talk to a Math professor because I was delving into territory that he didn’t have expertise in. Dr. Cowan was teaching me Linear Algebra at the time, and jokingly said: “What are you doing with this ancient logic when the real research is in computer science?” Dr. Rutland had us study Data General for his class, and I was reading Tracy Kidder’s book The Soul of a New Machine.  The quirky nature of the technical people in the book appealed to me (I was playing in a band at the time, and read legal journals and philosophy papers in my spare time, yes, I was pretty quirky myself) and I loved the elegance of logic, and the questioning nature of philosophy. I agreed to try out a computer science class, and ended up getting sucked into an internship at a software company not too long after learning to code.

The software company reminded me of what I had read in Kidder’s book Soul of a New Machine, particularly the interesting, intelligent technical people. I enjoyed hanging out with them, and working on really difficult problems. I loved the fast paced, work-hard, play-hard culture of a software startup. I’ve stayed here ever since, with a brief trip back to university to finish off my undergrad degree. Once in a while I am tempted to pursue the law degree again, but I am invariably sucked back into a great project or opportunity.

uTest: You were an early adopter of Agile and later started talking about “post-Agilism.” What are your thoughts on the Agile movement over the years? (Since you could probably write an entire book on this topic give us the dust jacket version.)

JK: I started looking into Agile methods when I was working on a team in 1999 that was trying a combination of Rapid Application Development (RAD), Open Source, and iterative/incremental models like Spiral. The lead developer sent us this article called “Chrysler Goes to Extremes” which described the very basics of Extreme Programming. We liked that it was open and free (we didn’t have to purchase some tool chain and coaching we couldn’t afford) and we started trying some of the ideas out. We then bought the book “Extreme Programming Explained” by Kent Beck, and started implementing XP as best we could. A year later I was introduced to Scrum, and we had good success with it. After a while, I started noticing failures on Agile projects, and people moving on and doing other things. That was fascinating to me. Sadly, instead of learning why there were failures, there was an overwhelming urge to suppress them, or worse, dehumanize people by blaming them for doing it wrong.

As for Agile methods in general, I am ambivalent.  I am glad that lightweight methodologies are much more common place than they were in the ‘90s, and we have benefited from creating a common language around practices, and our tools are so much better now than they were ten years ago. I enjoy working on many Agile projects, since they fit my process and personality, and how I like to work.

However, there is a lot of snake oil out there, with proponents claiming that merely adopting Agile methods will lead to a successful business (What about having a great product, great customer service, skilled people and strong financials? Don’t those sort of matter too?) That really turns me off.  Furthermore, for years, any Agile failure would inevitably involve blaming the victim: you did it wrong, you don’t get Agile, if you were really Agile it would have been a success, which often sounds like another variation on the “no true Scotsman” argument fallacy. Sure, sometimes people fail because they made some mistakes, or didn’t commit, but every Agile project that fails can’t be blamed on the people.

Read more …