got bugs?

Using Charles For Throttling Bandwidth During Load Testing Using JMeter

2011-11-15T18:47:00.001-08:00

Whenever I've been asked to do load testing for internet-connected apps, I've usually needed to simulate several lower speed internet connections. An example of this was the need to simulate the connections between classroom computers and web servers hosted at a school district. In some cases those connections were T-1 speeds and in others the connection speed was that of an ISDN connection. I needed to simulate all these speeds, along with 10BaseT/100BaseT connections. Rather than employ an expensive hardware solution, I opted to use Charles, an HTTP proxy / HTTP monitor / Reverse Proxy.

Charles can be used to adjust the bandwidth and latency of an internet connection, which enables you to simulate lower speed conditions using a high-speed connection. The bandwidth may be throttled to any arbitrary bytes per second. This enables any connection speed to be simulated. Latency may also be set to any arbitrary number of milliseconds. The latency delay simulates the latency experienced on slower connections, that is the delay between making a request and the request being received at the other end.

Here are the steps we used to throttle connection speeds between each JMeter server and the target server under load:

Install and launch Charles on a separate computer, i.e. not the JMeter server or target server.
Choose Proxy menu -> Throttle settings and Check Enable
Select the desired connection speed. If you need a speed greater than 256kbps, select 256K ISDN/DSL from the list, since it's the fastest connection available, and change the Download and Upload bandwidth (kbps) to the desired speed, e.g. 1500 for T-1.
Press OK
Disable Recording and Caching
Setup Reverse Proxy so that the JMeter servers point to the bandwidth throttle computer and the bandwidth throttle computer points to the target server under load

Testing on Agile Projects

2011-11-15T18:46:00.001-08:00

Do agile projects become the brunt of cubicle Dilbert jokes? Scott Ambler confronts Scott Adams.

Agile projects are both interesting and challenging, especially for testers. Too often these projects and their processes are misunderstood or misapplied. What's most uncomfortable to testers is what appears to be a complete abandonment of any sort of serial testing flow.Moreover, the lack of upfront specifications for all features and functionality seems to be a complete affront to not only testing best practices but also common sense. Suffice it to say that a lot is misunderstood about Agile testing.

Unfortunately this misunderstanding of what can be an effective development and testing process ends up being the brunt of cubicle jokes.

Now I wouldn't say that testers assigned to agile projects see things as comically as Dilbert, but it can be confusing for testers if the development processes don't include them. For this very reason, and based on some past experience, I was happy to find an article, titled Agile Testing Strategies, that spoke to agile testing specifically. Of all the articles I've read on agile development, this one made the most sense to me as a tester.

The author, Scott Ambler, does a great job of relating agile practices to testing and breaks down this subject into 5 parts:

This is a good read and reference document for any tester trying to make sense of their agile development process. I believe that if you spend the time needed to begin to understand the agile process and use Scott's advice and testing process overlays, you'll be able to not only understand your role as an agile tester, but graduate to driving the agile testing strategies for your projects.

Top 25 list of software errors

2011-11-15T18:45:00.000-08:00

It certainly helps the software testing and QA community when the NSA and the Director of National Intelligence highlight the Top 25 bugs. This list, which resides on the Common Weakness Enumeration (CWE) website, fully describes each error in an understandable taxonomy. My favorite reference to this list comes from an article on Network World's website: "Will the Top 25 worst software errors list released Monday be able to rescue customers from rotten software?" It's a good question for software developers and testers to consider. We'd certainly view this list as a starting point for any security testing project. For a complete list of the Top 25 sofware errors, read more at the jump...

Software security
The Top 25 programming errors
1. Improper input validation
2. Improper encoding or escaping of output
3. Failure to preserve SQL query structure (SQL injection)
4. Failure to preserve Web page structure (cross-site scripting)
5. Failure to preserve operating system command structure (OS command injection)
6. Cleartext transmission of sensitive information
7. Cross-site request forgery
8. Race condition
9. Error message information leak
10. Failure to constrain operations within the bounds of a memory buffer
11. External control of critical state data
12. External control of file name or path
13. Untrusted search path
14. Failure to control generation of code (code injection)
15. Download of code without integrity check
16. Improper resource shutdown or release
17. Improper initialization
18. Incorrect calculation
19. Porous defenses
20. Use of a broken or risky cryptographic algorithm
21. Hard-coded password
22. Insecure permission assignment for critical resource
23. Use of insufficiently random values
24. Execution with unnecessary privileges
25. Client-side enforcement of server-side security

Is It Possible To Achieve 99% Cumulative Defect Removal Efficiency?

2011-11-15T07:00:00.000-08:00

I'm finding that many articles on testing continue to be pessimistic about outcomes. Furthermore, they contribute to the idea that "you can't get there from here" - you'll never find all the bugs so quit trying to use engineering rigor and metrics. In our recent article, How Important Is Coverage When Testing Software?, the experts that were interviewed, in answering the question "When is testing done?", gave answers such as "never", "two months after development is done", and "when producers decide". I get what they are all saying and actually agree with them on some levels. But I'd rather see conversations like this address continuous process improvement and not tend towards capitulation. Indeed, the practical should win over the theoretical, but without discussions in the theoretical realm, we really can't advance that which we do as testers.

And there's research that supports this pessimistic view. In fact, most forms of testing are not very efficient, and find only about 25% to 40% of the bugs that are actually present[1]. No wonder the experts from the article mentioned above, when asked when testing is done, answered never. But what we're missing here is the investigation of "most forms of testing." What is that? Well it must be what we're doing if we all somehow agree that you can't find all the bugs in a product. The important thing is, that no matter how you attempt to answer this question, it's not something we want to reinforce! The question to ask (and answer) is "What other forms of testing must we do in order to find ALL the bugs in a product?" As it turns out, there are folks that spend time trying to answer that question.

Defect removal efficiency originated in IBM in the early 1970’s as a method for evaluating testing effectiveness. Because this metric provided information that could be used for inspections and static analysis, as well as testing, it has become a general way of measuring the relative effectiveness of all forms of defect removal activity. Recent findings by companies such as IBM, Coverity, and Dovél Technologies show removal efficiency levels that often top 96%.

96% removal efficiency? Really? Yes, and there are expectations that defect removal efficiency can approach 99%. Now we're talking quality. How is this being done, i.e. what other forms of testing are being done to achieve this? As it turns out, there's no silver bullet. But there is an engineering process that contributes to this - coverage is expanded through different defect prevention and detection tests types. Here's the list of what those are:[2]

unit test
new function test
regression test
performance test
security test
usability test
system test
some form of external test with customers or clients, such as Beta test or acceptance test

The reason I say there's no silver bullet is because these are not new test types. The question "What other forms of testing are being done to achieve this?" I posed earlier was not about new forms of testing, but about combinations of testing types. The above tells me that, if you're not performing all of these tests, you have holes in your coverage and can expect to find less defects. What, then, keeps us from doing all these tests?

If you want to find out more, check out the article Predicting Test Cases and Test Team Size, by Capers Jones, President, Capers Jones & Associates LLC.

[1] Predicting Test Cases and Test Team Size, by Capers Jones
[2] ibid

Selenium+ Fitnesse: A QA Multiplier

2011-11-15T06:00:00.000-08:00

If you're interested in how to use domain-specific language (DSL) for creating easy to read test cases, then take a look at a demo Matt Krapivner and I did at Sauce Labs last year. We used a framework called Fitnesse in combination with Selenium to create a DSL for testing the Shutterfly web site.

Does Quality Matter To You?

2011-10-20T08:00:00.000-07:00

Why Don’t Customers Seem to Care About Quality?

We once had an experience with a large printing service that brought into question quality. We had requested that a print job be completed in a very short amount of time, to which we were asked "Does quality matter to you?". I suppose they were asking if we would be interested in trading quality for time. We were not. Quality did matter to us. Perhaps this printing service had customers that were not so interested in quality.

So, are there software customers out there that don't seem to care about quality? The answer is YES if you subscribe to what Watts Humphrey has to say about software quality. He says software customers don't seem to care about quality because "defective software works." [1]. How about that! So, as testers, should we care if customers don't seem to? Well the answers is that we do care and, in fact, it's because we do care that we create this entity that both works and is defective. How can this be? Well it's because of our use of models in the development of our testing coverage. Our "use models" are defined by our best guess as to how users will use our software. And because we generally guess well when creating our use models, and consequently test well against these models, the end product works. So, why does Watts then say it's "defective"? It's because the software works within the scope of the expected use models, or as Watts calls it, the "testing footprint." [2] The diagram below shows what this footprint looks like:
The yellow area is what we don't know about new uses for the software we develop, and the green is what we know or think we know about how our software will be used. Combined, they create this somewhat schizophrenic entity - "defective software that works." Users that stay within the use model will find that the software works, while users who start thinking of new ways to use the software will probably encounter serious problems.

A good example of "defective software that works" is manifest in contemporary security vulnerabilities. When forum software first came out, it was probably tested to ensure that the insertion of html code would work so that people could insert URL links. However, the use models used for development and testing (at the time) did not envision "new uses" for embedded html - cross site scripting or XSS. However, once that use model was considered, the testing footprint expanded to account for XSS.

Is there a solution to this dilemma? Do we just try harder? Well, if you read more about what Watts has to say about Quality, he doesn't sound all that optimistic and forewarns that "it will take a severe, disruptive, and highly public software failure to get people concerned about software quality.” [2] He believes that with this new concern will come the need for a change in our attitudes towards Quality. That may seem a little weak if you are looking for a "silver bullet" fix, but if trying harder isn't going to make a difference, then a wholesale change of how we develop and test software is probably needed. And the only way we are going to radically change our current software development processes is with a new attitude that begins with the belief that defect-free software is possible.

[1] Humphrey, W., “The Quality Attitude”,news@sei newsletter, Number 3, 2004.
[2] Humphrey, W., “Defective Software Works”, news@sei newsletter, Number 1, 2004.

Tips and Techniques for Using Apache JMeter for Load Testing

2011-09-15T18:49:00.000-07:00

Apache JMeter is a great tool for load testing and we use it for LAN-based products as well as web sites and hosted services. And lately we've used it in conjunction with Amazon's Elastic Cloud Compute (EC2) service. Along the way, we've collected a number of tips and techniques that are sometime difficult to find in the documentation.

Here's a list of changes that we've found to be necessary each time we use jmeter:

Change Memory Settings
The default memory settings are too low need to be increased to the maximum memory available. To do this:

1. Navigate to jmeter/bin.
2. Edit jmeter (Linux command-line: "pico jmeter", Windows: open jmeter.bat with wordpad)
3. Scroll to the line that reads: HEAP="-Xms256m -Xmx256"
4. Change the numbers to meet your system specs (first number is the minimum, second maximum). For example: HEAP="-Xms1024m -Xmx1024"
5. Save changes.

Retries & Timeouts
If you are doing load testing with jmeter, inevitably you'll place enough of a load on your target server that it may have trouble responding. In order to give the server a chance to respond you may want to change the retries and timeout values. To do this, edit the jmeter.properties file for retries = 3 and timeout = 120. Be sure to delete the #s at the start of the lines or they will remain commented out.
--Note: The generic HTTP Request does NOT support these features, and the HTTP Request HTTP Client sampler type must be used.
--Note: The HTTP Request HTTP Client sampler does NOT support automatic redirect so be sure to deselect this (its checked by default) when creating the JMeter tests.

Ports (Windows only)
If you've ever seen this error message when running jmeter "JMeter Exception: java.net.BindException: Address already in use: connect" you've run out of ports. To avoid running out of available ports on Windows (and thus seeing a Address Already In Use error), there's a great explanation and solution posted on the twit88.com blog at this article.

Software Testing Hall of Fame: Dr. Boris Beizer

2011-06-13T08:00:00.000-07:00

We buy their books, memorize their quotes and credit them in our footnotes and bibliographies. Those that we consider experts in the software development and testing field are an important part of our professional lives. Enjoy this biographical insight into a well known expert and add to your collection of notable quotes.

In this biographical article I pay homage to my favorite, Dr. Boris Beizer. I had the good fortune to sit in on one of his lectures right before he retired from the lecture circuit. Dr. Beizer received his PhD in computer science from the University of Pennsylvania in 1966. He has written twelve books, ranging from system architecture to his well-known pair on software testing -- Software Testing Techniques and Software System Testing and Quality Assurance -- both considered standard references on the subject. His latest book is Black Box Testing, an introduction to testing technology. He directed testing for the FAA's Weather Message Switching Center and several other large communications systems. He has been a speaker at many testing conferences and is also known for his seminars on testing. He consults on software testing and quality assurance with many organizations throughout the world.

Beizer's oft-referenced "Bug Taxonomy" is contained in Software Testing Techniques. Although the taxonomy is copyrighted, you can occasionally find it online. See SourcesOfBugs.

Here are some of his more notable quotes from SoftwareQuotes.com:

"First law: The pesticide paradox. Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffective. "- Boris Beizer - Chapter 1, Section 1.7. Boris notes that farmers solve this problem by planting sacrifice crops for the bugs to eat, and laments that programmers are unable to write sacrifice functions., Software testing techniques by Boris Beizer , ISBN: 0442206720

"More than the act of testing, the act of designing tests is one of the best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded - indeed, test-design thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design, coding and the rest."- Boris Beizer - Software Testing Techniques, Creating a Software Engineering Culture by Karl Eugene Wiegers , ISBN: 0932633331 , Page: 211

"Testing proves a programmer’s failure. Debugging is the programmer’s vindication." -- Boris Beizer, Software testing techniques by Boris Beizer , ISBN: 0442206720

“One of the saddest sights to me has always been a human at a keyboard doing something by hand that could be automated. It’s sad but hilarious.” -—Boris Beizer, Black-Box Testing: Techniques for Functional Testing of Software and System

How Important Is Coverage When Testing Software?

2011-06-06T08:00:00.000-07:00

I ran across what I thought would be an informative and thought provoking video on software testing coverage. The title of the video, "When is software testing done?", seemed interesting. And since the video was touted in this way:

Is software testing ever done? Listen to what the industry experts had to say at STARWEST.

I had expected a conversation about the importance of coverage and how to assess defect metrics to determine progress against a coverage model. Instead, I got something that struck me as a pessimistic view of software development that falls short of an engineering endeavor. Surprisingly, coverage was only mentioned once. See what you think.

Mathematics Curriculum Meets The "Negroponte Switch"

2011-05-31T08:00:00.000-07:00

The "Negroponte Switch", attributed to MIT's Nicholas Negroponte, posited that signals that historically traveled by wireless methods (television, for example) would soon be transported by landline technologies, and signals that historically traveled by landline methods (telephone, for example) would soon be transported by wireless technologies. This view is referred to as the Negroponte Switch because it suggests a reversal of traditional signal transport markets.[1] In the video below, Mathematics Professor, Arthur Benjamin, contends that Calculus should not be the pinnacle of our mathematics curriculum, and it should be replaced by statistics and probability. He makes the interesting point that since the world has changed from analog to digital, so too should our mathematics curriculum.

[1] David Roddy, Flipping Negroponte's Switch, 1/1/1998.

Keyword-Driven Testing HOWTO Part 4 - Dictionary and Test Case Script

2011-05-23T08:00:00.000-07:00

This is the final article on keyword-driven testing (KDT). The previous articles leading up to this one discussed creating a keyword list, preparing the keyword list for automation, and, test automation "handlers". Today we bring it all together with a final script that contains "handlers" for each keyword in our dictionary, as well as the test cases proposed earlier. I've implemented this KDT automation engine in AppleScript. The script is organized in two sections. The test cases themselves are at the top and the KDT dictionary handlers are below the test cases.

Enjoy!

--Test Case #1
Calculator("Run")
SelectView("Basic")
Calculate("10", "20", "*", "200")
Calculator("Quit")

--Test Case #2
Calculator("Run")
SelectView("Scientific")
PaperTape("Show")
Calculate("10.234", "20.322", "/", "0.50359216613")
Calculator("Quit")

--Test Case #3
Calculator("Run")
SelectView("Basic")
PaperTape("Show")
Calculate("10", "20", "*", "200")
WindowMinimize("Calculator", "Minimize")
Calculator("Quit")

-- Calculator Run|Quit
on Calculator(action)
if action = "Run" then tell application "Calculator" to activate
if action = "Quit" then tell application "Calculator" to quit
delay 1
end Calculator

-- Select View Basic|Scientific|Programmer
on SelectView(view)
if view = "Basic" then
tell application "System Events"
 set frontmost of process "Calculator" to true
 click menu item "Basic" of menu 1 of menu bar item "View" of menu bar 1 of application process "Calculator"
end tell
end if
if view = "Scientific" then
tell application "System Events"
 set frontmost of process "Calculator" to true
 click menu item "Scientific" of menu 1 of menu bar item "View" of menu bar 1 of application process "Calculator"
end tell
end if
if view = "Programmer" then
tell application "System Events"
 set frontmost of process "Calculator" to true
 click menu item "Programmer" of menu 1 of menu bar item "View" of menu bar 1 of application process "Calculator"
end tell
end if
delay 1
end SelectView

-- Paper Tape Show|Hide
on PaperTape(mode)
if mode = "Show" then
tell application "System Events"
 set frontmost of process "Calculator" to true
 click menu item "Show Paper Tape" of menu 1 of menu bar item "View" of menu bar 1 of application process "Calculator"
end tell
end if
if mode = "Hide" then
tell application "System Events"
 set frontmost of process "Calculator" to true
 click menu item "Hide Paper Tape" of menu 1 of menu bar item "View" of menu bar 1 of application process "Calculator"
end tell
end if
delay 1
end PaperTape

-- Calculate Operand1,Operand2,Operator,Result
on Calculate(op1, op2, operator, expectedResult)
tell application "System Events"
set frontmost of process "Calculator" to true
keystroke op1
keystroke operator
keystroke op2
keystroke "="
try
 set myResult to (get value of static text 1 of group 2 of window "Calculator" of process "Calculator")
end try
try
 set myResult to (get value of static text 1 of group 3 of window "Calculator" of process "Calculator")
end try
if expectedResult ≠ myResult then display dialog "Error"
end tell

end Calculate

-- Window Calculator|Paper Tape,Minimize|Unminimize
on WindowMinimize(which_window, minimize_state)
tell application "System Events"
set frontmost of process "Calculator" to true
if minimize_state = "Minimize" then click button 3 of window which_window of application process "Calculator"
if ((minimize_state = "Unminimize") and (which_window = "Paper Tape")) then click UI element "Paper Tape" of list 1 of application process "Dock"
if ((minimize_state = "Unminimize") and (which_window = "Calculator")) then click menu item "Calculator" of menu 1 of menu bar item "Window" of menu bar 1 of application process "Calculator"
end tell
end WindowMinimize

Keyword-Driven Testing HOWTO Part 3 - Test Automation Engine "Handlers"

2011-05-16T08:00:00.000-07:00

This week's article on KDT introduces a key concept for automating our keywords - handlers. In the previous article, Keyword-Driven Testing HOWTO Part 2 - Preparing The Keyword List For Automation, we finalized our keyword list and demonstrated their use in some preliminary test cases. Now we need to begin automating the keywords so that testers can begin writing and executing automated test cases without needing any programming skills.
To refresh our memories, here's the finalized keyword list:

• Calculator Run|Quit
• Select View Basic|Scientific|Programmer
• Paper Tape Show|Hide
• Calculate Operand1,Operand2,Operator,Result
• Window Calculator|Paper Tape,Minimize|Unminimize

Our job now is to write a handle for each of these keywords so that the test cases we discussed in the previous article, like the one below, will make sense to the automation test engine.

1. Perform calculation using the Basic calculator without using paper tape.

Calculator Run
SelectView Basic
PaperTape Hide
Calculate 10,20,X,200

But first we'll need to understand the concept of a handler and what it looks like to program one. And has I mentioned in the first KDT article, Using Keyword-Driven Testing To Understand The Benefits of Model-Based Testing (MBT), we're going to program our handlers in AppleScript. If we had chosen Visual Basic Script or some other language, we would probably implement our handlers as part of a CASE statement. But for Applescript, handlers are a fundamental part of the language, hence we won't need to add anything to our automation engine to make them work.

A handler in Applescript, as described in the AppleScript Language guide, is:

"...a collection of statements that can be invoked by name. Handlers are useful in scripts that perform the same action in more than one place. You can package statements that perform a specific task as a handler, give it a descriptive name, and call it from anywhere in the script. This makes the script shorter and easier to maintain. The definition for a handler specifies the parameters it uses, if any."

That's a perfect fit for what we intend to do with KDT: we're going to refer to things by specific names (our keywords) and, in some case, pass parameters along with the name, e.g. Calculator Run. And what's cool about AppleScript and how it employs handlers, is the way in which you call upon these handlers to do something. You simply type a handler's name and any parameters and the handler goes to work. Here's an example of handler that speaks any word you send it:

on speak(this)
 say this
end speak

and the way you call upon this handler to do its job is simply:

speak("Hello")

As you can see from this example, the handler's name is "speak" and the parameter you pass it is what you want to have spoken, in this case "Hello". Here's the Applescript code that includes both the invocation of the handler and the handler itself:

speak("Ray")

on speak(this)
 say this
end speak

So you can think of the "speak" handler as representing a keyword in our KDT dictionary and speak("Ray") as a test case. In next week's article I'll reveal each AppleScript handler needed for our dictionary and convert the sample test cases to work with the those handlers.

Keyword-Driven Testing HOWTO Part 2 - Preparing The Keyword List For Automation

2011-05-09T08:00:00.000-07:00

In last week's Keyword-Driven Testing (KDT) HOWTO article, Keyword-Driven Testing HOWTO Part 1 - Creating a Keyword List, we identified the keyword list for testing the Mac OS X Calculator. The list we came up with was:

Run Calculator
Quit Calculator
Select Basic View
Select Scientific View
Select Programmer View
Show Paper Tape
Hide Paper Tape
Calculate
Minimize Window
Un-minimize Window

Our goal this week is to normalize, parameterize and tokenize this list in preparation for automation. The end result will be two fold: 1) a KDT "dictionary" for non-programmer so that they can easily construct automated test cases; and, 2) a well defined set of inputs for the automation engineer for use in the creation of the test automation engine. In other words, the keywords work for both tester and automation engineer. So let's start by first normalizing and then parameterizing this list.

Several of the words in our list relate to each other, e.g. Run Calculator and Quit Calculator. We can normalize Run Calculator and Quit Calculator into one keyword: Calculator. To do that, we'll need to define parameters for the keyword Calculator. Those would be {"Run","Quit"}. So now we can refer to launching or quitting the Calculator as Calculator Run or Calculator Quit. Using this examples, here's how I would normalize the rest of the list:

Calculator Run|Quit
Select View Basic|Scientific|Programmer
Paper Tape Show|Hide
Calculate Operand1,Operand2,Operator,Result
Window Calculator|Paper Tape,Minimize|Unminimize

We've now taken a list of 10 keywords and reduced it to 5 by normalizing related keywords and adding parameters. Now let's tokenize this list to remove any confusion as to what is a key-word and what is a parameter. We'll do this by removing any spaces in the keywords. This also makes it easier for the automation engineer to parse.

Calculator Run|Quit
SelectView Basic|Scientific|Programmer
PaperTape Show|Hide
Calculate Operand1,Operand2,Operator,Result
Window Calculator|Paper Tape,Minimize|Unminimize

We've now normalized, parameterized and tokenized our original list of keywords and are ready to hand them off to the automation engineer for the development of the test automation engine. Before we do that, let's "test" our list by writing a few test cases to get a feel for how to use this KDT dictionary. Let's say you're interested in creating an automated regression test suite that exercises the Calculator using a few typical use-cases such as:

Perform calculation using the Basic calculator without using paper tape.
Perform calculation using the Basic calculator using paper tape, then quit.
After performing a calculation, minimize the Calculator window but not the paper tape window.

Here's what that would look like using our newly created KDT dictionary:

1. Perform calculation using the Basic calculator without using paper tape.

Calculator Run
SelectView Basic
PaperTape Hide
Calculate 10,20,X,200

2. Perform calculation using the Scientific calculator using paper tape, then quit.

Calculator Run
SelectView Scientific
PaperTape Show
Calculate 10.234,20.322,\,0.50359216613
Calculator Quit

3. After performing a calculation, minimize the Calculator window but not the paper tape window.

Calculator Run
SelectView Basic
PaperTape Show
Calculate 10,20,X,200
Window Calculator,Minimize

Next week we will begin the development of the test automation engine. We'll start by looking at what a "handler" is and how it relates to our key-words.

Keyword-Driven Testing HOWTO Part 1 - Creating a Keyword List

2011-05-02T08:00:00.000-07:00

Last week I introduced keyword-driven testing in "Using Keyword-Driven Testing To Understand The Benefits of Model-Based Testing (MBT)", and promised to provide another Testlabs.com HOWTO series on this methodology. But first I'd like to define the scope of this HOWTO series in terms of what keyword-driven testing (KDT) is.

If you've done any googling of keyword-driven testing, you may have concluded that it simply involves coming up with a list of keywords that resemble actions you'd find in a set of manual test cases. And you'd be right for the most part. It certainly can be made more complicated, and the tools vendors may lead you to believe keyword-driven testing is complicated enough to warrant the purchase of their customized tools. So to gain a middle-of-the-road perspective, I researched what others had to say about KDT and came upon this simple description, and one that matches the KDT we designed for HP in San Diego, from Ayal Zylberman at Qualitest:

Keyword-driven testing is actually a dictionary that provides the entire organization with a customized language to build test automation scripts.

Certainly there's no right answer when it comes to developing this dictionary and a lot depends on how the test designers within the organization envision the use of KDT. So having said that, I plan to keep this HOWTO series simple enough to convey the concepts of KDT, while providing a useful set of examples that will allow you to start using KDT in your organization.

In order to develop some keywords for our dictionary, we'll need an application to test. As I mentioned in my previous article on KDT, we are going to use the Calculator application in Mac OS X as our software under test. And to keep our dictionary small, I've chosen to create my first keyword list based on running/quitting the app, the main calculator views, numeric calculations and minimizing/un-minimizing the calculator window. Based on this, here's a first pass at my keyword list:

Run Calculator
Quit Calculator
Select Basic View
Select Scientific View
Select Programmer View
Show Paper Tape
Hide Paper Tape
Calculate
Minimize Window
Un-minimize Window

Next week we'll normalize, parameterize, and tokenize this list in preparation for automation.

Using Keyword-Driven Testing To Understand The Benefits of Model-Based Testing (MBT)

2011-04-25T08:00:00.000-07:00

Keyword-driven testing, also called action-based or action-word testing, has come up in several of my MBT presentations and I think it's worth exploring keyword-driven testing as a precursor to the more powerful MBT methodology. In a way, keyword-driven testing can be viewed as a piece of MBT in that it provides the components for the test execution engine needed for MBT. Although keyword-driven testing doesn't deliver the highly leveraged "automated test design" output that MBT gives us (see the Google Tech Talk article GTAC 2007: Automated Model-Based Testing of Web Applications), it can help organizations understand how to get more out of their automated testing efforts.

Proponents of keyword-driven testing emphasize that test development and test automation must be viewed as separate activities requiring separate skills sets. This emphasis goes so far as to assume that most testers don't have the technical skills to do the automation and should leave that up to an automation group. At the same time, the automation group is relegated to simply carrying out the tasks assigned to them without regard to test design and coverage. And from what I've seen in different organizations, that pretty much describes how things are. I don't think it has to be this way, particularly when organizations, such as Google and others, see the value of using automation not for test execution but for test design. If you leverage your automation efforts to generate test cases rather than simply carry them out, then you gain far more for your ROI.

Having said all this, I still think it's beneficial for some organizations, particularly those with separate test and automation groups, to start with keyword-driven testing before they graduate to MBT. In the next series of articles, I'll provide a HOWTO series on keyword-driven testing that will include some example code. I'll take an existing application (the Mac OS X Calculator) and build a keyword-driven testing framework (in Applescript) and provide several example keyword-driven based test cases. I look forward to your comments and thoughts in this area.

This first in this series is Keyword-Driven Testing HOWTO Part 1 - Creating a Keyword List.

[1] Keyword-driven testing from Wikipedia

A Software Testing Primer - An Introduction to Software Testing

2011-04-18T08:00:00.000-07:00

Nick Jenkins has written a nice introductory primer that covers the main concepts and terms of software testing. He does a great job of explaining the basics and provides the reader with a taxonomy of testing types and terminology. I plan to use this book as a reference when teaching software testing.

A Software Testing Primer

Finding Bugs In The Software Between Our Ears

2011-04-13T08:00:00.000-07:00

Think of Keith Barry as a hacker of the human brain -- writing routines that exploit its bugs and loopholes, and offering a revealing look at the software between our ears.

This video is from a past TED conference and features Keith Barry, a magician, who shows us how our brains can fool our bodies. In a way, Keith is a tester and his tricks are nothing more than proven "test cases" for the brain .

A moth, 5 dead patients, and the Berkeley finger daemon - Software Testing Is Serious Business

2011-04-12T08:00:00.000-07:00

Who's To Blame - Process or People?

It all started in 1945 when engineers found a moth in Panel F, Relay #70 of the Harvard Mark II system. Bugs in computers and software were born! And I'm sure a test engineer was questioned as to how this could have happened.

As to the dead patients and Berkeley finger daemon, you can find an explanation for those and other calamities caused by software bugs in a Wired article call History's Worst Software Bugs.

If you are a test engineer, you may read the Wired article and be thankful that the bugs you've missed in your career were not as serious those in the article. Natural reaction. But is it fair for QA teams and individual software test engineers to carry the burden of missed bugs? In the case of the Therac-25 bug, people were killed because of a kludged OS that enabled the operator to accidentally configure this radiation device to be lethal. How do you test for that? Do we fault the process or the people involved?

Unless you've never missed finding a serious bug or have never been asked by management why you've missed a critical software defect, then this discussion may not make sense to you. After all it's the job of QA and the software testing engineers to find every bug. It's their job! Well is it really just QA's job to find the bugs? What about unit testing, code review and code coverage instrumentation? These are just a few pieces to what should be a rigorous software engineering process. And if we all claim to follow and adhere to process, then how can we blame any one development or test engineer for any bug? If we follow a process, then we blame the process when it fails. More importantly, we make the necessary changes to that process to ensure such failures do not happen again.

But we can't find and fix process errors unless we have and follow a software engineering process to begin with. What's your company's software development and testing process like? We'd all benefit from knowing your thoughts on this subject - please leave a comment.

Exploratory Testing with James Whittaker

2011-04-05T08:00:00.000-07:00

Last week in this spot, we wrote about exploratory testing (An Exploratory Testing Renaissance?) and provided a 3 part video of Jonathan Kohl offering his take on this subject. After I first found these videos, I came across one on the same subject by James Whittaker. Most of you know him from his "How to Break Software"books. Well it turns out James is equally excited about exploratory testing. So much so, he has a book on the subject titled "Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design" coming out this fall.

In today's video, James Whittaker provides an overview of Exploratory Testing--the subject of his latest book. He talks about ways to explore your application with intent, strategy, and tactics that find bugs and validate functionality.

An Exploratory Testing Renaissance?

2011-04-04T08:00:00.000-07:00

The concept of ad hoc or exploratory testing has probably been around for as long as there have been things to test, whether that thing was a set of gears in a machine, tubes and Hollerith cards for early computers or modern day software. And I'm sure the veracity of this testing approach has been hotly debated from the beginning. Interestingly, it continues to be re-invented and offered as something new, e.g. heuristic testing, agile testing, etc.. I suppose those that have spent a career (or for some, one project) designing and writing endless test cases and test procedures, feel a sense of freedom by adopting new ways to apply ad hoc/exploratory testing. And maybe that's a good thing, but don't expect our engineering colleagues in the fields of civil, mechanical, aeronautical, chemical, etc. to honor us anytime soon for our asystematic approach to software engineering.

To give you an idea of what I'm talking about, particularly in the re-selling of ad hoc/exploratory testing, I encourage you to watch this three part series on Exploratory Testing with Jonathan Kohl. This interview takes place at Star East. I recommend you skip ahead by 45 seconds to get past the introductory music and graphics. I think you'll find this interview enlightening, and, depending on your point-of-view on exploratory testing, it may support your position on this subject one way or another.

Exploratory Testing - Jonathan Kohl - 2 of 3

Exploratory Testing - Jonathan Kohl - 3 of 3

When Bug Reports Were Fun(ny) - A Gallery Of Susan Kare's MacPaint Art From 1983

2011-04-01T08:00:00.000-07:00

I came across a couple of stories on "testing" of the original Macintosh. One of these stories featured a gallery of Susan Kare's art that was done using MacPaint. Susan Kare is most famous for having drawn the "Japanese Lady" (below) that showed up on a lot of early Macintosh marketing material, as well as on the front of the MacPaint manual and box. Apparently, she was pressed into service to use MacPaint for all sorts of things for the Mac development team, to include the "bug report" sheet to the left. Those were days when your bug reports were done on paper and handed in.

The annotations on this report are mostly cute, but I was entertained by the information requested of the tester:

- Describe the bug.

- Can you provoke its appearance?

- If possible, provide a code fragment that reveals the problem.

The article describes testing as "one of the most important activities during the last couple of months before shipping...". Even back then, the test cycle was an end-loaded activity. A couple of months to test the original Macintosh. I guess it worked out OK.

If you're interested in the entire article and seeing more of Susan Kare's artwork, check out MacPaint Gallery.

The 5 Stages of Bug Grief

2011-03-29T12:44:00.000-07:00

Stage 1. Denial: This bug does not happen for me.

Stage 2. Anger: Why do you keep reporting this bug - it doesn't happen for anyone but you!

Stage 3. Bargaining: Because this bug happens only on a few QA systems, can't we just close it?

Stage 4. Depression: I don't want to talk about this bug anymore - make it go away.

Stage 5. Acceptance: OH! Now that I read the steps for this bug, I can reproduce it. We need to fix it now!

Oh, Good Grief!

A Lesson In Defect Characterization - How A Twitter Security Enhancement Created a Twitgoo Problem

2011-03-28T08:00:00.000-07:00

Sometimes defects aren't what they appear to be. I'm sure developers have countless stories of how defects they set out to fix were not at all like the defect report described. We ran into just such a situation recently while testing a twitter-enabled iPhone App - Photo Tweet. Photo Tweet would mysteriously, and seemingly arbitrarily, fail to upload photos to Twitgoo. It was one of those situations where one day a feature works and the next day it fails. And, of course, the developer had not made any changes.

Most of the time when you find a bug and the developer claims not to have made any changes, it's always best to quiz the developer with "what-if" questions. That allows you, as a tester, to avoid calling the developer's integrity into question. I usually try questions like this:

What if during last night's build, an older version of a component was used?
Even though you didn't make a change for that feature, what if some other change caused this bug?

Usually this sort of brainstorming session can tease out needed clues that lead up to finding the root cause of a mysterious bug. If that doesn't work, then you're left trying to find out what else could have contributed to the defect. Such was the case with Photo Tweet and it's inability to upload photos after a period of time. And so, a bug was written up that said Photo Tweet would stop uploading photos arbitrarily.

Subsequent conversations with the developer, including some what-if questions, lead us to believe that no changes were made for this feature and the bug resided outside the developers code. Well since Photo Tweet uploads its photos to Photo Bucket's Twitgoo service, we suspected they had instituted some sort of throttling for uploads. This is reasonable to expect given the popularity of Twitgoo. Maybe they just didn't want rogue apps flooding their service, and apps like Photo Tweet were paying a penalty for the nefarious activities of others. We could not have been more wrong about the root cause of our defect.

We reported our defect to Twitgoo and asked if they had instituted an upload limitation. I was amazed at how fast these folks responded and how helpful they were. Initially they were stumped since this was the first they heard of this problem and HAD NOT put a limit on uploads. At that point, I expected that they could not help any further and we would be left with having to do more debugging. But instead, Michael P. Clark, their SVP of Technology and Partnerships rallied his technical folks to find out what was at the heart of this problem. The next day I got an email from Justin Hart, a Sr. Engineer for Photobucket, informing me that they discovered an unannounced change to Twitter's service. Twitgoo relies on Twitter for those users wanting to "tweet" the photos they upload to Twitgoo. It turns out that Twitter had made a security enhancement. Michael and Justin found this issue very quickly and pointed me to a threaded discussion concerning what Twitter did. Much to Twitter's credit, they quickly made this issue public and extended an apology and an offer for input from developers. Here's a snippet of their statement:

A change shipped last week that limited the number of times a user could access the account/verify_credentials method [1] in a given hour. This change proved hasty and short-sighted as pointed out by the subsequent discussion [2]. We apologize to any developer that was adversely affected. Given the problems, we want to fix this in a public and transparent manner.

We learned a lot from this experience. First, the folks at Twitgoo are fantastic and serious about providing a great customer and technology partner experience. Second, some defects are certainly not what they appear to be. Third, when testing products that use APIs that connect to services that connect to other services, it's hard to determine root causes no matter how well intentioned your defect characterization process is. Finally, as you investigate why a defect occurs, don't assume you know the root cause until you have all the facts.

Understanding The Differences Between Testers And Developers

2011-03-26T08:00:00.000-07:00

Are testers from Venus and developers from Mars? In an ACM article titled An Exploratory Research Study on Interpersonal Conflict between Developers and Testers in Software Development, the authors look at the difference between testers and developers. As stated in the article, the goal is to better understand the differences between thse two actors in order to produce better software. The authors' call to action is based on two reasons:

Because of a trend toward more agile software development, testers are in contact with developers earlier and more often
Conflict can have negative consequences not only in relation to the end product but also in relation to the job satisfaction of both developers and tester

This paper points to studies that clearly separate testers from developers in terms of their goals: developers seek to maximize efficiency while testers are all about effectiveness. Developers seek to do get their work done with the least effort while testers seek the highest quality. Those are clearly different goals that can easily create conflict if each group sub-optimizes. Unless upper management reconciles these different goals and helps align them to reach the broader objective of the company, failure may ensue.

In addition to understanding the goals of each actor, managers need to understand and work with the differences between testers in terms of what makes them good at what they do. The authors point to a list of the twelve traits that make good testers and developers (Pettichord, 2000):

If these are truly the traits that make for good testers and developers, then you can see why those bug review meetings tend to be a little tense at times, especially when one group or the other has to justify fixing or not fixing a bug. Unless management understands the differences between the respective goals and attributes of developers and testers, and fails to effectively manage the natural conflict, the risk of failing will always be high.

So how do you effectively manage the conflict? I suggest taking the list above and adding another column with notes on how to address the conflict when it arises. For example, what do you say when a tester points out a bug (what's observed) and the developer say it's a feature (how it's designed)? Do you blow it off and agree with the developer or do you take the issue on as something that needs to be resolved either as an issue with the design or the documentation? By thinking through examples for each conflicting pair of attributes, and writing down examples and responses, you'll be able to better manage the conflict.

You can read the entire article here.

Top 10 Tips For Testing iPhone Applications

2011-03-25T08:00:00.000-07:00

This article started out as a Top 5 list but quickly grew. Most of these tips came from tracking down memory-related problems, which often resulted in defects that were very difficult to capture.

1. Accurately report available memory. Many of the non-reproducible bugs you run into when testing iPhone apps are related to memory problems. It's critical that you know and report available free memory before launching an application. In all likelihood, the reproducibility of a crashing iPhone app bug is related to low memory conditions. Consequently, a crashing defect may disappear when there's plenty of free memory. In a previous article (Using Memory Sweep for iPhone App Testing) we described a tool that can be used to determine free memory.

2. Provide 'crash reporter' logs with your defect reports. Each time an iPhone application crashes, a .crash file is created on your iPhone. You can retrieve this file when you synch your iPhone with iTunes. Here's a link that describes where those files are stored: iPhone Crash Logs

3. Spy on the app from the console. iPhone apps will report application and system level warnings to the console. You can view these warnings in real-time using Apple's iPhone Configuration Tool. By knowing what's being reported when interacting with an app can help you refine the steps you need to reproduce tricky (and memory related) problems.

4. Test under low memory conditions. This relates to #1 above. You'll be able to tease additional crashing bugs if you force free memory to a very low level, e.g. < 2MB, before proceeding with your tests. One way to do this is to open several Safari windows before you start your testing.

5. Screenshots, screenshots, screenshots. Nothing makes a UI bug stand out for a developer than when you send screenshots. And with the iPhone's built-in screen capture, there's no excuse not to do this.

6. Provide useful defect characterization information. Developers always like to have help in their debugging process, and useful defect characterization helps them narrow down the root cause of a bug. If a crash happens under low memory conditions (see #1 and #4 above), then try it under conditions where there's lots of memory available, e.g. >40MB. If a problem occurs under iPhone OS 2.2.x, then try it under 3.x.

7. Create connectivity problems. If you're testing an iPhone app that depends on internet connectivity, then test for degraded or unavailable connectivity. It's easy to make connectivity unavailable by simply turning on Airplane Mode. To degrade connectivity, especially on Edge or 3G, employ some sort of metallic "shield" on top of your iPhone.

8. Boundary test data input. Wherever an iPhone app asks for text input, you have an opportunity to find a bug. My favorite technique for this is to copy a huge amount of text, then paste it into each text field. You'd be surprised at how this trips up some apps.

Additionally, we’ve been finding that application errors are generating when entering the following characters into text fields: !@#$%^&*()_. (Note: Holding down letters (A, E, I, O, etc) or symbols ($, !, &, etc) on the onscreen keyboard generates a keyboard popup that includes localized and 2-byte characters. These should also be entered into text fields.)

9. Gather up UDIDs (unique device identifiers) early. This is a simple logistics task but seems to be one that becomes critical as the first build approaches. And it's a hassle for the dev team to add new UDIDs and create new provisioning files as each new person wants to install an application during development. Get the UDIDs of all known devices that will be used during testing and set a cut-off date for the addition of any new devices. Check out Find UDID with a single click. You can also connect all your iPhones and iPods touches to your computer and use the iPhone Configuration Tool to collect UDIDs.

10. Employ background applications. But the iPhone can only have one application running at a time, right? That's true for those of us that develop and test applications, but not for Apple. Applications that continue running in the background on the iPhone are Safari, iPod and Mail. And what about reminders and push notifications? These "interrupters" can affect the behavior of an application under test.

Also, since iPhones and iPod touches are devices that users buy primarily to use as a phone or a music player, it’s important to test that an app can gracefully handle situations where the user receives a call or plays music from their music library (iTunes) while the app is running. We’ve seen issues here where apps aren’t smoothly multi-functioning in these areas.

iPhone Development Basics - Memory Management Part 5

2011-03-24T08:00:00.002-07:00

This week is the final video from Mark Johnson on iPhone memory management. In case you missed any of the previous videos, here's a list of the articles we've posted with Mark's first 4 memory management videos:

iPhone Development Basics - Memory Management Part 1
iPhone Development Basics - Memory Management Part 2
iPhone Development Basics - Memory Management Part 3
iPhone Development Basics - Memory Management Part 4

As in the previous articles, I've listed some suggested questions you might want to ask your development team in regards to memory management. Today's video offers specific situations when an app might crash because of memory problems. It would be particularly useful to ask your developers why this could happen - it may help them avoid these situations.

Suggested questions for Part 5:

(4:21min) What does it mean to call dealloc to the super class?
(4:41min) What is a leaking object?
(5:29min) Supposedly there are two types of memory bugs - releasing an object that is already deleted and forgetting to send a release. Why does the first one cause a crash and the second one just a memory leak?
(7:26min) What does it mean that there is only around 12MB of memory for an app before memory is increased?

iPhone Development Basics - Memory Management Part 4

2011-03-23T08:00:00.004-07:00

So far we've seen 3 videos by Mark Johnson on iPhone memory management:

iPhone Development Basics - Memory Management Part 1
iPhone Development Basics - Memory Management Part 2
iPhone Development Basics - Memory Management Part 3

And we've been using these videos to understand iPhone application memory management and gain some insight has to why iPhone apps crash. In today's video, Mark describes a memory management programming convention used by iPhone developers for managing objects - either they're in the Autorelease pool or they are not. It's a very technical video but worth listening to because Mark warns of things not to do otherwise an app will crash - something worth being curious about when talking to your dev team.

Suggested questions for Part 4:

(4:02min) Why does returning a reference to a deleted object cause a crash? Why would that happen?
(5:17min) What does it mean to "own" an object versus having it be part of an Autorelease pool?

How Much Is Your Offshore/Outsource Testing Defect Find/Fix Cycle Costing You?

2011-03-22T08:00:00.001-07:00

Update: Linda G. Hayes, who is the CTO of Worksoft, Inc., and the founder of three software companies including AutoTester, the first PC-based test automation tool, wrote an interesting article on this subject at StickyMinds.com. She wrote about the "promise of the same value proposition" and outlined some key factors for success or failure with offshoring. One of the more interesting, and relevant comments (at least as it pertains to this article), was on "Time to Value". Linda said:

Offshore resources are never a quick fix, especially for testing. A 2005 report from AMR Research found that it took between fourteen months and three years before the offshore testers had sufficient familiarity to be effective in finding the root cause of problems. Others have found that the lack of domain knowledge resulted in spurious defects being reported that actually increased development overhead due to review and response times. In fact, one organization identified that as many as 33 percent of reported issues were traceable to tester error.

There are many reasons to involve external testing services. Sometimes companies need the expertise of an interoperability lab, other times they are attempting to reduce testing staff costs. Whatever the reason, one cost that should be calculated based on a known process is the Find/Fix Cycle Cost for defects. It's important to evaluate these costs not just based on the hourly rate of the offshore/outsource testing service provider, but on the total cost involved in finding and fixing defects. Ultimately, this cost involves all QA and developer resource costs.

The find/fix cycle represents the time required to find and fix one defect and you should know what that cost is to your company. In order to calculate this cost, you need to collect information along the way. I found a nice source for this calculation, as well as a number of other interesting testing topics, at Software Quality Consulting, Inc. Food For Thought. Steve Rakitin does a very nice job in describing this cost in the table below.

Estimate the find/fix cycle Cost
T1 is time required to run a test that finds a potential defect	T1
T2 is time required to report a defect	T2
T3 is time required to determine that this is a defect	T3
T4 is time required to determine if the defect should be fixed	T4
T5 is time required to fix the defect	T5
T6 is time required to include fix in a build	T6
T7 is time required to run regression tests to determine if defect is really fixed and close out defect report	T7
Total time required…	TT = (T1 + … + T7)
LC represents fully loaded labor cost ($/hour) for engineering time (development and QA)	LC
C represents cost to find and fix one defect	*C = LC TT**
TC represents the total cost to find and fix the defects found and fixed in the last release (F1)	*TC = C F1**

As you can see from this table, the "round trip" cost of a defect incorporates 7 different time elements that add up pretty quickly, especially if you have any inherent time delays such as dealing with remote test teams, external test staff turnovers, re-requests for defect characterization information, etc. And the final cost is not just that of the test team, but also incorporates the dev team costs.

My guess would be that T1 and T5 would be the largest time elements, while times T3 and T4, which require more verbal communication, could very well match T1 and T5, depending on the teams involved. It would be interesting, and most likely revealing, to measure these times and factor them into the costs associated with each project's QA and dev team composition.

iPhone Development Basics - Memory Management Part 3

2011-03-21T08:00:00.001-07:00

In our previous articles on iPhone memory management:

iPhone Development Basics - Memory Management Part 1
iPhone Development Basics - Memory Management Part 2

we've seen how developers allocate and de-allocate memory. And we've created some questions, for developers, on how crashes may occur when memory is not handled correctly. In today's video, the author, Mark Johnson introduces a variation on these things called "Autorelease pools". In the video, he shows how an object that is part of an autorelease pool can be retained despite releasing the entire pool. Seems to me that this could cause problems if not handled correctly.

Suggested questions for Part 3:

I've heard about a way to retain individual UI objects when releasing memory using autorelease pools. Do we do that and can you explain why you would retain something when it's part of an autorelease pool?

iPhone Development Basics - Memory Management Part 2

2011-03-20T08:00:00.000-07:00

We introduced this series, created by Mark Johnson, last Monday (iPhone Development Basics - Memory Management Part 1) to help testers understand memory management from the developers point-of-view in order to develop some insight as to why applications crash because of memory problems. As mentioned last week, these videos should be used to develop questions for the developer team that may lead to more effective and reproducible testing.

Suggested questions for Part 2:

(0:52min)Does our iPhone application use something called "Autorelease pools"? I've heard that this method can help avoid bugs because you don't have to keep track of so many memory management calls.

Can autorelease pools cause crashes by releasing UI objects too early?

Software Testing and Quality Assurance Rules of Thumb

2011-03-19T08:00:00.000-07:00

New this week: THE NUMBER OF WHITE-BOX TESTS REQUIRED TO OBTAIN SUFFICIENT COVERAGE OF A MODULE IS ADEQUATELY DETERMINED BY ITS CYCLOMATIC COMPLEXITY.

Rule of thumb - The earliest citation comes from Sir William Hope’s The Compleat Fencing-Master, second edition, 1692, page 157: "What he doth, he doth by rule of thumb, and not by art." The term is thought to originate with wood workers who used the length of their thumbs rather than rulers for measuring things, cementing its modern use as an inaccurate, but reliable and convenient standard.
From Wikipedia

As testers, we often use rules of thumb throughout a project. For example, we sometimes use total number of expected defects during test planning and then compare actual defects per hour found versus what we would expect, during test execution. Each of these rules of thumb aids us in managing the information we deal with as testers and QA managers.

It would be nice (and useful) to have a collection of these rules of thumb in one place, each documented with examples. And that's my goal: to list as many software testing rules of thumbs in this article. I'll update it each week and re-post it. I look forward to reader comments and additions to this list and hope to make it a reference you each bookmark and refer to often.

THE NUMBER OF WHITE-BOX TESTS REQUIRED TO OBTAIN SUFFICIENT COVERAGE OF A MODULE IS ADEQUATELY DETERMINED BY ITS CYCLOMATIC COMPLEXITY. One common testing strategy, espoused for example by the NIST Structured Testing methodology, is to use the cyclomatic complexity of a module to determine the number of white-box tests that are required to obtain sufficient coverage of the module. In almost all cases, according to such a methodology, a module should have at least as many tests as its cyclomatic complexity; in most cases, this number of tests is adequate to exercise all the relevant paths of the function. [ NIST Special Publication 500-235: Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric .]

MORE THAN 40% OF ALL SOFTWARE APPLICATIONS ARE RELEASED WITH BETWEEN ONE AND 10 CRITICAL DEFECTS. And managers are fully aware. Three-quarters of deployed apps have between one and 10 critical errors, and again, bosses know about it. [Software Is Deployed, Bugs and All, SD Times.]

80% OF DEVELOPMENT COSTS ARE CONSUMED BY SOFTWARE DEVELOPERS IDENTIFYING AND CORRECTING DEFECTS. A study commissioned by the United States Department of Commerce's National Institute of Standards and Technology (NIST) found that software defects cost the U.S. economy almost $60 billion annually. The study also found that about 80 percent of development costs are consumed by software developers identifying and correcting defects. [The business value of software quality.]

ABOUT 7% OF DEFECT REPAIRS WILL THEMSELVES ACCIDENTALLY INJECT A NEW DEFECT. A bad fix is a secondary defect accidentally injected in a bug repair. In other words, a bad fix is a failed attempt to repair a prior bug that accidentally contains a new bug. On average, about 7 percent of defect repairs will themselves accidentally inject a new defect, although the range is from less than 1 percent to more than 20 percent bad fix injections. [Capers Jones. "Software Cost Estimating Methods for Large Projects©"]

DEVELOPERS INJECT FEWER DEFECTS WHEN DESIGNING (2.0 DEFECTS PER HOUR) THAN WHEN THEY DESIGN WHILE CODING (4.6 DEFECTS PER HOUR). The practice of designing while coding is error prone. From data on 3,240 programs written in Personal Software Process (PSP)SM courses, the SEI has found that experienced developers inject fewer defects when designing (2.0 defects per hour) than when they design while coding (4.6 defects per hour). If you want low-defect designs, you must produce those designs, instead of just creating them while coding. [WATTS S. HUMPHREY. "A Personal Quality Strategy", Stage 2 – Detailed-Design Quality.]

NONREPRODUCIBLE OR AD HOC TESTING IS OF LITTLE OR NO USE. One of several basic rules of testing compiled from Myers (1976), Priest (1988), and Pullum and Doyle (1998). [W.R. Blischke, D.N.P. Murthy. "Reliability", Section 9.5.1, pp. 311.]

CHAOS WITH NO MEASURES TO TELL THE ORGANIZATION WHAT'S GOING ON LEADS TO … 3 IN 10 PROJECTS BEING CANCELED, 5 IN 10 OVERRUNNING SCHEDULE AND/OR BUDGET BY NEARLY DOUBLE, AND ONLY 1.6 IN 10 MAKING THEIR DEADLINES AND BUDGETS. Many organizations struggling with process chaos claim that they don't have time for software measures, which are sometimes perceived as a bureaucratic Dilbert exercise. But getting metrics in the face of deadlines, and using them to manage that pressure and reduce the chaos, is key to successful software development. [Johnson, Jim. "Chaos: The Dollar Drain of IT Project Failures." Application Development Trends (January 1995), pp. 41-47.]

THE AVERAGE TIME TO FIND AND FIX DEFECTS IS 10 TO 20 HOURS. "One way to think about quality is to consider how the process would change as a function of defect fix times. For example, programmers generally think that it takes only a few minutes to fix defects in test. They base this on their experience with most of the defects they find in unit testing. In system test, however, the time to find and fix defects typically extends to many hours or even days. While most of these defects are fixed rather quickly, some take much longer. The average time to find and fix each defect is generally 10 to 20 or more hours." [Watts Humphrey, Learning from Hardware: Design and Quality, p.107]

ABOUT 80 PERCENT OF AVOIDABLE REWORK COMES FROM 20 PERCENT OF THE DEFECTS. That 80 percent value may be lower for smaller systems and higher for very large ones. Two major sources of avoidable rework involve hastily specified requirements and nominal-case design and development, in which late accommodation of off-nominal requirements causes major architecture, design, and code breakage. [Software Defect Reduction Top 10 List]

EXPECT BETWEEN 6.5 AND 9 DEFECTS PER KLOC DURING SYSTEM TEST. Based on data from the Jet Propulsion Laboratory (JPL) from an internal report that listed all the defects found in the system testing of three spacecraft. These data were taken after the programs were developed, compiled, unit tested, and integration tested. [Watts Humpherey]

THE RELATIVE COST OF FIXING DEFECTS DURING TESTING IS 15 TIMES GREATER THAN DURING DESIGN. As reported by the Systems Sciences Institute at IBM. Additionally, the cost to fix an error found after product release was four to five times as much as one uncovered during design, and up to 100 times more than one identified in the maintenance phase. [IBM Systems Sciences Institute]

REQUIREMENTS, DESIGN, AND CODE REWORK COSTS 40 TO 50 PERCENT OF THE TOTAL COST OF SOFTWARE DEVELOPMENT. "If you can prevent defects or detect and remove them early, you can realize a significant schedule benefit. Studies have found that reworking defective requirements, design, and code typically consumes 40 to 50 percent of the total cost of software development (Jones 1986). As a rule of thumb, every hour you spend on defect prevention will reduce your repair time from three to ten hours. In the worst case, reworking a software requirements problem once the software is in operation typically costs 50 to 200 times what it would take to rework the problem in the requirements stage (Boehm and Papaccio 1988). "[Steve McConell]

TESTERS WILL FIND, ON AVERAGE, 1 DEFECT EVERY 4 HOURS DURING BLACK BOX TESTING. Based on average defect detection rates for 6 different JPL projects as reported in "An Analysis of Defect Densities Found During Software Inspections," Proceedings of the Fifteenth Annual Software Engineering Workshop. More information can be found in Grady's Practical Software Metrics For Project Management And Process Improvement. [Grady]

THE NUMBER ONE CAUSE OF DEFECTS IS SPECIFICATIONS. "Software Testing", R .Patton, p.17. [Patton]

TESTING DONE WITHOUT MEASURING CODE COVERAGE TYPICALLY EXERCISES ONLY ABOUT 55 PERCENT OF THE CODE. Robert Grady of Hewlett-Packard reports that testing done without measuring code coverage typically exercises only about 55 percent of the code (1993). See page 615 of Code Complete. [McConnell]

iPhone Development Basics - Memory Management Part 1

2011-03-18T08:00:00.000-07:00

For those of us involved in iPhone applications testing, it's useful to understand how memory management works on the iPhone OS. Most of the problems we see that result in some sort of application crash or failure are related to memory problems. Understanding how these memory problems can occur can help with trapping memory related defects so you can report reproducible defects.

We've discussed memory management issues in previous articles. In this series of articles, which will be posted each Monday, we have video presentations from Mark Johnson that introduce the basics of iPhone memory management in Objective-C for programmers starting out with the iPhone. These videos are very instructive for both developers and testers.

Tips for Testers: if you've never done any programming, the terms discussed by Mark may be new for you. I suggest that you use this information to develop a curiosity about iPhone app memory management so that when you talk with the developers, you can ask probing questions. And, as developers answer your questions, it often helps them sort through the complexity they deal with, as well as giving you more insight as to what may be cause the bugs you're seeing.

Suggested developer question for Part 1:

(2:50min) What is a NSMutableString object and why would the app ever release this object after it has been deleted? Would this cause a crash?

Unit Testing Tips and Techniques for iPhone

2011-03-01T08:00:00.000-08:00

"...Unit tests help ensure low-level code correctness, reduce software development cycle time, improve developer productivity, and produce more robust software."

-Unit Test Frameworks

As important as unit tests are to modern development efforts, the iPhone SDK makes it difficult, if not impossible, to introduce unit tests. However, there are some useful resources online that can help developers shoe-horn a unit test framework into the SDK. One of these is based on the "Google Toolbox for Mac" project. They've extended their toolbox to provide iPhone Unit testing. You can find their well-written HOWTO at the iPhoneUnitTesting page.

The other resource I suggest you look at is from the folks that brought the OCUnit framework to Apple's Xcode. At the iPhone Unit Testing page of their website, you'll find the history of OCUnit and precise instruction on how to get it to work with the iPhone despite Apple having put OCUnit into iPhone SDK 2.2.!

Automated Web Services Testing - An Interview With Matt Krapivner of SmartPilot Software

2010-03-22T08:00:00.000-07:00

As more and more companies face decisions about automating their testing, it's important to understand some of the pitfalls of automation and explore methods that avoid those pitfalls. Likewise, we need to find mature open source testing tools that help bend the test automation curve in the right direction. In previous articles, we've written about model-based and keyword-driven testing, each promising to reduce test automation script maintenance costs while increasing effectiveness (coverage). As I've searched for answers in this areas, I've sought examples of how others have approached test automation as a way of benchmarking good ideas. Recently, I've had the good fortune to meet a practitioner who has pursued new and more effective approaches to test automation. His name is Matt Krapivner and his consultancy business has provided a very impressive test automation framework to a local Web 2.0 company.

We interviewed Matt recently and talked to him about the new approach he's using with his current client and how it's a step up from traditional test automation practices.

Q: Matt, you started out with a Computer Science academic background and since have been working in the area of Quality Engineering. What got you interested in that area versus software development and programming?

Ray, my first quality engineering engagement was actually as a developer on a quality engineering team. I believe some companies call this Software Engineer in Testing, but in a nutshell the team was developing testing software. I started as an individual contributor, but ended up managing the team of 7 engineers locally plus another 7 offshore. I saw QE as an oft-undervalued field with an opportunity to make a great impact on the perception of the company’s products in the marketplace. It is also quite fascinating because at different times you have to wear multiple hats, such as a developer at one time, a tough customer advocate at another, sometimes a product marketing/management hat, and so on. The opportunities for improving the product and making an impact are there, it’s just that many companies often overlook QE and do not get the deserved output from their teams, which in turn impacts their perception of QE overall.

There are two different approaches to testing: Quality Assurance and Quality Engineering. In my opinion, those are very different. One will give you an assessment of the state of the system AFTER that system is put together. The other will help you make sure that your system is ENGINEERED with the highest-quality possible. Let me give your readers an analogy. Say you are building a retaining wall and 2 years after it’s built the wall comes down in a mudslide. Using the traditional QA approach, you would state “the wall fell down because the pressure was too large”. If I was using QE methodologies, I would hire an engineer to build the wall according to the building code to begin with, to make sure that mudslides will not affect it. That’s a fundamental difference, because the further you get in the product lifecycle, the more expensive it will get to fix the mistakes. In my retaining wall example (which I actually just finished building in the backyard), I made a mistake of not putting in the drainage behind it. It cost me more than the initial total construction cost to make an opening in the wall, add the pipe, and close the wall. If I did what was necessary from the beginning, it would only add 5-10% to the construction cost. Instead, by trying to save a few dollars and rushing the construction crew I ended up spending double the amount allocated initially. The same can be said about software development – the longer you wait to find and address an issue, the more you will end up paying to correct the mistakes.

Q: Can you tell us about your first test automation project and what you learned from it?

My first automation engagement was at Palmsource, where I managed a team which was responsible for design and development of an automated test harness for the Palm OS API. In a way, our team played a role of 3rd-party developers early on in the software development lifecyle (SDLC). This was quite fascinating as we got to develop software using a cutting-edge OS before everyone else did. We also received a lot of visibility in the company since as “early adopters” we were able to make informed suggestions for features, and avert a few blunders along the way before they hit the market. As we drove internal adoption, our team started receiving praises from other groups which used our automated tests to validate their changes, small and large. Very quickly, the automated tests were used as a means of certifying customers’ devices for deployment field. This meant that no device could ship with Palm OS or bear the Palm OS logo until it successfully passed our automated tests.

The interesting observation I have made since then in most companies I have worked with, is that very often the testing software ends up being more complex than the actual application under test. There are multiple reasons for that, from the need to cover and verify various edge cases, to just being able to confirm the results. In some cases, especially when you have a hardware/software interaction, this is not trivial at all. One also has to learn where to draw the “line of trust”. Let me explain. Assume that you are testing a PDA application and you need to verify that certain text appeared in a certain position on the screen. While it sounds trivial, verifying the correct result often is anything but. There are typically several solutions. We can call an internal API to query it for the text at certain coordinates, or a text in a specific field. But can you “trust” this API? What if it doesn’t exist to begin with? What if the API itself is buggy? In that case, your test will be meaningless. Another option is to ask your developers to give you a custom “hook” into the display buffer, but most likely this request will not be on top of their priority list. Also, there is a chance it may introduce another point of failure. Yet another option is using a software simulator, if one exists. We actually had to write our own software simulator for testing purposes at one of the companies where I ran the QE group. The point is that there are often many ways of verifying the information in your application under test, and the “easiest” choice may not necessarily be the right one. The challenge then becomes to persuade upper management to invest time and money into building a test system, where an immediate ROI may not necessarily be realized, but rather will result in intangible benefits such as improved product quality, customer satisfaction, etc.

Q: I understand that you have recently provided test automation development services for several local companies. What tools have you used and what was your assessment of the long term maintenance of the frameworks and test scripts you've delivered?

It’s true that there are many test automation systems, and there is no “perfect choice” that would work anywhere. Over the course of the last few years, I have designed a house-grown test automation system; used a hybrid house-grown and commercial tool (Automated QA’s TestComplete); used a commercial tool in combination with open source ones (Squish and Selenium); and finally developed what I believe to be the future of automated testing – a system based on FitNesse.

I will not dive deeply into the benefits and challenges of using commercial tools, there are countless articles and books written on the topic. However, I would like to mention a few points from my experience as a manager/director of QE teams, and as a consultant working on implementing these systems. When selecting an automation tool, one of the main things a company should look at is whether or not a specific tool will work with their application. If it will not, all other benefits become a moot point. Another significant factor is price. In some cases, it will run into tens of thousands of dollars for the tool alone. As a QE director, I had to restrain my teams quite often because the price of the tool was just prohibitive given our budget. Most often though, there is no technical reason to select those tools anyway as there often is a less-expensive alternative. With any commercial tool though, a person responsible for test automation will have to know at least a scripting language, such as Perl or Python. Ideally, that person will be able to program in a higher-level language as well, such as C++ or Java. As a result, your automation system will most likely be built fairly well. The major limiting factor to the tool’s adoption then becomes its scalability, or how quickly can new tests be developed to ensure adequate test coverage. It is, of course, less of a factor for large companies with deep pockets as these companies can most likely just organically grow the test automation team. For startups and smaller companies though, this becomes a major hurdle and automation efforts sometimes may not succeed, or may not provide the desired ROI.

At my last consulting engagement, my client approached me with a similar problem and suggested that I implement a system based on FitNesse. FitNesse is a Wiki-based system where tests are written in natural English language in the form of a Wiki table. Underneath, there is a “fixture” (in Fit-speak) which does the “translation” between English and the underlying tool. That’s right, we can still plug in any test automation tool with FitNesse (well, as long as it exposes some API). With this system, the automated tests now can be written by anybody in the company, including manual testers, product management folks, even a VP of Engineering. It is also easy to answer questions such as “what’s our testing coverage for feature XYZ”, “why did the latest ABC test failed”, and so on. Granted, there still needs to be someone to integrate FitNesse with a testing tool of choice, but this approach provides companies with the much-desired scalability of their testing efforts and a tool with no price tag and ample documentation.

I have integrated FitNesse with Selenium Grid for my client’s web application, and with CruiseControl for continuous build integration. We can now simultaneously test over 30 various targets (different OS/browser combinations), and get results within 15-20 minutes for a test run that spans the entire product in 10 languages. It is easy for anyone now to add to the automated test collection.

Q: For those of us that have tried to bring more automation to testing, we've each run into the problem of "brittle" automated test cases, as well as the need to generate more test cases early on in the development cycle. How has FitNesse solved this problem?

FitNesse is actually a great tool for this, since it allows non-programmers to write test cases which are immediately automated by definition. Most likely, the product managers in this case will not even have to change their logic since they will write the test cases in the form of a “use case”, using more or less the same English language to define a feature. So when the feature is defined, it is put into FItNesse. When the feature is implemented, the test case can be executed to provide immediate response to the team. This becomes even less labor-intensive when a continuous build integration system is utilized. In fact, it is so easy for developers to utilize it to achieve the ever-elusive target of “test-driven development”, that there are no reasons for them NOT to use it…

Q: Given what you've said about FitNesse, particularly the ability for non-programmers to write test cases using a Wiki, how easy is it for someone to produce a test case?

As easy as it is for someone to edit a Wiki page. The tests literally look like:

|user clicks button|Submit|
|page reloads in less than|30|seconds|
|page contains text|Hi Ray|
…
If a person can write a manual test case, they can write a test case in Wiki. If a person can write a use case in Word, they can write a test case in Wiki. You get the idea.

Q: What would you say is the most compelling reason to adopt FitNesse and what challenges are there for organizations to take this path?

The most compelling reason for companies to adopt FitNesse is its ability to provide so much valuable output with so little effort. Granted, it is a relatively new tool and sometimes you will stumble but in my experience the hurdles one will face with FitNesse are not any greater than with any other commercial test automation tool. In fact, the major challenge will likely be organizational in nature as it will require a slight change in the way people operate and contribute to the testing efforts. A FitNesse endeavor is much more likely to be successful if folks from multiple departments contribute.

Q: How extensible is FitNesse, i.e. what if my web service uses a lot of AJAX or components that don't work well with Selenium?

As long as Selenium can handle it, so can FitNesse. Remember that FitNesse operates with “fixtures”, which provide translation between English (the language of FitNesse) and Selenium API.

I chose to use Java API for Selenium, because FitNesse is written in Java and it is technically easier to use Java for the fixture. I was comfortable with Java, so I chose not to change something I did not have to. Any language can be used though, and if your automation tool of choice can only speak Python, for example, then you simply write your fixture in Python.

AJAX is an interesting case though, and the limitations one will face will have nothing to do with FitNesse, but rather with Selenium’s ability to handle AJAX. In my case, we handle AJAX by a combination of standard Selenium APIs, custom Selenium APIs that I wrote specifically for my client’s application, and custom hooks that developers have provided on the server-side. However, all of this is hidden from the tester since their tests are written in plain English. That’s the beauty of FitNesse – it really hides implementation details from a tester who is only concerned with, for example, whether or not a text appeared in the right spot on the page. We provide a custom action for the testers, and they simply do not and should not care how we implement this action behind the scenes.

Q: What would you recommend to those that want to adopt FitNesse for their test automation projects and where can they go for help in getting a FitNesse framework up and running at their company?

First of all, find a test automation tool that you would use without FitNesse. Then, find out if this tool has an API that you can call from FitNesse. Some tools do, and some do not. Some tool companies will be willing to open up their API for you, and some may even be willing to write a sample fixture for you that you would be able to extend. Some may prohibit you from calling their tools from anything besides their own proprietary interface. When in doubt, just ask the tool vendor.

As for FitNesse, there is plenty of easily-digestible documentation that comes along with the distribution of FitNesse. If you search online, you may even find ready-to-use “fixtures” for the most popular open source automation tools, such as Selenium for example. Those fixtures will give you a very good start.

You can reach Matt Krapivner by email at matt@smartpilotsoft.com.

Google Chrome Not Ready For Prime Time Compatibility Testing Yet (Still)

2010-01-12T08:00:00.003-08:00

This Just In: CNET reports that Chrome has overtaken Safari in the latest usage stats (Chrome edges out Safari in browser usage). What exactly does this mean to developers and testers?
I think it's an interesting statistic but not significant (yet). Why? For one thing, Chrome doesn't ship as a standard browser on any system (yet). And until it does, a browser like Safari, which is the browser of choice for each new Mac user (until their friends somehow con them into using FireFox), will remain one of the two browsers (the other being Firefox) you must test on for the Mac OS. For a second reason, read the article below we posted last week.

December 9, 2009 8:00 AM PST
Even though Chrome has gained a 3.93% market share (as compared to Safari at 4.36%, IE at 63.62% and Firefox at 24.72%), it's not ready to be added to compatibility testing matrices yet. There are two reasons for this (besides still being in beta):

It doesn't run on Mac OS 10.4.x (only on Leopard)
Features are not equally supported on all platforms, in particular there's a lack of support for extensions on the Mac

So what's the big deal here? Well for starters, by not supporting the next-to-the-next-to-the-last version of Mac OS X (as opposed to support for the 7+ year old Windows XP) , the adoption by Mac users will be limited to those that don't like Safari and Firefox on their Leopard machines, which might as well be no adoption at all.

Furthermore, because extensions are not supported on the Mac (and Linux) versions of Chrome, testing on Chrome is only worthwhile on Windows. That makes Chrome a single-platform browser bet when it comes to compatibility testing.

With such limited prospects for use on the Mac platform, it's not worth QA testing Chrome on the Mac, and making a case for testing it on Windows alone will be hard to make given the bigger numbers for IE and Firefox.

Chrome doesn't warrant a line on the browser compatibility testing matrix yet.

USB 3.0 Adapters Available Now

2010-01-12T08:00:00.000-08:00

Update: USB 3.0 finally arrives on consumber laptops. CES: USB 3.0 arrives in HP laptop: Yes, it's fast.

November 25, 2009 8:00AM PDT

We may not have readily available computer systems with USB 3.0 yet, but there are USB 3.0 host cards and cables available now for compatibility testing of USB 3.0 supported devices. Several motherboard manufacturers have announced offerings with USB 3.0, but until we see them in mainstream computer systems, it will be hard to justify their use for testing USB 3.0 supported devices. Third party USB 3.0 adapters, on the other hand, are available to consumers and are a perfectly reasonable addition for your compatibility test matrices.

The adapters and cables listed below are available for purchase now:

USB 3.0 Super High Speed 2-Port PCI Express Card for Windows 7 and Vista

ExpressCard USB 3.0 2-Ports Hub

USB 3.0 Hi-Speed PCIe Card 2-Port

2-Port ExpressCard SuperSpeed USB 3.0 Card Adapter

USB 3.0 Super High Speed Cables

Throwing Hard Disks With USB 3.0?

It has been difficult to find shipping USB 3.0 adapters, but this research has not been without some humor. I ran across an interesting article titled "Buffalo wants to be the first ones in throwing hard disks with USB 3.0", and could not resist reading it. Given the title, the following sentence from the article seemed to make sense:

It is clear that he has not too much sense to acquire a disk like that if we have not a compatible groove with USB 3.0 in the computer.

I think anyone that throws hard disks with USB 3.0 probably "has not too much sense" to begin with.

Things You Should Know About Windows 7 - God Mode

2010-01-11T08:00:00.000-08:00

Take a tip from Bruce Almighty - use Windows 7 "God Mode" wisely. Windows 7God Mode provide shortcuts to various Windows 7 settings. God Mode is a folder naming trick and it's easy to do.

As the video below will show, you just create a new folder and rename it using one of the God Mode strings.

Special thanks to Marius Oiaga for providing this list of God Mode strings in the article Windows 7 and Windows 8 GodModes, the Complete List:

Action Center Microsoft.ActionCenter (Windows 7 and later only) {BB64F8A7-BEE7-4E1A-AB8D-7D8273F7FDB6}
Backup and Restore Microsoft.BackupAndRestore (Windows 7 and later only) {B98A2BEA-7D42-4558-8BD1-832F41BAC6FD}
Biometric Devices Microsoft.BiometricDevices (Windows 7 and later only) {0142e4d0-fb7a-11dc-ba4a-000ffe7ab428}
Credential Manager Microsoft.CredentialManager (Windows 7 and later only) {1206F5F1-0569-412C-8FEC-3204630DFB70}
Default Location Microsoft.DefaultLocation (Windows 7 and later only) {00C6D95F-329C-409a-81D7-C46C66EA7F33}
Desktop Gadgets Microsoft.DesktopGadgets (Windows 7 and later only) {37efd44d-ef8d-41b1-940d-96973a50e9e0}
Devices and Printers Microsoft.DevicesAndPrinters (Windows 7 and later only) {A8A91A66-3A7D-4424-8D24-04E180695C7A}
Display Microsoft.Display (Windows 7 and later only) {C555438B-3C23-4769-A71F-B6D3D9B6053A}
Getting Started Microsoft.GettingStarted (Windows 7 and later only) {CB1B7F8C-C50A-4176-B604-9E24DEE8D4D1}
HomeGroup Microsoft.HomeGroup (Windows 7 and later only) {67CA7650-96E6-4FDD-BB43-A8E774F73A57}
Infrared Microsoft.Infrared (Windows 7 and later only) {A0275511-0E86-4ECA-97C2-ECD8F1221D08}
Location and Other Sensors Microsoft.LocationAndOtherSensors (Windows 7 and later only) {E9950154-C418-419e-A90A-20C5287AE24B}
Notification Area Icons Microsoft.NotificationAreaIcons (Windows 7 and later only) {05d7b0f4-2121-4eff-bf6b-ed3f69b894d9}
Pen and Touch Microsoft.PenAndTouch (Windows 7 and later only) {F82DF8F7-8B9F-442E-A48C-818EA735FF9B}
Phone and Modem Microsoft.PhoneAndModem (Windows 7 and later only) {40419485-C444-4567-851A-2DD7BFA1684D}
Recovery Microsoft.Recovery (Windows 7 and later only) {9FE63AFD-59CF-4419-9775-ABCC3849F861}
Region and Language Microsoft.RegionAndLanguage (Windows 7 and later only) {62D8ED13-C9D0-4CE8-A914-47DD628FB1B0}
RemoteApp and Desktop Connections Microsoft.RemoteAppAndDesktopConnections (Windows 7 and later only) {241D7C96-F8BF-4F85-B01F-E2B043341A4B}
Sound Microsoft.Sound (Windows 7 and later only) {F2DDFC82-8F12-4CDD-B7DC-D4FE1425AA4D}
Speech Recognition Microsoft.SpeechRecognition (Windows 7 and later only) {58E3C745-D971-4081-9034-86E34B30836A}
Troubleshooting Microsoft.Troubleshooting (Windows 7 and later only) {C58C4893-3BE0-4B45-ABB5-A63E4B8C8651}

Using Selenium And BrowserMob For Load Testing

2010-01-07T08:00:00.000-08:00

Update: Who doesn't like FREE! The folks at BrowserMob now give you an instant free test. No signup necessary. Uses real browsers. See screenshots, DNS lookup times, time-to-first-byte, and more! It's worth checking out. Click to get your Free Instant Test.

November 30,, 2009 8:00AM PDT
We've written several articles on using Selenium for web testing, including a HOWTO series on "Automated Cross-Browser Testing With Selenium". We plan to expand beyond these functional testing articles by looking at how Selenium can be used for other types of testing. Today's video on BrowserMob shows how you can use Selenium for low-cost load testing service that uses real web browsers.

The video below takes you through recording a Selenium test case using the Selenium IDE and then running it using BrowserMob. I'm particularly interested in how easily you can set up your load tests and the great information provided in their dashboard.

Note: I've not figured out the pricing yet (seems a bit complicated). They claim that you can "Simulate 2,000 VUs for only $150" but if you look at their pricing table, it's difficult to immediately see that sort of pricing.

Latest DeployStudio Tips, Techniques And News

2010-01-06T08:00:00.000-08:00

If you're a DeployStudio fan, or are looking to learn what's behind the growing popularity of this tool, then be sure to attend the upcoming DeployStudio Seminar described in today's article.
LinkedIn Event: If you're a Mac Systems Administrator with more work on your plate than time to complete it you can't miss this seminar. Oliver Block of Skeleton Key will present a demonstration of DeployStudio, which enables you to streamline the deployment of multiple machines. This and similar technology has saved our customers hundreds of hours. DeployStudio Seminar

Here's a nice PDF that covers a lot of the basics of installing and using DeployStudio. DeployStudio Tips & Tricks

This blog has a number of tips and techniques for using DeployStudio. Creating a Master DeployStudio Image

Latest DeployStudio Server with new features that allow you to define an authentication domain when configuring a workflow Active Directory binding task; provides packages management (packages sets and basic ACLs); an option to disable wireless support from DeployStudio NetBoot / external drive systems; and, an option to force UDIF disk images to be converted a second time to the same format before being scanned (multicast configuration panel). DeployStudio Server 1.0rc16

A recent HOWTO article on using one master Deploy Studio server and host the images at different sites. Use one Deploy Studio server with images on many servers

Would You Like To Browsersize That?

2010-01-05T08:00:00.000-08:00

Ever wonder what appears on real users' browsers when they visit your website? If you want a certain percentage of users, let's 90%, to all see the same thing, then you're going to need a tool that tells you in what percentile your web pages fall.

The Google Browser Size is just that tool and is a must have for anyone creating websites. It provides a visualization of browser window sizes, as a percentage of people who visit Google, superimposed on your website. You just enter your website URL, and you get to see contours that match up with different precentages. For example, the "90%" contour means that 90% of people visiting Google have their browser window open to at least this size or larger. Here's how they position this tool at their website:

This is useful for ensuring that important parts of a page's user interface are visible by a wide audience. On the example page that you see when you first visit this site, there is a "donate now" button which falls within the 80% contour, meaning that 20% of users cannot see this button when they first visit the page. 20% is a significant number; knowing this fact would encourage the designer to move the button much higher in the page so it can be seen without scrolling.

And here's what this blog looks like using the Google Browser Size tool:

To view your own Web site with this same visualization overlaid on it, go to http://browsersize.googlelabs.com/, type your URL into the "Enter URL here" textbox at the top of the window and click Go.

Crawljax For Testing AJAX Applications

2010-01-04T08:00:00.000-08:00

Wouldn't it be nice to finally have a tool that solves all the problems we encounter with automated testing? So often we find test teams going down the path of automating individual test cases knowing that the cost to maintain each of those individual test cases will grow. And for those that have looked at advanced automated testing methodologies, such as model-based and keyword driven testing, they know there's a better way, yet they proceed with developing large suites of automated test cases that end up creating a maintenance nightmare later on when things changes.

In 2010, we will continue to work with our clients to use open source frameworks that solve some of the problems inherent with developing automated tests. In particular, we recommend FitNesse for anyone testing web applications and TestOptimal for those interested in using model-based testing. For those of you that have already invested in using Selenium, you should consider FitNesse as THE framework for delivering a more effective automated test development and execution capability.

As for that new tool that will solve all our problems? We haven't found it yet, but we will keep looking. And this year, as we develop more FitNesse automated test solutions for our clients, we'll be keeping an eye on new tools like Crawljax. Crawljax is yet another open source tool that has received great attention through Google's Test Autotmation Conferences. It is a tool for crawling any AJAX/GWT application (GWT = Google Web Toolkit). It uses WebDriver to navigate through the different states of a web application. It creates a state-flow graph of the dynamic DOM states and the transitions between them. This inferred state-flow graph forms a very powerful base for many types of automated web testing. With plugins and invariants Crawljax can be used to perform various automated tests. For example: security testing, regression testing, accessibility testing, performance testing, cross-browser testing.

And as with most new open source tools, it takes time to create polished marketing and support material - the developers wisely spend their time making the tool better and better. The video below has a lot of information and detail on Crawljax. The audio is not great, but if you're motivated to see if Crawljax can solve some of your automated testing problems, then this video is worth watching. And you'll probably want to supplement what you watch with more info at this link.

University Of Utah Mac Managers Share Their DeployStudio Experience

2009-12-21T08:00:00.000-08:00

DeployStudio University! Mac Managers from the University of Utah share their DeployStudio secrets on iTunes in a presentation from their monthly Mac Managers meeting. Each month they upload their presentations to iTunes U. The video below is a comprehensive presentation on DeployStudio they gave in June 2009:

All of the slides shown in this video presentation are available as a PDF document here. Additionally, all of their presentations are available on iTunes U. Here's the link for all the Mac Manager Meetings presentations.

Using Sauce OnDemand To Run Automated Cross-Browser Tests In The Cloud

2009-12-18T08:00:00.000-08:00

Update: What has the original creator of Selenium (Jason Huggins) been cooking up lately? Hint: it has to do with Sauce.

Jason and his colleagues at Sauce Labs have been very busy since last we wrote about their product offerings. They've renamed their cloud-hosted service to Sauce OnDemand (formerly called "SauceRC") and announced a Sauce Labs enhanced version of Selenium RC (versions for both PC and Mac) which they are calling Sauce RC. To find out more about Sauce RC, here's a link to a screencast video that describes the product and how to get started using it.

August 21, 2009 8:00AM PDT

In May we wrote an article about a service we discovered called Sauce OnDemand (formerly called "SauceRC") from Sauce Labs (Selenium RC In The Cloud - SauceRC). We found this service interesting because it allows you to do browser compatibility testing without the hassle (let alone cost and energy) of configuring local machines and installing browsers. Instead, you push your tests into the Sauce Labs cloud. Simply create test scripts using the Selenium IDE plug-in for FireFox, designate the browser you want to test on (see code below) and let Sauce OnDemand do all the work. Repeat for each browser you want to test on, then go to the Sauce Labs website to retrieve the results. And the cool part is that you get a movie, as shown in the graphic above, of an actual browser session running your tests. Talk about convenience!

It's best if you know how to use Selenium RC before you start using Sauce OnDemand, but, according to the developer, some customers have successfully used the service without previous use of Selenium RC. In our case, we had already written scripts for Selenium RC, so it was simply a matter of changing 1 line in our script using the code below:

DefaultSelenium selenium = new DefaultSelenium(
                "saucelabs.com",
                4444,
                "{\"username\": \"rvizzone\"," +
                "\"access-key\": \"xxxxxxxxxxxxxxxxxxx\"," +
                "\"os\": \"Windows 2003\"," +
                "\"browser\": \"firefox\"," +
                "\"browser-version\": \"3.\"}",
                "http://www.google.com/");

Note the browser and version designated in red.

After we made this change, our scripts ran as they had using Selenium RC on our localhost, with the actually processing taking place at Sauce Labs, and the results, both a movie and log, available to use anytime through links accessible through our Sauce Labs account.

Here's a set of high-level HOWTO steps for using Sauce OnDemand:

1. Record and edit your test script with the Selenium IDE plug-in for Firefox.
2. Export your script and make sure it runs with Selenium RC on your localhost.
3. Make the 1 line change to your script that includes your user name, access key, etc., as shown above.
4. Now run the script again (which will now be directed to the Sauce OnDemand server).
5. Retrieve results from with your Sauce Labs account

If you're looking for your investment in Selenium to pay off even more, I suggest you try Sauce OnDemand. Sauce Labs offers both a subscription plan and pre-paid minutes. And if you sign up during their public beta program, the minutes you buy will be doubled, e.g. buy 250 minutes for $10 and you'll get 250 minutes free.

Automated Testing Institute

2009-12-15T08:00:00.000-08:00

While doing some article research this week I came across a website on automated testing called the Automated Testing Institute. Here's a synopsis of the content:

The ATI Online Reference is the web’s foremost comprehensive online resource for obtaining, exchanging and certifying industry standard test automation knowledge, information, and techniques. With key elements such as automation articles, automation specific news, searchable tools and book, discussion forums, free video tutorials, other automation specific videos, and more, this free, member-based site is the most comprehensive resource for helping you excel at test automation.

From my initial review of the main sections of this website, I was impressed with the variety of articles, video, tips and techniques offered. There's a wide range of articles, from those that appeal to the beginner (How Do I Get Started With Test Automation?), to more advanced topics on ROI, frameworks, etc. Check it out here.

Selenium 2.0 and Beyond!

2009-12-14T08:00:00.000-08:00

For those of you that are fans of Selenium, you'll probably want to hear what Simon Stewart (Google) and Jason Huggins (Sauce Labs) have to say about Selenium 2.0 and beyond.
In particular, the merging of efforts by Huggins (Selenium Core) and Stewart (WebDriver) is very important to understand where Selenium is headed.

Using The National Vulnerability Database During Security Testing

2009-12-11T08:00:00.000-08:00

The National Vulnerability Database (NVD) is a comprehensive cyber security vulnerability database
that integrates all publicly available U.S. Government vulnerability resources and provides references to industry resources. NVD's mission involves warning the public about vulnerabilities in computer systems. NVD provides this information using a search engine while integrating all publicly available U.S. government vulnerability resources. All of this information is given away for free with no licensing restrictions through XML and RSS feeds.

The NVD is easy to use and should be used during security testing. In fact, depending on which security tool you use, the results will most likely reference items in the NVD. I tried searching the NVD with some familiar software we use and the results were impressive. I did a search on the open source defect tracking system Bugzilla, and found numerous security flaws. I did this search by first going to this URL:

http://web.nvd.nist.gov/view/vuln/search?execution=e1s1

And then entering "bugzilla". The result was a list of 88 vulnerabilities that show up in an easy to read list with links that give you more specific details on each item:

Here's a closer look at some of the results from the list above:

CVE-2009-1213
Summary: Cross-site request forgery (CSRF) vulnerability in attachment.cgi in Bugzilla 3.2 before 3.2.3, 3.3 before 3.3.4, and earlier versions allows remote attackers to hijack the authentication of arbitrary users for requests that use attachment editing.
Published: 04/01/2009
CVSS Severity: 6.8 (MEDIUM)

CVE-2008-6098
Summary: Bugzilla 3.2 before 3.2 RC2, 3.0 before 3.0.6, 2.22 before 2.22.6, 2.20 before 2.20.7, and other versions after 2.17.4 allows remote authenticated users to bypass moderation to approve and disapprove quips via a direct request to quips.cgi with the action parameter set to "approve."
Published: 02/09/2009
CVSS Severity: 4.0 (MEDIUM)

Searching for vulnerabilities is just one feature of the NVD. There are number of checklists, statistics and other information available at the main NVD URL:

http://nvd.nist.gov/home.cfm

Add These Website Performance Test Tools To Your Toolkit

2009-12-10T08:00:00.000-08:00

I've listed some performance testing resource links that are worth looking at if you do any sort of website performance testing.
One of the more difficult tasks when conducting load and stress testing is identifying bottlenecks and suggesting areas on which developers should concentrate. Often this type of testing uses tools that are tuned to put a load on a website (e.g. JMeter) and consequently generates a lot of test data, primarily because these tools simulate lots of users. It helps to have another perspective to corroborate the data from your load tests. This is where these website performance tools may help:

Speed Tracer. Speed Tracer is a Google Chrome extension that helps you identify and fix performance problems in your web applications. It visualizes metrics that are taken from low level instrumentation points inside of the browser and analyzes them as your application runs. Using Speed Tracer you are able to get a better picture of where time is being spent in your application.

YSlow. This tool analyzes web pages and why they're slow based on Yahoo!'s rules for high performance web sites. YSlow uses Yahoo!'s Smush.it service.

Page Speed. An open-source Firefox/Firebug Add-on. Webmasters and web developers can use Page Speed to evaluate the performance of their web pages and to get suggestions on how to improve them.

Google Web Toolkit (with Speed Tracer). Google Web Toolkit (GWT) is a development toolkit for building and optimizing complex browser-based applications. GWT is used by many products at Google, including Google Wave and Google AdWords. It's open source, completely free, and used by thousands of developers around the world.

Web Performance Best Practices. When you profile a web page with Page Speed, it evaluates the page's conformance to a number of different rules. These rules are general front-end best practices you can apply at any stage of web development. We provide documentation of each of the rules here, so whether or not you run the Page Speed tool — maybe you're just developing a brand new site and aren't ready to test it — you can refer to these pages at any time. We give you specific tips and suggestions for how you can best implement the rules and incorporate them into your development process.

Vint Cerf - The Internet Today

2009-12-07T08:00:00.000-08:00

In today's video, Vint Cerf, often referred to as "the Father of the Internet", gives a lecture at Singularity University (www.singularityu.org) that first provides a history of the internet,
then covers the state of the Internet today, and what issues are arising as it continues to evolve. He talks about technology platforms - IPv6, cloud computing, etc. - that are sure influence how we evolve our test labs.

Open Web Application Security Project Top 10 Risks For 2010 Is Available

2009-12-03T08:00:00.000-08:00

The Open Web Application Security Project (OWASP) Top Ten has been a great source of information for security testing over the years that we've used it at Recommended Test Labs. They last published their list in 2007 (and 2004 before that). As stated at their web site:

The OWASP Top Ten provides a powerful awareness document for web application security. The OWASP Top Ten represents a broad consensus about what the most critical web application security flaws are. Project members include a variety of security experts from around the world who have shared their expertise to produce this list. There are currently versions in English, French, Japanese, Korean and Turkish. A Spanish version is in the works. We urge all companies to adopt this awareness document within their organization and start the process of ensuring that their web applications do not contain these flaws. Adopting the OWASP Top Ten is perhaps the most effective first step towards changing the software development culture within your organization into one that produces secure code.

As of Nov 13, 2009, the 2010 list has been released for comment through Dec 31,2009. The document is available at http://www.owasp.org/index.php/File:OWASP_T10_-_2010_rc1.pdf. Within that document, you will find the following Top 10 Risks:

A1 –Injection
A2 –Cross Site Scripting (XSS)
A3 –Broken Authentication and Session Management
A4 –Insecure Direct Object References
A5 –Cross Site Request Forgery (Cross site request forgery)
A6 –Security Misconfiguration(NEW for 2010)
A7 –Failure to Restrict URL Access
A8 –Unvalidated Redirects and Forwards (NEW for 2010)
A9 –Insecure Cryptographic Storage
A10 -Insufficient Transport Layer Protection

Below is a video that covers the Top 10 list from 2007:

Browser War Links - We May Not Know Where We're Headed, But We're Making Good Time

2009-12-02T08:00:00.000-08:00

Update: Adrian Kingsley-Hughes's article "XP's usage share down, Win 7, Firefox & IE8 up" describes a change afoot for OS and browser market share that's worth considering for compatibility test planning.
The latest data shows that XP is finally falling off and giving way to both Vista and Windows 7. As for browsers, Firefox continues to grab market share, while Chrome is closing in on Safari.

Firefox spoils an Internet Explorer Trifecta, while Safari and Chrome trail far behind. What the Latest Figures from the Browser Wars Tell Us

It's Netscape deja vu all over again. Andreessen working on a stealth browser project? Netscape founder re-enlists in browser wars

Speaking of Netscape: "1995 called and it wants its business plan back from Google." If this article holds any truth, then who's the next Microsoft? Is Google the next Netscape?

Now here's a battlefield in the browser war that may be the beginning of a tipping point. IE 8 is 5 times slower than Safari and Chrome? Yikes! Browser wars: Macs beat PCs on web speed

The mobile browser war appears to be over, or at least there's an armistice. Google Android (and Chrome), Apple Safari (and the iPhone), and now RIM move the mobile browser center-of-gravity in favor of WebKit. RIM acquires WebKit vendor for mobile browsing

Using DeployStudio For Imaging and Restoring OS X Setups for Testing

2009-12-01T08:00:00.000-08:00

Update: DeployStudio is very popular in schools. I found some very detailed documents on how one administrator setup DeployStudio in a K-12 school environment. The name of the article is Desktop Maintenance - DeployStudio QuickStart Documents.

For years we've used NetRestore for imaging and restoring our OS X test computers. Mike Bombich did a great job in writing and supporting this useful product, but it has now reached its end of life. Consequently, we set out to find a replacement and have settled on DeployStudio.

This feature set won us over:

Deploy & image Macs & PCs
Advanced computers reconfiguration (Mac only)
Full automation support
Realtime deployment monitoring
Freeware!

The way we employ it is on an external drive with OS X and DeployStudio server software installed. This server also acts as the image repository. When we need to reimage, we plug the external drive into the target machine and restart onto that OS. Then we use the DeployStudio Runtime to reimage the local partitions as desired.

It has the ability to run shell scripts after the restore is complete. This could be used to make images more convenient with less user interaction required. For example, you could cause the script to automatically install a commonly used testing application after being reimaged.

While their documentation on their homepage isn't the best, their forums are active and the creators seems to respond fairly quickly to questions. They also have a wiki. It's very new so not much in it, but presumably will have more information as time passes.

Testing Android Apps - The Compatibility Testing Matrix Continues To Grow

2009-11-24T08:00:00.000-08:00

Android, an open source operating system, must avoid the fate of J2ME, an open source mobile applications platform. Open source is great, until everyone splinters off into their own world. That’s what happened to J2ME, and a number of frustrated Android developers are now saying that there is a risk Android will follow the same path.
-Michael Arrington, A Chink In Android’s Armor, 10/11/2009

Android developers continue to voice their concerns, and in a more recent article, "Android's rapid growth has some developers worried", there's even more evidence that development and testing compatibility issues have a potential to grow out of control. Developers identify a slew of compatibility concerns:

3 versions of the OS (Android 1.5, 1.6 and 2.0)
Differences across handsets
Custom user interfaces, e.g. HTC Sense, Motorola Blur, etc.
Phones with and without camera flash
Keyboard differences (physical vs. GUI)
Different carrier ROMs
Non-standard commands for triggering standard functions, e.g. GPS on Hero vs. other Android devices
Uncooperative graphics drivers

These differences have created significant compatibility issues, so much so that developers are creating separate builds and maintaining separate code for different Android devices. The best analogy for the problems faced by Android developers can be found in Michael Arrington's article:

Imagine if Windows developers had to build different versions of their applications for different PC manufacturers. Or even different versions for various models by a single manufacturer. That’s what some Android developers are saying they are facing now.

I'm not so sure Android development and testing is any different than with most software development - witness the issues developer face with supporting their web apps across different browsers. But I do think that their expectations are reasonable: an OS developer (in this case Google) needs to do all they can to standardize how the application layer will behave. Certainly Apple's iPhone has set the bar high in this case and ultimately developers will decide how much they are willing to put up with.

Intelligent Automated Testing - A Look At FitNesse

2009-11-23T08:00:00.000-08:00

For those of you that read our interview with Matt Krapivner, you probably gained additional insight on the FitNesse automated testing framework. Having met with Matt recently, I've seen just how useful FitNesse can be, and so I wanted to share some videos I've found about FitNesse. This first one is a quick tutorial showing how to download and run FitNesse. This includes a demo of the FitNesse acceptance tests. The steps are straightforward, and when you're done, you'll have a fully functional FitNesse setup - ready for hacking and learning.

Model-Based Testing For Web Applications With TestOptimal - (Part 4) Reports

2009-11-20T08:00:00.000-08:00

Previous articles on Model-Based Testing For Web Applications With TestOptimal:

Part 1 - Preparing The Model
Part 2 - mScript
Part 3 - Generating Sequences

Today we conclude this series of articles on using TestOptimal for web applications testing by looking at the reporting features. TestOptimal provides detailed information on all aspects of each model-based testing project. The main reporting elements are:

Execution Status
Execution Progress
Testing Coverage: includes State/Transition traversals and # of each that are covered or left un-covered
Transition Sequences: a description of each state-to-state transition
Execution Details: includes exceptions and transition response times in milliseconds, as well as statistical analysis of the 3-sigma range for each response time on state/transitions based on historical MBT executions
Debugging: this information is provided in several panes - Console, mScript Log, Server Log and Selenium Log

Below is a report-by-report look at the information TestOptimal provides, along with a brief description for each report.

This is the Monitor Tab, which provides information on execution and coverage:

The Sequence Log tab shows the test sequence (state and transition traversal sequence) generated from the model with the selected sequencer. Use this transition log to find when and where the MBT exception occurred. You may also use this transition log to capture the test sequence and feed it to your testing frame work:

The Stats Tab shows MBT execution stats collected by TestOptimal, both in-progress MBT execution or historical executions:

The client log messages are written to this console log window. You can adjust the debug level using the drop-down list on top of the console:

The mScript Log window displays the log messages generated by your mScript and errors and exceptions triggered by the mScript:

The Server Log dislays the log generatd by TestOptimal sever. Ths is the place where you can find the errors and exception stack traces:

If you are using the bundled Seleninum RC server, this log window will let you access Selenium RC sever log file:

This concludes our 4 part series on using TestOptimal for model-based testing of web applications. Although we covered the main steps that relate to model-based testing - create the model, customize the mScript, generate and execute test sequences, and analysis results - there's a lot more to be discovered with TestOptimal. For more information on other features (e.g. mCase sequences, java support), different editions, and how to extend TestOptimal, visit their website here. I highly recommend that you look at their various tutorials and examples to see just how extensive this tool is for model-based testing.

Windows 7 In The News - A Month After Launch

2009-11-19T08:00:00.000-08:00

Windows 7 has been out for a month now and most everyone agrees that it's an improvement over Vista. As for testing with Windows 7, it looks like there's more to this new version of Windows than simply a better implementation of Vista. Aside from the security issues described in the links below, and the fact that the federal government cannot use Windows 7 yet, there are some pretty interesting unknowns. The two that impressed me most were the hidden tool for examining energy usage (might be useful during testing) and the way it handles crashes - Windows 7 actually modifies how it allocates memory after several crashes, which would affect how reproducible a bug might be. See if you agree:

In an inauspicious beginning to the week, the first zero-day bug for Windows 7 has emerged. The bug touches on Microsoft's Server Message Block (SMB) program -- specifically, SMBv1 and SMBv2 on Windows 7 and Windows Server 2008 R2. Microsoft has issued a security advisory describing workarounds, but says most users would be protected from attacks by blocking two ports at the firewall. Windows 7 Gets Its First Bug

Windows 7 has a hidden tool that will help you by examining your energy use and suggesting what you can do in order to improve it. To get a power efficiency report, Computerworld’s Preston Gralla provided the following steps... Tips and Tricks for Windows 7

If an application crashes a few times, Windows will automatically apply a shim that intercepts its memory operations. FTH will over-allocate memory, and keep a copy of freed memory so that attempts to re-read it will succeed. Windows 7's dirty secrets revealed

More than a dozen federal agencies, including the White House and all branches of the military, are testing Windows 7, according to Microsoft. But it may be another six months before agencies can move ahead with Windows 7 deployment because a government-mandated security standard hasn't been finalized. Government Use Of Windows 7 Hinges On Security Spec

Spotty support for networked all-in-one printers in particular may be a testing, QA, and certification issue; some notebook drivers aren't yet available either. Windows 7's Spotty Driver Support

Last week, security vendor Sophos published a blog post in which it said that Windows 7 was vulnerable to 8 our of 10 of the most common viruses. Microsoft has responded to these test results, which are a classic case of "scare 'm and they'll fall in line". Microsoft: Sophos Windows 7 Malware Test 'Sensationalised'

Why Would Anyone Want To Implement Automated Testing?

2009-11-18T08:00:00.000-08:00

Moreover, why does anyone even want to test their software at all? They just want it to work. But most know better, and so they test. But does management care if testing is manual or automated? Should testers care if their testing is manual or automated? My guess is that management cares about automated testing if they think testing is going to cost less and provide faster results. Tester's will consider automated testing if it increases coverage while making them more efficient.

The reason I'm writing about this subject is because of articles posted on Bj Rollison's blog about calculating ROI for automated testing. In Rollison's first article on the subject, Measuring Test Automation ROI, he says this about ROI:

I have 2 essential problems with ROI calculations within the context of test automation. First, if the business manager doesn’t understand the value of automation within a complex software project (especially one which will have multiple iterations) they should read a book on managing software development projects. I really think most managers understand that test automation would benefit their business (in most cases). I suspect many managers have experienced less than successful automation projects but don’t understand how to establish a more successful automation effort. I also suspect really bright business managers are not overly impressed with magic beans.

Harsh comments but probably a realistic view of things. But it makes things difficult for anyone wanting to bring automated testing into their QA processes, given that there's a need to explain benefits versus cost. The follow-on Part 2, further condemns ROI calculations to being a silly waste of time. What Rollison does offer is why he automates tests, as well as points to a set of questions useful in deciding when to automate. Rollison identifies 4 reasons he chooses to automate tests:

Free up my time,
Re-verify baseline assessments (BVT/BAT, regression, acceptance test suites)
Increase test coverage (via increased breadth of data or state variability),
Accomplish tasks that can’t easily be achieved via manual testing approaches.

In addition to his list, he praises Brian Marick's paper on When Should a Test Be Automated. Rollison lists Marick's 3 key questions one should ask before automating:

Automating this test and running it once will cost more than simply running it manually once. How much more?
An automated test has a finite lifetime, during which it must recoup that additional cost. Is this test likely to die sooner or later? What events are likely to end it?
During its lifetime, how likely is this test to find additional bugs (beyond whatever bugs it found the first time it ran)? How does this uncertain benefit balance against the cost of automation?

I think these questions and the process by which one answers them, is an effective way to inform one's decision as to automate or not. Neither Rollison or Marick offer answers, nor should they. As Rollison puts it: "Unfortunately, there is no single cookie-cutter approach in deciding what tests to automate. Different projects have different requirements and expectations, and, of course, not all tests are equal."

What I take away from these questions is a process for making logical decisions about automated tests with the ability to explain why the automation effort will benefit the organization's testing goals - if it improves testing it should be worth the investment, so long as that improvement fits within your goals.

USB Interoperability And Compatibility Testing Must Have - P55

2009-11-17T08:00:00.000-08:00

If you conduct USB interoperability testing, you'll want to add systems that use the Intel P55 Express chipset. We're finding out that the P55's USB controller can be problematic with certain devices. Here are a number of reports worth looking into:

Intel's quality control team who seem to have somehow missed an issue between Apple's app carrier deluxe and the P55 Express chipset's USB controller. iPhone and Windows 7 don't play nice, Intel P55 chipset to blame.

The Intel P55 Express chipset snafu that caused iPhones to lose their syncing minds has now been remedied -- at least by one motherboard maker. Gigabyte fixes iPhone sync issue with BIOS update

Asus and MSI, which also use the P55 express on their motherboards, haven’t issued any updates yet, but hopefully will soon. Busted Windows 7 + Intel P55 Express iPhone Sync Fixed for Gigabyte Motherboards — Asus and MSI Pending

"Our leading theory is a BIOS or system configuration issue, but we are still investigating," Intel said Monday. Intel comments on iPhone sync glitch

iTunes 9 on Windows 7 x64 Has Problems with iPhone

Getting Started With TestOptimal

2009-11-16T08:00:00.000-08:00

For the last few weeks we've been writing about model-based testing using TestOptimal. Today we have a video tutorial on getting started with TestOptimal. The video covers:

Installation
Screen Navigation
Main Features, and
Working with Demo Apps.

Model-Based Testing For Web Applications With TestOptimal - (Part 3) Generating Sequences

2009-11-13T08:00:00.000-08:00

So far we've prepared our TestOptimal project with the model we created in Part 1 (Model-Based Testing For Web Applications With TestOptimal - (Part 1) Preparing The Model) and the mScript we crafted in Part 2 (Model-Based Testing For Web Applications With TestOptimal - (Part 2) mScript). Now it's time to see how this all comes together by generating our test sequences.

TestOptimal offers 5 sequence modes. They are:

Optimal - uses the proven optimization algorithms to generate the optimal test sequence to achieve 100% coverage of your application. Because only the minimum number of test steps have to be repeated, the test execution completes in least amount of time.
Random - generates the test sequence by randomly selecting a transition from the current state. You can set conditions when to stop the test execution
Greedy - works the same way as Random except that it prefers un-traversed (un-tested) transitions over already traversed transitions. This helps to achieve the desired coverage with less number of test steps.
mCase Optimal - generates the optimal test sequences to cover all states and transitions contained in mCases. Use mCaseSequencer if you just want to test certain scenarios or parts of your application. You may also use mCaseOptimal with virtual users to simulate concurrent users running certain test scenarios.
mCase Serial - is a variation from mCaseOptimal. It differs from mCaseOptimal by preserving the order of transitions/states in mCase(s) in the test sequences it generates.

Each of these sequence types describe how TestOptimal will traverse our graph, and consequently how much coverage we will achieve. I tried out the first three sequence types (mCase Serial and Optimal won't be covered in this article) and took them for a "dry run" to see what was produced.

For each sequence type I selected, I received a report on Testing Coverage from the Monitor tab:

Greedy

Random

Optimal

As you can see, the Monitor tab shows how many traversals were done for each state and transition, as well as how many states/transitions were covered or not. Since I set up the sequencer to seek 100% coverage, there were no un-covered states/transitions. I was surprised to see the Random sequence achieve a greater efficiency over the Greedy sequence. And I was very impressed to see how efficient the Optimal sequencer was.

In addition to the information above provided in the Monitor tab, TestOptimal provides several other views for each run. There's a Sequence Log, which shows the last 1000 traversals and lists each state and transition traversed:

And a Stats tab, which provides state and transition coverage information with performance times:

After trying out these sequences in "dry-run" mode, I decided to run the test sequences against the actual target - maps.google.com. It worked flawlessly. After pressing the Run button, TestOptimal launched a browser window and began running each of the several hundred test sequences. It was fun to watch and, more importantly, demonstrated the power of model-based testing. Next week we'll take a look at some of the details within TestOptimal reports and analyze our results.

Learning How To Test Mobile Device Applications (Part 4) - iPhone

2009-11-12T08:00:00.000-08:00

Learning How To Test Mobile Device Applications (Part 1) - Palm webOS
Learning How To Test Mobile Device Applications (Part 2) - Android
Learning How To Test Mobile Device Applications (Part 3) - BlackBerry

Today we conclude this HOWTO series with the iPhone. Like the other mobile device platform companies, Apple provides everything we need to create a simple app so that we will have an understanding of how to test those apps. And it meets our cost goal - the SDK is free (if you know where to look) and you can create and test your iPhone app without having to buy an iPhone.

If you visit the iPhone Developer Program page, you'll end up being asked to join the program for a fee. And that makes sense if you want to be able to create and publish your iPhone app, but we just want to learn what it takes to develop and test an app by downloading the SDK for free. What you want to do is sign up as a Registered iPhone Developer. And if you have already have an iTunes, Apple Online Store or MobileMe account, you can use your existing Apple ID to sign in and become a registered iPhone developer. It's easy and takes a few seconds to do. Once you're registered, then you can access the iPhone Dev Center to download the SDK.

After you register and then log in to the iPhone Dev Center, you can download the SDK here. Once downloaded and installed, then you're ready to create your app and run it on the iPhone simulator. And luckily, there's a tutorial for that - Tutorial: Hello, World! This tutorial takes through these steps:

Create the Project
Write the Code
Run the Application

Seems simple enough, and fortunately the tutorial has pictures and clear descriptions for each step. When you're done, I recommend you setup Apple's iPhone Configuration Tool and run your app with the console window open. There you'll be able to log application and system level warnings. By knowing what's being reported when interacting with an app can help you refine the steps you need to reproduce tricky (and memory related) problems.

Model-Based Testing For Web Applications With TestOptimal - (Part 2) mScript

2009-11-06T08:00:00.000-08:00

This week we will create the scripting component of our TestOptimal model-based testing project. But before we do, I need to refer back to last week's article and point out that my initial model contained a "bad state" - one that was inaccessible from any other state. I found this out by importing my model into TestOptimal and having it check out the many states and transitions of my model. You can do a dry run using TestOptimal, which will generate a test sequence that covers all transitions in your model. If you have any isolated states or states that are not reachable (as was the case for me), an error popup will be displayed, which indicates you have missed some transitions thus making some states not reachable. TestOptimal found my "bad state" and I quickly fixed it. It turns out that a typo contributed to this error. The new (and correct) model has been posted to last week's article (Model-Based Testing For Web Applications With TestOptimal - (Part 1) Preparing The Model).

TestOptimal uses a scripting format called mScript. mScript is a simple, XML based scripting to drive your web applications. It has only 14 tags and is very easy to learn and use for writing your testing scripts. It's also very powerful. Here's a list of the 14 tags:

action - executes the action specified in the code attribute, e.g. to click the button named "Order", you would enter code="click('name=Order')". See mScript function. You may also create your own user defined functions.
assert - asserts the condition specified. if condition fails throw the MBTException.
dataset - defines a dataset. Its body contains tags that populates the dataset.
db - defines a jdbc data source to be used by sql tag.
field - defines a field in a row and sets its value.
if - performs condition check and if evaluates to true then execute the command tags in the body.
log - logs a message to TestOptimal server log file.
mbt - container tag to hold one or many scripts to control MBT execution.
rowdata - adds a row to the data set. Must be within a dataset tag and contain at least one field tag.
script - defines a script method to be executed at state entry/exit or transition prep, action or verify. The body of the script tag is the command tags. This tag is equivalent to the method declaration in a java class.
sql - declares the sql statement to populate dataset.
state - container tag to hold scripts for a state including transition.
transition - container tag to hold scripts for a transition.
while - executes the script in its body if the condition specified is true.

To write your mScript, you'll need an XML editor. I use Notepad++ (Windows) or TextWrangler (Mac OS). Here are the steps I used to create my mScript:

1. Generate a skeleton mScript. To do so, just click on mScriptTab, the mScript displayed is the mScript we will use. Right-mouse click on the script content and select "View Source" option, copy and paste the code into your XML editor.

2. Gather UI identifiers from the target web interface (google maps). I used the FireFox Selenium IDE to do this. By recording each action from my model, e.g. click Search maps, the Selenium IDE captured the identifiers I needed.

3. Add the specific script actions that pertain to each transition in the model. There were 11 transitions in my model, so I had 11 scripts to write. Within the skeleton script generated in Step #1 above, there are 2 sets of function stubs under each state and 3 sets of function stubs under each transition. To keep things simple, I did not use any of the state functions, instead concentrating on the transition function actions. The transitions for my model involved only two actions: clicking on objects (e.g. buttons) or selecting from a pop-down menu. Hence I needed to write two action scripts:

  <action code="$click(element_locator)"/>
  <action code="$selectOption(element_locator,label)"/>

"element_locator" is a string to locate the link on the page. Again, I used the Selenium IDE FireFox plug-in to find the identifiers for each element locator. To refresh our memory on what we are testing, here's a screenshot of the limited UI we selected for these articles:

Referring to the above graphic, here are the mScript action codes I used for the transitions in my model:

Click Show options

<action code="$click('id=showss')"/>

Click Remove options

<action code="$click('id=hidess')"/>

Select option 'All Results'

<action code="$selectOption('id=mrtbox','label=All results')"/>

Select option 'Locations'

<action code="$selectOption('id=mrtbox','label=Locations')"/>

Select option 'Businesses'

<action code="$selectOption('id=mrtbox','label=Businesses')"/>

Select option 'User-created content'

<action code="$selectOption('id=mrtbox','label=User-created content')"/>

Select option 'Related maps'

<action code="$selectOption('id=mrtbox','label=Related maps')"/>

Select option 'Mapped web pages'

<action code="$selectOption('id=mrtbox','label=Mapped web pages')"/>

Select option 'Real estate'

<action code="$selectOption('id=mrtbox','label=Real estate')"/>

Click Search Maps

<action code="$click('id=q-sub')"/>

Click Google maps logo

<action code="$click('id=logo')"/>

4. Modify the skeleton mScript code with the action codes from above (and remove any unneeded tags as desired).

5. Deploy your mScript code. To do so, copy or save to the /script/ folder in the TestOptimal home directory using the same name as the model name, e.g. "google_maps.xml".

We're now ready to exercise our model using the mScript that has been tailored to our test target (I've included the script below). Next week we'll generate test sequences and start testing.

<xml version="1.0">
 <mscript>
  <state id="n1">
   <transition event="show_options">
    <script type="action">
     <action code="$click('id=showss')" />
    </script>
   </transition>
  </state>
  <state id="n2">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n3">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n4">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n5">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n6">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n7">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
  <state id="n8">
   <transition event="remove_options">
    <script type="action">
     <action code="$click('id=hidess')" />
    </script>
   </transition>
   <transition event="select_all">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=All results')" />
    </script>
   </transition>
   <transition event="select_locations">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Locations')" />
    </script>
   </transition>
   <transition event="select_bus">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Businesses')" />
    </script>
   </transition>
   <transition event="select_user">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=User-created content')" />
    </script>
   </transition>
   <transition event="select_related">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Related maps')" />
    </script>
   </transition>
   <transition event="select_mapped">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Mapped web pages')" />
    </script>
   </transition>
   <transition event="select_real">
    <script type="action">
     <action code="$selectOption('id=mrtbox','label=Real estate')" />
    </script>
   </transition>
   <transition event="search_maps">
    <script type="action">
     <action code="$click('id=q-sub')" />
    </script>
   </transition>
   <transition event="click_google_maps">
    <script type="action">
     <action code="$click('id=logo')" />
    </script>
   </transition>
  </state>
 </mscript>
</xml>

Learning How To Test Mobile Device Applications (Part 3) - BlackBerry

2009-11-05T08:00:00.000-08:00

In this HOWTO series, we've covered two popular mobile device platforms, webOS and Android, in these articles:

Learning How To Test Mobile Device Applications (Part 1) - Palm webOS
Learning How To Test Mobile Device Applications (Part 2) - Android

Today we'll look at BlackBerry. As we've stated in previous articles, our goal is learn enough from developing simple apps on each platform so that we have a better understanding of the platform as testers. And we want to accomplish this without having to buy each mobile device or any development software. Fortunately, developer offerings from RIM meet these requirements, albeit with less hipness than webOS and Android.

Their "Getting Started" info is somewhat buried on their BlackBerry Java Development Environment page, and most of the info is provided as PDFs. Which is OK, but not as convenient as having online reference web pages, particularly since some of these PDFs load slowly. Basically, here's what you'll need to get started:

Intel® Pentium® 4 Processor (minimum 3 GHz)
Microsoft® Windows VistaTM, or Windows® XP
Java® SE Development Kit (JDK) version 5 or version 6 Version 6 is required if you
are using BlackBerry MDS-CS for debugging
Eclipse IDE for Java Developers version 3.4.0
Blackberry tools (Java Application Development tools)

They do say you can develop on Linux or Mac OS X, but the "easiest and best supported is the Windows 32bit platform." Interesting, especially since the minimum Windows systems needs to be 3GHz.

Here are the step-by-step instructions for setup:

Download the JDK at http://java.sun.com/javase/downloads index.jsp. The current version is JDK 6 Update 12 which includes the JRE
Select required platform and language, agree to the licence agreement, and proceed with the download
Download the Eclipse IDE at http://www.eclipse.org/downloads/
Select Eclipse IDE for Java Developers and the version you require
Save the file on your computer
Download the BlackBerry tools at http://na.blackberry.com/eng/developers/resources/devtools.jsp. Here you will need to download the BlackBerry Plug-in for Eclipse, as well as the BlackBerry JDEs
Download all available JDEs (currently 4.3 to 4.7) so that you can test your applications on most current BlackBerry devices. You will need the plug- in and at least one JDE.
Once you click on the link you want to download you will have to sign in (or register if you are using this service for the first time)
Use this video to learn how to install and configure the BlackBerry plug-in for Eclipse.

The above will get you to the point where you can create your first app. RIM provides a detailed, step-by-step tutorial on creating a "Hello World" app. This includes instructions on how to run it on their BlackBerry simulator. Follow their instructions and you'll have your first app up and running and ready to test. And you'll find a wealth of testing and debugging information in their How to Debug and Optimize guide.

iPhone App Project Managers - How Did You Test Your App?

2009-11-04T08:30:00.000-08:00

iPhone application development is exploding (How Hot Is The iPhone Application Development Market?). And users are demanding better quality - no longer is it acceptable for an iPhone app to crash unexpectedly with the explanation that apps crash every so often because of the iPhone OS. Moreover, with more-and-more iPhone and iPod touch models to choose from, developers face a compatibility testing challenge. So how are project managers testing their iPhone apps?

We'd like to hear from iPhone application developers and managers who have navigated their way onto the App Store and can share with us the problems they faced in testing their apps, as well as tips and recommendations.

If you are interested in helping out, please leave a comment or email us. We will send you questions by email and publish your interview here.

Top 10 Software Engineering Risk List

2009-11-03T08:00:00.001-08:00

This list is from an interesting article in SoftwareTech News and represents project assessments over many years. The author, Donald J. Reifer, has had the benefit of conducting risk assessments over several decades and he put together a list that identified root causes and impacts. Here are the Top 10 Risks he identified:

Impossible Schedules
Inadequate Budgets
Poor Requirements
Staffing Shortfalls
Lack of Operational Concept Document
Failure to Emphasize Testing Early Enough in the Process
Poor Measurement
Subcontract management focuses on contract issues
Over-Reliance on Commercial Off-The-Shelf (COTS)
Licensing Headaches

On the testing side of things, I can definitely relate to several in the list, particularly about not testing early enough and poor requirements. QA and testing always suffers from those problems, and yet they continue to be at the top of the list. The impact listed for failure to test early emphasized "Not enough time and resources available to pull together a meaningful test program...".

For more on this article, visit Common Risk Patterns

Testing Applications In Window 7's XP Mode (HOWTO)

2009-11-02T08:00:00.001-08:00

A lot has been said about XP Mode for Windows 7. And I thought it was included in all versions of Windows 7. Turns out it's not. You have to install XP Mode and it only works on Ultimate, Professional and Enterprise. This video covers the key features with XP Mode as well as a brief instruction on how to obtain and install it. At least Microsoft provides a fully functional copy of Windows XP for free with the download.

Model-Based Testing For Web Applications With TestOptimal - (Part 1) Preparing The Model

2009-10-30T08:00:00.000-07:00

Last week we introduced a 4 part HOWTO series on using TestOptimal for model-based testing of web applications. In today's Part 1, we'll create a model using RTL's model generation tool, then convert that model for import into TestOptimal. This will then allow us to generate automated test cases using TestOptimal's various test generations capabilities.

The target web application I selected to model for Part 1 is Google Maps. And as we've discussed in previous model-based testing articles (Model-Based Testing HOWTO Part 1 - Describing The Model), it's important to work with small models - ones that represent a logical set of functionality within a manageable set of states. To that end, I selected the Google maps search bar as our target to model. Basically, we're going to model the search options functionality (Show/Remove search options) as well as the two buttons Search Maps and Google maps. That may not seem like much to model, but from a testing point of view, as you take into account the various options for searching Google Maps, the model starts to expand quickly.

To build a model, you need to determine your states and transitions. Once you identify those, you can begin to build your model, either manually or, as we do at RTL, using an automated tool. So let's start by identifying the states we want to use for our model. Based on the UI in the Google Maps search bar, and to keep things simple for this example, we'll only use the search options states. Here's how I defined the sub-states:

1. SEARCH OPTIONS - Shown | Removed
2. OPTIONS SETTING - All results | Locations | Businesses | User-created content | Related maps | Mapped web pages | Real estate

The transitions I chose to include in this model are:

1. Click Show search options
2. Click Remove Search options
3. Select the search option "All results"
4. Select the search option "Locations"
5. Select the search option "Businesses"
6. Select the search option "User-created content"
7. Select the search option "Related maps"
8. Select the search option "Mapped web pages"
9. Select the search option "Real estate"
10. Click "Search Maps"
11. Click Google maps

Using this limited set of states and transitions resulted in a model (directed state graph) with 8 states and 71 paths! Click the graphic below to view in full size.

When importing a model like the one described above into TestOptimal, it needs to be in the GraphML format. So the next step is to convert our model into the proper format for TestOptimal. I used TestOptimal's GraphML example listed here, which is based on a simple digraph found here. Using these examples, I modified the output from our model generation tool so that it would produce a proper GraphML file:


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<graphml>
<graph>
    <node id="n1">
    </node>
    <node id="n2">
    </node>
    <node id="n3">
    </node>
    <node id="n4">
    </node>
    <node id="n5">
    </node>
    <node id="n6">
    </node>
    <node id="n7">
    </node>
    <node id="n8">
    </node>
    <edge id="show_options" source="n1" target="n2">
    </edge>
    <edge id="remove_options" source="n2" target="n1">
    </edge>
    <edge id="select_all" source="n2" target="n2">
    </edge>
    <edge id="select_locations" source="n2" target="n3">
    </edge>
    <edge id="select_bus" source="n2" target="n4">
    </edge>
    <edge id="select_user" source="n2" target="n5">
    </edge>
    <edge id="select_related" source="n2" target="n6">
    </edge>
    <edge id="select_mapped" source="n2" target="n7">
    </edge>
    <edge id="select_real" source="n2" target="n8">
    </edge>
    <edge id="search_maps" source="n2" target="n2">
    </edge>
    <edge id="click_google_maps" source="n2" target="n1">
    </edge>
    <edge id="remove_options" source="n3" target="n1">
    </edge>
    <edge id="select_all" source="n3" target="n2">
    </edge>
    <edge id="select_locations" source="n3" target="n3">
    </edge>
    <edge id="select_bus" source="n3" target="n4">
    </edge>
    <edge id="select_user" source="n3" target="n5">
    </edge>
    <edge id="select_related" source="n3" target="n6">
    </edge>
    <edge id="select_mapped" source="n3" target="n7">
    </edge>
    <edge id="select_real" source="n3" target="n8">
    </edge>
    <edge id="search_maps" source="n3" target="n3">
    </edge>
    <edge id="click_google_maps" source="n3" target="n1">
    </edge>
    <edge id="remove_options" source="n4" target="n1">
    </edge>
    <edge id="select_all" source="n4" target="n2">
    </edge>
    <edge id="select_locations" source="n4" target="n3">
    </edge>
    <edge id="select_bus" source="n4" target="n4">
    </edge>
    <edge id="select_user" source="n4" target="n5">
    </edge>
    <edge id="select_related" source="n4" target="n6">
    </edge>
    <edge id="select_mapped" source="n4" target="n7">
    </edge>
    <edge id="select_real" source="n4" target="n8">
    </edge>
    <edge id="search_maps" source="n4" target="n4">
    </edge>
    <edge id="click_google_maps" source="n4" target="n1">
    </edge>
    <edge id="remove_options" source="n5" target="n1">
    </edge>
    <edge id="select_all" source="n5" target="n2">
    </edge>
    <edge id="select_locations" source="n5" target="n3">
    </edge>
    <edge id="select_bus" source="n5" target="n4">
    </edge>
    <edge id="select_user" source="n5" target="n5">
    </edge>
    <edge id="select_related" source="n5" target="n6">
    </edge>
    <edge id="select_mapped" source="n5" target="n7">
    </edge>
    <edge id="select_real" source="n5" target="n8">
    </edge>
    <edge id="search_maps" source="n5" target="n5">
    </edge>
    <edge id="click_google_maps" source="n5" target="n1">
    </edge>
    <edge id="remove_options" source="n6" target="n1">
    </edge>
    <edge id="select_all" source="n6" target="n2">
    </edge>
    <edge id="select_locations" source="n6" target="n3">
    </edge>
    <edge id="select_bus" source="n6" target="n4">
    </edge>
    <edge id="select_user" source="n6" target="n5">
    </edge>
    <edge id="select_related" source="n6" target="n6">
    </edge>
    <edge id="select_mapped" source="n6" target="n7">
    </edge>
    <edge id="select_real" source="n6" target="n8">
    </edge>
    <edge id="search_maps" source="n6" target="n6">
    </edge>
    <edge id="click_google_maps" source="n6" target="n1">
    </edge>
    <edge id="remove_options" source="n7" target="n1">
    </edge>
    <edge id="select_all" source="n7" target="n2">
    </edge>
    <edge id="select_locations" source="n7" target="n3">
    </edge>
    <edge id="select_bus" source="n7" target="n4">
    </edge>
    <edge id="select_user" source="n7" target="n5">
    </edge>
    <edge id="select_related" source="n7" target="n6">
    </edge>
    <edge id="select_mapped" source="n7" target="n7">
    </edge>
    <edge id="select_real" source="n7" target="n8">
    </edge>
    <edge id="search_maps" source="n7" target="n7">
    </edge>
    <edge id="click_google_maps" source="n7" target="n1">
    </edge>
    <edge id="remove_options" source="n8" target="n1">
    </edge>
    <edge id="select_all" source="n8" target="n2">
    </edge>
    <edge id="select_locations" source="n8" target="n3">
    </edge>
    <edge id="select_bus" source="n8" target="n4">
    </edge>
    <edge id="select_user" source="n8" target="n5">
    </edge>
    <edge id="select_related" source="n8" target="n6">
    </edge>
    <edge id="select_mapped" source="n8" target="n7">
    </edge>
    <edge id="select_real" source="n8" target="n8">
    </edge>
    <edge id="search_maps" source="n8" target="n8">
    </edge>
    <edge id="click_google_maps" source="n8" target="n1">
    </edge>
  </graph>
</graphml>

We're now ready to import our model into TestOptimal. In Part 2, we'll import this model and create a TestOptimal mScript.

Learning How To Test Mobile Device Applications (Part 2) - Android

2009-10-29T08:00:00.000-07:00

Last week we introduced this series on learning how to test mobile device applications based on the idea that developing a simple mobile app on each platform will help you understand how to test them (Learning How To Test Mobile Device Applications (Part 1) - Palm webOS). And we want to do this for free and without having to buy an actual device. That's worked for webOS from Palm and will work with this week's platform - Android.

And as we recommended in Part 1, testers should start by downloading and installing the SDK and building one of the included apps. For the Android platform, the steps for building your first app is explicitly laid out at Android Developer's SDK page. The instructions provided take you through the following steps:

Preparing your development computer
Downloading and installing the SDK starter package
Installing the ADT Plugin for Eclipse
Adding Android platforms to your SDK
Getting the latest documentation
Downloading other SDK components
Getting started with an application project

As with the webOS SDK, there's a Hello World app provided and it comes with a very nice tutorial. This tutorial explains how to set up a "virtual devices" for running your apps. As we mentioned at the beginning of this article, we want to be able to build and run an app without having to own a device. And that's possible because of AVDs or Android Virtual Devices. With an AVD, you're able to run the Android emulator that is used to run your apps.

Since we're building apps to learn how to test them, then you need to add some debugging tools into the mix once you build your app. And the Android SDK includes a fairly extensive set of tools to help you debug your programs. Two debugging tools I'd concentrate on are logcat and Android Log. All the Android debugging tools are describe here.

So there you have it. Another mobile platform that comes with free tools and the ability to develop and run apps without having to own a device, as well as debugging tools that will help during testing.

Plan For USB 3.0 Compatibility Testing In Late 2010

2009-10-27T08:00:00.000-07:00

Bad news for USB 3.0. Intel is pushing off it's plans to add USB 3.0 chipsets to 2011. This means that we won't see USB 3.0 in consumer-level systems until then. We can expect some systems with USB 3.0, but those will be limited to "a few high-end graphics workstations and consumer systems," according to an article at EE Times. This puts a damper on USB 3.0 testing plans. Although some manufacturers either have or will have products that support USB 3.0 next year, testing those products will be limited to custom configurations.

One manufacturer to look to for help might be Gigabyte. They plan a motherboard with USB 3.0 support. This board will have use a NEC chip for its USB 3.0 connector. And the on-again, off-again plans from ASUS regarding their debut of a motherboard with USB 3.0 support appears to be back on-again according to an this article. We previously cited a report of ASUS pulling this motherboard (USB 3.0 Compatibility Testing Is Coming Soon - Maybe), so we'll have to wait and see.

Model-Based Testing For Web Applications With TestOptimal

2009-10-23T08:00:00.000-07:00

We published an introductory article on TestOptimal earlier this year (TestOptimal - A Model-Based Testing Automation Tool For Web Applications). Since then, the TestOptimal developers have been busy adding new features, tutorials and examples. We spoke with the developers and asked them how we might integrate our model generation tool with TestOptimal. They anticipated a need like this and had already built a capability to import models created using external tools (rather than enter the model using TestOptimal). We were pleased to discover this capability and feel it's time now to put what we know about TestOptimal to work. For the next few weeks, I'll provide another of our HOWTO series on TestOptimal.

This HOWTO series will be presented in 4 parts and will cover:

Creating a model for import into TestOptimal
Creating a TestOptimal mScript
Generating Test Sequences
Analyzing Reports

Next week we'll publish Part 1 of this HOWTO series where we will create a model, based on a popular website, using RTL's model generation tool, then convert that model for import into TestOptimal.

Windows 7 And Unintended Consequences

2009-10-22T08:00:00.000-07:00

I went to Best Buy this weekend to buy a laptop. I didn't care what OS was on it, I just needed to get a laptop for a belated birthday gift. I thought I might find a low-end Dell or HP that had Vista and I could then upgrade to Windows 7 later. What I found was NOTHING. The shelfs were literally bare. I asked the sales person if they were going out of business. That was my first thought and he confirmed that others thought the same way. He assured me that they were not going out of business, but they had to remove all their display laptops and desktops to prepare for the Windows 7 debut on Thursday, 10/22. I was happy for him, but wanted a laptop and he had none to sell. In fact, the ones they did put on display the next day could not be sold before 10/22. I wasn't happy about that and almost bought another computer with a very different OS. But since this was a gift for someone that was used to Windows, I was stuck and ultimately ordered one from Best Buy online with a promised delivery date of 10/22. We'll see.

Because Microsoft restricted sales prior to October 22nd, and Best Buy was then forced to remove product from sale for almost a week, I wonder how many people considered, for the first time, an alternative to a Windows computer. Moreover, how many organizations, who may have stayed with Windows XP and are now facing upgrade paths that mean wiping their hard drives, are now forced to think through an upgrade strategy that may not include Windows 7. Certainly my inconvenience was a non-event for Best Buy and Microsoft, and I'll get over it, but this new Windows version, as good as it might be, may create a set of unintended consequences for Microsoft. I looked around to see if there's any merit to this supposition, and what I found were a number of articles that speak directly to this subject on several different levels.

Windows 7 will drive virtual desktop adoption, VMware says A decision to upgrade employee desktops to Win 7 may also make a CIO think more broadly about a desktop strategy, Balkansky (VMware's Bogomil Balkansky, vice president of product marketing) says. "[Windows 7] is very topical and puts the question on the table," he says. "I think it really forces the subject of virtual desktops to become an agenda item in many companies."

IBM Teams With Ubuntu For Windows 7 Alternative IBM and Ubuntu Linux developer Canonical on Tuesday jointly unveiled an open standards-based desktop architecture they said cuts 50 percent off the cost of deploying desktop PCs and provides an alternative to migrating to Microsoft (NSDQ:MSFT)'s Windows 7 operating system. ...comes at a time when customers are contemplating when to move to Windows 7, and even whether or not to make that move.

How Apple stole Microsoft’s Windows 7 glory Coming out of their most profitable quarter ever, Apple has gone ahead and stolen Microsoft's Windows 7 release-day thunder by announcing an all new lineup of iMacs and upgrading their range of MacBooks - two days before Microsoft releases their newest OS. ...offering consumers an alternative option to the soon-to-be-released Windows 7.

Microsoft's Windows 7 getting positive reviews, but PC market faces challenges Has Microsoft built a faster train while the rest of the industry is making planes and automobiles?

History suggests Windows 7 launch could boost Mac sales First reported by Fortune Brainstorm Tech, the findings show how Mac sales figures have consistently grown every time a new Windows operating system launched over the last 10 years. Mac sales spiked following the debut of Windows 98, Windows 200, Windows XP and Windows Vista.

The Changing Client Computing Landscape: Windows 7 and PC Alternatives ...I was surprised by how many seem to be considering hosted virtual desktops. The concept has been around for some time, but many more people seem to be taking it seriously now.

Economy May Slow Windows 7 Adoption in Enterprises The dark chapter of Windows Vista will close next week as Microsoft rolls out Windows 7, but enterprises may not have the money for full deployment. Unlike Vista's introduction, Softchoice said 88 percent of today's PCs can run Window 7. But with IT departments doing more with less, an analyst said it could take three years to fully deploy Windows 7.

If you're interested in a little CIO humor about Windows 7, check out this article: Windows 7 Launch: Top 10 Things Steve Ballmer Will Not Say

Top 5 .NET Groups On LinkedIn

2009-10-21T08:00:00.000-07:00

In previous articles, we posted the Top 10 LinkedIn Groups for Software Testing and QA and iPhone Development:

Top 10 Software Testing and QA Groups On LinkedIn
Top 10 iPhone Groups On LinkedIn

Today we have the Top 5 .NET developer groups on LinkedIn. Based on membership, the Top 5 are:

.Net Professionals (3,411 members): For all professionals who use the Microsoft .Net tools and related products.
Senior .NET Developers (8,478 members): Group of senior .NET developers working with latest and greatest Microsoft technologies including WCF, WF, WPF, AJAX, SQL Server 2005/2008, Silverlight, LINQ, BizTalk Server, MSMQ, IIS, ASP.NET, CardSpace, etc. Share ideas, ask questions, provide answers, share knowledge and news.
.NET Developers (18,869 members): A network for developers working with .NET technologies, including C#, VB.NET, ASP.NET, LINQ, etc. Connect with other like-minded software engineers.
.NET People (26,479 members): Are you working on Microsoft technologies like C#,ASP.NET,VB.NET and any other .NET related technologies? This is the place to meet other.NET Proffesional around the world. Join this group to connect and share knowledge on .NET. Please post discussions on .NET Technologies.
Linked.NET Users Group (LIDNUG) (31,163 members): Professionals that are using or interested in the Microsoft .NET technologies and the C# language in particular. The group is for developers, architects, designers (Silverlight), managers, consultants, Visual Studio software engineers, QA and Program Managers and anyone else who is interested.

Learning How To Test Mobile Device Applications (Part 1) - Palm webOS

2009-10-20T08:00:00.000-07:00

I recently visited Palm and our friends there gave us a Palm webOS book - signed by the author Mitch Allen (VP, Software CTO at Palm)! Very cool. While reading this book I thought I would use this opportunity to start a HOWTO series on learning to test mobile device apps, and model it after the process we used in our labs - develop apps to learn how to test them. Basically, we took testers through the process of setting up a development environment, compiling a simple app and then installing and testing that app.

We'll start with Palm webOS and then cover the iPhone and Android platforms. The place to start for any of these platforms is the software development toolkit, or SDK. And in the case of webOS, as the book points out, the SDK is available for free. It's called the Mojo SDK and it's available at http://developer.palm.com or can be directly downloaded here.

This SDK provides you development tools, sample code, along with tutorials and reference documents. All great resources for a developer and serves to inform and educate testers as well. We recommend that testers download and install the SDK and then use any of the example apps as their first project. That keeps the fear factor down since no code is written. And since each of the mobile device developers provide an emulator, you don't even need an actual device to learn how to compile and test applications.

Most people start out with something called the "Hello World" application that each SDK invariably offers as the starter application. And webOS is no different. In fact, there's an entire tutorial that describes how to download, compile and install their Hello World app. Once you get the Hello World app to compile and install on your emulator (again, the tutorial explains how to run the emulator), then you can set about to try some tests and learn about the tools that will help you test on actual devices.

On page 30 of the book, there's a paragraph about Testing and Debugging (yes, just a paragraph - but this is a developer book), where the author reinforces the idea of using the emulator during development and testing, and for debugging. Many of the tools described in the book and at Palm's developer website are mainly for developers, but testers should become proficient with any tool that logs information, if for nothing else than to provide a log file with each reported bug. In particular, get to know the palm-log tool. It displays log messages from an application on the emulator or USB-connected webOS device. You can also use palm-log to display the installed applications, which is useful for getting the ID of the application to log.

Here's a checklist I'd recommend for any tester wanting to learn more about webOS apps and how to test them:

Download the Palm webOS SDK
Install it on your favorite OS
Compile the Hello World application and run it in the emulator
Setup palm-log to understand how to capture log files

It's actually very straightforward and best of all, you'll be able to say you've compiled, run and tested a webOS app, as well as collected its log file, without spending any money and without even owning an actual device.

Next we'll look at how to do this for the iPhone.

Windows 7 Upgrade Helper

2009-10-16T08:00:00.000-07:00

Did you know that upgrading from Windows XP to Windows 7 requires you to erase your hard drive before you can upgrade? Doesn't sound like an upgrade to - more like a fresh install. Sure, you can back up your programs, files and settings first and then transfer them back. But all your applications will be nuked and you'll have to reinstall them (plus all updates). Again, sounds like a fresh install to me.

For testers, this shouldn't be a problem since fresh installs should always be done before you start testing. But what if your testing objective is to follow the same upgrade path as an end user might follow? That end user is most likely going to save programs, files, settings, and applications, and then restore them after the install. Consequently, you'll need to test for that installation use case. It's going to be a hassle either way, but at least there's a tool that can make it somewhat easier. And it's available until October 22nd for only $15. It's called PCmover Windows 7 Upgrade Assistant from Laplink Software Inc.

If you're interesting in finding out more, here's a video from WSJ's Katherine Boehret who reviewed the product:

Is All Testing Essentially Exploratory In Nature?

2009-10-15T08:00:00.000-07:00

Having just been to STARWEST 2009, and actually seeing James Whittaker in person, exploratory testing is naturally on my mind. And today, as I was browsing my favorite testing blogs, I ran across this quote, supposedly from Boris Beizer:

All testing is essentially exploratory in nature.

Makes a great title for this article, and also calls into question of why exploratory testing is a separate test type. I mean, after all, if all testing is essentially exploratory, then should there really be an exploratory test type?

In my quest to shed some light on this subject and find an answer, I dug up some articles from well known testing experts that have shared their opinions on exploratory testing, e.g. Kaner, Bach, Robinson, Bj Rollison, and others. But first stop is Wikipedia:

Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution. Cem Kaner, who coined the term in 1983, now defines exploratory testing as "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."... Exploratory testing

So naturally, we can't go any further until we hear what Caner has to say:

I coined the phrase "exploratory testing" 24 years ago, to describe a style of skilled work that was common in Silicon Valley. Naturally, the concept
has evolved since then, primarily as a way of focusing on how testers , p y y glearn about the product under test and its risks throughout the product's
lifecycle....A Tutorial in Exploratory Testing

At the risk of starting an argument, I would think that James Bach may want to take some credit in popularizing exploratory testing:

Exploratory software testing is a powerful approach, yet widely misunderstood. In some situations, it can be orders of magnitude more productive than scripted testing. All testers practice some form of exploratory testing, unless they simply don’t create tests at all. Yet few of us study this approach, and it doesn't get much respect in our field. This attitude is beginning to change as companies seek ever more agile and cost effective methods of developing software....Exploratory Testing Explained

Now many of you know Harry Robinson to be an expert on model-based testing and probably have attended one of his presentations at STARTWEST or STAREAST. But did you know he has a fond appreciation for exploratory testing?

The mission of the Lewis and Clark expedition was to explore and map the territory of the Pacific Northwest. William Clark was the chief mapmaker for the expedition. He spent months perfecting his mapmaking skills before the expedition left St. Louis. As it turned out, later explorers and settlers used Clark’s maps for the next thirty years until newer, more detailed maps were drawn up. The mapmaking was an essential part of the trip. It would have made little sense for Lewis and Clark to return home with collections of trinkets and souvenirs of the trip but no map of the territory that would tell others about the land and how to best navigate through it. On the other hand, it would have been equally foolish to send scores of settlers into that unexplored territory -- much better to send a handful of scouts to survey the territory and report back. Think about exploratory and automated testing from this same perspective.... Exploratory Modeling

We certainly can't leave out James Whittaker, who has just released a book on exploratory testing. You can find plenty about this subject in his book, but here's a blog post that revealed a little more about what James thinks:

I just got finished talking (actually the conversation was more like a debate) to a colleague, exploratory testing critic and a charter member of the plan-first-or-don’t-bother-testing-at-all society....explaining exploratory testing

I'm not sure that Bj Rollinson is a big advocate of exploratory testing. I've seen him take some digs at this subject before, but this article clearly shows an internalization of exploratory testing that transcends its most common use:

Much of the information about exploratory testing focuses on testing from an end-user perspective. Pundits of exploratory testing claim the approach is also useful from a white box test design approach, but I have yet to see any practical discussion or examples. But, professional testers use exploratory testing approaches all the time from a white box perspective to explore the code for untested paths. Professional testers learn about areas of the code that are at risk, and reactively design effective tests to evaluate previously untested or under-tested areas of the code.... Exploratory testing inside the box

So what do you think - does exploratory testing warrant all this attention as a separate test type, or, as Beizer says, is all testing exploratory?

Become A Windows 7 Compatibility Testing Expert Now

2009-10-13T08:00:00.000-07:00

If you're waiting to get your hands on a copy of Windows 7 before you can start your compatibility testing, you're going to be behind from the start. That's because you can test now for both system and application compatibility issues. Microsoft provides tools for testing your system and your applications. There are three tools you should become familiar with: Windows 7 Upgrade Advisor, Microsoft Application Compatibility Toolkit 5.5, and the Windows 7 Software Logo Toolkit.

The Windows 7 Upgrade Advisor is important because it will tell you if your current test systems can run Windows 7. Now's the time to find out if the test machines you're currently counting on upgrading to Windows 7 will work. If not, you're going to be further behind in your compatibility testing as you wait to procure new systems.

The Microsoft Application Compatibility Toolkit 5.5 contains tools and documentation to evaluate application compatibility issues before deploying Windows 7. You can find out more information about the toolkit, as well as where to download it, here.

The final tool is part of the Windows 7 logo program. The Windows 7 Software Logo
Toolkit provides you a quick and agile, automated, light weight tool to validate your application’s compliance with the Windows 7 Software Logo requirements.

None of these tools require Windows 7, but each provides you valuable compatibility information. By using these tools now, you'll have an almost complete pictures of your compatibility issues before Windows 7 even ships.

How Hot Is The iPhone Application Development Market?

2009-10-09T08:00:00.000-07:00

We've just returned from the STARWEST 2009 software testing conference. While there we talked to hundreds of testing and QA people. And as we shared with them our outsource testing services and our new product, myQMT, we asked them what their iPhone development plans were. Since we both develop and test iPhone applications, we were interested in how many were developing for the iPhone market. What we found out was that most companies were planning an iPhone app. And this information was coming from their QA folks.

Today I have some links to articles that provide some insight on how hot this market is becoming:

39% average growth was experienced by most developers from adding mobile application development to their business models. NATIONAL SURVEY RESULTS: Developers Shift Business Model To Ride iPhone App Wave Developers and entrepreneurs change course and thrive as iPhone app development increases

University of Utah researchers created new iPhone programs. iPhone the body electric

The exploding iPhone Application development community. iPhoneAppQuotes Forges Strategic Partnership with Appular

American heavy metal band Metallica has launched a new iPhone application to allow fans to listen to its live show recordings on the move. Metallica launches iPhone application for fans

The App Store taps the creative energy of entrepreneurial developers. From cars to TVs, apps are spreading to the real world

Mercedes-Benz joins other consumer-oriented companies, including Pizza Hut, Kraft, and Whole Foods, in developing apps for customers. Mercedes-Benz Launches iPhone App

STARWEST 2009 EXPO - Day 2

2009-10-08T08:00:00.000-07:00

Live from the floor of the STARWEST 2009 Expo. It's Day 2 here in the Disneyland Hotel. The STARWEST folks have certainly done a great job again with this conference and EXPO. Throughout yesterday we've seen testers and QA managers from all over the world attending presentations and visiting the vendor booths. We've been able to demo our new QA Management Toolkit (myQMT) to a number of different companies and comments have been very positive. Plus we've received a number of good ideas on what to add next, such as the configuration management system subversion. It's certainly been exciting and worth the trip.

The readers of this blog know that we provide ample coverage of different testing techniques, and one of our favorite methodologies is model-based testing (MBT). We wrote about a MBT tool in May when we intoduced Testoptimal. While visiting the other booths here at the EXPO we came across another interesting MBT tool called Conformiq Qtronic. I spent some time with the developers talking about their product and it was very enlightening. As with Testoptimal, Qtronic requires that a model be developed before using their tool, but once you provide that as input, the generation of hundres of test cases is automatic and painless.

The two features that really stood out for me was Qtronic's integration with Eclipse and the support for user-defineable output formats for test case output. Eclipse integration gives you a familiar and powerful development environment, and being able to output test cases in any format your test environment supports is huge. Here's a screen shot from the demo they gave me:

If your organization is looking for a tool to support the use of MBT, take a look at this product at the CONFORMIQ website.

STARWEST 2009 EXPO - Day 1

2009-10-07T08:00:00.000-07:00

We're at STARWEST 2009 in Anaheim this week, and by "Anaheim" I mean the Disneyland Hotel. Today is the first day of the EXPO and we are showing our new open source QA management product - myQMT. myQMT runs on the Amazon cloud and offers software developers and QA managers a turn-key solution for defect tracking, test case management, Wiki and forums.

This year's STARWEST looks to be exciting and all the major testing experts are here: Lee Copeland, James Bach, James Whittaker, Michael Bolton, Jonathan Kohl, Harry Robinson, Rex Black, etc.

Our booth at the EXPO is next to Microsoft and across from HP and Oracle. We look forward to lots of visitors. In between demos of our product, I plan to visit some of the other vendor booths and find out what's new and interesting. In particular, I'm interested in talking to the folks at the American Software Testing Qualification Board and the Better Software Magazine. They both have booths at the EXPO this year.

The Vendor Technical Presentations look interesting this year. HP's presentation is "Finding the Blue Sky - Performance Testing in the Cloud". That's an area of interest to RTL, and we've been using the Amazon cloud to do load testing using JMeter. Another presentation that looks worth attending is from Borland - "Testing AJAX Applications - What Does It Take?"

Look for my update in tomorrow's STARWEST 2009 EXPO - Day 2.

Putting Together An Open Source QA Management System (Part 5) - Information Center

2009-10-06T08:00:00.000-07:00

Last week we posted the first 4 parts to this series, which covered Defect Tracking, Test Case Management, Wiki and Forums. In this, the last article, we are going to look at the information center that ties all of the other components together. In order to provide a "Home" page and tie each of the different open source applications together, we needed a malleable content management system that would provide project managers a way to customize the look-and-feel of their QA management system, as well as offer a way to create a dashboard that could pull information from each of the other applications. Joomla proved to be the answer.

With Joomla, we got a content management system with the capability of creating web pages, customized UI, and online applications. And since it's open source, it fit right in with the rest of the applications selected. Even though our initial evaluation convinced us to go with Joomla, the more we used it and discovered more about the underlying architecture, the more it proved to exceed our expectations. We anticipated that Joomla would serve as a flexible web site tool and had expected that links to the other applications would probably spawn new pages, but we found out that Joomla, and it's extensions, allowed us to keep the UI together and maintain a single page view for most of our needs. Moreover, as we shared this QA management system with others, we were continually asked about having a dashboard feature that would pull data from Bugzilla, Testopia, MediaWiki and Forums. As it turns out, the "article" capability of Joomla, along with some PHP and MySQL calls, we could provide some nice dashboard "widgets".

The dashboard "widgets" we created for defect tracking includes a snapshot of Assigned Defects and a Bug Summary from Bugzilla.

For test case management, a widget was created that showed a summary of the current state of test cases development. This information was directly extracted from Testopia.

And to provide a convenient way for testers to monitor forum posts, we created a widget that extracted out the title of the latest articles posted to the forums.

Likewise, we found a way to monitor what articles have been written and placed on the Wiki. This particularly useful given the non-hierarchical nature of the wiki - daily notifications with this widget keeps everyone informed about the work going on in the wiki.

In addition to the dashboard widgets, we found that we could create Joomla articles that provided other information that testers should see. This included an area for daily instructions, Top 5 defects, Top 5 contributors, and a list of the most recent defects.

On top of the ability to create dashboard widgets and articles, Joomla provided a very convenient way to link in all the other open source applications. It allowed us to provide navigation links via a left-hand navigation panel, as well as tabs along the top, complete with a "bread crumbs" mechanism for navigating back to the home page. All-in-all, Joomla proved to be a very powerful application that neatly integrated all the other components. A snapshot of the home page is provided below and at this link.

We will be demoing this system, called My QA Management Toolkit, or myQMT, at our booth at STARWEST 2009 in Anaheim. And we expect to offer myQMT, via Amazon's Web Services, by the end of October.

Putting Together An Open Source QA Management System (Part 4) - Forums

2009-10-02T08:00:00.000-07:00

In the last article we covered the knowledge management piece of our open source QA management system - the Wiki. To complement the Wiki and provide a place for conversation threads and the ability for testers to create their own topics for discussions, we chose to add forums to our QA management system. With forums, managers, test leads and testers can create topical threads to discuss issues, concerns and explore ideas. We chose a forum system (Agora Forums) that would integrate well and had all the things you'd expect in a forums package: public and private posting, moderators, WYSIWYG editor, stats, user participation reports, polls, search, etc.

At RTL we use our forums to create discussions on everything from current projects to important client information to information specific to our different labs. Our forums are organized by major topic, such as Test Engineering or Inside RTL, and testers customize their online persona with their own icons and graphics. Addtionally, we use the forums to link to Wiki articles. Sometimes discussions that ensue are easily answered by pointing others to articles that reside in our Wiki.

Tuesday, we'll post the final article of this series and talk about the "information center" that integrates all the components into a single landing page and offers a dashboard with summary information from each of the components.

Putting Together An Open Source QA Management System (Part 3) - Wiki

2009-10-01T08:00:00.000-07:00

Parts 1 and 2 of this series introduced the core components for our open source QA management system: Defect and Test Case Management. In this article we introduce the "knowledge and content management system". We wanted to include a way to share information that should be persistent, such as tips, techniques and procedures. Likewise, we wanted a repository for training articles that pertain to testing in general and customer-specific technologies. For these needs we selected MediaWiki.

MediaWiki is written in the PHP programming language, and can use either the MySQL or PostgreSQL relational database management system. Since our defect tracking (Bugzilla and Mantis) and test case management (Testopia and TestLink) systems use MySQL, we opted to stick with MySQL for MediaWiki.

We've found that having a separate system for sharing knowledge has been very helpful in keeping technology lists current, such as OS lists. And because each Wiki user can edit any document, collectively we are able to keep these lists current without having to assign that responsibility to any one person. Here's an example of one of the OS lists we maintain on our wiki:

Since wikis are somewhat non-hierarchical, most people search on keywords to find articles, which is pretty much how Wikipedia is used. But there are times when you want to know the latest articles added to your wiki. For that, we created some dashboard widgets that list the latest articles each time you log into the QA management system. I'll describe these widgets in the last article in this series. In Part 4, we'll describe the forums component we included this integrate, open source QA management system.

Putting Together An Open Source QA Management System (Part 2) - Test Case Management

2009-09-30T08:00:00.000-07:00

In Part 1 of this series, we introduced the defect tracking systems we selected as part of our integrated, open source QA management system. In this article, we'll identify the test case management systems we paired with the defect tracking systems: Bugzilla and Mantis.

For Bugzilla, we found a convenient test case management system in Testopia. Testopia is a test case management extension for Bugzilla, and it's designed to integrate bug reporting with test case run results. Projects in Testopia consist of a test plan, test cases, test runs, test run environments, builds and test-case runs. And since Testopia is an extension to Bugzilla, it relies on many of the same objects that Bugzilla uses to track bugs. In order to use Testopia you first need to set up products and versions using the Bugzilla product editor. Testopia makes use of Bugzilla's target milestones feature as well.

INTEGRATION POINTS

Testopia integrates with Bugzilla products, components, versions, and milestones to allow a single management interface for high level objects.
Testopia allows users to login to one tool and uses Bugzilla group permissions to limit access to modifying test objects.
Testopia allows users to attach bugs to test case run results for centralized management of the software engineering process.

We paired Mantis with TestLink for test case management. TestLink provides the capability to create and manage Test cases as well as organize them into Test plans. These Test plans allow team members to execute Test cases and track test results dynamically, generate reports, trace software requirements, prioritize and assign tasks. The tool has web based interface with PHP and background database MySQL, Postgres or MS-SQL.

Both of these test case management systems integrate well with their respective defect tracking system. In the next article we'll cover the Wiki system we integrated into our open source QA management system.

Putting Together An Open Source QA Management System (Part 1) - Defect Tracking

2009-09-29T08:00:00.000-07:00

Often we are asked to recommend QA tools, in particular defect tracking systems. And given that there are now very good open source tools, we recommend that our clients take a look at defect tracking systems such as Bugzilla or Mantis. We've found that is usually takes some time before they are able to install and set up one of these systems, and often they find it difficult to provision dedicated servers within their organization for hosting. Moreover, these companies also end up needing a test case management, as well as some way to collaborate and share articles. Even though they are happy about the costs of open source QA tools, the effort to get these systems installed and running is much bigger than they expected.

So this year we set out to create for our clients an easier path to a fully integrated, open source QA management system. Over the course of the next several articles, I'll describe the systems we adopted and ultimately integrated into a turnkey system that runs on a popular cloud computing platform. In this first article, we'll cover the two defect tracking systems we've included in the two editions of our QA management system. They are Bugzilla and Mantis.

We chose these two open source defect tracking systems because they've been around for awhile and are very popular. They are also strongly supported by their development communities. Finally, they both integrate well with some open source test case management systems (which we'll cover in Part 2 of this series).

The opensourcetesting.org site is a great resource for researching open source testing tools. Here's what they have to say about Bugzilla:

Bugzilla has matured immensely, and now boasts many advanced features. These include: integrated, product-based granular security schema, inter-bug dependencies and dependency graphing, advanced reporting capabilities, a robust, stable RDBMS back-end, extensive configurability, a very well-understood and well-thought-out natural bug resolution protocol, email, XML, console, and HTTP APIs, available integration with automated software configuration management systems, including Perforce and CVS (through the Bugzilla email interface and checkin/checkout scripts), too many more features to list.

Mantis is the other very popular (almost 1 million downloads - the highest download count at opensourcetesting.org) defect tracking system we adopted for our open source QA management system. It too has a wealth of features and, very importantly, integrates with a popular test case management system. Here's a brief description of Mantis:

MantisBT is a free popular web-based bugtracking system (feature list). It is written in the PHP scripting language and works with MySQL, MS SQL, and PostgreSQL databases and a webserver. MantisBT has been installed on Windows, Linux, Mac OS, OS/2, and others. Almost any web browser should be able to function as a client. It is released under the terms of the GNU General Public License (GPL).

We've worked with these two defect tracking system for most of this year and find them to be very stable and useful for tracking defects. And both run very well on top of Ubuntu linux (the OS we chose for implementation on the cloud platform we selected). Part 2 of this series will introduce the test case management systems with which these two defect tracking systems integrate.

Car 2.0

2009-09-28T08:00:00.000-07:00

How radically different do cars need to become in order to have a significant impact on the cost of fuel and emissions? Hybrid's are great, but Shai Agassi is all in on electric cars. His company, Better Place, is already taking us towards all electrics. And best of all, the trade-offs don't involve any guilt or inconvenience, just a recognition that what you used to pump into your car (gasoline), now plops (charged battery). And it gets better - overtime your "eMile" costs goes down.

Here's the video of Shai Agassi's presentation at TED:

Clean System Testing Using Sandboxie

2009-09-25T08:00:00.001-07:00

Sandboxie is one of those unique utilities that varies in description depending on how you intend to use it. To some people, it's a security tool because it isolates your local drive from any modification when running selected applications, particularly ones you don't necessarily trust. To others it's an anonymity tool that allows you to easily erase any files when web surfing. To me it's a great clean system utility and investigation tool for testing.

At RTL, we always prepare our test configurations with clean OS installs. This ensures we are able to reproduce defects because we control the setups and don't have to contend with "dirty" systems - systems where OS and preference files have been changed because of previous testing. So each time we begin a set of tests, we install a clean OS. But there are times when you'd like to restart your testing without having to re-install the OS. That's where Sandboxie can come in handy. Sandboxie creates a private sandbox for your application under test, to include your browser when doing web testing. Here's how it's described at the Sandboxie website:

Think of your PC as a piece of paper. Every program you run writes on the paper. When you run your browser, it writes on the paper about every site you visited. And any malware you come across will usually try to write itself into the paper.

Traditional privacy and anti-malware software try to locate and erase any writings they think you wouldn't want on the paper. Most of the times they get it right. But first the makers of these solutions must teach the solution what to look for on the paper, and also how to erase it safely.

On the other hand, the Sandboxie sandbox works like a transparency layer placed over the paper. Programs write on the transparency layer and to them it looks like the real paper. When you delete the sandbox, it's like removing the transparency layer, the unchanged, real paper is revealed.

I've used Sandboxie a number of times and I'm always amazed at what I find in the sandbox. All files, included those created by an installer, are written to the drive in the sandbox. Here's an example of what a sandbox looks like to give you and idea of what I'm talking about:

The graphic above shows that any file created by any sandboxed program, in this case NotePad, will be placed in a sandbox folder corresponding to the real folder where it would have been placed on your local drive. When I saved my file "test1.txt", it was actually saved to the sandboxed C: drive as shown, but not to the local drive. So if you use Sandboxie during testing, any application you install, while using the sandbox, ends up in the sandbox, along with any pref, config and other files. Then once you're done, you throw away the sandbox and your system remains "clean".

I've also used Sandboxie for investigative work. Referring again to the graphic above, you can see that each and every file that is created and would have been written to your local drive, is identified and listed by location - even changes to the Registry (the All Files and Folders folder typically contains RegHive files which represent the sandboxed copy of the Windows registry). So if you're interested in knowing exactly what's going on when you install and run an application under test, you can find out by looking at the contents of the sandbox.

Sandboxie only runs on Windows 2000, XP, 2003, Vista, 7; 32-bit only. And not all applications can be sandboxed. The Help & FAQ page is has all the information you need to determine if it will work for you.