Thought this was a thread worth following

http://tech.groups.yahoo.com/group/agile-testing/message/18026

As I stated in the thread, it is a tough decision to hire developers to fill Agile Tester spots. Actually more to the point, it is tough to fill Agile Tester spots in general You are really looking for someone with good technical skills but with a true passion for testing. When you look at hiring a developer to fill that spot you are going to be getting those technical skills but there are times you will find that the candidate in question does not have the true passion for testing. If this is the case then it is a possibility that the job that they are asked to do will not be up to the needs of the team.

This will be the last of my posts on this specific topic for now, though I am sure many more posts will be made along the same lines in the not to distant future.  For this one I want to look at safely using InnerHTML.  Many people do not realize how vastly different browser can treat HTML.  Specifically HTML created dynamically as part of a web 2.0 application.  Below you will find some generalized examples of html in 3 different browsers.

–IE 7

<option value=fl selected>Florida</option>

–IE 8

<option selected value=fl>Florida</option>

-FireFox

<option value=“fl” selected>Florida</option>

Notice how the selected attribute moves around and that the value sometimes has quotes and sometimes does not.  These are all things to be expected when dealing with HTML elements. Unfortunately many times I see people write automation against one browsers specific interpretation of the HTML which can cause issues down the line.   So now the question is how do we avoid these issues and create expressions to best deal with them.  Below are three examples of different ways to write expressions that relate to the above HTML.  Each uses a different technique to best find the element.  Obviously these techniques can be combined for even better expressions.

1. Use Parent Element

|parentElement.id=csState;value=fl;selected=true|OPTION|

- Notice how by using parentElement we can identify the parent of the control and use that to create a stronger expression.  You can find more on parentElement here (http://ulti-swat.wikispaces.com/Expressions+Explained).

2. Use Regular Expressions

|innerHTML:Florida[\w\W]+Georgia[\w\W]+New Jersey|SELECT|

- The above expression would be used to find the order of items in a select list.  Instead of using the entire innerHTML of the element as many people do, you can instead just use an expression such as this one to look for the pertinent data.

3. Make good use of attributes

|value=fl;innerHTML=Florida;selected=false|OPTION|

- Attributes are your friend.  Here we are using all of the available attributes to really get the best handle possible on the element.

Hopefully these examples will help you create a strong, more reliable SWAT test.

I did a presentation internally at Ultimate Software on improving the Reliability and Efficiency of tests and I thought it would be worth posting on here as well.  I will also be making a few more posts to cover all the topics I outlined in the presentation.

Improving SWAT\UI Test Reliability and Efficiency

One of my biggest flaws with automation in my eyes is that when you run most UI automation (excluding bitmap comparisons that are more trouble then they are worth) the UI can change but the test will not catch it.  This is actually a very common scenario and usually when people discuss solutions they go back to bitmap comparisons.  However I have heard (but not confirmed) that Google has a better way.

What Google has done is to run their UI automation on very public monitors.  Then when people are walking by, hopefully if they see a defect they will go let the responsible team know and it will be rectified.  I think this is an absolutely great idea and one I hope to implement some time in the near future.

At a presentation I did earlier this week one of my coworkers made the comment

Automation is for validation, testing requires hands, eyes, and a brain

I have heard similar comments before but I thought that this was worth repeating.  When you create and run automation, you are validating against what you already know.  The script (whether its above or below the UI) was based on some known state and values.  For the most part the automation can only tell you when something expected did NOT happen.  The majority of scripts will not tell you when something unexpected did happen.

For example if I have a login page that takes a user name and password.   When I fill in the user name and password, and press a button I log in.  Now if someone were to add an additional step to my login process, any automation I have would obviously catch that.  However if someone added something that did NOT affect login to the login page then most likely my scripts would NOT catch that.  These things could include changes to the UI, popups, new text, or possibly something more malignant.

This is where the basic difference between automation and testing come from.  When you test you not only are working with base data and expected results, but you have the ability to comprehend the unexpected. What extensive automation really does is allow testers more time to look into the unexplored areas of the system.  More time can be spent on exploratory testing since your regression suite is handling all of the expected tests and scenarios.  By spending more time doing this you are liable to find all of those edge case scenarios that your customers are guaranteed to find.

So in sum it all up, automation is a hugely important part of the development process, but to help identify that what is working hasn’t broke not to “test”. When it comes to testing, it takes someone (or maybe one day something) that can identify the unexpected and can appropriately identify what to do about it.

Were now up to part 3, and I am really thinking about ditching the numbers, I think this list is going to get very long very…

But on to part 3 and this will be a short one.  AssertElementDoesNotExist was one of the first commands created for SWAT.  Unfortunately many people do not know some of the dirty and dark secrets of this command.

You dont always know what you are looking for…

The most common mistake I see with this command is that people match it with a very complicated expression where many factors could make the command pass (which means the expression was NOT matched.)  In essence you get a false positive.

Example:

HTML

<a href="www.google.com" id="myLink">Google</a>

SWAT

|AssertElementDoesNotExist|Expression|id:myLink;href:google@|A|

Notice that I made a mistake with the href.  However because of that, this command would actually pass.  Which is one of the dangers of this command.

Its Slowwwwwwwwwwwwwwwwww

As you may or may not know, in your config file for SWAT you can set the default timeouts of many of the commands.  In almost all cases, as soon as SWAT finds an element it moves on to the next command.  However in the case of AssertElementDoesNotExist, SWAT will keep on searching through the length of the timeout.  Because of this, using a large amount of AssertElementDoesNotExist commands can drastically alter the time it takes to run your test.

Example

7 second timeout

x 10 uses of the command

——————————

70 second minimum test run

So those are the dark secrets of AssertElementDoesNotExist.  What I try to tell people when it comes to this command, as with all commands, it should be thought about before using it.  I have seen many tests with 10 or 20 of these commands used in a row when it may have been possible to make do with less.  And in some cases, half of those commands were false to begin with.

This post is written more for a SWAT user since it has to do specifically with how SWAT users check the values of SELECT boxes.  However the ideas here could be applied to other UI testing tools as well.

For this post we will be using the example Select Box below:

<SELECT id="mySelectBox">
  <OPTION value="A">Alpha></OPTION>
  <OPTION value="b">Beta</OPTION>
</SELECT>

The Problem

What we have here is a select box with 2 values and B selected.  Typically we have seen the test for checking the selected value to be done like this:

|AssertElementExists|Expression|id=mySelectBox;innerHTML:OPTION value=”B” selected|SELECT|

Unfortunately each browser handles select boxes a little bit differently.  For instance IE7 and IE8 reverse the order of the value and selected

IE7 Example:

OPTION value=”B” selected

IE8 Example:

OPTION selected value=”B”

Firefox also treats this differently as well.  Many times it removes\adds quotation marks in different places.

The Solution

There are 2 different ways to solve this.  The first way is slightly simpler but we have found that in some cases it can lend itself to mistakes.  The 2nd is the much stronger way, and also allows for multi select boxes.

Solution 1

|AssertElementExists|Expression|id=mySelectBox;value=B|SELECT|

Solution 2

|AssertElementExists|Expression|parentElement.id=mySelectBox;value=B;selected=true|OPTION|

Notice that in solution 2 we are no longer referencing the Select box.  Instead we are referencing the Option itself.  What we are in essence saying is that we want SWAT to find an Option whose parent is the select box (located by parentElement.id), whose value is B and is selected.  By doing it this way you are drilling down to the specific option you want and avoiding browser compatibility issues.  You can also use this method for just checking options inside a select box.  The example below checks that both values are in the select box:

|AssertElementExists|Expression|parentElement.id=mySelectBox;value=B;innerHTML=Beta;selected=true|OPTION|

|AssertElementExists|Expression|parentElement.id=mySelectBox;value=A;innerHTML=Alpha;selected=false|OPTION|

Hopefully this helps clear up some of the confusion with Select boxes.

This is the start of a series of posts I am going to be writing on how to improve both the reliability and efficiency of SWAT tests.  Any ideas I present here can be applied to any UI testing tool (SWAT, Selenium, WatiN, QTP, etc…).  The most important concept to take away from these posts will be that tests can be made both reliable and efficient with just a few simple techniques.

For the first post we will be looking at the Sleep command (http://ulti-swat.wikispaces.com/SwatCommandsSleep).  Almost all UI testing frameworks contain some version of this command.  This command is also sometimes known as a “Wait” Command.

Lets start out with why this command is bad:

• The command will slow down a test suite
• No matter what, the time you set for the sleep is always too slow or to long.  Either it will be too long, and your test will just sit there waiting for nothing, or it will be too short and the test will move on before it should.

Assuming you are still reading this, and agree that sleeps are bad, you are probably asking what you can do to avoid sleep commands and make your test both stronger and more reliable.

In my experience I have found there are 3 typical reasons why a sleep command is most commonly used (there are others but I feel at least for the moment that these are edge case scenarios):

• An asynchronous event is happening on the page that takes longer then SWAT’s default timeout
• An event on the page is waiting for a SQL event (such as a save to finish)
• A browser takes longer to load (and set its title bar) then SWATS default timeout

With SWAT, 3 commands are provided to deal with these situations

• AssertElementExistsWithTimeout
• AssertDBRecordExistsWithTimeout
• AssertBrowserExists

Here is some more on these commands

AssertElementExistsWithTimeout is meant to allow users to avoid sleeps when they are doing a check on the page.  Many times either a page is a slow loading page, or an asynchronous action is happening on the page, and testers will drop a sleep on the page to attempt to determine when the action has completed.  The standard AssertElementExists command has an adjustable timeout (configured in the config file) but is usually set for just 5-10 seconds.  Because of this people often drop a sleep command to wait for the page.  Instead a tester can use the AssertElementExistsWithTimeout command and set a timeout in milliseconds to tell SWAT to keep polling and checking if this element exists yet.  No matter how long you set the timeout for, as soon as the element is found, SWAT move on.  In my own experience I usually set the timeout from 60000 (1 minute) to 360000 (6 minutes) with the comfort and knowledge that as soon as the criteria is met, the test will move on, but not before it is ready.

AssertDBRecordExistsWithTimeout is meant for any time the page needs to wait for a DB event to finish.  The timeout is again in milliseconds.  I typically use this any time I am waiting for a SQL transaction such as a save to finish.

AssertBrowserExists is meant for times when a popup takes so long to load, that the default timeout for AttachToWindow times out.  This will allow SWAT to keep on waiting for a longer time.

In the next post I will examine some other best practices for increasing test efficiency and reliability.

UPDATE: Got some great feedback already. Sleeps do have their uses. However the important part to take away from this post is that they are easily over used or used incorrectly.

For part 2 we are going to look at a few different container types and tables.

First some container types you may experience:

• DIV
• SPAN
• P

Each of these are common containers you may come across.  They each have some common similar attributes:

• InnerHTML – This is the area between the open and close tag.
  • Example: <DIV>This is the innerHTML </div>
• InnerHTML is ALL of the HTML between the tags, including pieces of other tags
  • Example:<DIV><SPAN>This is some text</SPAN></DIV>
    • The innerHTML of the div is “<SPAN>This is some text</SPAN>”
    • The innerHTML of the span is “This is some text”
• Id
• Name
• Class

What is important about any container is that it usually contains additional tags.  Below is an example you may come across:

Containers are prevalent through any application but with CSS (style sheets) make them even more important.

Another type of container is a Header container.  There are a couple you may come across but the most common are:

• H1
• H2
• H3

These are commonly used to create common types of headers on a page.  When it comes to attributes they are treated the same as the containers we looked at above.

The next type of HTML we will look at are tables.

This has been bothering me for a while and wanted to bring it up.  When writing tests I have already mentioned the concept of 200 test steps.  By limiting yourself to only 200 steps, you will find tests easier to write, maintain, and run.  However that really does not take into account test execution time.  I am sure we could easily find an example of a test that is only 10 steps in length but that takes 20 minutes.  I believe optimum length is less than 10 minutes, but preferably 5.

I really do not have a reason for that time, other than gut feeling but the math seems to work in my favor.  If a test takes 5 minutes to run, and 5 minutes to fix a bug.  It means at minimum you are going to need 15 minutes to fix any bug (5 to verify it, 5 to fix, and 5 to verify the fix.)  Once we start getting over that 15 minute threshold I believe we are really setting ourselves up for failure.

UPDATE:   So just spent over an hour working on a test, starting to think that 5 minutes may even be too long…

Does anyone else have tips or tricks, or experience with this?