Ways to improve the reliability and efficiency of SWAT/UI tests (Part 4 – Safely Using InnerHTML)

This will be the last of my posts on this specific topic for now, though I am sure many more posts will be made along the same lines in the not to distant future.  For this one I want to look at safely using InnerHTML.  Many people do not realize how vastly different browser can treat HTML.  Specifically HTML created dynamically as part of a web 2.0 application.  Below you will find some generalized examples of html in 3 different browsers.

–IE 7

<option value=fl selected>Florida</option>

A shortfall of UI automation and a creative way to deal with it

One of my biggest flaws with automation in my eyes is that when you run most UI automation (excluding bitmap comparisons that are more trouble then they are worth) the UI can change but the test will not catch it.  This is actually a very common scenario and usually when people discuss solutions they go back to bitmap comparisons.  However I have heard (but not confirmed) that Google has a better way.

What Google has done is to run their UI automation on very public monitors.  Then when people are walking by, hopefully if they see a defect they will go let the responsible team know and it will be rectified.  I think this is an absolutely great idea and one I hope to implement some time in the near future.

Automation is validation

At a presentation I did earlier this week one of my coworkers made the comment

Automation is for validation, testing requires hands, eyes, and a brain

I have heard similar comments before but I thought that this was worth repeating.  When you create and run automation, you are validating against what you already know.  The script (whether its above or below the UI) was based on some known state and values.  For the most part the automation can only tell you when something expected did NOT happen.  The majority of scripts will not tell you when something unexpected did happen.

For example if I have a login page that takes a user name and password.   When I fill in the user name and password, and press a button I log in.  Now if someone were to add an additional step to my login process, any automation I have would obviously catch that.  However if someone added something that did NOT affect login to the login page then most likely my scripts would NOT catch that.  These things could include changes to the UI, popups, new text, or possibly something more malignant.

This is where the basic difference between automation and testing come from.  When you test you not only are working with base data and expected results, but you have the ability to comprehend the unexpected. What extensive automation really does is allow testers more time to look into the unexplored areas of the system.  More time can be spent on exploratory testing since your regression suite is handling all of the expected tests and scenarios.  By spending more time doing this you are liable to find all of those edge case scenarios that your customers are guaranteed to find.

So in sum it all up, automation is a hugely important part of the development process, but to help identify that what is working hasn’t broke not to “test”. When it comes to testing, it takes someone (or maybe one day something) that can identify the unexpected and can appropriately identify what to do about it.

Ways to improve the reliability and efficiency of SWAT/UI tests (Part 3 – The truth behing AssertElementDoesNotExist)

Were now up to part 3, and I am really thinking about ditching the numbers, I think this list is going to get very long very…

But on to part 3 and this will be a short one.  AssertElementDoesNotExist was one of the first commands created for SWAT.  Unfortunately many people do not know some of the dirty and dark secrets of this command.

Ways to improve the reliability and efficiency of SWAT/UI tests (Part 2 – Reliably checking select boxes)

This post is written more for a SWAT user since it has to do specifically with how SWAT users check the values of SELECT boxes.  However the ideas here could be applied to other UI testing tools as well.

For this post we will be using the example Select Box below:

<SELECT id="mySelectBox">
  <OPTION value="A">Alpha></OPTION>
  <OPTION value="b">Beta</OPTION>
</SELECT>

Ways to improve the reliability and efficiency of SWAT/UI tests (Part 1 – Sleeps are evil)

This is the start of a series of posts I am going to be writing on how to improve both the reliability and efficiency of SWAT tests.  Any ideas I present here can be applied to any UI testing tool (SWAT, Selenium, WatiN, QTP, etc…).  The most important concept to take away from these posts will be that tests can be made both reliable and efficient with just a few simple techniques.

For the first post we will be looking at the Sleep command (http://ulti-swat.wikispaces.com/SwatCommandsSleep).  Almost all UI testing frameworks contain some version of this command.  This command is also sometimes known as a “Wait” Command.

Lets start out with why this command is bad:

• The command will slow down a test suite
• No matter what, the time you set for the sleep is always too slow or to long.  Either it will be too long, and your test will just sit there waiting for nothing, or it will be too short and the test will move on before it should.

Assuming you are still reading this, and agree that sleeps are bad, you are probably asking what you can do to avoid sleep commands and make your test both stronger and more reliable.

HTML Part 2 (Containers)

For part 2 we are going to look at a few different container types and tables.

First some container types you may experience:

• DIV
• SPAN
• P

Each of these are common containers you may come across.  They each have some common similar attributes:

• InnerHTML – This is the area between the open and close tag.
  • Example: <DIV>This is the innerHTML </div>
• InnerHTML is ALL of the HTML between the tags, including pieces of other tags
  • Example:<DIV><SPAN>This is some text</SPAN></DIV>
    • The innerHTML of the div is “<SPAN>This is some text</SPAN>”
    • The innerHTML of the span is “This is some text”
• Id
• Name
• Class

Starting from the beginning, an introduction to HTML

To start teaching web UI automation techniques using any DOM based (SWAT, Selenium, WATIN) automation tools I think we all need to take a step back and start by looking at HTML.  A great reference for HTML is the W3Schools tutorial here.

Wikipedia definition of HTML (http://en.wikipedia.org/wiki/Html)

HTML, which stands for Hyper Text Markup Language, is the predominant markup language for web pages. It provides a means to create structured documents by denoting structural semantics for text such as headings, paragraphs, lists etc as well as for links, quotes, and other items. It allows images and objects to be embedded and can be used to create interactive forms. It is written in the form ofHTML elements consisting of “tags” surrounded by angle brackets within the web page content. It can include or can load; scripts in languages such as JavaScript which affect the behavior of HTML processors like Web browsers; and Cascading Style Sheets (CSS) to define the appearance and layout of text and other material. The W3C, maintainer of both HTML and CSS standards, encourages the use of CSS over explicit presentational markup.^[1]

Hyper Text Markup Language(HTML)is the encoding scheme used to create and format a web document.A user need not be a expert programmer to make use of HTML for creating hypertext documents that can be put on the internet.

Put more clearly, HTML is the language of the Web. Every web page is created using HTML. Below you will find an example of some very simple HTML

QTP vs open source tools (and what may be the greatest comment ever)

Came across this article thanks to Jason Huggins (http://twitter.com/jhuggins) and got a great kick out of it.

http://www.qtp10.com/2009/08/qtp-vs-selenium.html

UPDATE: The original post this came from is here:http://functionaltestautomation.blogspot.com/2008/10/sno-features-qtp-selenium-explanation_22.html

My favorite line is now and will forever be:

Tests developed using selenium can be easily part of the development project. Using tools like cruise control Continuous Integration is easier with Selenium. But don’t get too caught up with this feature, it’s really not that important to be integrated with the development process but it’s nice to have.

Whats the point of doing automation if its not part of the development process?  What are we gaining from all the work we put in if is just and after the fact artifact?

Agile 2009 – Applying modern software development techniques to automating the web UI

Chris Taylor (http://agiletester.wordpress.com/) and I have been working on this presentation for the past year.  The synopsis is that we need to take the lessons learned from development and apply them to automated testing.

Two quotes have really influenced us in this:

• “Automated testing done right is Software Development” –Elfriede Dustin and Marcus Borch @ GTAC 2008
• “Recorders are the training wheels of the automation world” –Loosely translated from Jason Huggins @ AA-FTT 2009

The presentation went off really well.  The best part were the questions at the end and the realization that we need to start providing tools for new people to really learn automation techniques.  Over the next month I am going to start creating smaller presentations on how to create automation from the ground up and how to avoid different pitfalls.

You can find the slides for the presentation here : http://ulti-swat.wikispaces.com/file/view/Agile+2009+-+Applying+modern+software+development+techniques+to+automating+the+web+UI.pptx

Below are also some of the pictures from the event (thanks John for taking these for us)