SWAT 3.0 Released

We are very excited to announce the release of SWAT 3.0.  The biggest announcements are the new support of Chrome on the Windows OS and Safari on OSX.  In addition we now support calling SWAT from the command line independent of any test runner tool.   Finally we have made a roughly 30% improvement on the speed of SWAT running against the IE browser.  As usual we have also fixed a number of bugs and smaller issues that are detailed in the complete release notes.

New Features and Updates

1. New Browsers Supported
   1. Chrome
      1. New custom built plugin for Chrome allows SWAT to easily control the Chrome Browser
      2. http://ulti-swat.wikispaces.com/Setting+up+Chrome
   2. Safari
      1. Custom app for OSX now allows SWAT to stream commands to an OSX based system.
         1. http://ulti-swat.wikispaces.com/Setting+Up+Mac
         2. http://ulti-swat.wikispaces.com/Setting+up+Safari
2. SWAT Console
   1. Tests can now be run independent of any test runner.
   2. As of now this feature is for SWAT only and does not support any custom fitnesse fixtures.
   3. http://ulti-swat.wikispaces.com/SWAT+Console+User+Information
3. New Commands
   1. UI
      1. AssertJSDialog
         1. Added an optional timeout to allow users to select how long the command will search for a dialog box
      2. AssertTopWindow
         1. Allows users to verify if a window is the top most window on the screen
         2. http://ulti-swat.wikispaces.com/AssertTopWindow
   2. Database
      1. RestoreAllTables
         1. Users can now restore all of the tables they have saved at once instead of one at a time
4. Defects Resolved
   1. CORE – AttachToBrowser always finds the about:Blank window
   2. CORE – DB Connect now automatically closes the old connection when it creates new ones
   3. CORE – FF – AssertElementExists will find an element when the identifier is empty
   4. CORE – FF – PressKeys does not ignore CancelEvent in Javascript
   5. CORE – FF – Special characters above the standard 128 return correctly when
   6. CORE – FF – The AttachToWindow failure due to “OK” being added to title has been fixed
   7. CORE – IE – attachToModalDialog is now setting the currentIEBrowser
   8. CORE – IE – Browser screenshots work correctly in IE9
   9. CORE – IE – KillAllOpenBrowsers is now respecting AbandonTest
   10. CORE – IE – OnBeforeUnload JSDialogs in IE9 are compatible with SWAT
   11. CORE – IE – PDF pages open automatically in IE9
   12. CORE – Screenshots don’t throw a “Not attached to window” exception.
   13. CORE – SQL checks are passing in the editor and not in Fitnesse
   14. CORE – SWAT waits for all frames to load when waiting for the browser
   15. EDITOR – Editor is respecting the -c attribute for included files
   16. EDITOR – Values of 0 can now be saved in the Delay between commands option

SWAT’s International Release

Really enjoyed doing some data mining recently on SWAT’s international reach. Below is some of the download data from 2/1/2010 until 8/13/2010.

1.	United States	615
2.	India	326
3.	United Kingdom	86
4.	Unknown	84
5.	Germany	55
6.	China	50
7.	Korea	49
8.	Canada	46
9.	France	33
10.	Brazil	29
11.	Australia	23
12.	Viet Nam	23
13.	Indonesia	21
14.	Israel	21
15.	Romania	21
16.	Italy	20
17.	Philippines	18
18.	Turkey	16
19.	Netherlands	16
20.	Russia	15

South Florida Code Camp 2010 Presentations are up

I have posted the presentations Chris Taylor and I did at the South Florida Code Camp.  You can find them here

http://ulti-swat.wikispaces.com/SwatIntroDemos

Specifically

An Introduction to UI testing using SWAT

Applying modern software development techniques to automating the web UI

Please let me know any feedback or questions you might have.

Internal Presentation on Reliabilty and Efficiency of UI tests

I did a presentation internally at Ultimate Software on improving the Reliability and Efficiency of tests and I thought it would be worth posting on here as well.  I will also be making a few more posts to cover all the topics I outlined in the presentation.

Improving SWAT\UI Test Reliability and Efficiency

A shortfall of UI automation and a creative way to deal with it

One of my biggest flaws with automation in my eyes is that when you run most UI automation (excluding bitmap comparisons that are more trouble then they are worth) the UI can change but the test will not catch it.  This is actually a very common scenario and usually when people discuss solutions they go back to bitmap comparisons.  However I have heard (but not confirmed) that Google has a better way.

What Google has done is to run their UI automation on very public monitors.  Then when people are walking by, hopefully if they see a defect they will go let the responsible team know and it will be rectified.  I think this is an absolutely great idea and one I hope to implement some time in the near future.

Automation is validation

At a presentation I did earlier this week one of my coworkers made the comment

Automation is for validation, testing requires hands, eyes, and a brain

I have heard similar comments before but I thought that this was worth repeating.  When you create and run automation, you are validating against what you already know.  The script (whether its above or below the UI) was based on some known state and values.  For the most part the automation can only tell you when something expected did NOT happen.  The majority of scripts will not tell you when something unexpected did happen.

For example if I have a login page that takes a user name and password.   When I fill in the user name and password, and press a button I log in.  Now if someone were to add an additional step to my login process, any automation I have would obviously catch that.  However if someone added something that did NOT affect login to the login page then most likely my scripts would NOT catch that.  These things could include changes to the UI, popups, new text, or possibly something more malignant.

This is where the basic difference between automation and testing come from.  When you test you not only are working with base data and expected results, but you have the ability to comprehend the unexpected. What extensive automation really does is allow testers more time to look into the unexplored areas of the system.  More time can be spent on exploratory testing since your regression suite is handling all of the expected tests and scenarios.  By spending more time doing this you are liable to find all of those edge case scenarios that your customers are guaranteed to find.

So in sum it all up, automation is a hugely important part of the development process, but to help identify that what is working hasn’t broke not to “test”. When it comes to testing, it takes someone (or maybe one day something) that can identify the unexpected and can appropriately identify what to do about it.

Ways to improve the reliability and efficiency of SWAT/UI tests (Part 1 – Sleeps are evil)

This is the start of a series of posts I am going to be writing on how to improve both the reliability and efficiency of SWAT tests.  Any ideas I present here can be applied to any UI testing tool (SWAT, Selenium, WatiN, QTP, etc…).  The most important concept to take away from these posts will be that tests can be made both reliable and efficient with just a few simple techniques.

For the first post we will be looking at the Sleep command (http://ulti-swat.wikispaces.com/SwatCommandsSleep).  Almost all UI testing frameworks contain some version of this command.  This command is also sometimes known as a “Wait” Command.

Lets start out with why this command is bad:

• The command will slow down a test suite
• No matter what, the time you set for the sleep is always too slow or to long.  Either it will be too long, and your test will just sit there waiting for nothing, or it will be too short and the test will move on before it should.

Assuming you are still reading this, and agree that sleeps are bad, you are probably asking what you can do to avoid sleep commands and make your test both stronger and more reliable.

HTML Part 2 (Containers)

For part 2 we are going to look at a few different container types and tables.

First some container types you may experience:

• DIV
• SPAN
• P

Each of these are common containers you may come across.  They each have some common similar attributes:

• InnerHTML – This is the area between the open and close tag.
  • Example: <DIV>This is the innerHTML </div>
• InnerHTML is ALL of the HTML between the tags, including pieces of other tags
  • Example:<DIV><SPAN>This is some text</SPAN></DIV>
    • The innerHTML of the div is “<SPAN>This is some text</SPAN>”
    • The innerHTML of the span is “This is some text”
• Id
• Name
• Class

Starting from the beginning, a re-introduction to HTML

To start teaching web UI automation techniques using any DOM based (SWAT, Selenium, WATIN) automation tools I think we all need to take a step back and start by looking at HTML.  A great reference for HTML is the W3Schools tutorial here.

Wikipedia definition of HTML (http://en.wikipedia.org/wiki/Html)

HTML, which stands for Hyper Text Markup Language, is the predominant markup language for web pages. It provides a means to create structured documents by denoting structural semantics for text such as headings, paragraphs, lists etc as well as for links, quotes, and other items. It allows images and objects to be embedded and can be used to create interactive forms. It is written in the form ofHTML elements consisting of “tags” surrounded by angle brackets within the web page content. It can include or can load; scripts in languages such as JavaScript which affect the behavior of HTML processors like Web browsers; and Cascading Style Sheets (CSS) to define the appearance and layout of text and other material. The W3C, maintainer of both HTML and CSS standards, encourages the use of CSS over explicit presentational markup.^[1]

Hyper Text Markup Language(HTML)is the encoding scheme used to create and format a web document.A user need not be a expert programmer to make use of HTML for creating hypertext documents that can be put on the internet.

Put more clearly, HTML is the language of the Web. Every web page is created using HTML. Below you will find an example of some very simple HTML

UI testing isn’t brittle…its the test that is

This morning I had the chance to chat with Andrew Chen from Auto GUI Test Service.  The conversation we were having was about the different types of automation tools that are on the market today.  And it was during this conversation that I began to form a thought in my head.  It’s not that UI testing is brittle, its that the tests themselves are.

What I mean is that traditionally people have done record and playback type automated testing.  Every test is independent of another, thus when one feature changes all of the tests break.  I believe much has been to propogate this idea.  But the reality is that if your test is componetized then this test is no more brittle then a unit test.

An example

Traditional way – I have a 3 test cases where as part of them I have to navigate from Point A to Point B.  6 months after I have created the 3 unique tests we add a Point C between A and B.  At this point all 3 tests have to be updated and in many cases completely rescripted.

Component Way – All three tests share a common function that navigates from A -> B.  When 6 months later Point C is added, it is only this function that needs to be updated.

Now I believe many people will say that they already do this, but the tests are still brittle.  But then I ask the question, if you make a change to the system, and the test breaks, how is that any different then a xUnit or Fitnesse test breaking for the same reason?  I think the answer is that it’s not, it is just the conception that is.