devXero's blog

a blog about agile, development, and automation

Posts Tagged ‘Selenium’

Ways to improve the reliability and efficiency of SWAT/UI tests (Part 2 – Reliably checking select boxes)

Posted by Mike Longin on October 6, 2009

This post is written more for a SWAT user since it has to do specifically with how SWAT users check the values of SELECT boxes.  However the ideas here could be applied to other UI testing tools as well.

For this post we will be using the example Select Box below:

<SELECT id="mySelectBox">
  <OPTION value="A">Alpha></OPTION>
  <OPTION value="b">Beta</OPTION>
</SELECT>

Read the rest of this entry »

Advertisements

Posted in Uncategorized | Tagged: , , , , , | Leave a Comment »

Ways to improve the reliability and efficiency of SWAT/UI tests (Part 1 – Sleeps are evil)

Posted by Mike Longin on October 4, 2009

This is the start of a series of posts I am going to be writing on how to improve both the reliability and efficiency of SWAT tests.  Any ideas I present here can be applied to any UI testing tool (SWAT, Selenium, WatiN, QTP, etc…).  The most important concept to take away from these posts will be that tests can be made both reliable and efficient with just a few simple techniques.

For the first post we will be looking at the Sleep command (http://ulti-swat.wikispaces.com/SwatCommandsSleep).  Almost all UI testing frameworks contain some version of this command.  This command is also sometimes known as a “Wait” Command.

Lets start out with why this command is bad:

  • The command will slow down a test suite
  • No matter what, the time you set for the sleep is always too slow or to long.  Either it will be too long, and your test will just sit there waiting for nothing, or it will be too short and the test will move on before it should.

Assuming you are still reading this, and agree that sleeps are bad, you are probably asking what you can do to avoid sleep commands and make your test both stronger and more reliable.

Read the rest of this entry »

Posted in Uncategorized | Tagged: , , , , , , , , | Leave a Comment »

HTML Part 2 (Containers)

Posted by Mike Longin on October 4, 2009

For part 2 we are going to look at a few different container types and tables.

First some container types you may experience:

  • DIV
  • SPAN
  • P

Each of these are common containers you may come across.  They each have some common similar attributes:

  • InnerHTML – This is the area between the open and close tag.
    • Example: <DIV>This is the innerHTML </div>
  • InnerHTML is ALL of the HTML between the tags, including pieces of other tags
    • Example:<DIV><SPAN>This is some text</SPAN></DIV>
      • The innerHTML of the div is “<SPAN>This is some text</SPAN>”
      • The innerHTML of the span is “This is some text”
  • Id
  • Name
  • Class

Read the rest of this entry »

Posted in Uncategorized | Tagged: , , , , , , | Leave a Comment »

Test steps vs Test execution time

Posted by Mike Longin on September 30, 2009

This has been bothering me for a while and wanted to bring it up.  When writing tests I have already mentioned the concept of 200 test steps.  By limiting yourself to only 200 steps, you will find tests easier to write, maintain, and run.  However that really does not take into account test execution time.  I am sure we could easily find an example of a test that is only 10 steps in length but that takes 20 minutes.  I believe optimum length is less than 10 minutes, but preferably 5.

I really do not have a reason for that time, other than gut feeling but the math seems to work in my favor.  If a test takes 5 minutes to run, and 5 minutes to fix a bug.  It means at minimum you are going to need 15 minutes to fix any bug (5 to verify it, 5 to fix, and 5 to verify the fix.)  Once we start getting over that 15 minute threshold I believe we are really setting ourselves up for failure.

UPDATE:   So just spent over an hour working on a test, starting to think that 5 minutes may even be too long…

Does anyone else have tips or tricks, or experience with this?

Posted in Uncategorized | Tagged: , , , | Leave a Comment »

Starting from the beginning, a re-introduction to HTML

Posted by Mike Longin on September 19, 2009

To start teaching web UI automation techniques using any DOM based (SWAT, Selenium, WATIN) automation tools I think we all need to take a step back and start by looking at HTML.  A great reference for HTML is the W3Schools tutorial here.

Wikipedia definition of HTML (http://en.wikipedia.org/wiki/Html)

HTML, which stands for Hyper Text Markup Language, is the predominant markup language for web pages. It provides a means to create structured documents by denoting structural semantics for text such as headings, paragraphs, lists etc as well as for links, quotes, and other items. It allows images and objects to be embedded and can be used to create interactive forms. It is written in the form ofHTML elements consisting of “tags” surrounded by angle brackets within the web page content. It can include or can load; scripts in languages such as JavaScript which affect the behavior of HTML processors like Web browsers; and Cascading Style Sheets (CSS) to define the appearance and layout of text and other material. The W3C, maintainer of both HTML and CSS standards, encourages the use of CSS over explicit presentational markup.[1]

Hyper Text Markup Language(HTML)is the encoding scheme used to create and format a web document.A user need not be a expert programmer to make use of HTML for creating hypertext documents that can be put on the internet.

Put more clearly, HTML is the language of the Web. Every web page is created using HTML. Below you will find an example of some very simple HTML

Read the rest of this entry »

Posted in Uncategorized | Tagged: , , , , , , | Leave a Comment »

QTP vs open source tools (and what may be the greatest comment ever)

Posted by Mike Longin on September 17, 2009

Came across this article thanks to Jason Huggins (http://twitter.com/jhuggins)
and got a great kick out of it. http://www.qtp10.com/2009/08/qtp-vs-selenium.html
UPDATE: The original post this came from is here:http://functionaltestautomation.blogspot.com/2008/10/sno-features-qtp-selenium-explanation_22.html
My favorite line is now and will forever be:

Tests developed using selenium can be easily part
of the development project. Using tools like cruise control
Continuous Integration is easier with Selenium. But don’t get too
caught up with this feature, it’s really not that important to be
integrated with the development process but it’s nice to
have.

Whats the point of doing automation if its
not part of the development process? What are we gaining from
all the work we put in if is just an after the fact
artifact?

Posted in Uncategorized | Tagged: , , , , | 1 Comment »

Agile 2009 – Selenium Open Space

Posted by Mike Longin on August 30, 2009

The last day I was in Chicago I attended the Selenium Open Space sponsored by Adam Goucher (http://adam.goucher.ca/).  There was no real pupose to the meet up, but I wanted to spend some more time talking to Adam and Jason Huggins (creator of Selenium) and to continue the debate of the future of the Selenium IDE.

A couple of my favorite points from the conversation were:
Read the rest of this entry »

Posted in Uncategorized | Tagged: , , , | Leave a Comment »

Agile 2009 – Applying modern software development techniques to automating the web UI

Posted by Mike Longin on August 28, 2009

Chris Taylor (http://agiletester.wordpress.com/) and I have been working on this presentation for the past year.  The synopsis is that we need to take the lessons learned from development and apply them to automated testing.

Two quotes have really influenced us in this:

  • “Automated testing done right is Software Development” –Elfriede Dustin and Marcus Borch @ GTAC 2008
  • “Recorders are the training wheels of the automation world” –Loosely translated from Jason Huggins @ AA-FTT 2009

The presentation went off really well.  The best part were the questions at the end and the realization that we need to start providing tools for new people to really learn automation techniques.  Over the next month I am going to start creating smaller presentations on how to create automation from the ground up and how to avoid different pitfalls.

You can find the slides for the presentation here : http://ulti-swat.wikispaces.com/file/view/Agile+2009+-+Applying+modern+software+development+techniques+to+automating+the+web+UI.pptx

Below are also some of the pictures from the event (thanks John for taking these for us)

Posted in Uncategorized | Tagged: , , , , , , , | 1 Comment »

The test step sweet spot

Posted by Mike Longin on August 27, 2009

As part of Chris McMahons (http://chrismcmahonsblog.blogspot.com/) presentation “History of a Large Test Automation Project using Selenium” he mentioned the concept of a test step sweet spot.  To him the general rule should be about 200 test steps.  For those that use Fitnesse that would be 200 assertions.  I have been thinking along these steps for a while and his comments finally gave rise to my inner dialog.  Once a test starts getting to long it does 2 things.

  1. It starts testing much more then an intial test case and creates many more failure points
  2. It becomes unmanageable to support.  If a failure happens later in the test run, it will be next to impossible to debug and deal and solve the issue

Every person writing tests should be looking for that sweet spot.  Looking at my test suite, I can see that for 200 Fitnesse\SWAT tests that I have the average assertion count is about 210.  However my most solid tests are all closer to around 150.  It is interesting to me that my numbers are so close to Chris’s.   From now on I am going to start paying more attention to a test and force myself to reexamine the test once I am above 150 and try to keep myself below 200.

Posted in Uncategorized | Tagged: , , , , , | Leave a Comment »

AA-FTT meetup in Chicago (an overview)

Posted by Mike Longin on August 24, 2009

I really wanted to post this yesterday but a little social interaction got in the way.

I spent the day at the AA-FTT meetup (http://www.eventbee.com/view/qualitytree/event?eventid=59531) in Chicago yesterday and had a really great time.  I had the opportunity to demonstrate two tools that we have developed interally at Ultimate Software.  The first is a web driver known as SWAT that I have messaged before.  The 2nd is a windows based UI wrapper for running Fitnesse tests.  I hope to be able to post some more on that in the future.

The day was an open space presentation style that was high energy the whole day.  I started by attending a lightning talk showing off a number of tools people are actively using.  Cucumber (http://cukes.info/), Twist (http://studios.thoughtworks.com/agile-test-automation), and the Robot Framework (http://code.google.com/p/robotframework/) were just some of the tools shown off.  iTest2 (http://itest2.com/) was also shown off and really goes a long way in my mind to show off the cababilities of an IDE for functional testing tools.  As the project lead for SWAT, one of my biggest pushes has been to take the tools I get in Visual Studios and bring them to the functional testing world.  At least for Watir, iTest2 is there.

From there we moved onto a conversation about running tests quicker.  Jason Huggins (from Selenium fame) was giving a presentation on cloud computing and Selenium.  This was a really great conversation a point of which I will create a 2nd entry for.  As part of this though we really looked at automation and how we can get tests running quicker and get feedback times down into the minutes.   I dont believe we came up with any specific answers but the conversations them selves really opened my mind to some new ideas.

Finally as we were nearing the end I got into a second conversation with Jason discussing the Selenium IDE.  For those who do not know, the Selenium IDE is a basic plugin for FireFox that does recording for Selenium.  Jason is a big fan of deprecating the plugin since he feels that it is continually teaching bad automation techniques.  By now the 5 readers of this blog who also most likely know me personally know that I totally would like to see record\playback go away.  However I also realize that we can not just remove this functionality since so many people are currently dependent on it.  In fact Jason says that at least 40% of the current downloads for Selenium are for the IDE.

Some of the ideas that we came up with are:

  • Rename Selenium IDE to Selenium Trainer to help promote the idea that recording really isnt the end all for automation
  • Create some kind of complexity meter on the recorder that shows when your recording is getting so complex that maybe its time to learn new automation techniques
  • Recorders are the training wheels of the automated world and we need to start treating them as so

I think both ideas are really going a long way in trying to promote the concept that automation is really software development.  I think this is a place where SWAT really is going to shine since we are doing a great deal to continually improve our IDE to help teach the newest of new QA good practices in creating automation.

As these conversations winded down, a number of us started discussing the current state of tools.  We have numerous drivers, frameworks, and runners out there and now maybe is the time to start looking at integration techniques.  How do we take all of these great tools and start combining them.  I think that is a great question and something that we really need to start thinking about.  (And where I will leave off this overview of my time at the AA-FTT.)

Posted in Uncategorized | Tagged: , , , , | 1 Comment »