Posted by Mike Longin on August 24, 2009
I have spent the past year dreaming of my (and quite a few others) holy grail of the automation world. In my mind this has always been a large grid (or cloud) based system that can run all of your functional tests at once. In more realistic terms it would at least turn an 12 hours suite into an 40-80 minute run. I have talked about this numerous times, and felt that Ultimate was really going to get there with the recent aquisition of lab manager. Jason Huggins (http://twitter.com/jhuggins) said something at AA-FTT that really made me stop in my tracks.
(Loosely remembered) Even if we can run our full suite in a much shorter time span, the programmers still wont be able to do that BEFORE every checkin
Him saying that really just blew me away because I really had not thought about that before.
Today I had the opportunity to hear J.B. Rainesbergers speak about the inherent flaws in integration testing and how we need to move to micro tests that only test one object at a time. To do this requires extensive mocking, which I have not always been a fan of after having to debug\modify others code like this and feeling the pain of doing so. However the past two days of comments really make me realize just how long these integration tests take to run, and the absolute need to have as few as possible.
I am hoping that over the next few days I can attend more sessions such as these 2 and see where my thought path lines up on what my personal holy grail of automation should be and maybe some steps to lead me down that path.
Posted in Uncategorized | Tagged: AAFTT, Agile 2009, Automated Testing | Leave a Comment »
Posted by Mike Longin on August 24, 2009
I really wanted to post this yesterday but a little social interaction got in the way.
I spent the day at the AA-FTT meetup (http://www.eventbee.com/view/qualitytree/event?eventid=59531) in Chicago yesterday and had a really great time. I had the opportunity to demonstrate two tools that we have developed interally at Ultimate Software. The first is a web driver known as SWAT that I have messaged before. The 2nd is a windows based UI wrapper for running Fitnesse tests. I hope to be able to post some more on that in the future.
The day was an open space presentation style that was high energy the whole day. I started by attending a lightning talk showing off a number of tools people are actively using. Cucumber (http://cukes.info/), Twist (http://studios.thoughtworks.com/agile-test-automation), and the Robot Framework (http://code.google.com/p/robotframework/) were just some of the tools shown off. iTest2 (http://itest2.com/) was also shown off and really goes a long way in my mind to show off the cababilities of an IDE for functional testing tools. As the project lead for SWAT, one of my biggest pushes has been to take the tools I get in Visual Studios and bring them to the functional testing world. At least for Watir, iTest2 is there.
From there we moved onto a conversation about running tests quicker. Jason Huggins (from Selenium fame) was giving a presentation on cloud computing and Selenium. This was a really great conversation a point of which I will create a 2nd entry for. As part of this though we really looked at automation and how we can get tests running quicker and get feedback times down into the minutes. I dont believe we came up with any specific answers but the conversations them selves really opened my mind to some new ideas.
Finally as we were nearing the end I got into a second conversation with Jason discussing the Selenium IDE. For those who do not know, the Selenium IDE is a basic plugin for FireFox that does recording for Selenium. Jason is a big fan of deprecating the plugin since he feels that it is continually teaching bad automation techniques. By now the 5 readers of this blog who also most likely know me personally know that I totally would like to see record\playback go away. However I also realize that we can not just remove this functionality since so many people are currently dependent on it. In fact Jason says that at least 40% of the current downloads for Selenium are for the IDE.
Some of the ideas that we came up with are:
- Rename Selenium IDE to Selenium Trainer to help promote the idea that recording really isnt the end all for automation
- Create some kind of complexity meter on the recorder that shows when your recording is getting so complex that maybe its time to learn new automation techniques
- Recorders are the training wheels of the automated world and we need to start treating them as so
I think both ideas are really going a long way in trying to promote the concept that automation is really software development. I think this is a place where SWAT really is going to shine since we are doing a great deal to continually improve our IDE to help teach the newest of new QA good practices in creating automation.
As these conversations winded down, a number of us started discussing the current state of tools. We have numerous drivers, frameworks, and runners out there and now maybe is the time to start looking at integration techniques. How do we take all of these great tools and start combining them. I think that is a great question and something that we really need to start thinking about. (And where I will leave off this overview of my time at the AA-FTT.)
Posted in Uncategorized | Tagged: AAFTT, Agile 2009, Automated Testing, Selenium, SWAT | 1 Comment »