Automating UI tests (was Re: Art of Unit Testing) 
Author Message
 Automating UI tests (was Re: Art of Unit Testing)

Quote:

> I've found little value in writing unittests for GUI apps

This is a subject that's been on my mind lately, and since the
subject has come up here, maybe some one can give me some pointers
on this matter:

How exactly would one go about automating unit tests of GUI code?
Specifically, I'm working on some UI stuff in Tkinter.
At first I thought, "well, I can just write tests that call
my Tk callback methods directly with appropriate values", but
then I realized that that can't work because my code
depends on the widget state (eg might call event.widget.get()
for a key-release event, or something like that).

So that means I'd need to write code that actually drives
the Tk widgets "from outside", as it were, either by synthesizing
events somehow at the OS/window-system level, or by knowing
enough about the Tk internals to convince the widgets that they're
being clicked on, typed at, etc. And at that point it looks like
a lot more effort than it's worth.

Can anyone suggest a workable approach to this problem?
Do wxPython or PyQt make this kind of thing any easier?
(I've looked at them both briefly, but am so used to
Tk that I would prefer not to switch with out a
compelling reason.)

Thanks,

--
# Joe Knapka
# "You know how many remote castles there are along the
#  gorges? You can't MOVE for remote castles!" - Lu Tze re. Uberwald
# 2nd Lbl A + 1 = 2nd Pause 2nd Prt A
--
# Joe Knapka
# "You know how many remote castles there are along the
#  gorges? You can't MOVE for remote castles!" - Lu Tze re. Uberwald
# 2nd Lbl A + 1 = 2nd Pause 2nd Prt A



Wed, 04 Feb 2004 02:33:46 GMT  
 Automating UI tests (was Re: Art of Unit Testing)
Quote:
----- Original Message -----

Newsgroups: comp.lang.python
Sent: Friday, August 17, 2001 2:33 PM
Subject: Automating UI tests (was Re: Art of Unit Testing)

> So that means I'd need to write code that actually drives
> the Tk widgets "from outside", as it were, either by synthesizing
> events somehow at the OS/window-system level, or by knowing
> enough about the Tk internals to convince the widgets that they're
> being clicked on, typed at, etc. And at that point it looks like
> a lot more effort than it's worth.

Depending on platform there may be tools that do exactly this -- Rational
Visual Test and NuMega TestPartner on Windows come to mind. I've never
looked for corresponding tools on other platforms...

// Brett g Porter * Lead Engineer, Development Practices

// Art & Logic * Custom software solutions for hardware products
// Windows * MacOS * Embedded Systems



Wed, 04 Feb 2004 04:23:21 GMT  
 Automating UI tests (was Re: Art of Unit Testing)


    ...

Quote:
> So that means I'd need to write code that actually drives
> the Tk widgets "from outside", as it were, either by synthesizing
> events somehow at the OS/window-system level, or by knowing
> enough about the Tk internals to convince the widgets that they're
> being clicked on, typed at, etc. And at that point it looks like
> a lot more effort than it's worth.

That's basically what we did at work for our capture-and-playback
based testing system -- knowing enough about the internals of
our own UI-engine architecture, of course (it's based on MFC, but
wraps it in a script-friendly way), so that events can be captured
and played back at different levels of abstraction (depending on
whether we're testing the UI-engine, or the actual application).

There are commercial solutions based on OS-level C&P, but of
course they're much more oriented to testing the 'externals' -- a
little change in UI layout or whatever, and your database of
captured sessions for regression-testing playbacks is toast.

I can't think of a reasonably complete solution that's worth the
effort for one user of a GUI toolkit -- the test-harness should,
it seems to me, be part of the GUI toolkit itself (alter sys.path
so a guitest directory is prepended, and rather than the actual
widgets &c what you're getting is some kind of test-harness
version).  But I don't know of any GUI kit, commercial or free,
for any language[s], that offers such a precious feature:-(.  It's
not easy to architect, of course:-(.

Alex



Wed, 04 Feb 2004 04:08:52 GMT  
 Automating UI tests (was Re: Art of Unit Testing)

Quote:



>     ...
> > So that means I'd need to write code that actually drives
> > the Tk widgets "from outside", as it were, either by synthesizing
> > events somehow at the OS/window-system level, or by knowing
> > enough about the Tk internals to convince the widgets that they're
> > being clicked on, typed at, etc. And at that point it looks like
> > a lot more effort than it's worth.

> That's basically what we did at work for our capture-and-playback
> based testing system -- knowing enough about the internals of
> our own UI-engine architecture, of course (it's based on MFC, but
> wraps it in a script-friendly way), so that events can be captured
> and played back at different levels of abstraction (depending on
> whether we're testing the UI-engine, or the actual application).

> There are commercial solutions based on OS-level C&P, but of
> course they're much more oriented to testing the 'externals' -- a
> little change in UI layout or whatever, and your database of
> captured sessions for regression-testing playbacks is toast.

> I can't think of a reasonably complete solution that's worth the
> effort for one user of a GUI toolkit -- the test-harness should,
> it seems to me, be part of the GUI toolkit itself (alter sys.path
> so a guitest directory is prepended, and rather than the actual
> widgets &c what you're getting is some kind of test-harness
> version).  But I don't know of any GUI kit, commercial or free,
> for any language[s], that offers such a precious feature:-(.  It's
> not easy to architect, of course:-(.

Possibly the Anygui project could offer a backend that,
rather than instantiating Tkinter or wxPython controls,
or whatever, instead instantiates objects that present a
programmatic interface for the use of unit test code.

--
# Joe Knapka
# "You know how many remote castles there are along the
#  gorges? You can't MOVE for remote castles!" - Lu Tze re. Uberwald
# 2nd Lbl A + 1 = 2nd Pause 2nd Prt A



Wed, 04 Feb 2004 05:33:14 GMT  
 Automating UI tests (was Re: Art of Unit Testing)
[posted to both comp.lang.python and the anygui list,

the latter -- feel free to override if you think c.l.p needs
more discussion of this, of course]



    ...

Quote:
> > I can't think of a reasonably complete solution that's worth the
> > effort for one user of a GUI toolkit -- the test-harness should,
> > it seems to me, be part of the GUI toolkit itself (alter sys.path
> > so a guitest directory is prepended, and rather than the actual
> > widgets &c what you're getting is some kind of test-harness
> > version).  But I don't know of any GUI kit, commercial or free,
> > for any language[s], that offers such a precious feature:-(.  It's
> > not easy to architect, of course:-(.

> Possibly the Anygui project could offer a backend that,
> rather than instantiating Tkinter or wxPython controls,
> or whatever, instead instantiates objects that present a
> programmatic interface for the use of unit test code.

Yes, I think that does make sense (for a _second_ release
of anygui -- better get a first one out first:-).  Maybe two
backends -- one which wraps any other real backend but as
a side effect also writes a log of anygui-level events, and
one that replays such events (and logs gui-related program
actions in textual form, for easier checking, rather than
drawing to the screen).  That's not exactly "programmatic
control", but it would give easy "capture & playback" test
ability to any anygui program, and finer-grained program
control would then be easier to build as an add-on (e.g.
by generating the logs to be replayed).

Alex



Wed, 04 Feb 2004 15:27:30 GMT  
 Automating UI tests (was Re: Art of Unit Testing)
On Fri, 17 Aug 2001 18:33:46 GMT, Joseph Andrew Knapka

Quote:


>> I've found little value in writing unittests for GUI apps

>This is a subject that's been on my mind lately, and since the
>subject has come up here, maybe some one can give me some pointers
>on this matter:

>How exactly would one go about automating unit tests of GUI code?
>Specifically, I'm working on some UI stuff in Tkinter.
>At first I thought, "well, I can just write tests that call
>my Tk callback methods directly with appropriate values", but
>then I realized that that can't work because my code
>depends on the widget state (eg might call event.widget.get()
>for a key-release event, or something like that).

>So that means I'd need to write code that actually drives
>the Tk widgets "from outside", as it were, either by synthesizing
>events somehow at the OS/window-system level, or by knowing
>enough about the Tk internals to convince the widgets that they're
>being clicked on, typed at, etc. And at that point it looks like
>a lot more effort than it's worth.

It's probably easier than you think: look at the Tk 'event' command.
It can deliver an arbitrary keyboard mouse or virtual event to any Tk
widget.

In your UI code,  setup a dictionary which contains the widgets
that you want to test by sending events to, so that you can refer to
then in the testng code. Then in the testing code, use the event
command to send thngs like <Button-1> events to those widgets.

Note that if you are testing Dialogs (popped up windows with a grab)
you probably have to use the Tk 'after' command to queue up the events
before instantiating the Dialog.

Mike.



Thu, 12 Feb 2004 07:32:49 GMT  
 Automating UI tests (was Re: Art of Unit Testing)

Quote:

> It's probably easier than you think: look at the Tk 'event' command.
> It can deliver an arbitrary keyboard mouse or virtual event to any Tk
> widget.

> In your UI code,  setup a dictionary which contains the widgets
> that you want to test by sending events to, so that you can refer to
> then in the testng code. Then in the testing code, use the event
> command to send thngs like <Button-1> events to those widgets.

> Note that if you are testing Dialogs (popped up windows with a grab)
> you probably have to use the Tk 'after' command to queue up the events
> before instantiating the Dialog.

> Mike.

In PyQt, I use a a small extra framework to test whether signals arrive.
This way, I can check whether the signals that the underlying logic sends
fit what the GUI will expect. That can be extended to send signals to the
GUI and check whether the GUI components do the right thing on getting them
- it's a bit comparable to what you describe, but still no substitute for
user tests, I'm afraid.

--
Boudewijn | http://www.valdyas.org



Fri, 13 Feb 2004 16:44:38 GMT  
 
 [ 7 post ] 

 Relevant Pages 

1. Test::Unit::Mock: Mock objects for testing with Test::Unit

2. using test::unit for C++ unit tests

3. Test::Unit::UI::Console::TestRunner suggestions

4. Test::Unit::UI::Fox::TestRunner

5. Tcl Automated Unit Test

6. Art of Unit Testing: Part 2

7. Art of Unit Testing: Part 2

8. Art of Unit Testing

9. Art of Unit Testing

10. Looking for SmallTalk UI Regression Tester (Whether to test the UI)

11. Automating VW2.0 testing with MICROSOFT TEST

12. Test order in Test::Unit

 

 
Powered by phpBB® Forum Software