Unit Testing Style 
Author Message
 Unit Testing Style

I know I should have written my tests first, but I didn't, so now I'm
trying to do them after the fact.  I'm running into a few style
questions, though, not being used to the Test::Unit style of testing:

1) I have several test cases that do multiple assert_* calls.  Is that
   a Good Thing, a Bad Thing, or not something that's even worth
   worrying about?  Usually it's things like "Was the file created?
   Does it contain data of the right format?  Is the data correct?"

   In one case, however, the test case is "run a command remotely that
   logs to yet another remote database, and verify the command was
   run, output was generated, logged to the right database with the
   right timestamp, from the right machine, etc., etc., etc."

2) How do people generally feel about interactivity in test cases?  I
   have a method that should send email to a random address.  Since
   I'd like everybody on my team to be able to run my tests, I can't
   use a specific address, as that address might not (probably won't)
   exist on their machine, and depending on which OS they're running,
   the mail spool dirs will be in a different location.

   My idea is to have the test ask for the address by stdin, and also
   ask the user to verify the contents manually.  The user's response
   will determine if the test passes or fails.

Other than that, I'm enjoying Test::Unit quite a lot.

-=Eric
--
Come to think of it, there are already a million monkeys on a million
typewriters, and Usenet is NOTHING like Shakespeare.
                -- Blair Houghton.



Sat, 19 Nov 2005 03:57:29 GMT  
 Unit Testing Style

Quote:
> 1) I have several test cases that do multiple assert_* calls.  Is
> that
>    a Good Thing, a Bad Thing, or not something that's even worth
>    worrying about?  Usually it's things like "Was the file created?
>    Does it contain data of the right format?  Is the data correct?"

>    In one case, however, the test case is "run a command remotely
> that
>    logs to yet another remote database, and verify the command was
>    run, output was generated, logged to the right database with the
>    right timestamp, from the right machine, etc., etc., etc."

In the general sense, I have a number of tests (in java, mind you)
that have multiple asserts.

__________________________________
Do you Yahoo!?
Yahoo! Calendar - Free online calendar with sync to Outlook(TM).
http://calendar.yahoo.com



Sat, 19 Nov 2005 04:11:13 GMT  
 Unit Testing Style
see inline...

Quote:
> I know I should have written my tests first, but I didn't, so now I'm
> trying to do them after the fact.  I'm running into a few style
> questions, though, not being used to the Test::Unit style of testing:

> 1) I have several test cases that do multiple assert_* calls.  Is that
>    a Good Thing, a Bad Thing, or not something that's even worth
>    worrying about?  Usually it's things like "Was the file created?
>    Does it contain data of the right format?  Is the data correct?"

>    In one case, however, the test case is "run a command remotely that
>    logs to yet another remote database, and verify the command was
>    run, output was generated, logged to the right database with the
>    right timestamp, from the right machine, etc., etc., etc."

It seems to me like you are trying to do too much in a single test.  
I would have tests for the item that you are testing remotely.  Since
those test would actually be locally, you can then break them down
into small simple tests, and then when that is tested you only have
to simulate your remote procedure call in a test.

Quote:
> 2) How do people generally feel about interactivity in test cases?  I
>    have a method that should send email to a random address.  Since
>    I'd like everybody on my team to be able to run my tests, I can't
>    use a specific address, as that address might not (probably won't)
>    exist on their machine, and depending on which OS they're running,
>    the mail spool dirs will be in a different location.

This too could be broken down.  I have test that tests
sending/reading of an email class.  That test does in fact use a mail
server.  Once you have done that and you know that your email class
works, it looks like you need a test that tests the random email
address to send to.  If you actually need to test sending an email,
you can then use the class interface to simulate the sending of an
email (Google mock object).

Quote:
>    My idea is to have the test ask for the address by stdin, and also
>    ask the user to verify the contents manually.  The user's response
>    will determine if the test passes or fails.

In general I think it is a bad idea to require input.  One of the key
things to automated unit tests, is that they are, well... automated.  
They just run and tell you success or failure.  For me that's why I
like them.

Quote:

> Other than that, I'm enjoying Test::Unit quite a lot.

cool.

Walt
*****************************************************
Walter Szewelanczyk
IS Director

259 Front St.            Phone : (207) 442-7994 x 128
Bath, ME 04530           Fax   : (207) 443-6284
*****************************************************



Sat, 19 Nov 2005 04:26:58 GMT  
 Unit Testing Style

Quote:

> I know I should have written my tests first, but I didn't, so now I'm
> trying to do them after the fact.  I'm running into a few style
> questions, though, not being used to the Test::Unit style of testing:

> 1) I have several test cases that do multiple assert_* calls.  Is that
>    a Good Thing, a Bad Thing, or not something that's even worth
>    worrying about?  Usually it's things like "Was the file created?
>    Does it contain data of the right format?  Is the data correct?"

Just remember that the test method will terminate with the first failing
assert_*.  So if you have a lot of them in a single method, you could
have multiple failures hiding behind a single failure.

Quote:
>    In one case, however, the test case is "run a command remotely that
>    logs to yet another remote database, and verify the command was
>    run, output was generated, logged to the right database with the
>    right timestamp, from the right machine, etc., etc., etc."

I try to keep the external environment as simple and self contained as
possible.  I want to run the unit tests and KNOW everything is ok.  If
my tests depend on a lot of external resources, then failure may mean a
problem with the code or a problem with the environment.  I would rather
spend time working on the code than tracing down environment problems.

Sometimes you DO need to talk to external resources (e.g. databases,
servers, etc.).  I usually isolate these tests in their own area
(usually a separate Rake/Make target).  Then I can run the unit tests to
check the code as often as I want and get a lots of "feel good" karma.
And then I can run the environment dependent tests separately to make
sure I still can talk to the outside world.

Quote:
> 2) How do people generally feel about interactivity in test cases?  

Evil!  Don't do this!  Never (at least in unit tests).

If you REALLY need some interactivity, I would move it outside the unit
tests.

Quote:
> Other than that, I'm enjoying Test::Unit quite a lot.

Great!

--

---------------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)



Sat, 19 Nov 2005 09:30:00 GMT  
 Unit Testing Style

Quote:

> Sometimes you DO need to talk to external resources (e.g. databases,
> servers, etc.).  I usually isolate these tests in their own area
> (usually a separate Rake/Make target).  Then I can run the unit tests to
> check the code as often as I want and get a lots of "feel good" karma.
> And then I can run the environment dependent tests separately to make
> sure I still can talk to the outside world.

 So Jim, do you have a short example of using rake to
 manage unit tests?
 I am thinking about moving from make to rake so my tests
 will be environment independent (at least it will be easier
 to install rake than make).

--
Jim Freeze
----------
New systems generate new problems.



Sat, 19 Nov 2005 10:15:59 GMT  
 Unit Testing Style

Quote:


> > [...] (usually a separate Rake/Make target). [...]

>  So Jim, do you have a short example of using rake to
>  manage unit tests?
>  I am thinking about moving from make to rake so my tests
>  will be environment independent (at least it will be easier
>  to install rake than make).

I was wondering if anyone would catch the Rake reference.

A dirt simple example follows.  Assume all of your unit tests are in a
directory named "test" and that your source code is in a directory named
"lib".

  task :test do                 # Run the Unit Tests
    Sys.ruby %{-Ilib -Itest } +
      %{-e 'Dir["test/test*.rb"].each { |fn| require fn }'}
  end

Truthfully, I'm still learning to use Rake effectively.  I've been
limited by Make for so long, it is taking me a while to see some of the
possibilities.  

For example, I'm playing around with a Builder object that is
knowledgable about how a, for example, Ruby project is built.  All you
do is specify a few special directories or actions and the Builder will
construct the make targets for you programmatically.  I'm watching the
discussion on "standard" project directories closely, because if we can
standardize the directory structure, then more knowledge can be moved
into the Builder.

--

-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)



Sat, 19 Nov 2005 11:16:50 GMT  
 Unit Testing Style

Quote:

> I know I should have written my tests first, but I didn't, so now I'm
> trying to do them after the fact.

The best way to do this (in my exalted opinion) is to point the tests at an
empty project, use the testless project as a reference, and write tests that
force you to copy tiny bits of code from the testless project into the empty
project.

Quote:
> I'm running into a few style
> questions, though, not being used to the Test::Unit style of testing:

> 1) I have several test cases that do multiple assert_* calls.  Is that
>    a Good Thing, a Bad Thing, or not something that's even worth
>    worrying about?  Usually it's things like "Was the file created?
>    Does it contain data of the right format?  Is the data correct?"

Dave Astels (who I suspect wrote a book about TDD) and I pretended to have a
vicious fight over this subject.

He claimed to only do one (1) assert per test case. I made him admit this is
a goal, not a rule.

If you try to approach that goal, you will maximize re-use and encapsulation
of your tests' setup code. It grows into resplendent fixtures and resources
in the test case.

I test GUIs, which optimize for displaying a window that I never give a
chance to display. So the toolkit wastes a lot of cycles getting ready to
display, and I throw all these cycles away. Hence, after every setup and
test call, I pack in as many assertions as I can think of.

I defer to Astels' style as (potentially) peer-reviewed, but I still think
this topic awaits a more subtle metric or rule-of-thumb. I suggested 3
assertions, to provide triangulation within R3 space.

Quote:
>    In one case, however, the test case is "run a command remotely that
>    logs to yet another remote database, and verify the command was
>    run, output was generated, logged to the right database with the
>    right timestamp, from the right machine, etc., etc., etc."

Yup. Don't run that one over and over again at test time.

If your tests get slower than the traditional time a UI event should take,
split the folder they run in, and keep going. Put another way, the time
tests take to run are a good metric for the number of source files you
should keep in a folder.

Quote:
> 2) How do people generally feel about interactivity in test cases?  I
>    have a method that should send email to a random address.  Since
>    I'd like everybody on my team to be able to run my tests, I can't
>    use a specific address, as that address might not (probably won't)
>    exist on their machine, and depending on which OS they're running,
>    the mail spool dirs will be in a different location.

I hope nobody's as tired of the following anecdote as I am of telling it:

The first project I ran full-bore test-first, it sent faxes. So I got the
numbers of all the fax machines in our offices, and put them in my test
resources. So, as I developed, every 30-90 seconds I'd hit the test button,
and 20 to 40 bogus faxes would come out of a machine somewhere in the
offices.

I came in one day and found they'd installed a new fax machine, in my cube.

I then signed up for an online free fax service, and sent it faxes. They
came back as e-mails, each with an ad attached, of course. Then one night I
left my tests running over a huge number of records, to test a new
high-volume phone card they put on the server. I flooded the fax service
with ~45,000 faxes in a couple hours, and they cancelled my account.

Uh... what was your question again?

Quote:
>    My idea is to have the test ask for the address by stdin, and also
>    ask the user to verify the contents manually.  The user's response
>    will determine if the test passes or fails.

Nix. Send to a common office account, or a Yahoo account, or something. Or
register each tester, and let them put their favorite addy in a
configuration file. Tests >must< run unattended. Everyone needs the minimum
possible excuses not to run all the tests, all the time.

Quote:
> Other than that, I'm enjoying Test::Unit quite a lot.

It's a trip!

--
  Phlip
    http://www.c2.com/cgi/wiki?TestFirstUserInterfaces



Sat, 19 Nov 2005 11:29:17 GMT  
 Unit Testing Style

Quote:



> > > [...] (usually a separate Rake/Make target). [...]

> >  So Jim, do you have a short example of using rake to
> >  manage unit tests?
> >  I am thinking about moving from make to rake so my tests
> >  will be environment independent (at least it will be easier
> >  to install rake than make).

> I was wondering if anyone would catch the Rake reference.

> A dirt simple example follows.  Assume all of your unit tests are in a
> directory named "test" and that your source code is in a directory named
> "lib".

>   task :test do                    # Run the Unit Tests
>     Sys.ruby %{-Ilib -Itest } +
>       %{-e 'Dir["test/test*.rb"].each { |fn| require fn }'}
>   end

Thanks for the example.

Quote:
> Truthfully, I'm still learning to use Rake effectively.  I've been
> limited by Make for so long, it is taking me a while to see some of the
> possibilities.  

> For example, I'm playing around with a Builder object that is
> knowledgable about how a, for example, Ruby project is built.  All you
> do is specify a few special directories or actions and the Builder will
> construct the make targets for you programmatically.  I'm watching the
> discussion on "standard" project directories closely, because if we can
> standardize the directory structure, then more knowledge can be moved
> into the Builder.

I think the Standard Project Directories discussion has died
down. Either we reached a consensus, or wore everyone out. ;)
Below is the final state it reached:

------------------
Below is a suggested directory hierarchy.
All of the directories, except possibly test/ ;),
are optional. Most of the filenames listed
are optional but are listed for consistent
spelling and capitalization.

CHANGELOG
INSTALL
LICENSE
Manifest
README     # if multi lang, README.en, README.ja, etc
TODO       # optional
install.rb # place holder
setup.rb   # place holder
  bin/
     # ruby executables
     # other executables
  contrib/
     # third party contributions
  doc/  # If multi lang, then use doc.en, doc.ja, etc
     man/       # for unix man
     api/       # rdoc output
     rd/        # if rd based docs
     tutorial/
  example/
  ext/
     ext_name/
       extconf.rb
       ... other source
  lib/
     # .rb files that go into RUBY_LIB
     # may want to use parallel directory structure below
     # site_ruby/<rev>/
  src/
     <other_lang_name>/
  test/
     # test suites. Best if dirs mirror lib/.

--
Jim Freeze
----------
Common sense is instinct, and enough of it is genius.
                -- Josh Billings



Sat, 19 Nov 2005 11:39:12 GMT  
 Unit Testing Style

Quote:


> > I know I should have written my tests first, but I didn't, so now I'm
> > trying to do them after the fact.

> The best way to do this (in my exalted opinion) is to point the tests at an
> empty project, use the testless project as a reference, and write tests that
> force you to copy tiny bits of code from the testless project into the empty
> project.

Unfortunately, that works out to effectively rewriting the code from
scratch, which is (alas) not terribly feasable at the moment.

Quote:
> Dave Astels (who I suspect wrote a book about TDD) and I pretended to have a
> vicious fight over this subject.

> He claimed to only do one (1) assert per test case. I made him admit this is
> a goal, not a rule.

That certainly was the goal I went for.  But it seems to me that if
I've got a method which is supposed to create a file with a specified
format and data, isn't it reasonable to first assert() the file
exists, then to assert() its format is correct, and finally to
assert() the data in it is what you expect?  I'm not sure how one
would make these separate tests.

Or rather, I can guess how you might do that, but it seems a bit
absurd for the sake of correctness to run the exact same code three
times just so you can say you didn't run more than one assert() per
test case.

Quote:
> If you try to approach that goal, you will maximize re-use and encapsulation
> of your tests' setup code. It grows into resplendent fixtures and resources
> in the test case.

Well, this was exactly my goal; it's just that given the above
situation, I didn't see any (to my mind) reasonable way around it.

Quote:
> I hope nobody's as tired of the following anecdote as I am of telling it:

<snip illuminating fax story>

Quote:
> Uh... what was your question again?

Heh.  I can certainly create a mock object to pretend to be a close of
TMail, which will solve 80% of my problem, but this still doesn't
answer two important questions:

1) Has the interface to tmail changed?  How can I tell?
2) Is my mailserver config screwed up?

I think I care less about 2), because if the mailserver on my test
host is screwed up, I'm going to have a number of problems that should
shortly become obvious.  I'm not sure how to test 1) without actually
sending an email.

Quote:
> Nix. Send to a common office account, or a Yahoo account, or
> something.

But then, how can the test verify that the email got there?  All it
can know is if it thinks it was sent properly.  Again though, given my
previous paragraph, I think I most likely don't care.

Quote:
> Or register each tester, and let them put their favorite addy in a
> configuration file.

And if the configuration file isn't set up?  I assume the test
fails... and then you get into much harder problems, as someone could
easily set up an email address in their configuration file that I
could have no way of knowing how to check.

Quote:
> Tests >must< run unattended. Everyone needs the minimum
> possible excuses not to run all the tests, all the time.

I know, that's why I *really* didn't like my 'solution', but couldn't
think of any way around it.  But using a mock object for the email
will solve most of my problems, as my concern when testing my TestItem
class should not be whether or not the mailserver works, but whether
TestItem is doing what it thinks it should.

Quote:
> > Other than that, I'm enjoying Test::Unit quite a lot.

> It's a trip!

Yeah, it reminds me a bit of when I first learned OO-- it took a month
or two to truly understand the difference between an object and a
struct with function pointers.  (Which is to say, on the one hand:
none.  On the other: everything.)  If I force myself to write
test-first in the future, it'll be a lot easier in the long run.

-=Eric
--
Come to think of it, there are already a million monkeys on a million
typewriters, and Usenet is NOTHING like Shakespeare.
                -- Blair Houghton.



Sun, 20 Nov 2005 00:30:58 GMT  
 
 [ 9 post ] 

 Relevant Pages 

1. Test::Unit::Mock: Mock objects for testing with Test::Unit

2. using test::unit for C++ unit tests

3. Test order in Test::Unit

4. Test order in Test::Unit

5. Automating UI tests (was Re: Art of Unit Testing)

6. Unit testing data; What to test

7. Test Tool for Unit Tests?

8. COM and unit tests

9. Report on Unit Testing MVP Presenters.

10. Newbie: Unit testing presenters?

11. Unit Test guidelines

12. ANSI Standard Messages Unit Tests

 

 
Powered by phpBB® Forum Software