this seems like an appropriate forum in which to mention an
observation i've made concerning programming language advocacy.

it is not unusual for a programmer of language L to believe
that L is superior to other languages, modulo chosen restrictions
on the domain of application.  the best way to test whether that
belief is well-founded is to compare L solutions to solutions in
other languages.

the problems should be smaller than your average dissertation
topic, but larger than 'hello world'.  the comparison metric
should involve lines of code, readability, performance, and
possibly other considerations introduced on an ad hoc basis.
ideally, the problems should bear some visible relation to the
kinds of problems application programmers deal with in their
daily work.

i think it is important that participants really do believe that
their language of choice is "better" than competing languages,
as long as this attitude is accompanied by a sincere desire to
discover whether that belief is well-founded.  programmers do
their best work when they hold their tools in high regard, but
they learn to do good work by cultivating the habit of learning
from others.

with the exception of the occasional contest, such as the ICFP
functional language contest sponsored by MIT, none of the news
groups i'm familiar with provide a framework for such informal
inter-language comparisons.  perhaps there is such a site, but
i haven't been able to find it.

as a partisan for a particular language myself, i enjoy this
kind of competition, although i detest the rancorous exchanges
which typically disfigure cross-language discussions on the
primary language groups.

one thing that is rarely presented in either the "language L is
better IMO" missives or the "I looked an N languages and L came
out on top" studies that I have looked at is that the presenter
rarely addresses the issue of the skill level and objectives
of the coder. for example, I develop and support a set of
engineering codes written in fortran 77, as well as a data
visualization code in ANSI C. Uncle Sam distributes source only,
to a specific set of academics and industry people. my "customers"
range from guys who were never fully weaned from things like
backward gotos to guys who can argue the relative merits of
automatic vs. manual GC.

since many of the codes in my field tend to exist in the gray area
between prototype stage and production stage (as are most engineering
codes IMHO) and the manhours per kloc is nowhere near what folks
in other fields consider adequate, language choice is very much driven
by simplicity, which is why Fortran still dominates despite all the
bad press it gets.

I would hazard a guess that this stems from the subconcious attitude
that "L works perfectly for me, and I get my work done, so if you
aren't getting good results with L there must be something wrong
with you."  this is sort of a pet peeve, as I'm a firm believer
in adapting the tool to the user, not vice versa.

--
---------------------------------------------------------------------------

"There are lies, damned lies, and statistics" -- Benjamin Disraeli
"...and benchmarks" -- Garry Hodgson
---------------------------------------------------------------------------

This is the main killer issue that prevents objective comparisons
of languages. To be statistically significant such a comparison
would need to control for all significant factors that may affect
the result of the evaluation: the familiarity of the subjects with
the language, the tool support available, the human support available,
etc, but the huge differences between the skill levels of different
programmers is the biggest such factor.

To average out these effects, a thorough evaluation would need to repeat
the experiment many times over with many different programmers, in the
expectation that the differences between the *average* skill levels of
two randomly chosen groups of programmers will be much smaller than
the differences between the skill levels of two randomly chosen programmers.
However, this puts the experiment beyond the financial reach of almost
all organizations. In particular, beyond the reach of research projects
working on new languages.

Department of Computer Science, University of Melbourne, AUSTRALIA

What would be the result of such an experiement, even if it could be
done?  A statement something like "average programmers on an average
problem using an average development process produce code that works in
an average test case in the average least time with language X"?  I
think it's pretty obvious that no-one would take much notice ("we don't
write average code", "but what about maintenance?" etc.).

The whole idea of one language being simply "better" than another makes
way too many simplifying assumptions.  A much better approach is to look
at a range of languages, identify features that are useful in that
language (used in an appropriate context) and see how they translate to
other languages and other contexts.  In this way you slowly build up
experience with languages, patterns, and application areas - producing,
finally, better code.  That still doesn't give you a "best language" -
there simply isn't such an animal.  Instead (I guess) you eventually
reach a state of higher consciousness where you can turn to your
(similarly elevated) co-worker and say something like "Hmmm - we're in a
company whose engineers will be happiest supporting P, but we can use
pattern Q from language R in component S and T the U with a V from X in
a Y paradigm under Z..."  ;-)

Andrew

http://www.andrewcooke.free-online.co.uk/index.html (home)
http://www.andrewcooke.free-online.co.uk/andrew/lang.html (intro to prog
languages)

PS Incidentally, most pages I have looked at that "choose" a best
language do at least give the chooser's own background and target area.

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.

i think you're confusing what i want with something much grander.  i'm
not interested in large-scale formal experiment, since, as you and others
have pointed out, the complexity of the criteria for real-world decision-
making would make such a project enormously difficult.

in particular, i'm not interested in how average programmers perform
on average problems in average development environments.  clearly, informal
competition does not constitute an average environment (although in some
respects it might approximate one more closely than many people will admit),
nor should we expect that the type of person who enters such a competition
on his own time and for no compensation an average specimen.

i know that this view has achieved wide acceptance among programmers,
although i have my doubts that it is as true as people like to think.
still, let me give you the point that no language is better simpliciter
than any other language.  that is not inconsistent with assertions that
fit the scheme,

    L is better than L' w.r.t. ... p ... for tasks of kind ... k ...

for example, C is better than Pascal w.r.t. performance for writing
device drivers.  there must be some way to rationally decide between
languages.  it seems to me that even if competitions of the sort i'm
interested in don't provide information for conclusively deciding,
they can give us certain useful insights.

even if i am absolutely wrong about the value of this activity, i'm
still puzzled why there isn't more of it.

I've always wondered why the big CS departments don't use their
cannon fodder/er, I mean undergraduates, for this kind of testing.
for example, break the 3rd year students into 3 or 4 groups; group A
uses pascal for some particular class (data structures for instance),
group B uses lisp, group C uses C, etc. if the students were randomly
assigned to the groups, and the problems solved during the semester
were the same, then there would be a body of results that could yield
useful data.  for quantitative analysis, I guess labor hours per
assignment could be recorded, and code could be inspected for the
usual bugs. qualitative analysis could be done from a survey.

--
---------------------------------------------------------------------------

"There are lies, damned lies, and statistics" -- Benjamin Disraeli
"...and benchmarks" -- Garry Hodgson
---------------------------------------------------------------------------

Something like

http://academic.uofs.edu/faculty/beidler/levy/

Or in a commercial environment

http://www.rational.com/sitewide/support/whitepapers/dynamic.jtmpl?do...

There is also the article about the course which utilized the
programming of a train control system to teach programming. This was a
similar experiment to yours, but serially rather than parallel.
However, the URL escapes me.

It would be something like

        "For programmers with this particular kind of skill level, on this
        particular set of problems, using this given development process,
        we have found it takes on average X1 months and Y1 dollars to produce
        code that meets this given set of quality requirements in language L1,
        whereas for languages L2 and L3 the comparable figures were X2 and X3
        months and Y2 and Y3 dollars respectively."

People who work on similar problems with similarly skilled programmers *ought*
to pay attention to such a study (whether they will is another matter :-().
For other people the study is not relevant.

Nobody who has seriously thought about the issue would make a blanket
statement like that. However, it ought to be possible to make objective
comparisons between languages if the parameters (application domain etc)
are limited enough. Unfortunately, while making the limits tighter makes
the experiment more feasible, it also reduces the usefulness of the results.
There ain't such thing as a free lunch.

Department of Computer Science, University of Melbourne, AUSTRALIA

Definitely.

I just give a lecture which drew a distinction between languages oriented
towards flexibility (C, C++, Lisp), which take the programmer at his/her word,
and languages oriented towards safety (Ada, Haskell, Mercury) which reduce
the programmer's choices to a safer set of alternatives and apply a lot more
static checks. Languages of the first sort are much more likely to shine in
experiments using small programs, while languages of the second sort are
much more likely to shine in experiments using large programs.

I did not list this as the main problem because the problem of achieving
statistical significance kills the idea in most people who think of it;
this other problem just makes it even more dead.

Anecdotal evidence is of course useful, but not anywhere as useful
as the results of controlled experiments would be. However, as you
point out, the choice is not really between anecdotes and experiments;
it is between anecdotes and wild-ass guesses. In that situation, relying
on anecdotes is the correct choice, so long as one recognizes the limitations
of the method.

Department of Computer Science, University of Melbourne, AUSTRALIA

Sorry, I misunderstood you.

I've been trying to think what other examples you could have rather than
device drivers that suggest an obvious best language that's not simply
"my favourite language".

I guess there's the distinction between designed/written/maintained by
similarly qualified people and designed by wizards/implemented by
monkeys.  In the latter case you could make an argument for enforced
encapsulation, contracts, static types, etc. to try and catch as many
errors at compile-time as possible (the argument being that the
implementors don't understand the design, so may do silly things).

Then there's code size - the distinction betwen scripting languages and
"the rest".

At first glance, logical languages are particularly suited to certain
tasks, but I suspect some people would argue that they are also suitable
for general programs, so to pick them out for expert systems, for
example, may be unfair.

Apart from that, and what toolkits/support is available, I think you get
back to general problem solving, which is where having as much knowledge
as possible comes in...

Andrew
http://www.andrewcooke.free-online.co.uk/index.html (home)
http://www.andrewcooke.free-online.co.uk/andrew/lang.html (intro to prog
lang)

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.

In article

This was interesting.  I was surprised by the comment that
"Though Pascal is still taught by most of the colleges in the
United States (Reid 95) ...."; I would have guessed from all
the criticism that it receives (including in this article) that it had
been abandoned.

--
MJSR

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.

If the article was published in 1995, it might have been written in 1994
and based on data from 1993. We're talking quite a few years ago. Since that
article was written a lot of places will have switched to Java in any case.

There is also a time lag to consider whereby the top research universities
move with the trend, lower ranking ones are a bit behind, and teaching in
pre-university institutions is quite a few years behind. By the time Pascal
reached the latter it was already being abandoned by the former.

Matthew Huntbach