language advocacy and language comparison 
Author Message
 language advocacy and language comparison

this seems like an appropriate forum in which to mention an
observation i've made concerning programming language advocacy.

it is not unusual for a programmer of language L to believe
that L is superior to other languages, modulo chosen restrictions
on the domain of application.  the best way to test whether that
belief is well-founded is to compare L solutions to solutions in
other languages.

the problems should be smaller than your average dissertation
topic, but larger than 'hello world'.  the comparison metric
should involve lines of code, readability, performance, and
possibly other considerations introduced on an ad hoc basis.
ideally, the problems should bear some visible relation to the
kinds of problems application programmers deal with in their
daily work.

i think it is important that participants really do believe that
their language of choice is "better" than competing languages,
as long as this attitude is accompanied by a sincere desire to
discover whether that belief is well-founded.  programmers do
their best work when they hold their tools in high regard, but
they learn to do good work by cultivating the habit of learning
from others.

with the exception of the occasional contest, such as the ICFP
functional language contest sponsored by MIT, none of the news
groups i'm familiar with provide a framework for such informal
inter-language comparisons.  perhaps there is such a site, but
i haven't been able to find it.

as a partisan for a particular language myself, i enjoy this
kind of competition, although i detest the rancorous exchanges
which typically disfigure cross-language discussions on the
primary language groups.



Thu, 14 Feb 2002 03:00:00 GMT  
 language advocacy and language comparison

Quote:

> this seems like an appropriate forum in which to mention an
> observation i've made concerning programming language advocacy.

> it is not unusual for a programmer of language L to believe
> that L is superior to other languages, modulo chosen restrictions
> on the domain of application.  the best way to test whether that
> belief is well-founded is to compare L solutions to solutions in
> other languages.

> the problems should be smaller than your average dissertation
> topic, but larger than 'hello world'.  the comparison metric
> should involve lines of code, readability, performance, and
> possibly other considerations introduced on an ad hoc basis.
> ideally, the problems should bear some visible relation to the
> kinds of problems application programmers deal with in their
> daily work.

one thing that is rarely presented in either the "language L is
better IMO" missives or the "I looked an N languages and L came
out on top" studies that I have looked at is that the presenter
rarely addresses the issue of the skill level and objectives
of the coder. for example, I develop and support a set of
engineering codes written in fortran 77, as well as a data
visualization code in ANSI C. Uncle Sam distributes source only,
to a specific set of academics and industry people. my "customers"
range from guys who were never fully weaned from things like
backward gotos to guys who can argue the relative merits of
automatic vs. manual GC.

since many of the codes in my field tend to exist in the gray area
between prototype stage and production stage (as are most engineering
codes IMHO) and the manhours per kloc is nowhere near what folks
in other fields consider adequate, language choice is very much driven
by simplicity, which is why Fortran still dominates despite all the
bad press it gets.

- Show quoted text -

Quote:
> i think it is important that participants really do believe that
> their language of choice is "better" than competing languages,
> as long as this attitude is accompanied by a sincere desire to
> discover whether that belief is well-founded.  programmers do
> their best work when they hold their tools in high regard, but
> they learn to do good work by cultivating the habit of learning
> from others.

> with the exception of the occasional contest, such as the ICFP
> functional language contest sponsored by MIT, none of the news
> groups i'm familiar with provide a framework for such informal
> inter-language comparisons.  perhaps there is such a site, but
> i haven't been able to find it.

> as a partisan for a particular language myself, i enjoy this
> kind of competition, although i detest the rancorous exchanges
> which typically disfigure cross-language discussions on the
> primary language groups.

I would hazard a guess that this stems from the subconcious attitude
that "L works perfectly for me, and I get my work done, so if you
aren't getting good results with L there must be something wrong
with you."  this is sort of a pet peeve, as I'm a firm believer
in adapting the tool to the user, not vice versa.

--
---------------------------------------------------------------------------

"There are lies, damned lies, and statistics" -- Benjamin Disraeli
"...and benchmarks" -- Garry Hodgson
---------------------------------------------------------------------------



Thu, 14 Feb 2002 03:00:00 GMT  
 language advocacy and language comparison

Quote:

>one thing that is rarely presented in either the "language L is
>better IMO" missives or the "I looked an N languages and L came
>out on top" studies that I have looked at is that the presenter
>rarely addresses the issue of the skill level and objectives
>of the coder.

This is the main killer issue that prevents objective comparisons
of languages. To be statistically significant such a comparison
would need to control for all significant factors that may affect
the result of the evaluation: the familiarity of the subjects with
the language, the tool support available, the human support available,
etc, but the huge differences between the skill levels of different
programmers is the biggest such factor.

To average out these effects, a thorough evaluation would need to repeat
the experiment many times over with many different programmers, in the
expectation that the differences between the *average* skill levels of
two randomly chosen groups of programmers will be much smaller than
the differences between the skill levels of two randomly chosen programmers.
However, this puts the experiment beyond the financial reach of almost
all organizations. In particular, beyond the reach of research projects
working on new languages.


Department of Computer Science, University of Melbourne, AUSTRALIA



Sun, 17 Feb 2002 03:00:00 GMT  
 language advocacy and language comparison

What would be the result of such an experiement, even if it could be
done?  A statement something like "average programmers on an average
problem using an average development process produce code that works in
an average test case in the average least time with language X"?  I
think it's pretty obvious that no-one would take much notice ("we don't
write average code", "but what about maintenance?" etc.).

The whole idea of one language being simply "better" than another makes
way too many simplifying assumptions.  A much better approach is to look
at a range of languages, identify features that are useful in that
language (used in an appropriate context) and see how they translate to
other languages and other contexts.  In this way you slowly build up
experience with languages, patterns, and application areas - producing,
finally, better code.  That still doesn't give you a "best language" -
there simply isn't such an animal.  Instead (I guess) you eventually
reach a state of higher consciousness where you can turn to your
(similarly elevated) co-worker and say something like "Hmmm - we're in a
company whose engineers will be happiest supporting P, but we can use
pattern Q from language R in component S and T the U with a V from X in
a Y paradigm under Z..."  ;-)

Andrew

http://www.andrewcooke.free-online.co.uk/index.html (home)
http://www.andrewcooke.free-online.co.uk/andrew/lang.html (intro to prog
languages)

PS Incidentally, most pages I have looked at that "choose" a best
language do at least give the chooser's own background and target area.



Quote:

> >one thing that is rarely presented in either the "language L is
> >better IMO" missives or the "I looked an N languages and L came
> >out on top" studies that I have looked at is that the presenter
> >rarely addresses the issue of the skill level and objectives
> >of the coder.

> This is the main killer issue that prevents objective comparisons
> of languages. To be statistically significant such a comparison
> would need to control for all significant factors that may affect
> the result of the evaluation: the familiarity of the subjects with
> the language, the tool support available, the human support available,
> etc, but the huge differences between the skill levels of different
> programmers is the biggest such factor.

> To average out these effects, a thorough evaluation would need to
repeat
> the experiment many times over with many different programmers, in the
> expectation that the differences between the *average* skill levels of
> two randomly chosen groups of programmers will be much smaller than
> the differences between the skill levels of two randomly chosen
programmers.
> However, this puts the experiment beyond the financial reach of almost
> all organizations. In particular, beyond the reach of research
projects
> working on new languages.

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.


Sun, 17 Feb 2002 03:00:00 GMT  
 language advocacy and language comparison


Quote:

> What would be the result of such an experiement, even if it could be
> done?  A statement something like "average programmers on an average
> problem using an average development process produce code that works in
> an average test case in the average least time with language X"?  I
> think it's pretty obvious that no-one would take much notice ("we don't
> write average code", "but what about maintenance?" etc.).

i think you're confusing what i want with something much grander.  i'm
not interested in large-scale formal experiment, since, as you and others
have pointed out, the complexity of the criteria for real-world decision-
making would make such a project enormously difficult.

in particular, i'm not interested in how average programmers perform
on average problems in average development environments.  clearly, informal
competition does not constitute an average environment (although in some
respects it might approximate one more closely than many people will admit),
nor should we expect that the type of person who enters such a competition
on his own time and for no compensation an average specimen.

Quote:

> The whole idea of one language being simply "better" than another makes
> way too many simplifying assumptions.  A much better approach is to look
> at a range of languages, identify features that are useful in that
> language (used in an appropriate context) and see how they translate to
> other languages and other contexts.  In this way you slowly build up
> experience with languages, patterns, and application areas - producing,
> finally, better code.  That still doesn't give you a "best language" -
> there simply isn't such an animal.  Instead (I guess) you eventually
> reach a state of higher consciousness where you can turn to your
> (similarly elevated) co-worker and say something like "Hmmm - we're in a
> company whose engineers will be happiest supporting P, but we can use
> pattern Q from language R in component S and T the U with a V from X in
> a Y paradigm under Z..."  ;-)

i know that this view has achieved wide acceptance among programmers,
although i have my doubts that it is as true as people like to think.
still, let me give you the point that no language is better simpliciter
than any other language.  that is not inconsistent with assertions that
fit the scheme,

    L is better than L' w.r.t. ... p ... for tasks of kind ... k ...

for example, C is better than Pascal w.r.t. performance for writing
device drivers.  there must be some way to rationally decide between
languages.  it seems to me that even if competitions of the sort i'm
interested in don't provide information for conclusively deciding,
they can give us certain useful insights.

even if i am absolutely wrong about the value of this activity, i'm
still puzzled why there isn't more of it.

Quote:

> Andrew

> http://www.andrewcooke.free-online.co.uk/index.html (home)
> http://www.andrewcooke.free-online.co.uk/andrew/lang.html (intro to prog
> languages)

> PS Incidentally, most pages I have looked at that "choose" a best
> language do at least give the chooser's own background and target area.




> > >one thing that is rarely presented in either the "language L is
> > >better IMO" missives or the "I looked an N languages and L came
> > >out on top" studies that I have looked at is that the presenter
> > >rarely addresses the issue of the skill level and objectives
> > >of the coder.

> > This is the main killer issue that prevents objective comparisons
> > of languages. To be statistically significant such a comparison
> > would need to control for all significant factors that may affect
> > the result of the evaluation: the familiarity of the subjects with
> > the language, the tool support available, the human support available,
> > etc, but the huge differences between the skill levels of different
> > programmers is the biggest such factor.

> > To average out these effects, a thorough evaluation would need to
> repeat
> > the experiment many times over with many different programmers, in the
> > expectation that the differences between the *average* skill levels of
> > two randomly chosen groups of programmers will be much smaller than
> > the differences between the skill levels of two randomly chosen
> programmers.
> > However, this puts the experiment beyond the financial reach of almost
> > all organizations. In particular, beyond the reach of research
> projects
> > working on new languages.

> Sent via Deja.com http://www.deja.com/
> Share what you know. Learn what you don't.



Sun, 17 Feb 2002 03:00:00 GMT  
 language advocacy and language comparison

Quote:


> >one thing that is rarely presented in either the "language L is
> >better IMO" missives or the "I looked an N languages and L came
> >out on top" studies that I have looked at is that the presenter
> >rarely addresses the issue of the skill level and objectives
> >of the coder.

> This is the main killer issue that prevents objective comparisons
> of languages. To be statistically significant such a comparison
> would need to control for all significant factors that may affect
> the result of the evaluation: the familiarity of the subjects with
> the language, the tool support available, the human support available,
> etc, but the huge differences between the skill levels of different
> programmers is the biggest such factor.

> To average out these effects, a thorough evaluation would need to repeat
> the experiment many times over with many different programmers, in the
> expectation that the differences between the *average* skill levels of
> two randomly chosen groups of programmers will be much smaller than
> the differences between the skill levels of two randomly chosen programmers.
> However, this puts the experiment beyond the financial reach of almost
> all organizations. In particular, beyond the reach of research projects
> working on new languages.

I've always wondered why the big CS departments don't use their
cannon fodder/er, I mean undergraduates, for this kind of testing.
for example, break the 3rd year students into 3 or 4 groups; group A
uses pascal for some particular class (data structures for instance),
group B uses lisp, group C uses C, etc. if the students were randomly
assigned to the groups, and the problems solved during the semester
were the same, then there would be a body of results that could yield
useful data.  for quantitative analysis, I guess labor hours per
assignment could be recorded, and code could be inspected for the
usual bugs. qualitative analysis could be done from a survey.

--
---------------------------------------------------------------------------

"There are lies, damned lies, and statistics" -- Benjamin Disraeli
"...and benchmarks" -- Garry Hodgson
---------------------------------------------------------------------------



Sun, 17 Feb 2002 03:00:00 GMT  
 language advocacy and language comparison

Quote:

>I've always wondered why the big CS departments don't use their
>cannon fodder/er, I mean undergraduates, for this kind of testing.
>for example, break the 3rd year students into 3 or 4 groups; group A
>uses pascal for some particular class (data structures for instance),
>group B uses lisp, group C uses C, etc. if the students were randomly
>assigned to the groups, and the problems solved during the semester
>were the same, then there would be a body of results that could yield
>useful data.  for quantitative analysis, I guess labor hours per
>assignment could be recorded, and code could be inspected for the
>usual bugs. qualitative analysis could be done from a survey.

Something like

http://academic.uofs.edu/faculty/beidler/levy/

Or in a commercial environment

http://www.rational.com/sitewide/support/whitepapers/dynamic.jtmpl?do...

There is also the article about the course which utilized the
programming of a train control system to teach programming. This was a
similar experiment to yours, but serially rather than parallel.
However, the URL escapes me.



Sun, 17 Feb 2002 03:00:00 GMT  
 language advocacy and language comparison

Quote:

>What would be the result of such an experiement, even if it could be
>done?  A statement something like "average programmers on an average
>problem using an average development process produce code that works in
>an average test case in the average least time with language X"?

It would be something like

        "For programmers with this particular kind of skill level, on this
        particular set of problems, using this given development process,
        we have found it takes on average X1 months and Y1 dollars to produce
        code that meets this given set of quality requirements in language L1,
        whereas for languages L2 and L3 the comparable figures were X2 and X3
        months and Y2 and Y3 dollars respectively."

Quote:
> I
>think it's pretty obvious that no-one would take much notice ("we don't
>write average code", "but what about maintenance?" etc.).

People who work on similar problems with similarly skilled programmers *ought*
to pay attention to such a study (whether they will is another matter :-().
For other people the study is not relevant.

Quote:
>The whole idea of one language being simply "better" than another makes

Nobody who has seriously thought about the issue would make a blanket
statement like that. However, it ought to be possible to make objective
comparisons between languages if the parameters (application domain etc)
are limited enough. Unfortunately, while making the limits tighter makes
the experiment more feasible, it also reduces the usefulness of the results.
There ain't such thing as a free lunch.


Department of Computer Science, University of Melbourne, AUSTRALIA



Mon, 18 Feb 2002 03:00:00 GMT  
 language advocacy and language comparison

Quote:

>all of this may be true.  indeed, i think the problem is even more
>difficult than you suggest.  in my experience, toy problems teach us
>little about the characteristics of a language in "extreme" (which
>is to say, normal) circumstances.

Definitely.

I just give a lecture which drew a distinction between languages oriented
towards flexibility (C, C++, Lisp), which take the programmer at his/her word,
and languages oriented towards safety (Ada, Haskell, Mercury) which reduce
the programmer's choices to a safer set of alternatives and apply a lot more
static checks. Languages of the first sort are much more likely to shine in
experiments using small programs, while languages of the second sort are
much more likely to shine in experiments using large programs.

I did not list this as the main problem because the problem of achieving
statistical significance kills the idea in most people who think of it;
this other problem just makes it even more dead.

Quote:
>since,
>for the reasons you enumerate, decision makers in this area are forced
>to rely on anecdote and casual recommendations, information about how
>different programmer-language combinations have fared on a range of
>interesting problems probably more useful than you suggest.

Anecdotal evidence is of course useful, but not anywhere as useful
as the results of controlled experiments would be. However, as you
point out, the choice is not really between anecdotes and experiments;
it is between anecdotes and wild-ass guesses. In that situation, relying
on anecdotes is the correct choice, so long as one recognizes the limitations
of the method.


Department of Computer Science, University of Melbourne, AUSTRALIA



Mon, 18 Feb 2002 03:00:00 GMT  
 language advocacy and language comparison


Quote:


> > The whole idea of one language being simply "better" than another
makes
> > way too many simplifying assumptions.  A much better approach is to
look
> > at a range of languages, identify features that are useful in that
> > language (used in an appropriate context) and see how they translate
to
> > other languages and other contexts.  In this way you slowly build up
> > experience with languages, patterns, and application areas -
producing,
> > finally, better code.  That still doesn't give you a "best language"
-
> > there simply isn't such an animal.  Instead (I guess) you eventually
> > reach a state of higher consciousness where you can turn to your
> > (similarly elevated) co-worker and say something like "Hmmm - we're
in a
> > company whose engineers will be happiest supporting P, but we can
use
> > pattern Q from language R in component S and T the U with a V from X
in
> > a Y paradigm under Z..."  ;-)

> i know that this view has achieved wide acceptance among programmers,
> although i have my doubts that it is as true as people like to think.
> still, let me give you the point that no language is better
simpliciter
> than any other language.  that is not inconsistent with assertions
that
> fit the scheme,

>     L is better than L' w.r.t. ... p ... for tasks of kind ... k ...

> for example, C is better than Pascal w.r.t. performance for writing
> device drivers.  there must be some way to rationally decide between
> languages.  it seems to me that even if competitions of the sort i'm
> interested in don't provide information for conclusively deciding,
> they can give us certain useful insights.

Sorry, I misunderstood you.

I've been trying to think what other examples you could have rather than
device drivers that suggest an obvious best language that's not simply
"my favourite language".

I guess there's the distinction between designed/written/maintained by
similarly qualified people and designed by wizards/implemented by
monkeys.  In the latter case you could make an argument for enforced
encapsulation, contracts, static types, etc. to try and catch as many
errors at compile-time as possible (the argument being that the
implementors don't understand the design, so may do silly things).

Then there's code size - the distinction betwen scripting languages and
"the rest".

At first glance, logical languages are particularly suited to certain
tasks, but I suspect some people would argue that they are also suitable
for general programs, so to pick them out for expert systems, for
example, may be unfair.

Apart from that, and what toolkits/support is available, I think you get
back to general problem solving, which is where having as much knowledge
as possible comes in...

Andrew
http://www.andrewcooke.free-online.co.uk/index.html (home)
http://www.andrewcooke.free-online.co.uk/andrew/lang.html (intro to prog
lang)

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.



Mon, 18 Feb 2002 03:00:00 GMT  
 language advocacy and language comparison
In article

Quote:


> >I've always wondered why the big CS departments don't use their
> >cannon fodder/er, I mean undergraduates, for this kind of testing.
> >for example, break the 3rd year students into 3 or 4 groups; group A
> >uses pascal for some particular class (data structures for instance),
> >group B uses lisp, group C uses C, etc. if the students were randomly
> >assigned to the groups, and the problems solved during the semester
> >were the same, then there would be a body of results that could yield
> >useful data.  for quantitative analysis, I guess labor hours per
> >assignment could be recorded, and code could be inspected for the
> >usual bugs. qualitative analysis could be done from a survey.

> Something like

> http://academic.uofs.edu/faculty/beidler/levy/

This was interesting.  I was surprised by the comment that
"Though Pascal is still taught by most of the colleges in the
United States (Reid 95) ...."; I would have guessed from all
the criticism that it receives (including in this article) that it had
been abandoned.

--
MJSR

Sent via Deja.com http://www.deja.com/
Share what you know. Learn what you don't.



Mon, 18 Feb 2002 03:00:00 GMT  
 language advocacy and language comparison

Quote:

> This was interesting.  I was surprised by the comment that
> "Though Pascal is still taught by most of the colleges in the
> United States (Reid 95) ...."; I would have guessed from all
> the criticism that it receives (including in this article) that it had
> been abandoned.

If the article was published in 1995, it might have been written in 1994
and based on data from 1993. We're talking quite a few years ago. Since that
article was written a lot of places will have switched to Java in any case.

There is also a time lag to consider whereby the top research universities
move with the trend, lower ranking ones are a bit behind, and teaching in
pre-university institutions is quite a few years behind. By the time Pascal
reached the latter it was already being abandoned by the former.

Matthew Huntbach



Mon, 18 Feb 2002 03:00:00 GMT  
 
 [ 13 post ] 

 Relevant Pages 

1. Choice Of Language (was: Python Advocacy)

2. Smalltalk Language Comparison - 1995

3. Comparison of J & K languages

4. FP in a larger scale (Re: Comparison of functional languages)

5. Language Comparisons

6. APL and Language Comparison

7. Language comparison

8. Smalltalk Language Comparison - 1995

9. I need your experience - classification and comparison of languages

10. KOT (kinda off topic):yet another language comparison

11. Language Comparisons

12. Language comparisons-Enfin smalltalk vs others

 

 
Powered by phpBB® Forum Software