mjc4y@brunelleschi.cs.virginia.edu: python object model 
Author Message
 mjc4y@brunelleschi.cs.virginia.edu: python object model

This could be the start of a very interesting discussion on the future
of objects in Python.  I am forwarding it because the tail of the
message says I can decide if I want to...  Later I'll comment on it
myself.


<URL: http://www.*-*-*.com/ ;

------- Forwarded Message

Date:    Tue, 23 Aug 1994 14:34:18 -0400



Subject: python object model

Guido,
        Introduction
        --------------------------------------------
        this mail starts with an excerpt from a discussion that Tommy
        Burnette, Rob DeLine and I have been having on the subject of
        the extensibility of the Python object system. Tommy tells me
        you (eventually) intend to lift the restrictions of the
        builtin objects to allow subclassing. I think this discussion
        has direct bearing on such an effort, and points to a
        mechanism that might make Python *much* more powerful and
        flexible without sacrificing the farm.

        This discussion isn't a plea for more features, but more of a
        "what if" between Python admirers. I thought it a sufficiently
        interesting line of speculation that it was worth letting you
        in on it. I'd be delighted to hear any thought or reactions
        you have, but I understand you're a busy guy. :)

        We start with a discussion of a feature I hacked into the
        python interpreter....I write to Rob:


   |
   | :  [[ Interesting footnote: I've successfully hacked Python's dictionary
   | :     slot access to allow the user to specify a SUIT-interest-like callba
ck
   | :     whenever a dictionary element is changed. Using this on any class
   | :     instance's self.__dict__, you can trap assignments like fred.x = 10
   | :     to call a function of your choosing. This function will be given
   | :     fred.__dict__ (the dict being changed), "x" (the key being
   | :     changed), and fred.x oldval and new val. Works great....
   |
   |
   | Hmmmmm....
   |
   | Isn't it the case that if a dictionary were implemented as a class
   | with setting and getting as methods, then you could get the behavior
   | you want just by subclassing?
   |    class Dictionary:
   |        def __getitem__(self, key):
   |            # some implementation
   |        def __setitem__(self, key, value):
   |            # some implementation
   |
   |    class DictionaryWithNotification(Dictionary):
   |        def register(self, func):
   |            self.func = func
   |        def __setitem__(self, key, value):
   |            oldValue = self.__getitem__(self, key)
   |            self.func(self, key, oldValue, value)
   |            Dictionary.__setitem__(self, key, value)

My reply:

Objects in Python
- ----------------------------

Yes and No.  Unfortuately, in Python, there are objects, and then
there are Objects.  :) As you know, lists, tuples, dictionaries,
strings, integers, floats, etc. are all "objects" in the Python model,
and objects like lists have "methods" associated with them like "sort"
and "append". None of these "builtin" object types support subclassing
or subtyping in any way. There is a fair bit of machinery in Python to
support two builtin object types: classobjects and instanceobjects
(ok, and methodobjects). These are the only objects that
participate in the subclassing mechanism.  

You can create a class Dictionary like you have outlined here, but
that doesn't help you for implementing the "notification" facility on
instance object slot access. Classes and instances store their slot
values in *regular* dictionaries, a decision that's been hardcoded in
C. From here, it's not clear how to proceed. There are any
number of options (hacking the original source, adding a new classtype
and kluging out a way to instantiate them...), but all have some
serious drawbacks. The rest of this mail hopes to outline a possible
solution, taken from languages like CLOS.

The problem here is that there are no "hooks" into the implementation
from Python. Perhaps this is intentional on Guido's part, so that he
can stay free to reimplement slot access and assignment in any way,
perhaps without using dicts at all at some future time. Whatever the
reason, it seems we have a clear case where "the implementation shows
through" in bad ways, even though it's supposed to be helping us with
the power of abstraction. [see Kiczales and his metaobject protocol
stuff]

Some days I wish computers weren't trying to be so damn helpful.  :)

What we want is to be able to override the *definition of a
classobject* and instanceobject itself, and to do so in such a way
that you can use the default behavior of classes when you want
to. (Rob: CLOS folks will recognize this idea as MOP, yes?). I'm going to
think out loud here, dreaming up what a mechanism might
look like, and imagining what it might take to actually implement
it. If it's not too hard, maybe Guido can be convinced to extend the
language.

Classobjects Reconsidered
- -------------------------

So we need to redefine what it means to be a classobject. Whether the
implementation of this new class object happens in C or in Python is
really no big deal to me. Clearly having it in Python is better for
exploring new ideas and doesn't require a recompile of the
interpreter, but I understand the motivation for putting it in C for
speed reasons. Let's assume that it happens in C for now.

   [[ side note: my terminology might be wrong here, but we're talking
   about metaclasses here, yes? a metaclassobject has things like
   getitem and setitem methods, and when instantiated, a metaclass
   creates classes, not instances. It serves as a container for the
   "protocol" to which all "instances" of a 'real' class must
   adhere. Python currently supports a single instance of the ethereal
   metaclassobject; it's called a classobject. Cutting to the chase,
   what I'm about to propose is that Python allow *many* instances of
   the metaclass object -- many different kinds of classes and
   therefore many different kinds of instances. ]]

A proposed Syntax for metaclasses
- ----------------------------------------

Let's invent some syntax and introduce a new keyword:

        metaclass

Here's an example of what it might look like in python:

metaclass SmartClass:
        def __init__(self):
            # assume that the attributes of an object are held in a
            # field called __attrs__
            self.__attrs__ = SmartDict()

        def __getslot__(self, key):
            # do something              

        def __setslot__(self, key, value):
            # do something

        def __callmethod__(self, meth, *args):
            print "calling method", meth, "with", args
            # really call the method
            apply(meth, self + args)

        etc...fleshing out this list will
        be an interesting exercise. Does the init
        take args, for example? :)

This defines all the operations on a class instance. Now to create a
class of this kind we do this:

class SmartRect is SmartClass:
        # SmartClass.__init__(self) gets called here.

        def __init__(self, color):
                self.color = color

        def getcolor(self):
                retrun self.color

Now when the SmartRect is instantiated,

        myrect = SmartRect("red")

you get a "smartinstance" called myrect. Instead of a dict in its
belly, myrect has a SmartDict (see the SmartClass spec). when you call
getcolor

        print "color is ----->",  myrect.getcolor()

you get

        calling method <method myrect.getcolor of myrect instance at 11b558> wi
th args None
        color is -----> red  

where the first line comes from __callmethod__ and the second line
comes directly from the getcolor method of SmartRect.

The C implementation of this seems to be pretty
straightforward. Metaclasses have well-defined methods (getslot,
setslot, callmethod, etc) and Class objects get pointers to these
methods when the class is created. On each of the relevant
opertations, these methods are called through those pointers (just
another table like exists for typeobjects).

With this, we have all the power we need for modifying the semantics
of class instances. What's nice here is that the mechanism is
completely general and yet has fine granularity: one class can
implement method call one way and another class can do it another way,
all within the same program.

Back to the Notification Class
- ----------------------------------
Now, implementing the mechanism we talked about at the beginning is
easy. To implement a class that calls a user callback whenever
instance data is accessed, you :
        - make the class's metaclass define a method that sets up the
        callback
        - override the __setslot__ function to perform the assignment
        and to then call the callback.

That's it. In case you think that all this trouble is the facilitate
this one favorite feature of mine, rest assured. This metaclass
facility is something that could be used to implement the long awaited
access objects, read only classes, for restricting the attrs that can
be attached to an instance, before and after methods, and other
wonderful things that we can't even think of yet.

Conclusion
- ---------------------
Whew.

Guido,
Sorry for being so long winded. I hope this discussion sparks some
interest and some discussion. Does this look like something you could
build into the language, given your (rumored) desire to fold the
builtin object types into a more general object system?  

I have no idea if this stuff would be of general use to the
newgroup. I don't want to spark a "feature war" discussion, so I'll
leave it to you, Guido, if you want to forward this to the mailing
list.

Cheers,
        matt

PS for Rob DeLine:
1. You can forward ths off to Gregor Kiczales if you think he'd be
interested.  
2. What else is the metaclass mechanism often used for? Is there a
"standard list of cool things" that metaclasses are good for?

------- End of Forwarded Message



Sun, 09 Feb 1997 21:16:28 GMT  
 mjc4y@brunelleschi.cs.virginia.edu: python object model
Metaclasses!  This is great!  OK, here are some issues:

1. How can these metaclasses actually help me implement the smart
dictionary given as an example?  I mean, beyond just intercepting the
__getslot__ and __setslot__ meta-methods (meta-methods?).  How can I
get this Bold-New-Python to parse "{'one':1, 'two':2}" and create a
smart dictionary instead of the standard not-so-smart dictionary?
(Display is easy: we already have __str__ and __repr__).
  I'm talking about extending the syntax, and so extending the parser.
Can this be done through a metaclass specification?  (This question is
starting to sound like a request for C++ operator overloading
features.)  The issue here is: how to maintain readability.  One of
Python's strengths is it's brevity, and to give that up in favor of
a more universal (and flexible) syntax will cause the language to look
more like Lisp or Scheme.

2. From the example/proposed metaclass syntax:
        metaclass SmartClass:
          def __init__(self):    ...
It occurs to me that metaclasses can also be inherited, like so:
        metaclass SmartClass(StandardClass):
          def __init__(self):    ...
So we get all the advantages of inheritance when defining metaclasses.

3. Inheritance and metaclass conflicts.  The example/proposed new
class syntax is:
        class SmartRect is SmartClass:
          def ...
What would the restrictions be when classes are inherited?  Can I do
this:
        class SmartRect is SmartClass:
          def ...
        class SmartPoly(SmartRect) is DumbClass:
          def ...
I can imagine arguments both for and against.  I haven't worked with
metaclasses, so I'm asking out of ignorance.  And, of course, to stir
up the hornet's nest.

4. Readability vs. Metaclass Proliferation: Will the ability to monkey
with class operation lead to less readable Python code?  I mention
readability again because it is one of Python's strengths, and there
are plenty of languages that are less so.  Imagine:
        dict1['key'] = 1
        dict2['key'] = 1
where "dict1" and "dict2" are of different classes and metaclasses.
They (hopefully) will do something similar, but may cause different
side effects.  Is this desirable?  Is this behavior similar to one of
C++'s good features, or one of it's bad features?
  In the US, if I may delve into a figurative argument, we have a
saying: "Guns don't kill people, people kill people."  Which is to say
that programmers should be responsible for their own confusion.
Although, a variation is "Guns don't kill people, bullets kill
people," which is an obfuscated way of saying "Tough shit", except
that people make bullets, so whose responsibile for that?  Then
there's "Slogans don't confuse issues, people confuse issues..."

5. How metaclasses will help me: one of my peeves about (some)
object-oriented languages is the unnecessary distinction between
instance variable and method.  Why must I write:
        instance.calculated_value()
to cause some calculated value to be actually calculated from other
instance variables?  It's a value!  It may already *be* calculated!
What does the caller care how or when it comes into being?  I would
prefer to leave the parentheses off, and leave the implementation,
whether it be dictionary look-up or method call, to the class
implementation.  (This can be accomplished with something similar to
the notify-on-change functionality discussed in this thread.)
  Well, that's what my code will look like in the future.  Do you want
to read it?
  How to differentiate between calling the method and returning the
method as a first-class object?  Use "instance.calculated_value.im_func"

5. Metaclasses: do we need this?  It makes the language more complex.
I was reading the Dylan language spec yesterday, and one of the design
goals specified was, "Make the language as simple as possible, and no
simpler."  Well, I don't think they've accomplished that, and I can
use Python as an example of how things *could* be simpler.  Should this
potentially trite phrase be part of Python's design goal?
--
Craig Lawson



Mon, 10 Feb 1997 04:13:54 GMT  
 mjc4y@brunelleschi.cs.virginia.edu: python object model
If the object model is redesigned to allow meta-programming (which I'd
consider a good idea), I propose to make reflection - i.e. the ability
of the system to inspect itself - as complete as possible. The various
problems of persistence IMO demponstrated clearly that there should be
as much regularity as feasable. The ideal might be a completely speci-
fied virtual machine, which can inspect itself in every corner, except
that some primitive functions [that's enough; data is only accessed by
functions, though possibly with syntactic sugar] have a representation
which consists only of names needed to access them in any interpreter,
provided that they are present (or dynamically linkable - but probably
this should already be constructed from more basic functionality, that
is invoked when a kind of "autoload-object" is evaluated). As far as I
can tell from the description, the Self language/implementation serves
as a good example of what I'm thinking of (for Self documentation look
in /pub/Self at self.stanford.edu). For the meta-functionality, Lisp's
various MOPs are certainly good sources of information (in addition to
CLOS, EuLisp has a somewhat different protocol, that is supposed to be
easier to optimise; look at ftp.bath.ac.uk:pub/eulisp. Some postscript
documents, including the CLOS specification, can be found in /pub/mops
on parcftp.xerox.com).

------------------------------------------------------------------------------
   *   wonder everyday   *   nothing in particular   *   all is special   *



Mon, 10 Feb 1997 07:17:30 GMT  
 mjc4y@brunelleschi.cs.virginia.edu: python object model
But (unless I've missed something): why do we really need a 'metaclass'
feature at all?   Wouldn't it be enough to add new special methods for
catching method/data-member fetch/set/call's to the current 'class'
datatype?  Borrowing from the example:

        __setslot__(self, name, newvalue)
        __getslot__(self, name)
        __callmethod__(self, name, *args)

or something similar.  If these methods are defined for an instance (in
its class, or one of its base classes), the interpreter would call these
to handle the access event.

Because these methods would get inherited by subclasses (just like special
methods such as __len__() do now), they'd be visible from instances-- a
normal class could be used like a 'metaclass' by defining these methods,
and letting other 'real' classes be derived from it.  I.e., any class
could potentially be a 'metaclasses', by defining or over-riding these
special methods.

It seems that this would be alot simpler to implement, and might avoid
a major can-of-worms, design-wise.  There's no new 'metaclass' feature
(only a set of new 'special' method names), and no syntax changes.

It also seems like this would be simpler to use, since there's no special
syntax or semantics for meta-classes.  For example, it would use the usual
'class' resolution, for multiple inheritance ambiguity.

I concur that catching class item accesses could be a very useful
feature, in general.  The only concerns I'd raise:

- Impact on class performance:  in general, classes are already pretty
  slow, and I'd rather see their perfomance optimized than worsened.
  The extra overhead may be trivial here.

- Potential for misuse: I suspect it might get used in cases where
  it would be simpler to define wrapper classes, use normal class
  inheritance, etc.  For example, data is usually wrapped in get/set
  members in C++ coding, for better encapsulation; avoiding this
  step alone doesn't justify the feature (I realize there's other
  reasons).

- Potential for abuse: exposing the guts of class access like this
  could lead to alot of hard-to-understand tricky code (but so can
  anything else :-).  

But I agree: the feature has clear utility, as already pointed out.
I'd just like to see it kept simple.

Mark Lutz



Sun, 09 Feb 1997 18:12:15 GMT  
 mjc4y@brunelleschi.cs.virginia.edu: python object model

   Metaclasses!  This is great!  OK, here are some issues:

First off, I'm a Python newbie, but I'm familiar with metaclasses from
CLOS (where else?). For anyone who is completely baffled by what a
metaclass is, and why anyone would care, a metaclass is just a class
whose instances are class objects. Clear as mud, I know.

Check out The Art of the Metaobject Protocol, by Kiczales, des
Rivieres, and Bobrow (MIT Press, 1991), or significant sections of
Object-Oriented Programming / The CLOS Perspective, by Paepcke (MIT
Press, 1993), for a description and discussion of the CLOS meta-object
protocol (MOP).

   1. How can these metaclasses actually help me implement the smart
   dictionary given as an example?  I mean, beyond just intercepting the
   __getslot__ and __setslot__ meta-methods (meta-methods?).  How can I
   get this Bold-New-Python to parse "{'one':1, 'two':2}" and create a
   smart dictionary instead of the standard not-so-smart dictionary?
   (Display is easy: we already have __str__ and __repr__).
     I'm talking about extending the syntax, and so extending the parser.
   Can this be done through a metaclass specification?  (This question is
   starting to sound like a request for C++ operator overloading
   features.)  The issue here is: how to maintain readability.  One of
   Python's strengths is it's brevity, and to give that up in favor of
   a more universal (and flexible) syntax will cause the language to look
   more like Lisp or Scheme.

In CLOS, there's no way to do this -- the parser (reader, in Lisp-ese)
always reads a list as a list, a string as a string, etc. There's no
way to say, "All lists should now be 'super-nifty-lists', instead".

{ A minor note regarding CLOS -- first off, you can't inherit from
built-in classes like list or string, so this can't really be done
anyway; and the Lisp reader is actually completely programmable, so
you actually COULD get it to read (1 2 3) and produce a
super-nifty-list instead of a plain old list, if that was what you
wanted; but not easily. }

   2. From the example/proposed metaclass syntax:
           metaclass SmartClass:
             def __init__(self):    ...
   It occurs to me that metaclasses can also be inherited, like so:
           metaclass SmartClass(StandardClass):
             def __init__(self):    ...
   So we get all the advantages of inheritance when defining metaclasses.

Yes, of course; a metaclass is a class like any other, and so there's
an inheritance hierarchy of metaclasses. Note also that there's a
class Metaclass, of which metaclasses are instances...

   3. Inheritance and metaclass conflicts.  The example/proposed new
   class syntax is:
           class SmartRect is SmartClass:
             def ...
   What would the restrictions be when classes are inherited?  Can I do
   this:
           class SmartRect is SmartClass:
             def ...
           class SmartPoly(SmartRect) is DumbClass:
             def ...
   I can imagine arguments both for and against.  I haven't worked with
   metaclasses, so I'm asking out of ignorance.  And, of course, to stir
   up the hornet's nest.

In CLOS, unless DumbClass inherited from SmartClass, I don't think
this is allowed. This makes sense to me, but I don't know if it's
absolutely required, or if it could be relaxed.

   4. Readability vs. Metaclass Proliferation: Will the ability to monkey
   with class operation lead to less readable Python code?  I mention
   readability again because it is one of Python's strengths, and there
   are plenty of languages that are less so.  Imagine:
           dict1['key'] = 1
           dict2['key'] = 1
   where "dict1" and "dict2" are of different classes and metaclasses.
   They (hopefully) will do something similar, but may cause different
   side effects.  Is this desirable?  Is this behavior similar to one of
   C++'s good features, or one of it's bad features?
     In the US, if I may delve into a figurative argument, we have a
   saying: "Guns don't kill people, people kill people."  Which is to say
   that programmers should be responsible for their own confusion.
   Although, a variation is "Guns don't kill people, bullets kill
   people," which is an obfuscated way of saying "Tough shit", except
   that people make bullets, so whose responsibile for that?  Then
   there's "Slogans don't confuse issues, people confuse issues..."

It is always true that providing more power also provides more
opportunities for shooting yourself in the foot. While this doesn't
mean that you should restrict the capabilities of a language, it might
mean that you shouldn't make it trivial to use facilities that make it
easy to shoot yourself (keep guns legal, but make the licensing
requirements strict, and require a waiting period before you can get
one).

For example, in CLOS, there's actually a lot of mucking around with
various peculiar methods that you have to do to make your metaclass
work at all, most of which is highly unobvious. This is usually
considered a flaw, but maybe it's really a feature?

   5. How metaclasses will help me: one of my peeves about (some)
   object-oriented languages is the unnecessary distinction between
   instance variable and method.  Why must I write:
           instance.calculated_value()
   to cause some calculated value to be actually calculated from other
   instance variables?  It's a value!  It may already *be* calculated!
   What does the caller care how or when it comes into being?  I would
   prefer to leave the parentheses off, and leave the implementation,
   whether it be dictionary look-up or method call, to the class
   implementation.  (This can be accomplished with something similar to
   the notify-on-change functionality discussed in this thread.)
     Well, that's what my code will look like in the future.  Do you want
   to read it?
     How to differentiate between calling the method and returning the
   method as a first-class object?  Use "instance.calculated_value.im_func"

Yes, you could do this with a metaclass. Other nifty uses: generalized
persistent objects; specialized implementations for performance
optimization; classes that automatically track (keep a list of) their
instances; and classes that automatically provide a trace of all
accesses.

   5. Metaclasses: do we need this?  It makes the language more complex.
   I was reading the Dylan language spec yesterday, and one of the design
   goals specified was, "Make the language as simple as possible, and no
   simpler."  Well, I don't think they've accomplished that, and I can
   use Python as an example of how things *could* be simpler.  Should this
   potentially trite phrase be part of Python's design goal?

The issue is whether the additional power and flexibility is worth the
additional costs in performance, maintainability, and learn-ability
(?). I'm comfortable with the idea of a MOP, so I kind of like the
idea; on the other hand, I can easily see where this could be a source
of confusion to novice users, and my primary attraction to Python is
as an extension language that people who are not experienced
programmers can use. So I'm stuck in the middle, myself.

--
-----------------------------------------------------------------------------
 Alan Geller                                            phone: (908)699-8285
 Bell Communications Research                             fax: (908)336-2953

 RRC 5G-110
 Piscataway, NJ 08855-1342



Sun, 16 Feb 1997 07:25:37 GMT  
 mjc4y@brunelleschi.cs.virginia.edu: python object model
I think Craig Lawson raises some good issues in his posting, but there
is one that i think is a bit of a red herring:

Quote:
> 4. Readability vs. Metaclass Proliferation: Will the ability to monkey
> with class operation lead to less readable Python code?  I mention
> readability again because it is one of Python's strengths, and there
> are plenty of languages that are less so.  Imagine:
>         dict1['key'] = 1
>         dict2['key'] = 1
> where "dict1" and "dict2" are of different classes and metaclasses.
> They (hopefully) will do something similar, but may cause different
> side effects.  Is this desirable?  Is this behavior similar to one of
> C++'s good features, or one of it's bad features?

You already have this complexity, without metaclasses.  Specifically,
dict1 and dict2 need not be of different metaclasses to behave
drastically differently - if they're of different classes, they could
have different __setitem__ methods, and so have drastically different
behaviors, including side effects.  ?

On the other hand, while we currently can safely presume that attribute
assignments, 'obj1.x = 22.33', will have the obvious and expected
immediate effect, metaclasses would probably change that aspect of the
ballgame completely.

(BTW, if anyone has a ready answer, i'm still curious what the reasons
are for distinguishing metaclass inheritance from regular class
inheritance.  Or are we doing away with that, and just introducing
meta-methods like __setslot__ slot assignment and __callmethod__
method invocation directly into the standard repertoire?)

Ken



Mon, 17 Feb 1997 02:33:57 GMT  
 
 [ 8 post ] 

 Relevant Pages 

1. mjc4y@brunelleschi.cs.virginia.edu: python object model

2. NOW SHOWING - wiki.cs.uiuc.edu/VisualWorks/Recent+Changes

3. CORRECTING -->NOW SHOWING - wiki.cs.uiuc.edu/VisualWorks/Recent+Changes

4. DEPRECATED: http://www.cs.uoregon.edu/research/tcl/

5. PATCHES: Patch Archive Updated http://www.cs.uoregon.edu/research/tcl/patch/

6. NO PYTHON: http://cm.bell-labs.com/cm/cs/who/bwk/interps/pap.html

7. vwnc@cs.uiuc.edu

8. How to connect to Smalltalk Archive (st-www.cs.uiuc.edu)

9. anarres.cs.berkeley.edu

10. news-relayFrom: pop@roo.cs.umass.edu (Robin Popplestone)

11. news-relayFrom: pop@roo.cs.umass.edu (Robin Popplestone)

12. news-relayFrom: pop@roo.cs.umass.edu (Robin Popplestone)

 

 
Powered by phpBB® Forum Software