Style for overwritten methods of abstract classes 
Author Message
 Style for overwritten methods of abstract classes

Hello all :)

As an example, I have an abstract base class CacheBase which provides
an interface to a persistent storage. All methods _could_ be either
overwritten (__init__, __del__) or _have to_ be provided (__getitem__,
__setitem__). Initially, I wrote

class CacheBase:
    '''Represents an abstract interface for the URL/value
    storage.

    Overwrite __init__, __getitem__, __setitem__, and
    __del__ to customize the class.'''

    def __init__(self):
        pass

    def __getitem__(self, url):
        '''Return an abstract value from the cache for the
        given URL. Return None if the URL is not in the
        cache.'''
        return None

    def __setitem__(self, url, value):
        '''Set an abstract value (usable for comparisons with
        a remote URL) for the URL.'''
        pass

    def __del__(self):
        '''Clean up.'''
        pass

I personally like this as an overview of which methods should be
defined in derived classes. On the other hand, it may be considered
verbose to write all of the above given that the methods would have
to be overwritten anyway.

Another aspect is that the "noop implementations" of __getitem__ and
__setitem__ could actually obscure bugs if somebody forgets to overwrite
the methods in derived classes.

So I can think of three different approaches:
1. write the code as above
2. merely describe the implementation/interfaces (e. g. in the class
   docstring or in a comment); write no code
3. use code but let the abstract methods raise NotImplementedError
   instead of providing a default behaviour which might be useless
(4. Are there others?)

In the light of 2., __del__ might be a special case because it may be
very important for clean-up and at the same time might go unnoticed
for a while because python will use its default behaviour regardless
if __del__ had been defined in the abstract class or not.

Which approach would you prefer and why? Can you think of criteria/cases
where you would use one approach and in another case another approach?

Thanks in advance for your replies.

Stefan



Tue, 22 Jun 2004 02:45:02 GMT  
 Style for overwritten methods of abstract classes


[snip example interface]

Quote:

> I personally like this as an overview of which methods should be
> defined in derived classes. On the other hand, it may be considered
> verbose to write all of the above given that the methods would have
> to be overwritten anyway.

> Another aspect is that the "noop implementations" of __getitem__ and
> __setitem__ could actually obscure bugs if somebody forgets to overwrite
> the methods in derived classes.

> So I can think of three different approaches:
> 1. write the code as above [with pass/return None]

i would do this only for methods where a noop is save

Quote:
> 2. merely describe the implementation/interfaces (e. g. in the class
>    docstring or in a comment); write no code

you get an exception for some methods, but not for __init__ or __del__
raising an explicit NYI exception seems cleaner to me.

Quote:
> 3. use code but let the abstract methods raise NotImplementedError
>    instead of providing a default behaviour which might be useless

i prefer this one. this way you can't forget to implement something. this
comes close to javas or c++ "abstract".
things like __init__ or __del__ can use "pass" or just documentation,
because some implementation may work with empty methods for them.
(if __init__ must take some arguments i would go for "pass")

Quote:
> (4. Are there others?)

> In the light of 2., __del__ might be a special case because it may be
> very important for clean-up and at the same time might go unnoticed
> for a while because Python will use its default behaviour regardless
> if __del__ had been defined in the abstract class or not.

most classes don't need a __del__ as python has a good memory management.
you _must have_ it only when you allocate external recources (e.g. files
are closed if all references to them are deleted, but if you open a serial
port as file it will be closed but the original settings are not
restored.).

Quote:

> Which approach would you prefer and why? Can you think of criteria/cases
> where you would use one approach and in another case another approach?

> Thanks in advance for your replies.

> Stefan

--



Tue, 22 Jun 2004 05:21:44 GMT  
 Style for overwritten methods of abstract classes

Quote:

> class CacheBase:
>     '''Represents an abstract interface for the URL/value storage.

>     Overwrite __init__, __getitem__, __setitem__, and __del__ to
>     customize the class.'''

>     def __init__(self):
>         pass

>     def __getitem__(self, url):
>         '''Return an abstract value from the cache for the given URL.
>         Return None if the URL is not in the cache.''' return None
> [snip]

I suspect that what you've got above is only a little more useful than
documentation on it's own. Atleast it's more likely to get updated if the
interface changes. Whenever I need to do this I always go for your third
option (raise an exception).

I also tend to use that trick when I'm fleshing out an interface and think
I'm being too lazy to write all my unit tests up front; it's like an extra
safety net, even if I don't need to overwrite stuff later. A bit like an
in-line post-it note, telling me exactly where to come back to implement
the next bit of functionality.

Quote:
> 3. use code but let the abstract methods raise NotImplementedError
>    instead of providing a default behaviour which might be useless

I also like it to tell me exactly what was missed (okay, you can find it
in the traceback, but I find this slows down my edit/run cycle less):

  def spam(self):
      raise NotImplementedError, "%s has not implemented spam()" % \
                                 self.__class__

---
Graham



Tue, 22 Jun 2004 05:50:45 GMT  
 Style for overwritten methods of abstract classes

Quote:
> Hello all :)

> As an example, I have an abstract base class CacheBase which provides
> an interface to a persistent storage. All methods _could_ be either
> overwritten (__init__, __del__) or _have to_ be provided (__getitem__,
> __setitem__). Initially, I wrote
    ...
> So I can think of three different approaches:
> 1. write the code as above
> 2. merely describe the implementation/interfaces (e. g. in the class
>    docstring or in a comment); write no code
> 3. use code but let the abstract methods raise NotImplementedError
>    instead of providing a default behaviour which might be useless
> (4. Are there others?)

Yes, there are.  You could have a metaclass that does ensure the
behavior you desire, and specify specialmethods that MAY or MUST
be provided in class-attributes.

It's easier in 2.2, of course, where you can make your metaclass
just by inheriting from type and tweaking it slightly, so, for
example, here's a metaclass to implement "must-checking", only:

class AbstractionError(Exception): pass

class me(type):
    def __init__(self, name, bases, classdict):
        type.__init__(self, name, bases, classdict)
        if bases == ():
            missing = self.__must_methods =
classdict.get('_must_methods_',())
        else:
            missing = []
            for x in self.__must_methods:
                if not hasattr(self, x): missing.append(x)
        if missing:
            errmsg = "Can't instantiate (%s), missing methods (%s)" % (
                name, ' '.join(missing))
            class dontcallme(me):
                def __call__(*args, **kwds):
                    raise AbstractionError, errmsg
            self.__class__ = dontcallme
        else:
            class callme(me):
                __call__ = type.__call__
            self.__class__ = callme

def tryMaking(klass):
    try: x = klass()
    except AbstractionError, noway:
        print noway
    else:
        print "made", x

class AbstractBase:
    __metaclass__ = me
    _must_methods_ = '__getitem__ __setitem__'.split()
tryMaking(AbstractBase)

class SemiConcrete(AbstractBase):
    def __getitem__(self, index): pass
tryMaking(SemiConcrete)

class ConcreteClass(SemiConcrete):
    def __setitem__(self, index, value): pass
tryMaking(ConcreteClass)

Of course, you may choose to be even more thorough in your checks, e.g.,
check for compatibility of signature rather than just for equal names.
Adding the "may" (raising errors if a class using me as its meta ever
dares define a class-attribute not in an allowed-list) is left as an
exercise to the reader -- it's not hard, of course.

The (tiny) advantage of this approach, just like for any "declaration",
is that you get your error at the earliest possible moment -- when a
class that doesn't define all methods you "know" it MUST define (thus,
an abstract class) is *instantiated*.  If you didn't want to allow partial
definition such as in SemiConcrete, you could get the error at the time
such a class is *defined*, of course.

You pay the price in terms of runtime only when a class is defined --
that's when me.__init__ runs.  The larger price is exactly in the
extra checks that this approach lets you do...

That price is hefty indeed.  Now, suddenly, even if you never use nor
need (e.g.) __getitem__, it becomes an error not to have it: you're
not within Python's wonderfully-smooth world of signature based
polymorphism anymore.

But, to each its own.  Python gives you all the power you need to
shoot yourself in the foot, in this and several other ways.  While
still not the best language for "{*filter*} and Discipline Programming",
Python does enjoy the advantage (?) that you may easily devise and
implement your own cruel and unusual punishments for yourself
along these lines.

Alex



Wed, 23 Jun 2004 00:49:25 GMT  
 Style for overwritten methods of abstract classes

Quote:

> To be honest, I haven't understood everything of this but it inspired
> my curiosity regarding the "type" class. I didn't find anything about
> it in the Python 2.2 docs (though I won't say it's not there ;-) ).

The type class is just what used to be the type type in earlier Python
versions, i.e. <type 'type'>, see

http://www.python.org/doc/current/lib/bltin-type-objects.html

It is just that the builtin type() function is not a function anymore,
but the type object itself.

Quote:
> The PEPs 252 and 253 seemed rather to be aimed at C programmers who
> want to define their own types. The most useful document (for me) seems
> to be the tutorial "Unifying types and classes in Python 2.2" at
> http://www.python.org/2.2/descrintro.html . If I haven't overlooked it
> that also doesn't mention the type class, does it?

The type class is nothing new in Python, it was always there.

Quote:
> - Where can I find more information on metaclass programming in Python?

At the moment: Use the source, Luke. Volunteers to write documentation
are highly encouraged to do so.

Quote:
> - (How) are the built-in function type and the class named "type" (that
>   didn't exist prior to Python 2.2?) related?

The builtin is not a function anymore, it is a type. It existed
before, as types.TypeType. Since types are callable now, many builtins
have changed from functions to types (like int, float, str).

Quote:
> - Should I be very concerned with metaclass programming?

No.

Quote:
>   On the other hand, I suppose that would not have been added to
>   Python if it weren't useful (contrasted to beeing merely
>   "nice"). Are metaclasses rather for experts?

The prime rationale for adding them was the ability to have types
implemented in C to be used as base classes for Python-defined
classes; this was not possible before. The prime rationale for that,
in turn, was the desire to get rid of the ExtensionClasses
infrastructure, in particular inside Zope.

So unless you have written ExtensionClasses before, you don't need to
worry about meta-programming. If you have written extension classes,
you should consider moving to 2.2 style types/classes ASAP, to let us
better understand how usable the existing API is.

Please expect that many things will change in the future in this area;
there is a desire to add "proper syntax" for many of the features
which are currently done through conventions only (e.g. __slots__,
__metaclass__).

Regards,
Martin



Fri, 25 Jun 2004 00:58:03 GMT  
 Style for overwritten methods of abstract classes
        ...

Quote:
> [Interesting code example of metaclass snipped]

> To be honest, I haven't understood everything of this but it inspired
> my curiosity regarding the "type" class. I didn't find anything about
> it in the Python 2.2 docs (though I won't say it's not there ;-) ).

I don't think the type type is yet covered in the docs -- I see it still
mentioned there as a built-in function (which it isn't any more; try:


Python 2.2 (#1, Dec 23 2001, 20:09:01)
[GCC 2.96 20000731 (Mandrake Linux 8.1 2.96-0.62mdk)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Quote:
>>> type(type)
<type 'type'>

> to be the tutorial "Unifying types and classes in Python 2.2" at
> http://www.python.org/2.2/descrintro.html . If I haven't overlooked it
> that also doesn't mention the type class, does it?

http://www.python.org/2.2/descrintro.html#metaclasses
"The built-in 'type' is the most common metaclass; it is the metaclass of
all built-in types."

Quote:
> - Where can I find more information on metaclass programming in Python?

http://www.python.org/doc/essays/metaclasses/

but it's not really recommended -- in 2.2, things are simpler, as explained
in the above-referenced paragraph of the descrintro.html.

Quote:
> - (How) are the built-in function type and the class named "type" (that
>   didn't exist prior to Python 2.2?) related?

Many entities that were built-in functions (acting as factories) up to 2.1
have become types in 2.2.  type, int, str ...

Quote:
> - Should I be very concerned with metaclass programming? I can't really
>   estimate how useful they are. On the other hand, I suppose that would
>   not have been added to Python if it weren't useful (contrasted to beeing
>   merely "nice"). Are metaclasses rather for experts?

Yes, they're still intended for experts, although they've become much
easier to use with 2.2 wrt 2.1.  I've never felt the need for actual use of
metaclasses in production code, myself... yet.  Most of what you can
accomplish with a metaclass, you can also do with clever metaprogramming in
__getattr__ and friends.  However, doing it in a metaclass has potential
advantages: the magic happens at class-construction time, rather than on
each attribute access -- so, errors may be diagnosed earlier (making them
easier to ferret out in tests, and easier to correlate with what's actually
wrong with the code) and performance overhead may be less.

However, rather than thinking of them as something that's been added to
Python, I see them as part of Python's general stance that *the underlying
mechanics are exposed* -- you don't HAVE to mess with them, but, if you
want, you can... they're very instructive to study and experiment with,
although using them in production code may not be appropriate.

Back when I was a motorcycle enthusiast, I carefully studied every detail
of my machine's "low-level" behavior.  I never actually needed to do any
serious maintenance myself (excellent motorcycle mechanics are not scarce
in the hometown of Ducati &c:-), but I felt the understanding and
familiarity with the "underlying mechanisms" made me a better driver
anyway, besides getting me ready should an emergency ever call for detailed
knowledge.  To be honest, these days I drive a Honda Civic (or, more often,
just ride a bus:-) and I have very vague ideas about what exactly keeps it
ticking in fine detail, just like most drivers (and bus passengers).  But
should I ever be bitten by the motorcycle bug again, I think that once
again I will appreciate having all underlying mechanisms detailed and
exposed... it's just a different mindset.  Not all bikers share it, mind
you -- it's optional.   You CAN approach a motorbike, just like you
approach a car, as a "black box" -- it's a very convenient and practical
vehicle even then, after all.  But, you do have the option of going deeper.

Quote:
> No, I don't like B&D programming. If I would, I probably wouldn't use
> Python but rather C++ or Java. ;-) My question aimed more at a simple
> or elegant approach. (Please don't be upset if you consider your
> suggestion simple and/or elegant. To be more specifically: the usage
> of the so defined class may be simple but its implementation not so.
> ;-) ).

I make no claims about the quality of the solution I proposed (actually
hacked together somewhat fast) and wouldn't be surprised if it could be
substantially improved.

The fundamental idea however does seem quite simple to me.  When a class
statement executes, the metaclass is instantiated, so the metaclass's
__init__ executes.  At that time, I want this metaclass-instance (this
class) to become callable (with the normal semantics, that of producing
class-instances) if and only if no abstract methods are left; otherwise,
calling the class must raise an exception with a suitable error-message
about what methods have not been overridden (have remained abstract).  All
I do in that metaclass is try to enforce this idea, in order to parallel or
mimic the "abstract classes" (non-instantiable classes) of certain other
languages.

The whole setup seems reasonably elegant to me within the specified mindset
of "must override" and "may override".  How else is a class going to be
able to constrain its subclasses regarding what may or may not, must or
must not, be overridden?  I think the mindset itself is at fault, and the
normal Pythonic approach (let anybody override whatever, diagnose problems
at runtime only if conflicts arise) far superior.  But the ability to
constrain the 'may' and 'must' was the very essence of the thread, no?

One case where I think a similar use of metaclasses might be _productive_
is to emulate Haskell's typeclass construct.  In a Haskell typeclass,
'abstract' methods may be specified in apparently mutually-recursive terms,
for example (in Python syntax):

class OneOrMany:
    def doOne(self, one):
        self.doMany([one])
    def doMany(self, many):
        for one in many:
            self.doOne(one)

To implement a typeclass into an instantiable class, you have to override
some subset of its methods so that all dependency cycles are broken.  Here,
you might override either or both of doOne and doMany, otherwise the class
would not be implementable.  Now THIS is what I consider elegant: it
clearly specifies the intended semantic dependencies between the methods,
and doesn't arbitrarily define one as more fundamental than others, when
they're actually all on the same plane.  To translate this into a Python
metaclass, you'd have to automatically or explicitly identify the
dependencies (doOne->doMany, doMany->doOne), notice which dependencies are
broken by subclassing (overriding some subsets of the methods), and check
if the resulting dependency graph is acyclic -- otherwise, keep the class
non-instantiable until by further subclassing the graph does become acyclic.

Alex



Fri, 25 Jun 2004 01:37:27 GMT  
 Style for overwritten methods of abstract classes
Hello Alex

Quote:

> > - (How) are the built-in function type and the class named "type" (that
> >   didn't exist prior to Python 2.2?) related?

> Many entities that were built-in functions (acting as factories) up to 2.1
> have become types in 2.2.  type, int, str ...

I don't know if this question is related but it appeared to me some
weeks ago: I can define __int__ etc. in my classes to provide an
"opportunity" for int() to "convert" an instance of my class into an
int. I have not seen something like this for e. g. __list__, so I could
write something like

class ACollection:
    #...
    def __list__(self):
        '''Return a list object representing the contents of this object.'''
        # ...

obj = ACollection()
# do something with obj
L = list(obj)

Is it anyhow possible? Or, first, is it desirable and/or why was/is it
omitted from Python? (It's not that it hurts me, but to me it's an
asymmetry.)

Quote:
> > No, I don't like B&D programming. If I would, I probably wouldn't use
> > Python but rather C++ or Java. ;-) My question aimed more at a simple
> > or elegant approach. (Please don't be upset if you consider your
> > suggestion simple and/or elegant. To be more specifically: the usage
> > of the so defined class may be simple but its implementation not so.
> > ;-) ).

> I make no claims about the quality of the solution I proposed (actually
> hacked together somewhat fast) and wouldn't be surprised if it could be
> substantially improved.

> The fundamental idea however does seem quite simple to me.  When a class
> statement executes, the metaclass is instantiated, so the metaclass's
> __init__ executes.  At that time, I want this metaclass-instance (this
> class) to become callable (with the normal semantics, that of producing
> class-instances) if and only if no abstract methods are left; otherwise,
> calling the class must raise an exception with a suitable error-message
> about what methods have not been overridden (have remained abstract).  All
> I do in that metaclass is try to enforce this idea, in order to parallel or
> mimic the "abstract classes" (non-instantiable classes) of certain other
> languages.

> The whole setup seems reasonably elegant to me within the specified mindset
> of "must override" and "may override".  How else is a class going to be
> able to constrain its subclasses regarding what may or may not, must or
> must not, be overridden?  I think the mindset itself is at fault, and the
> normal Pythonic approach (let anybody override whatever, diagnose problems
> at runtime only if conflicts arise) far superior.  But the ability to
> constrain the 'may' and 'must' was the very essence of the thread, no?

Don't worry. :-) I did understand this from your previous posting.
I just could not figure out the details of the implementation. If I
hadn't understood your goals or your approach, I wouldn't have found
so much interest in it that I had asked those questions in response. :-)

- Show quoted text -

Quote:
> One case where I think a similar use of metaclasses might be _productive_
> is to emulate Haskell's typeclass construct.  In a Haskell typeclass,
> 'abstract' methods may be specified in apparently mutually-recursive terms,
> for example (in Python syntax):

> class OneOrMany:
>     def doOne(self, one):
>         self.doMany([one])
>     def doMany(self, many):
>         for one in many:
>             self.doOne(one)

> To implement a typeclass into an instantiable class, you have to override
> some subset of its methods so that all dependency cycles are broken.  Here,
> you might override either or both of doOne and doMany, otherwise the class
> would not be implementable.  Now THIS is what I consider elegant: it
> clearly specifies the intended semantic dependencies between the methods,
> and doesn't arbitrarily define one as more fundamental than others, when
> they're actually all on the same plane.  To translate this into a Python
> metaclass, you'd have to automatically or explicitly identify the
> dependencies (doOne->doMany, doMany->doOne), notice which dependencies are
> broken by subclassing (overriding some subsets of the methods), and check
> if the resulting dependency graph is acyclic -- otherwise, keep the class
> non-instantiable until by further subclassing the graph does become acyclic.

Yes, I think I understand. :-)

On the other hand, I don't know if I would really like it. C++ does so
many things automatically that it's not so easy to keep all these rules
in mind and consider them when it's necessary. I like the simplicity of
Python. :-)

Stefan



Fri, 25 Jun 2004 07:05:20 GMT  
 Style for overwritten methods of abstract classes

Quote:
> I don't know if this question is related but it appeared to me some
> weeks ago: I can define __int__ etc. in my classes to provide an
> "opportunity" for int() to "convert" an instance of my class into an
> int. I have not seen something like this for e. g. __list__, so I could
> write something like

> [...sample omitted...]

> Is it anyhow possible?

Well, yes...  If you implement __getitem__() or __iter__(),
then list() and dict() will use those.  For example,

  import os

  class Dir:
      def __init__(self, path):
          self.path = path
      def __iter__(self):
          return iter(os.listdir(self.path))

  d = Dir("c:\\")
  x = list(d)
  print x

Of course, hypothetical __list__() or __dict__() methods
would have been another possible design.  But I think the
existing behavior is precisely what you'd normally want.

I've never used __int__() or __float__(); I use __str__()
all the time, but mostly for debug reasons.

## Jason Orendorff    http://www.jorendorff.com/



Fri, 25 Jun 2004 13:19:22 GMT  
 Style for overwritten methods of abstract classes

    ...

Quote:
> I don't know if this question is related but it appeared to me some
> weeks ago: I can define __int__ etc. in my classes to provide an
> "opportunity" for int() to "convert" an instance of my class into an
> int. I have not seen something like this for e. g. __list__, so I could

Define __iter__ instead: that's what list(x) calls, as well as any
"for item in x" loops.  It makes no difference for list(), but, for
loops, it may be a crucial saving of resources to be able to generate
the items one by one, each only as needed, rather than having to
build the whole list in memory at the same time.

Quote:
> On the other hand, I don't know if I would really like it. C++ does so
> many things automatically that it's not so easy to keep all these rules
> in mind and consider them when it's necessary. I like the simplicity of
> Python. :-)

I agree that C++'s utter complexity is C++'s key defect, and that much of
the complexity comes from the desire to offer "convenient" automatic/default
behavior in so many different and slightly overlapping cases.  And I share
your appreciation for the simplicity achieved with Python.

But I don't see how these consideration apply to Haskell-ish typeclasses
(which are not offered in C++, anyway).  Isn't their superiority over C++'s
"abstract classes" rather obvious?  And what diminution of simplicity do
you see in Haskell typeclasses when compared to C++ abstract classes?

In a C++ abstract class you basically *have* to over-constrain -- decide
at abstract-class-design time which methods are "more fundamental" than
others, when in fact there may be no such constraint in the abstract
system you're modeling.  Where's the simplicity gain in this?

Alex



Fri, 25 Jun 2004 16:56:36 GMT  
 Style for overwritten methods of abstract classes
Hi Alex

["substitute" for non-existing __list__]

Quote:
> Define __iter__ instead: that's what list(x) calls, as well as any
> "for item in x" loops.  It makes no difference for list(), but, for
> loops, it may be a crucial saving of resources to be able to generate
> the items one by one, each only as needed, rather than having to
> build the whole list in memory at the same time.

hm, rather flexible :-)

Quote:
> But I don't see how these consideration apply to Haskell-ish typeclasses
> (which are not offered in C++, anyway).  Isn't their superiority over C++'s
> "abstract classes" rather obvious?  And what diminution of simplicity do
> you see in Haskell typeclasses when compared to C++ abstract classes?

I didn't want to compare Haskell with C++. Indeed, I know nothing about
Haskell. It might have been an unconscious reflex to "compare" them
because of the association by doing things implicitly. Since I do know
practically nothing about Haskell, I clearly can't judge about it. :-)

The interaction of the method-for-one and the method-for-many though is
really interesting because the two are often (or often enough to trigger
a feeling of familarity) needed both.

Stefan



Sat, 26 Jun 2004 03:36:09 GMT  
 
 [ 18 post ]  Go to page: [1] [2]

 Relevant Pages 

1. Update method in new style classes

2. New-style classes and special methods

3. parser API style - methods take path or file vs sep methods for path and file

4. new-style classes VS classic-classes

5. method gets overwritten?!

6. accessing a instance method from a class method

7. How do I call a method from another method within the same class

8. Class methods vs Instance methods

9. Class method vs instance method???

10. Class method vs instance method???

11. Class methods in ab???.clw files and derived methods

12. same method as class and instance methods?

 

 
Powered by phpBB® Forum Software