ANN: experimental patch to allow importing from file-like objects 
Author Message
 ANN: experimental patch to allow importing from file-like objects

Motivation: Write a customized import that reads data from encrypted
files. Both execfile() and exec() are limited, if only that that they
don't mimic the import syntax.

Problem: imp.load_module does expect a file object, but you can only
pass "real" files, not file-like objects. The internal C-code expects
native FILE* stuff.

Solution: An experimental patch to the python22.dll that makes this
possible. Using file-like objects for import opens up a whole new set
of possibilities, for example you could fetch the code at runtime from
a database or a socket.

Limitations:
- Currently, Win32 only.
- Currently, you can only import source from file-like objects, not
precompiled binaries.
- Currently, error handling is completely undefined.

Sample Code:

import imp, os, rotor

# sample code ***************************************
source = """
print 'Inside encrypted code'
variable = 123
"""
# generate encrypted file ***************************
file = open("test.raw","wb")
rt = rotor.newrotor('key', 12)
file.write( rt.encrypt(source) )
file.close()

# sample file class, that reads data from encrypted file
class my_file_class:
    def __init__(self,filename):
        rt = rotor.newrotor('key', 12)
        self.lines = rt.decrypt(open(filename).read()).split('\n')
        print "read", len(self.lines), "lines from encrypted file"

    def read(self,size):
        if self.lines:
            line = self.lines[0]
            # must make sure line ends with \n
            if not line:
                line = '\n'
            elif not (line[-1] == '\n'):
                line += '\n'
            self.lines = self.lines[1:]
            return line

# redirect import hook ******************************
old_import_func = __import__

def my_import_func(name, globals=None, locals=None, fromlist=None):
    global old_import_func

    if name == "encrypted_test":
        return imp.load_module(name,
                               my_file_class("test.raw"),
                               "test.raw",
                               ('.raw','rb',imp.PY_FILEOBJECT_SOURCE))

    # use original code    
    return old_import_func( name, globals, locals, fromlist )

__builtins__.__import__ = my_import_func

# test code
import encrypted_test
print encrypted_test.variable

Download:
http://www.*-*-*.com/

More technical details:
http://www.*-*-*.com/ (at the bottom)

Now what? PEP-stuff? Or completely uninteresting?



Wed, 18 Aug 2004 02:50:34 GMT  
 ANN: experimental patch to allow importing from file-like objects

Quote:

> Motivation: Write a customized import that reads data from encrypted
> files. Both execfile() and exec() are limited, if only that that they
> don't mimic the import syntax.

> Problem: imp.load_module does expect a file object, but you can only
> pass "real" files, not file-like objects. The internal C-code expects
> native FILE* stuff.

> Solution: An experimental patch to the python22.dll that makes this
> possible. Using file-like objects for import opens up a whole new set
> of possibilities, for example you could fetch the code at runtime from
> a database or a socket.

Most people doing something like this don't use imp.load_module.

First you need a code object for the module. If it's source:

  co = compile(open(srcfile, 'r').read()+'\n', srcfile, 'exec')

If it's already compiled (and still fresh etc.)

  stuff = open(bytecodefile, 'rb').read()
  co = marshal.loads(stuff[8:])

Then you need a module object

  mod = imp.new_module(nm)

(You need to set up mod.__file__, mod.__name__ etc. here).
And finally

  sys.modules[fqname] = mod
  exec co in mod.__dict__

This is not to say that I'm against your patch (didn't look
at it), just that there's another way around the problem.

-- Gordon
http://www.mcmillan-inc.com/



Wed, 18 Aug 2004 20:30:42 GMT  
 ANN: experimental patch to allow importing from file-like objects


Quote:
>Most people doing something like this don't use imp.load_module.

>First you need a code object for the module. If it's source:

>  co = compile(open(srcfile, 'r').read()+'\n', srcfile, 'exec')

>If it's already compiled (and still fresh etc.)

>  stuff = open(bytecodefile, 'rb').read()
>  co = marshal.loads(stuff[8:])

>Then you need a module object

>  mod = imp.new_module(nm)

>(You need to set up mod.__file__, mod.__name__ etc. here).
>And finally

>  sys.modules[fqname] = mod
>  exec co in mod.__dict__

Hey, thats' cool! I didn't know that.

I must say, though, that the documentation on this in the python
reference manual is *quite* rudimentary. Well, at the very least I
learned something about python internals and PyTokenizer ;)

But I still feel that the python core should be more "pythonesque".

There are a gazillion (ok, more like 40) places in ./Include, that use
a "FILE *" as argument. As this is the external interface of python, I
think it would be better if it expected PyFile-like objects instead.
For example, look at marshal.dump(). You cannot pass a file-like
object if you wanted to (because, alas, the implementation expects
FILE*). You have to use dumps() and write that.

Is this just a decision dictated by "the need for speed", or is there
any motivation behind it?

On a related note, (I've asked this before to no avail) - how do I
successfully subclass files & open them?

The problem is - where to I subclass a file returned from open()?
open() always returns the builtin file class. Attempting to do this

----------- (snip here) -------------

raw_file = open(...)
my_file = my_file_class()
raw_file.read = my_file.read

----------- (snip here) -------------

I'll get an error that read is a read-only method.



Thu, 19 Aug 2004 03:25:17 GMT  
 ANN: experimental patch to allow importing from file-like objects

Quote:

> On a related note, (I've asked this before to no avail) - how do I
> successfully subclass files & open them?

> The problem is - where to I subclass a file returned from open()?
> open() always returns the builtin file class. Attempting to do this

> ----------- (snip here) -------------

> raw_file = open(...)
> my_file = my_file_class()
> raw_file.read = my_file.read

> ----------- (snip here) -------------

> I'll get an error that read is a read-only method.

I don't know much about 2.2 yet, but I think the
preferred method is subclassing file, then opening a
file through the subclass's constructor (rather than
through open()). For example:

class NoisyFile(file):
    def __init__(self, *args, **kwargs):
        file.__init__(self, *args, **kwargs)
        print "Bwahahaha!"

f = NoisyFile("c:/python/test/2.2/test-file-1.py", "r")
print f.readlines()
f.close()

HTH,

--
Hans (base64.decodestring('d3VybXlAZWFydGhsaW5rLm5ldA=='))
# decode for email address ;-)
The Pythonic Quarter:: http://www.awaretek.com/nowak/



Thu, 19 Aug 2004 09:32:12 GMT  
 ANN: experimental patch to allow importing from file-like objects

Quote:


[...]
>I don't know much about 2.2 yet, but I think the
>preferred method is subclassing file, then opening a
>file through the subclass's constructor (rather than
>through open()). For example:

open *is* file. (Just take a closer look.)

So you should subclass file (i.e. open) -- and of course you'd then
use the constructor of that class, not the constructor of your
superclass (which would be file or open, which are one and the same).

--
Magnus Lie Hetland                                  The Anygui Project
http://hetland.org                                  http://anygui.org



Sat, 21 Aug 2004 01:05:44 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. experimental patch to allow importing from file-like objects

2. patch to allow dump/load to work on DATA objects

3. Patch allowing filechooser to select multiple files

4. experimental patch for poplog for v15.53 on Redhat 9

5. ANN: eGenix.com mx EXPERIMENTAL Package 0.7.0

6. ANN: eGenix.com mx EXPERIMENTAL Package 0.6.0

7. ANN: eGenix.com mx EXPERIMENTAL Package 0.5.0

8. ANN: eGenix.com mx EXPERIMENTAL Package 0.4.0

9. ANN: eGenix.com mx EXPERIMENTAL Extension Package 0.3.0

10. ANN: mxNumber -- Experimental Number Types, Version 0.2.0

11. ANN: rjab-connection 0.1.0 experimental

12. ANN: eGenix.com mx EXPERIMENTAL Package 0.7.0

 

 
Powered by phpBB® Forum Software