Reading a comma delimited file 
Author Message
 Reading a comma delimited file

I am creating a system to load existing flat files that are not "truly"
comma delimited from the web onto a server, run a macro on them and "fix"
them, and then dump the data into a database.  I've got the files coming
over, running the macro ok, and the data looks good.  I am getting the data
out and trying to get it into a list, tuple structure in order to use the
executemany() call in ODBC.  Here's my code so far:

 dbfile = open(newname,'r')
  table = []
  lines = dbfile.readline()
  newtable = []
  for line in lines:   *Split the .csv file into rows and take off the
newline character
   line = line[:-1]
   r = string.split(line,',')
   table.append(r)
  for x in table:    *Take each list of strings within the list and change
the numbers from strings to dump into the table
   t=[]
   for all in x:
    if all.isalpha() == 0:
     num = float(all)
    else:
     num = all
   t.append(num)
   newtable.append(tuple(t))  **Make a new table with the list, tuple
structure

Here's what it looks like when I'm done:

[('Cotulla',  28.449999999999999,  99.216700000000003,  1960.0,  2.0,  11.1,
20.0,  0.0,  167.30000000000001),
 ('Cotulla',  28.449999999999999,  99.216700000000003,  1960.0,  3.0,
6.0999999999999996,  13.9,  0.0,  239.0)]

Just what I need except I don't want all those decimal places.  I've tried
searching the help and I'm sure there is a very simple call to fix this.
Can someone help me out on this.  Also, I'm new at python and am not sure
this is the best way to accomplish my task, and any suggestions as to how I
might increase speed by doing it another way would be great.   One .csv file
could and many times will have up to 14000 rows of data to be dumped into
the database.

Thanks,
Kristen Zander



Mon, 26 Apr 2004 23:34:59 GMT  
 Reading a comma delimited file


Quote:
> Here's what it looks like when I'm done:

> [('Cotulla',  28.449999999999999,  99.216700000000003,  1960.0,
2.0,  11.1,
> 20.0,  0.0,  167.30000000000001),
>  ('Cotulla',  28.449999999999999,  99.216700000000003,  1960.0,
3.0,
> 6.0999999999999996,  13.9,  0.0,  239.0)]

> Just what I need except I don't want all those decimal places.

Look up 'format % values' formatting.

Terry J. Reedy



Tue, 27 Apr 2004 00:04:38 GMT  
 Reading a comma delimited file

    Kristen> Here's what it looks like when I'm done:

    Kristen> [('Cotulla',  28.449999999999999,  99.216700000000003,  1960.0,  2.0,  11.1,
    Kristen> 20.0,  0.0,  167.30000000000001),
    Kristen>  ('Cotulla',  28.449999999999999,  99.216700000000003,  1960.0,  3.0,
    Kristen> 6.0999999999999996,  13.9,  0.0,  239.0)]

    Kristen> Just what I need except I don't want all those decimal places.

You're being victimized by floating point (im)precision.  Details here:

    http://python.sourceforge.net/devel-docs/tut/node14.html
    http://www.python.org/cgi-bin/faqw.py?req=show&file=faq04.098.htp

--

http://www.mojam.com/
http://www.musi-cal.com/



Tue, 27 Apr 2004 00:08:32 GMT  
 Reading a comma delimited file
That cleared up alot becuase it was really becoming frustrating.

I just figured it out.

import fpformat
a = 97.33532356246
b = fpformat.fix(a,2)
print b

97.33

Great!  Thanks.


Quote:

>     Kristen> Here's what it looks like when I'm done:

>     Kristen> [('Cotulla',  28.449999999999999,  99.216700000000003,

1960.0,  2.0,  11.1,
Quote:
>     Kristen> 20.0,  0.0,  167.30000000000001),
>     Kristen>  ('Cotulla',  28.449999999999999,  99.216700000000003,
1960.0,  3.0,
>     Kristen> 6.0999999999999996,  13.9,  0.0,  239.0)]

>     Kristen> Just what I need except I don't want all those decimal
places.

> You're being victimized by floating point (im)precision.  Details here:

>     http://python.sourceforge.net/devel-docs/tut/node14.html
>     http://www.python.org/cgi-bin/faqw.py?req=show&file=faq04.098.htp

> --

> http://www.mojam.com/
> http://www.musi-cal.com/



Tue, 27 Apr 2004 01:42:19 GMT  
 Reading a comma delimited file
Well, I thought I had it but the last thing I did with the fpformat made
them string again.

Does the %f work just like in C?

%.2f,var


Quote:

>     Kristen> Here's what it looks like when I'm done:

>     Kristen> [('Cotulla',  28.449999999999999,  99.216700000000003,

1960.0,  2.0,  11.1,
Quote:
>     Kristen> 20.0,  0.0,  167.30000000000001),
>     Kristen>  ('Cotulla',  28.449999999999999,  99.216700000000003,
1960.0,  3.0,
>     Kristen> 6.0999999999999996,  13.9,  0.0,  239.0)]

>     Kristen> Just what I need except I don't want all those decimal
places.

> You're being victimized by floating point (im)precision.  Details here:

>     http://python.sourceforge.net/devel-docs/tut/node14.html
>     http://www.python.org/cgi-bin/faqw.py?req=show&file=faq04.098.htp

> --

> http://www.mojam.com/
> http://www.musi-cal.com/



Tue, 27 Apr 2004 01:46:56 GMT  
 Reading a comma delimited file

Quote:
> That cleared up alot becuase it was really becoming frustrating.

> I just figured it out.

> import fpformat
> a = 97.33532356246
> b = fpformat.fix(a,2)
> print b

> 97.33

> Great!  Thanks.

Oops. It probably won't matter, as you seem to have values which are very
close to 2DP anyway, but the example you give above would normally be
expected to be shown as 97.34 (because 97.33532356246 is closer to 97.34
than to 97.33).

The "%.2f" notation should provide this desired behavior, and indeed the
documentation for the fpformat module states: """The fpformat module defines
functions for dealing with floating point numbers representations in 100%
pure Python. Note: This module is unneeded: everything here could be done
via the % string interpolation operator."""

Quote:
>>> "%.2f" % 97.3353
'97.34'
>>> "%.2f" % 97.3333

'97.33'

look-after-the-places-and-the-decimals-will-look-after-themselves-ly y'rs  -
steve
--
http://www.holdenweb.com/



Tue, 27 Apr 2004 02:15:18 GMT  
 Reading a comma delimited file

    Kristen> Does the %f work just like in C?
    Kristen> %.2f,var

Close.  Try this:

    >>> val = 1.1
    >>> val
    1.1000000000000001
    >>> "%.2f" % val
    '1.10'

In the common case (the one most like C), it takes a string containing
typical %-format stuff on the left and a tuple of strings on the
right:

    >>> "%.2f %.3f %.4f" % (val, val, val)
    '1.10 1.100 1.1000'

The one-element case can be reduced to a string on the right as in my
first example.

The cooler case takes a dict on the right and slightly different
format characters on the left:

    >>> "%(val).2f %(val).3f %(val).4f" % {"val": val}
    '1.10 1.100 1.1000'

This is generally helpful where you are interpolating a large string
with many %-expressions and would find it difficult to match up the
%-format elements with the index positions of the tuple.

Normally, the values you want to interpolate are local variables, so
you see it often spelled as

    >>> "%(val).2f %(val).3f %(val).4f" % locals()
    '1.10 1.100 1.1000'

People have also developed tricks for interpolating with multiple
dictionaries at once and interpolating expressions on-the-fly.  Search
the recent archives of the list (last two months or so) for
"interpolat" to find example code.

Skip



Tue, 27 Apr 2004 02:18:05 GMT  
 Reading a comma delimited file

Quote:
> Here's what it looks like when I'm done:

> [('Cotulla',  28.449999999999999,  99.216700000000003,  1960.0,  2.0,  11.1,
> 20.0,  0.0,  167.30000000000001),
>  ('Cotulla',  28.449999999999999,  99.216700000000003,  1960.0,  3.0,
> 6.0999999999999996,  13.9,  0.0,  239.0)]

> Just what I need except I don't want all those decimal places.  

Are you not happy with the number of decimal places?

If so, play around with this:

Quote:
>>> x='1.234567'
>>> float(x)
1.234567
>>> import fpformat
>>> fpformat.fix(float(x),2)
'1.23'
>>> fpformat.fix(float(x),0)

'1'

Hope that helps...
--B



Tue, 27 Apr 2004 04:11:24 GMT  
 Reading a comma delimited file
import the math module, round, floor, ceil or do whatever you want to
get rid of the extraneous digits; cast to integer; also, you could
just using a good ole %d string replacement (you can specify number of
digits before and after the decimal point).  Basically, you want to
convert the way you see the result.  I'm not sure how you want the
numbers (rounded vs. floored, etc.) but look at it as nothing more
than a formatting issue and you should find your own way to make the
data appear exactly like you would like it.  What I would most likely
do is if 'num' is a number:
  num = int(math.round(num))
or the like.  Just a quick idea.  Good luck and play with the modules
and documentation if you still can't find what you want.
Quote:

> I am creating a system to load existing flat files that are not "truly"
> comma delimited from the web onto a server, run a macro on them and "fix"
> them, and then dump the data into a database.  I've got the files coming
> over, running the macro ok, and the data looks good.  I am getting the data
> out and trying to get it into a list, tuple structure in order to use the
> executemany() call in ODBC.  Here's my code so far:

>  dbfile = open(newname,'r')
>   table = []
>   lines = dbfile.readline()
>   newtable = []
>   for line in lines:   *Split the .csv file into rows and take off the
> newline character
>    line = line[:-1]
>    r = string.split(line,',')
>    table.append(r)
>   for x in table:    *Take each list of strings within the list and change
> the numbers from strings to dump into the table
>    t=[]
>    for all in x:
>     if all.isalpha() == 0:
>      num = float(all)
>     else:
>      num = all
>    t.append(num)
>    newtable.append(tuple(t))  **Make a new table with the list, tuple
> structure

> Here's what it looks like when I'm done:

> [('Cotulla',  28.449999999999999,  99.216700000000003,  1960.0,  2.0,  11.1,
> 20.0,  0.0,  167.30000000000001),
>  ('Cotulla',  28.449999999999999,  99.216700000000003,  1960.0,  3.0,
> 6.0999999999999996,  13.9,  0.0,  239.0)]

> Just what I need except I don't want all those decimal places.  I've tried
> searching the help and I'm sure there is a very simple call to fix this.
> Can someone help me out on this.  Also, I'm new at Python and am not sure
> this is the best way to accomplish my task, and any suggestions as to how I
> might increase speed by doing it another way would be great.   One .csv file
> could and many times will have up to 14000 rows of data to be dumped into
> the database.

> Thanks,
> Kristen Zander



Tue, 27 Apr 2004 04:54:52 GMT  
 
 [ 9 post ] 

 Relevant Pages 

1. VW code to read comma-delimited text files??

2. Fortran 77: read second number on a line in a comma delimited text file

3. Export Clarion .DAT files to ASCII comma delimited files

4. Export Clarion .DAT files to ASCII comma delimited files

5. getting fields NOT comma delimited with commas inside

6. Reading Comma delimited, Quoted String records

7. READ comma delimited data

8. Parsing Comma delimited files in J

9. matching records in a comma delimited file

10. Comma delimited file problem

11. 2.01 Comma delimited ASCII file

12. Import comma delimited text file

 

 
Powered by phpBB® Forum Software