Serialize/Persist large array 
Author Message
 Serialize/Persist large array

I have many 10 MB arrays of single data type that I wish to persist by
saving to the local hard drive.

I have benchmarked this using three separate ways in VB.NET SP2, but
the speed is at most only 20% of what I can achieve in VB6 using the
PUT function.

In VB.Net I have used

A) FilePut to write the entire array at once. (what I expected to be
fast)
B) Serialize to write the entire array as an object. (also should be
fast)
C) FilePut to write the array using a loop. (same speed as A!)

Please advise how to at least achieve the same levels of speed that
PUT gets (oops on the pun) in VB6.

Thanks,

Jaffray



Mon, 31 Jan 2005 00:19:26 GMT  
 Serialize/Persist large array

we will investigate the issue of FilePut in VB.Net when used for arrays,

in the mean time, to speed up your work, you could use something like the
following:

Module Module1
    Sub Main()
        Dim Ary() As Integer

        Ary = New Integer(2500000) {}

        WriteArrayToFile(Ary, "c:\temp4.bin")

        Ary = Nothing
        Ary = GetArrayFromFile("c:\temp4.bin")
    End Sub

    Sub WriteArrayToFile(ByVal Ary As Integer(), ByVal FileName As String)
        Dim str As IO.Stream = New IO.StreamWriter(FileName).BaseStream
        Dim formatter As New
System.Runtime.Serialization.Formatters.Binary.BinaryFormatter()
        formatter.Serialize(str, Ary)
        str.Close()
    End Sub

    Function GetArrayFromFile(ByVal FileName As String) As Integer()
        Dim str As IO.Stream = New IO.StreamReader(FileName).BaseStream
        Dim formatter As New
System.Runtime.Serialization.Formatters.Binary.BinaryFormatter()

        Dim temp As Integer() = formatter.Deserialize(str)
        str.Close()

        Return temp
    End Function
End Module

we noticed that this is actually about 2 times faster than even VB6 Put

You said you used serialization. In what way was it different from the above
?

thanks,
VB Team

--
This posting is provided "AS IS" with no warranties, and confers no rights.


Quote:
> I have many 10 MB arrays of single data type that I wish to persist by
> saving to the local hard drive.

> I have benchmarked this using three separate ways in VB.NET SP2, but
> the speed is at most only 20% of what I can achieve in VB6 using the
> PUT function.

> In VB.Net I have used

> A) FilePut to write the entire array at once. (what I expected to be
> fast)
> B) Serialize to write the entire array as an object. (also should be
> fast)
> C) FilePut to write the array using a loop. (same speed as A!)

> Please advise how to at least achieve the same levels of speed that
> PUT gets (oops on the pun) in VB6.

> Thanks,

> Jaffray



Mon, 31 Jan 2005 05:15:12 GMT  
 Serialize/Persist large array
Thanks for the quick response. First I want to preface this posting
with the following:

Other than I/O operations I have been *very* impressed with VB.NET for
my work. I have actually done considerable heavy memory-only
number-crunching benchmarking VB6 versus VB.NET and found VB.NET to be
1.25 to 4 times faster (depending on the specific task) after
optimizing both the VB6 code and the VB.NET code. (And I have much
more experience optimizing VB6 code...)

So, on the current topic of seraliazing reasonably large arrays...

I actually was using something similar to tbe VBDotNet team code to
benchmark the Serialize method in VB.NET. I benchmarked your code and
got the same results I have been getting with my initial benchmarks.

On my new Pentium 4, 2.4GHz system I ran benchmarks comparing VB.NET
to VB6 for writing 10MB arrays to disk.

With the code provide by the VBDotNet team using VB.NET with all
optimizations checked the same sized file (2.5 million .NET
*integers*) takes between 1 and 2 seconds to write. Why so inexact?
Well, for some reason it takes less time to do one iteration compared
to 2+ iterations. I don't have an explanation for that, it is
counter-intuitive to me. In the timing I even excluded everything but
the one line with the Serialize call.

On the same machine, writing to the same directory using the "Put"
function it takes VB6 0.28 seconds to write a 10MB file of 2.5 million
VB6 *longs*.

Well, my serialization code was somewhat different, but at this point
I don't believe it significant for me to differentiate between your
code and mine because the benchmarked results are virtually the
same....

Further input on this situation would be greatly appreciated.

-Jaffray


Quote:
> we will investigate the issue of FilePut in VB.Net when used for arrays,

> in the mean time, to speed up your work, you could use something like the
> following:

> Module Module1
>     Sub Main()
>         Dim Ary() As Integer

>         Ary = New Integer(2500000) {}

>         WriteArrayToFile(Ary, "c:\temp4.bin")

>         Ary = Nothing
>         Ary = GetArrayFromFile("c:\temp4.bin")
>     End Sub

>     Sub WriteArrayToFile(ByVal Ary As Integer(), ByVal FileName As String)
>         Dim str As IO.Stream = New IO.StreamWriter(FileName).BaseStream
>         Dim formatter As New
> System.Runtime.Serialization.Formatters.Binary.BinaryFormatter()
>         formatter.Serialize(str, Ary)
>         str.Close()
>     End Sub

>     Function GetArrayFromFile(ByVal FileName As String) As Integer()
>         Dim str As IO.Stream = New IO.StreamReader(FileName).BaseStream
>         Dim formatter As New
> System.Runtime.Serialization.Formatters.Binary.BinaryFormatter()

>         Dim temp As Integer() = formatter.Deserialize(str)
>         str.Close()

>         Return temp
>     End Function
> End Module

> we noticed that this is actually about 2 times faster than even VB6 Put

> You said you used serialization. In what way was it different from the above
> ?

> thanks,
> VB Team

> --
> This posting is provided "AS IS" with no warranties, and confers no rights.



> > I have many 10 MB arrays of single data type that I wish to persist by
> > saving to the local hard drive.

> > I have benchmarked this using three separate ways in VB.NET SP2, but
> > the speed is at most only 20% of what I can achieve in VB6 using the
> > PUT function.

> > In VB.Net I have used

> > A) FilePut to write the entire array at once. (what I expected to be
> > fast)
> > B) Serialize to write the entire array as an object. (also should be
> > fast)
> > C) FilePut to write the array using a loop. (same speed as A!)

> > Please advise how to at least achieve the same levels of speed that
> > PUT gets (oops on the pun) in VB6.

> > Thanks,

> > Jaffray



Tue, 01 Feb 2005 01:39:32 GMT  
 Serialize/Persist large array
Well, we were using a machine with a different configuration and that might
explain the difference between our results.
we will investigate this issue

thanks for bringing it to our attention,

thanks,
VB Team

--
This posting is provided "AS IS" with no warranties, and confers no rights.


Quote:
> Thanks for the quick response. First I want to preface this posting
> with the following:

> Other than I/O operations I have been *very* impressed with VB.NET for
> my work. I have actually done considerable heavy memory-only
> number-crunching benchmarking VB6 versus VB.NET and found VB.NET to be
> 1.25 to 4 times faster (depending on the specific task) after
> optimizing both the VB6 code and the VB.NET code. (And I have much
> more experience optimizing VB6 code...)

> So, on the current topic of seraliazing reasonably large arrays...

> I actually was using something similar to tbe VBDotNet team code to
> benchmark the Serialize method in VB.NET. I benchmarked your code and
> got the same results I have been getting with my initial benchmarks.

> On my new Pentium 4, 2.4GHz system I ran benchmarks comparing VB.NET
> to VB6 for writing 10MB arrays to disk.

> With the code provide by the VBDotNet team using VB.NET with all
> optimizations checked the same sized file (2.5 million .NET
> *integers*) takes between 1 and 2 seconds to write. Why so inexact?
> Well, for some reason it takes less time to do one iteration compared
> to 2+ iterations. I don't have an explanation for that, it is
> counter-intuitive to me. In the timing I even excluded everything but
> the one line with the Serialize call.

> On the same machine, writing to the same directory using the "Put"
> function it takes VB6 0.28 seconds to write a 10MB file of 2.5 million
> VB6 *longs*.

> Well, my serialization code was somewhat different, but at this point
> I don't believe it significant for me to differentiate between your
> code and mine because the benchmarked results are virtually the
> same....

> Further input on this situation would be greatly appreciated.

> -Jaffray




- Show quoted text -

Quote:

> > we will investigate the issue of FilePut in VB.Net when used for arrays,

> > in the mean time, to speed up your work, you could use something like
the
> > following:

> > Module Module1
> >     Sub Main()
> >         Dim Ary() As Integer

> >         Ary = New Integer(2500000) {}

> >         WriteArrayToFile(Ary, "c:\temp4.bin")

> >         Ary = Nothing
> >         Ary = GetArrayFromFile("c:\temp4.bin")
> >     End Sub

> >     Sub WriteArrayToFile(ByVal Ary As Integer(), ByVal FileName As
String)
> >         Dim str As IO.Stream = New IO.StreamWriter(FileName).BaseStream
> >         Dim formatter As New
> > System.Runtime.Serialization.Formatters.Binary.BinaryFormatter()
> >         formatter.Serialize(str, Ary)
> >         str.Close()
> >     End Sub

> >     Function GetArrayFromFile(ByVal FileName As String) As Integer()
> >         Dim str As IO.Stream = New IO.StreamReader(FileName).BaseStream
> >         Dim formatter As New
> > System.Runtime.Serialization.Formatters.Binary.BinaryFormatter()

> >         Dim temp As Integer() = formatter.Deserialize(str)
> >         str.Close()

> >         Return temp
> >     End Function
> > End Module

> > we noticed that this is actually about 2 times faster than even VB6 Put

> > You said you used serialization. In what way was it different from the
above
> > ?

> > thanks,
> > VB Team

> > --
> > This posting is provided "AS IS" with no warranties, and confers no
rights.



> > > I have many 10 MB arrays of single data type that I wish to persist by
> > > saving to the local hard drive.

> > > I have benchmarked this using three separate ways in VB.NET SP2, but
> > > the speed is at most only 20% of what I can achieve in VB6 using the
> > > PUT function.

> > > In VB.Net I have used

> > > A) FilePut to write the entire array at once. (what I expected to be
> > > fast)
> > > B) Serialize to write the entire array as an object. (also should be
> > > fast)
> > > C) FilePut to write the array using a loop. (same speed as A!)

> > > Please advise how to at least achieve the same levels of speed that
> > > PUT gets (oops on the pun) in VB6.

> > > Thanks,

> > > Jaffray



Tue, 01 Feb 2005 03:52:11 GMT  
 Serialize/Persist large array
VBDotNet Team,

Please post your progress. This is a very serious issue with VB.Net.

You said that you would investigate the speed of FilePut for writing
large arrays.

You also said that you would investigate the speed of using the
Serialize Method with large arrays.

It would be ridiculous for VB6 to outperform VB.Net writing large
arrays to the local disk, especially if the difference is a factor of
five as I have very consistently found to be case!



Wed, 09 Feb 2005 00:44:48 GMT  
 
 [ 5 post ] 

 Relevant Pages 

1. How to serialize (persist) usercontrol

2. Discussion: Persisting Large classes to disk

3. Managing/processing a large number of large arrays

4. Freeing memory from large array 2

5. Freeing memory from large Array

6. Large Variable Arrays

7. Large arrays

8. Help: Loading very large arrays from disk (quickbasic4.5)

9. Very Large Arrays

10. How to edit large array of data

11. Large Arrays!

12. Large arrays ?

 

 
Powered by phpBB® Forum Software