VB.Net, Large Variable Arrays

Large Variable Arrays

Author	Message
Jonathan Alle #1 / 5	Large Variable Arrays If the data is processed sequentially and discretely, dump them into MSMQ Messages. If the data has to be accessed all at once, use database tables. Either way, you probably don't want 900 MB of data in memory at one time. -- Jonathan Allen Quote: > We are building an application that will have some very > large arrays (as much as 900MB of stuff) used in > intensive computations (run time as much as several > days). What is the best way to set up those arrays in > vb.net and is there a way that they can interop with VB6. > Thanks
Mon, 14 Jun 2004 15:53:26 GMT

Warre #2 / 5	Large Variable Arrays Thanks Jonathan, But you must have missed my statement that this data is used in intensive processing that can take several hours or days (even on a 2GHz Pentium 4 with a GB of memory). Keeping the data all in memory is the only way to have acceptable performance. And the data is processed randomly and accessing the data for processing via database tables would probably change the "several hours or days" to several years (with the data access overhead) - not acceptable. I was wondering if there was something such as data structures, etc that might be usuable for direct memory access among various vb.net programs. Thanks again. Warren Quote: > If the data is processed sequentially and discretely, dump them into MSMQ > Messages. > If the data has to be accessed all at once, use database tables. > Either way, you probably don't want 900 MB of data in memory at one time. > -- > Jonathan Allen > > We are building an application that will have some very > > large arrays (as much as 900MB of stuff) used in > > intensive computations (run time as much as several > > days). What is the best way to set up those arrays in > > vb.net and is there a way that they can interop with VB6. > > Thanks
Wed, 16 Jun 2004 00:37:13 GMT

Jonathan Alle #3 / 5	Large Variable Arrays Quote: > But you must have missed my statement that this data is used in intensive > processing that can > take several hours or days No I didn't, which is why I used the word "If" when discussing alternatives and the scenarios that made them applicable. Quote: > Keeping the data all in memory is the only way to have acceptable > performance. In that case, I would go for a C-only project. That way you can get the best possible performance from the system. I know this isn't what you were asking for, but I cannot recommend VB or VB.Net for this scenario. With a dataset that large, even the most trivial overhead can result in minutes or even hours of additional processing time. Though I fear I am being presumptuous, I can't help but wonder if there is a better way to handle your situation. Could you explain the process that leads up to the massive calculations. Perhaps someone in this newsgroup has experienced it before and can suggest an alternative architecture. -- Jonathan Allen Quote: > Thanks Jonathan, > But you must have missed my statement that this data is used in intensive > processing that can > take several hours or days (even on a 2GHz Pentium 4 with a GB of memory). > Keeping the data all in memory is the only way to have acceptable > performance. And the data is processed randomly and accessing the data for > processing via database tables would probably change the "several hours or > days" to several years (with the data access overhead) - not acceptable. I > was wondering if there was something such as data structures, etc that might > be usuable for direct memory access among various vb.net programs. > Thanks again. > Warren > > If the data is processed sequentially and discretely, dump them into MSMQ > > Messages. > > If the data has to be accessed all at once, use database tables. > > Either way, you probably don't want 900 MB of data in memory at one time. > > -- > > Jonathan Allen > > > We are building an application that will have some very > > > large arrays (as much as 900MB of stuff) used in > > > intensive computations (run time as much as several > > > days). What is the best way to set up those arrays in > > > vb.net and is there a way that they can interop with VB6. > > > Thanks
Wed, 16 Jun 2004 03:03:14 GMT

<colin> #4 / 5	Large Variable Arrays I have a 1Gb dataset of Logitude/Latitude information (decimal(18,10)) with some trig functions as calcs and I find It takes longer to build an array or populate/Walk the array than to process the data on the sql server where the data resides. I create Stored procedures or user defined functions on the sql-server that accept or calculate input parameters and return a result. as an example; the first time a calcs runs takes an Hour after which if this is repeated the same calc takes 20 minutes (sql server has optimised/Cached my request for me) This method, if you can utilise it would allow you to use any development that can execute stored procedues Colin Quote: > We are building an application that will have some very > large arrays (as much as 900MB of stuff) used in > intensive computations (run time as much as several > days). What is the best way to set up those arrays in > vb.net and is there a way that they can interop with VB6. > Thanks
Wed, 16 Jun 2004 22:47:53 GMT

Dale Roberson (Microso #5 / 5	Large Variable Arrays Considering the other responses already posted, there is nothing special to do set up the arrays. If your design requires arrays this large, then set them up and use them. Other than that, if you have specific concerns or problems, let us know. Thank You, Dale Roberson Microsoft Developer Support This posting is provided "AS IS" with no warranties, and confers no rights. You assume all risk for your use. ? 2001 Microsoft Corporation. All rights reserved.
Wed, 23 Jun 2004 04:13:59 GMT

Page 1 of 1

[ 5 post ]