Data Replication 
Author Message
 Data Replication

Anyone know of any case studies or articles on how others have
implemented data replication systems?  I've read articles on the
issues that need addressing and the technical tools available, but
have seen nothing on actual systems implemented.

I'm specifically referring to business type questions probably best
explained with examples.  For example, I have a central database with
many clients.  I would need to prevent remote users from re-adding a
client that already exists.  Client names are not good duplicate
checks as clients may actually have the same name or typos can occur.
Also, one remote user updates a client's address, and another updates
it as well with different abbreviations.  How have others resolved
such types of conflicting data issues?

Just curious as to what others have done.

Thanks...Brett



Mon, 16 Nov 1998 03:00:00 GMT  
 Data Replication

Sorry, this may be a repeat, but I got an error on my transmission.

Anyone know of any case studies or articles on how others have
implemented data replication systems?  I've read articles on the
issues that need addressing and the technical tools available, but
have seen nothing on actual systems implemented.

I'm specifically referring to business type questions probably best
explained with examples.  For example, I have a central database with
many clients.  I would need to prevent remote users from re-adding a
client that already exists.  Client names are not good duplicate
checks as clients may actually have the same name or typos can occur.
Also, one remote user updates a client's address, and another updates
it as well with different abbreviations.  How have others resolved
such types of conflicting data issues?

Just curious as to what others have done.

Thanks...Brett



Tue, 17 Nov 1998 03:00:00 GMT  
 Data Replication

Quote:

>Newsgroups: comp.databases.xbase.fox
>Subject: Data Replication
>Anyone know of any case studies or articles on how others have
>implemented data replication systems?
>For example, I have a central database with
>many clients.  I would need to prevent remote users from re-adding a
>client that already exists.  Client names are not good duplicate
>checks as clients may actually have the same name or typos can occur.

The problems of duplicate entries or conflicting updates seem
orthogonal to the problems associated with data replication; two
users in a non-replicated environment might just as well attempt to
create duplicate clients.

As to duplicate entries, different application will have different
criteria for detecting duplicates. To really solve the problem, each
entry must contain something unique, such as a Social Security number,
or a client account number, and duplicate insertion attempts are rejected
on the spot.

Of course, sometimes a unique ID is simply unavailable. For many name 'n
address applications a reasonable compromise is to do these comparisons
declaring a match if all compare exactly equal, case insensitive:
 - the first and last tokens of the name
 - the zip code
 - the first numeric token of the street address

Not perfect, but this is quick, and easy to code. It catches
neary all duplications. It falsely matches on same name, same street
number, same zip: rare enough that most can live with it. For most
lists of "free-form" addresses, additional checks will result in
noticeably more misses of real duplicates.

One could point out that detection of fraud or other deliberate
attempts to fool the application require a far more sophisticated
approach, usually when there's money to be stolen by getting
duplicates slipped into the data.

Quote:
>Also, one remote user updates a client's address, and another updates
>it as well with different abbreviations.  How have others resolved
>such types of conflicting data issues?

The last one wins?  It works well enough for a large part of the world.

The point of the above is that both updates are applied against the
same record, and that one tries to avoid consideration of abreviations,
middle names and initials, etc in determining sameness.

Dan Hepner
Not a statement of the Hewlett Packard Company



Wed, 18 Nov 1998 03:00:00 GMT  
 Data Replication

Access 7 already has built-in some form of replication. I'm checking this
with Visual FP myself and will let you know if something comes up.
Right now I'm using Sybase SQL Anywhere through ODBC as database. Please
let me know if you hear of something from your side.

Jorge Ferreira



Wed, 18 Nov 1998 03:00:00 GMT  
 Data Replication


Quote:

>Anyone know of any case studies or articles on how others have
>implemented data replication systems?  I've read articles on the
>issues that need addressing and the technical tools available, but
>have seen nothing on actual systems implemented.

>I'm specifically referring to business type questions probably best
>explained with examples.  For example, I have a central database with
>many clients.  I would need to prevent remote users from re-adding a
>client that already exists.  Client names are not good duplicate
>checks as clients may actually have the same name or typos can occur.
>Also, one remote user updates a client's address, and another updates
>it as well with different abbreviations.  How have others resolved
>such types of conflicting data issues?

>Just curious as to what others have done.

>Thanks...Brett

The only luck I've had is to produce a pick list that sorts on name,
address, zip, etc. then the user clicks in the field, then letting the
user look thru the list before deciding to add the new record.  They
would first click on last name & check there, then click on address &
check there, etc.  That cut out our duplication by over 95%.

Bill



Fri, 20 Nov 1998 03:00:00 GMT  
 Data Replication

Quote:

>  Sorry, this may be a repeat, but I got an error on my transmission.

>  Anyone know of any case studies or articles on how others have
>  implemented data replication systems?  I've read articles on the
>  issues that need addressing and the technical tools available, but
>  have seen nothing on actual systems implemented.

>  I'm specifically referring to business type questions probably best
>  explained with examples.  For example, I have a central database with
>  many clients.  I would need to prevent remote users from re-adding a
>  client that already exists.  Client names are not good duplicate
>  checks as clients may actually have the same name or typos can occur.
>  Also, one remote user updates a client's address, and another updates
>  it as well with different abbreviations.  How have others resolved
>  such types of conflicting data issues?

>  Just curious as to what others have done.

>  Thanks...Brett

Brett-

Ah yes, the age old question of validation......

Two thoughts come to mind:

1) allow your system to include 'branches' in the sense of CLIENT ABC - DOWNTOWN, CLIENT ABC - EAST SIDE etc.  The users will (gaurenteed) enter incorrect
data with typos and stuff but you can just deal with it during the update.

2) have the remote system use a lookup feature of valid company names, if not found in the list (assuming they look up correctly) then anything not found in the list is
new. Of course this means keeping each remote site updated with a master list - a major pain.

Good luck,

Matthew S. Jarvis

c/o Eagle Crest MIS
    821 S. 6th Street
    Redmond, Oregon  97756
    541/923-0807



Fri, 20 Nov 1998 03:00:00 GMT  
 Data Replication

Quote:

> Anyone know of any case studies or articles on how others have
> implemented data replication systems?  I've read articles on the
> issues that need addressing and the technical tools available, but
> have seen nothing on actual systems implemented.

> I'm specifically referring to business type questions probably best
> explained with examples.  For example, I have a central database with
> many clients.  I would need to prevent remote users from re-adding a
> client that already exists.  Client names are not good duplicate
> checks as clients may actually have the same name or typos can occur.
> Also, one remote user updates a client's address, and another updates
> it as well with different abbreviations.  How have others resolved
> such types of conflicting data issues?

> Just curious as to what others have done.

> Thanks...Brett

I always use the phone number field as my lookup to avoid dup's. It's not
perfect, but it's close and it's easy.

Scott M. Ritter



Fri, 20 Nov 1998 03:00:00 GMT  
 Data Replication

Quote:

> >Anyone know of any case studies or articles on how others have
> >implemented data replication systems?  I've read articles on the
> >issues that need addressing and the technical tools available, but
> >have seen nothing on actual systems implemented.

Sounds like you'll have to do some sort of seek on the customer's surname.
Do a compare of both tables using the '$' operator. This might help you somewhat - not
an easy situation to resolve!

We are busy implementing a replication program and it is proving to be very, very
complicated. Basically every time a user makes a change to the table, the old and new
values are written to a 'Change' table. This table contains the old & new value, a
primary key indicator (a must!), and other stuff. This takes place at the field level.
IOW, any field change produces a record. Also, records that never change should be
avoided to go through this mechanism, as 1 _new_ record with 20 fields will generate 20
records. You can see how disastrous this could become with a largish update! IOW, rather
send the entire record if the record will never change.

Our agents download the 'new' or 'changed' information (constantly being updated) and
they then upload the changes they have made. Obviously if changes are made to the same
field (same record) then these have to be resolved using some rather {*filter*} coding
algorithms!

The above method will meet our needs, but is difficult to implement and maintain. I wish
a table write could 'trigger' a procedure automatically in FPW. Rest assured something
like this exists using SQL Server or another C/Server database!

I would really like to hear from anyone who has implemented something along these lines!

--
Jonathan Kimber

+27-21-4487400 (w)
+27-21-5543179 (h)

Disclaimer:
My views are my own and do not necessarily reflect the ideas or opinions
of my company.



Fri, 20 Nov 1998 03:00:00 GMT  
 Data Replication

: Anyone know of any case studies or articles on how others have
: implemented data replication systems?  I've read articles on the
: issues that need addressing and the technical tools available, but
: have seen nothing on actual systems implemented.

This is no small task. For a prior job, I had the task of normalizing
client information to find duplications. Basically, I had to go through
and somehow guess whether the name was either a company or client and
then manipulate them to follow a prescibed format. I remember the program I
wrote had about 30 zillion CASE statements - and even then, it was at
best 95% accurate. I was still left with finding a good number of
duplicates by hand.

If I were designing a system from scratch, I would use two tables instead
of one. One table would be my master client file; the second file would
hold the "aliases" that users input into the system. When the system
detects the presence of a new alias, it should attempt to locate it in
the master table (using as much normalization as possible of course). If
the program is unsuccessful, then the job is up to the central office to
either manually locate a client or post the alias as a new client.

Note, to the view of the users, the alias table *IS* the client table.
Since a client can have multiple aliases, each remote branch can continue
to use their preferred alias. There is no need to update their database
to reflect correct clients; nor is there need to tell the branches to
start using another client account instead.



Sun, 22 Nov 1998 03:00:00 GMT  
 Data Replication

hi.
I wrote a replicator about a year ago that works great for our remote
office.  I was thinking about making it a shareware product but never
took the time.  it's in fp2.6 but it's just procedural code so it should
port just fine.  It supports _field_ level replication, is optimized for
dial-up connections, creates logs and other neet stuff.  it even allows
you to rollback your databases to another point in time if something
goes wrong or you get corruption.

you are welcome to the code as long as you are not going to use it to
make money.  keep in mind that you will have to re-write almost every
screen in your app - replace all gathers and appends with functions.

you will also have to add a field to all of you databases to support the
new locking scheme.

sound like fun yet?

Quote:

> >  Sorry, this may be a repeat, but I got an error on my transmission.

> >  Anyone know of any case studies or articles on how others have
> >  implemented data replication systems?  I've read articles on the
> >  issues that need addressing and the technical tools available, but
> >  have seen nothing on actual systems implemented.

> >  I'm specifically referring to business type questions probably best
> >  explained with examples.  For example, I have a central database with
> >  many clients.  I would need to prevent remote users from re-adding a
> >  client that already exists.  Client names are not good duplicate
> >  checks as clients may actually have the same name or typos can occur.
> >  Also, one remote user updates a client's address, and another updates
> >  it as well with different abbreviations.  How have others resolved
> >  such types of conflicting data issues?

> >  Just curious as to what others have done.

> >  Thanks...Brett

--
Go Climb a Mountain! __/\
Mark.             __/:/. \_



Tue, 24 Nov 1998 03:00:00 GMT  
 
 [ 11 post ] 

 Relevant Pages 

1. Data Replication

2. Data Replication

3. Automated Replication for FoxPro?

4. Replication Time Delay Between Publisher & Distributer

5. VFP 6.0/SQL 7.0 Replication & Reporting Application

6. Replication with Foxpro

7. replication

8. Replication

9. Foxpro replication across TCP/IP?

10. SQL to Foxpro ODBC Replication?

11. Replication and synchronizaton of tables over the web.

12. Replication

 

 
Powered by phpBB® Forum Software