[PATCH] Conflict between several SAME/SIMILAR mappings (was: Fake conflict)

[tuju]

I run msynctool normally and it falls to --slow-sync.

During the slow sync, it detects a conflict and rises it to the user to decide what should be done with it, even the two cards are exactly the same:

Entry 2:
Member: 1 (file-sync)
UID: 5DAIcMTISl
<?xml version="1.0"?>
<contact>
  <EMail Type="Internet">
    <Content>doe@example.com</Content>
  </EMail>
  <FormattedName>
    <Content>John Doe</Content>
  </FormattedName>
  <Name>
    <LastName>Doe</LastName>
    <FirstName>John</FirstName>
  </Name>
  <Organization>
    <Name>Example O&#xFC;</Name>
    <Unit></Unit>
  </Organization>
  <Revision>
    <Content>20081103T092103Z</Content>
  </Revision>
  <Role>
    <Content>Product Manager</Content>
  </Role>
  <Telephone Location="Work">
    <Content>+372123456</Content>
  </Telephone>
  <Telephone Type="Cellular">
    <Content>+3725551234</Content>
  </Telephone>
  <Url>
    <Content>http://www.example.com/</Content>
  </Url>
</contact>

Entry 1:
Member: 2 (syncml-obex-client)
UID: 5                        
<?xml version="1.0"?>         
<contact>                     
  <EMail Type="Internet">     
    <Content>doe@example.com</Content>
  </EMail>                                
  <FormattedName>                         
    <Content>John Doe</Content>      
  </FormattedName>                        
  <Name>                                  
    <LastName>Doe</LastName>          
    <FirstName>John</FirstName>          
  </Name>                                 
  <Organization>                          
    <Name>Example O&#xFC;</Name>        
    <Unit></Unit>                         
  </Organization>                         
  <Revision>                              
    <Content>20081103T092103Z</Content>   
  </Revision>                             
  <Role>                                  
    <Content>Product Manager</Content>    
  </Role>                                 
  <Telephone Location="Work">             
    <Content>+372123456</Content>        
  </Telephone>                            
  <Telephone Type="Cellular">
    <Content>+3725551234</Content>
  </Telephone>
  <Url>
    <Content>http://www.example.com/</Content>
  </Url>
</contact>

Which entry do you want to use? [1-9] To select a side, [D]uplicate:

This ticket is based on true story. Real names were changed with vim editor and no animals were hurt during the composition of this ticket.

[tuju]

This basically prevents using the opensync -> major severity.

[dgollub]

I can't reproduce this issue.

Could you try to reproduce this issues with export OSYNC_NOPRIVACY=1. It would be also helpful to get this reproduce with some dummy-contacts. Your last trace files contained 96 changes which were candidate for a conflict.

[chi70]

I was able to reproduce this using osynctool (r3860) with OSYNC_NOPRIVACY=1. As this was done on real data these traces are not attached to the bug. Data was send to Dgollub.

[chi70]

I did some more tests. It does not happen with a file-file sync using the data synced from syncml-obex-client to file.

At the moment this action duplicates one of my contacts and I assume these messages are the cause of the duplication:

Received an entry 99 from member 1 (syncml-obex-client). Changetype ADDED
element Url: Schemas validity error : Element 'Url', attribute 'Location': The attribute 'Location' is not allowed.
ERROR: XMLFormat validation failed.
EXIT_ERROR: osync_converter_invoke: XMLFormat validation failed.
EXIT_ERROR: osync_format_env_convert: XMLFormat validation failed.
Main sink of member 1 of type syncml-obex-client had an error: XMLFormat validation failed.
EXIT_ERROR: _osync_engine_receive_change: XMLFormat validation failed.
Received an entry 101 from member 1 (syncml-obex-client). Changetype ADDED

Entry 99 gets duplicated.

I do not see anything unusual with that entry from syncml-ds-tool dumping the database.

[dgollub]

Replying to chi70:

I did some more tests. It does not happen with a file-file sync using the data synced from syncml-obex-client to file.

file2file contains the same set of capabilities.

At the moment this action duplicates one of my contacts and I assume these messages are the cause of the duplication:

Received an entry 99 from member 1 (syncml-obex-client). Changetype ADDED
element Url: Schemas validity error : Element 'Url', attribute 'Location': The attribute 'Location' is not allowed.
ERROR: XMLFormat validation failed.
EXIT_ERROR: osync_converter_invoke: XMLFormat validation failed.
EXIT_ERROR: osync_format_env_convert: XMLFormat validation failed.
Main sink of member 1 of type syncml-obex-client had an error: XMLFormat validation failed.
EXIT_ERROR: _osync_engine_receive_change: XMLFormat validation failed.
Received an entry 101 from member 1 (syncml-obex-client). Changetype ADDED

Entry 99 gets duplicated.

xmlformat validation issue are vformat conversion bugs.

I do not see anything unusual with that entry from syncml-ds-tool dumping the database.

Hehe, that's quite easy: syncml-ds-tool dosen't drive mapping or does format conversion... it's just dumping the data.

[dgollub]

I found some interesting things in the Trace files from Tuju.

There are entries which got mapped osync_xmlformat_compare() two times! The first time was a mapping which resulted in to a 'SIMILAR' match (which would cause the conflict).

The second time was a mapping which resulted in to a 'SAME' match (which would explain the 'fake' conflict - since the entry was really the same).

Talking to Tuju about the setup turned out that the resource very likely contains very likely several "duplicates" (entries which look similiar but are not 1:1 the same). Which would explain how this scenario could happen.

Will prepare a testcase for the mapping-engine which consists:

• two members (1, 2)
• Entry A: identical entries on Member 1 and 2
• Entry B: slightly different to Entry A (stripping unimportant field)

This will hopefully also create two mappings:

• first: A2 and B1 - SIMILAR (setting conflict)
• second: A2 and A1 - SAME

Third required step would be to invalidate "frist" mapping. So Entry B(1) would be an unmapped entry, which gets the changetype ADDED. Expected result would be:

Member 1: Entry A (UUID1), Entry B (UUID2) Member 2: Entry A (UUID1), Entry B (UUID2)

[dgollub]

Introduction of regression-testcase r4295

[dgollub]

potential fix - prefer mappings which have compare result SAME

[dgollub]

Please try attached fix. (Can't commit right now to /trunk, due to bad testsuite-state)

[chi70]

Replying to dgollub:

Please try attached fix. (Can't commit right now to /trunk, due to bad testsuite-state)

I am pretty unsure if there is an improvment. First run, it doubled every entry. I recovered from Phone backup and tried again. Calanedar does not work at all anylonger.

With the recovered phone I did a config change (move the filesync dir for contacts). Now I saw, no conflicts were reported, the entries seems to be silently duplicated:

...
Main sink of member 1 of type syncml-obex-client committed all changes.
Sent an entry 14-new to member 2 (file-sync). Changetype ADDED
Sent an entry 34-new to member 2 (file-sync). Changetype ADDED
Sent an entry 64-new to member 2 (file-sync). Changetype ADDED
Sent an entry 90-new to member 2 (file-sync). Changetype ADDED
Sent an entry 91-new to member 2 (file-sync). Changetype ADDED
Main sink of member 2 of type file-sync committed all changes.
Sent an entry 176 to member 1 (syncml-obex-client). Changetype ADDED
Sent an entry 177 to member 1 (syncml-obex-client). Changetype ADDED
Sent an entry 178 to member 1 (syncml-obex-client). Changetype ADDED
Sent an entry 179 to member 1 (syncml-obex-client). Changetype ADDED
Sent an entry 180 to member 1 (syncml-obex-client). Changetype ADDED
...

Sorry, no traces form that session.

But the sync finished and the mobile did not complain.

Second run, again some more duplicates:

chris@urania:/opt/opensync/e71-fileContacts$ ls -l 14 14-*
-rw-r--r-- 1 chris users 236 Dec  7 22:58 14
-rw-r--r-- 1 chris users 236 Dec  7 22:58 14-new
-rw-r--r-- 1 chris users 236 Dec  7 22:58 14-new-new
chris@urania:/opt/opensync/e71-fileContacts$ cksum 14 14-*
2431554501 236 14
2431554501 236 14-new
2431554501 236 14-new-new

This time I have traces and they are attaced as ar incl. a copy and paste from console output.

[chi70]

traces from the silent duplication

[dgollub]

Could it be that those *-new-new files were present _already_ before you applied the fix?

[chi70]

Replying to dgollub:

Could it be that those *-new-new files were present _already_ before you applied the fix?

No. at least the -new-new were definitive not there.

[dgollub]

Replying to chi70:

Replying to dgollub:

Please try attached fix. (Can't commit right now to /trunk, due to bad testsuite-state)

I am pretty unsure if there is an improvment.

Just to be clear - the duplicates have very likely a different root-cause. That they "duplicate silently" now, could be "correct" behavior on the synchronization engine.

To clarify that i would ask for to create a new ticket to avoid mixing-up different issues. There will be no ticket "make OpenSync? perfect" - we have address different issue in separated tickets. This ticket is about "Fake conflict" - which is hopefully solved now.

First run, it doubled every entry. I recovered from Phone backup and tried again. Calanedar does not work at all anylonger.

This doesn't sound like a bug which got introduced by the provided patch. Sounds more like that no capabilities/merger got used and the engine duplicate those entries which doesn't have any mapping ...

Please reproduce the duplication issue in a different ticket and provide trace files (without sensitive information, preferred with a limited number of contacts).

With the recovered phone I did a config change (move the filesync dir for contacts). Now I saw, no conflicts were reported, the entries seems to be silently duplicated:

Put this information as well into the separated bug please. [...]

If there are no objections i will commit the patch during the week, as soon the testsuite got fixed in /trunk.

[dgollub]

The current patch breaks some testcases of unittest sync (.e.g sync_easy_conflict). The problem is that *mapping_engine get set to NULL with the next iteration and if the next sink_engine is the "own" sink_engine.

But the idea of the patch is basically the same - if the result is just "SIMILAR" keep looking and remember the first mapping_engine which fit at least somehow...

[dgollub]

new patch which address duplicates and failing unittest "sync"

[dgollub]

New patch - address testsuite issues "sync" and "mapping_engine" at the same time. This tests finally prefers "SAME"-mappings correctly.

[dgollub]

Patch v3 got committed with r4415. Please reopen if the issue is still reproducible.

[dgollub]

Fails again on x86_64 - not on x86-32.

[dgollub]

*mapping_engine_same_similar_conflict

• mapping_engine_same_similar_conflict_multi

Both are failing on Linux x86-64

[dgollub]

(In [5315]) Final fix of mapping_engine for similiar&same conflicts.

The change/entry_engine of the "similar" entry got overwritten by the "same" entry. So the similar&same was never fixed to 100%

This fixes tescases: mapping_engine_same_similar_conflict mapping_engine_same_similar_conflict_multi

(This was not a $ARCH problem as stated before)

fixes #883