Ticket #883 (closed defect: fixed)

Opened 5 years ago

Last modified 5 years ago

[PATCH] Conflict between several SAME/SIMILAR mappings (was: Fake conflict)

Reported by: tuju Owned by: dgollub
Priority: high Milestone: OpenSync 0.39
Component: OpenSync: Engine Version: 0.38
Severity: blocker Keywords:
Cc:

Description

I run msynctool normally and it falls to --slow-sync.

During the slow sync, it detects a conflict and rises it to the user to decide what should be done with it, even the two cards are exactly the same:

Entry 2:
Member: 1 (file-sync)
UID: 5DAIcMTISl
<?xml version="1.0"?>
<contact>
  <EMail Type="Internet">
    <Content>doe@example.com</Content>
  </EMail>
  <FormattedName>
    <Content>John Doe</Content>
  </FormattedName>
  <Name>
    <LastName>Doe</LastName>
    <FirstName>John</FirstName>
  </Name>
  <Organization>
    <Name>Example O&#xFC;</Name>
    <Unit></Unit>
  </Organization>
  <Revision>
    <Content>20081103T092103Z</Content>
  </Revision>
  <Role>
    <Content>Product Manager</Content>
  </Role>
  <Telephone Location="Work">
    <Content>+372123456</Content>
  </Telephone>
  <Telephone Type="Cellular">
    <Content>+3725551234</Content>
  </Telephone>
  <Url>
    <Content>http://www.example.com/</Content>
  </Url>
</contact>

Entry 1:
Member: 2 (syncml-obex-client)
UID: 5                        
<?xml version="1.0"?>         
<contact>                     
  <EMail Type="Internet">     
    <Content>doe@example.com</Content>
  </EMail>                                
  <FormattedName>                         
    <Content>John Doe</Content>      
  </FormattedName>                        
  <Name>                                  
    <LastName>Doe</LastName>          
    <FirstName>John</FirstName>          
  </Name>                                 
  <Organization>                          
    <Name>Example O&#xFC;</Name>        
    <Unit></Unit>                         
  </Organization>                         
  <Revision>                              
    <Content>20081103T092103Z</Content>   
  </Revision>                             
  <Role>                                  
    <Content>Product Manager</Content>    
  </Role>                                 
  <Telephone Location="Work">             
    <Content>+372123456</Content>        
  </Telephone>                            
  <Telephone Type="Cellular">
    <Content>+3725551234</Content>
  </Telephone>
  <Url>
    <Content>http://www.example.com/</Content>
  </Url>
</contact>

Which entry do you want to use? [1-9] To select a side, [D]uplicate:

This ticket is based on true story. Real names were changed with vim editor and no animals were hurt during the composition of this ticket.

Attachments

ticket883_fix_same_similar_conflicts.diff (1.0 KB) - added by dgollub 5 years ago.
potential fix - prefer mappings which have compare result SAME
t883.tar.gz (3.5 MB) - added by chi70 5 years ago.
traces from the silent duplication
ticket883_fix_same_similar_conflicts_v2.diff (1.4 KB) - added by dgollub 5 years ago.
new patch which address duplicates and failing unittest "sync"
ticket883_fix_same_similar_conflicts_v3.diff (5.2 KB) - added by dgollub 5 years ago.
New patch - address testsuite issues "sync" and "mapping_engine" at the same time. This tests finally prefers "SAME"-mappings correctly.

Change History

comment:1 Changed 5 years ago by tuju

  • Version changed from 0.37 to 0.38
  • Milestone changed from 0.40 to 0.39

comment:2 Changed 5 years ago by tuju

  • Severity changed from normal to major

This basically prevents using the opensync -> major severity.

comment:3 Changed 5 years ago by tuju

  • Priority changed from normal to high
  • Severity changed from major to blocker

comment:4 Changed 5 years ago by dgollub

  • Status changed from new to assigned
  • Summary changed from Fake conflict to [NEEDINFO] Fake conflict

I can't reproduce this issue.

Could you try to reproduce this issues with export OSYNC_NOPRIVACY=1. It would be also helpful to get this reproduce with some dummy-contacts. Your last trace files contained 96 changes which were candidate for a conflict.

comment:5 Changed 5 years ago by tuju

  • Owner changed from dgollub to tuju
  • Priority changed from high to normal
  • Status changed from assigned to new

comment:6 Changed 5 years ago by tuju

  • Status changed from new to assigned

comment:7 Changed 5 years ago by chi70

  • Summary changed from [NEEDINFO] Fake conflict to Fake conflict

I was able to reproduce this using osynctool (r3860) with OSYNC_NOPRIVACY=1. As this was done on real data these traces are not attached to the bug. Data was send to Dgollub.

comment:8 follow-up: ↓ 9 Changed 5 years ago by chi70

I did some more tests. It does not happen with a file-file sync using the data synced from syncml-obex-client to file.

At the moment this action duplicates one of my contacts and I assume these messages are the cause of the duplication:

Received an entry 99 from member 1 (syncml-obex-client). Changetype ADDED
element Url: Schemas validity error : Element 'Url', attribute 'Location': The attribute 'Location' is not allowed.
ERROR: XMLFormat validation failed.
EXIT_ERROR: osync_converter_invoke: XMLFormat validation failed.
EXIT_ERROR: osync_format_env_convert: XMLFormat validation failed.
Main sink of member 1 of type syncml-obex-client had an error: XMLFormat validation failed.
EXIT_ERROR: _osync_engine_receive_change: XMLFormat validation failed.
Received an entry 101 from member 1 (syncml-obex-client). Changetype ADDED

Entry 99 gets duplicated.

I do not see anything unusual with that entry from syncml-ds-tool dumping the database.

comment:9 in reply to: ↑ 8 Changed 5 years ago by dgollub

Replying to chi70:

I did some more tests. It does not happen with a file-file sync using the data synced from syncml-obex-client to file.

file2file contains the same set of capabilities.

At the moment this action duplicates one of my contacts and I assume these messages are the cause of the duplication:

Received an entry 99 from member 1 (syncml-obex-client). Changetype ADDED
element Url: Schemas validity error : Element 'Url', attribute 'Location': The attribute 'Location' is not allowed.
ERROR: XMLFormat validation failed.
EXIT_ERROR: osync_converter_invoke: XMLFormat validation failed.
EXIT_ERROR: osync_format_env_convert: XMLFormat validation failed.
Main sink of member 1 of type syncml-obex-client had an error: XMLFormat validation failed.
EXIT_ERROR: _osync_engine_receive_change: XMLFormat validation failed.
Received an entry 101 from member 1 (syncml-obex-client). Changetype ADDED

Entry 99 gets duplicated.

xmlformat validation issue are vformat conversion bugs.

I do not see anything unusual with that entry from syncml-ds-tool dumping the database.

Hehe, that's quite easy: syncml-ds-tool dosen't drive mapping or does format conversion... it's just dumping the data.

comment:10 Changed 5 years ago by dgollub

  • Priority changed from normal to high
  • Summary changed from Fake conflict to Conflict between several SAME/SIMILAR mappings (was: Fake conflict)

I found some interesting things in the Trace files from Tuju.

There are entries which got mapped osync_xmlformat_compare() two times! The first time was a mapping which resulted in to a 'SIMILAR' match (which would cause the conflict).

The second time was a mapping which resulted in to a 'SAME' match (which would explain the 'fake' conflict - since the entry was really the same).

Talking to Tuju about the setup turned out that the resource very likely contains very likely several "duplicates" (entries which look similiar but are not 1:1 the same). Which would explain how this scenario could happen.

Will prepare a testcase for the mapping-engine which consists:

  • two members (1, 2)
  • Entry A: identical entries on Member 1 and 2
  • Entry B: slightly different to Entry A (stripping unimportant field)

This will hopefully also create two mappings:

  • first: A2 and B1 - SIMILAR (setting conflict)
  • second: A2 and A1 - SAME

Third required step would be to invalidate "frist" mapping. So Entry B(1) would be an unmapped entry, which gets the changetype ADDED. Expected result would be:

Member 1: Entry A (UUID1), Entry B (UUID2) Member 2: Entry A (UUID1), Entry B (UUID2)

comment:11 Changed 5 years ago by dgollub

Introduction of regression-testcase r4295

Changed 5 years ago by dgollub

potential fix - prefer mappings which have compare result SAME

comment:12 follow-up: ↓ 13 Changed 5 years ago by dgollub

  • Summary changed from Conflict between several SAME/SIMILAR mappings (was: Fake conflict) to [NEEDINFO] Conflict between several SAME/SIMILAR mappings (was: Fake conflict)

Please try attached fix. (Can't commit right now to /trunk, due to bad testsuite-state)

comment:13 in reply to: ↑ 12 ; follow-up: ↓ 16 Changed 5 years ago by chi70

Replying to dgollub:

Please try attached fix. (Can't commit right now to /trunk, due to bad testsuite-state)

I am pretty unsure if there is an improvment. First run, it doubled every entry. I recovered from Phone backup and tried again. Calanedar does not work at all anylonger.

With the recovered phone I did a config change (move the filesync dir for contacts). Now I saw, no conflicts were reported, the entries seems to be silently duplicated:

...
Main sink of member 1 of type syncml-obex-client committed all changes.
Sent an entry 14-new to member 2 (file-sync). Changetype ADDED
Sent an entry 34-new to member 2 (file-sync). Changetype ADDED
Sent an entry 64-new to member 2 (file-sync). Changetype ADDED
Sent an entry 90-new to member 2 (file-sync). Changetype ADDED
Sent an entry 91-new to member 2 (file-sync). Changetype ADDED
Main sink of member 2 of type file-sync committed all changes.
Sent an entry 176 to member 1 (syncml-obex-client). Changetype ADDED
Sent an entry 177 to member 1 (syncml-obex-client). Changetype ADDED
Sent an entry 178 to member 1 (syncml-obex-client). Changetype ADDED
Sent an entry 179 to member 1 (syncml-obex-client). Changetype ADDED
Sent an entry 180 to member 1 (syncml-obex-client). Changetype ADDED
...

Sorry, no traces form that session.

But the sync finished and the mobile did not complain.

Second run, again some more duplicates:

chris@urania:/opt/opensync/e71-fileContacts$ ls -l 14 14-*
-rw-r--r-- 1 chris users 236 Dec  7 22:58 14
-rw-r--r-- 1 chris users 236 Dec  7 22:58 14-new
-rw-r--r-- 1 chris users 236 Dec  7 22:58 14-new-new
chris@urania:/opt/opensync/e71-fileContacts$ cksum 14 14-*
2431554501 236 14
2431554501 236 14-new
2431554501 236 14-new-new

This time I have traces and they are attaced as ar incl. a copy and paste from console output.

Changed 5 years ago by chi70

traces from the silent duplication

comment:14 follow-up: ↓ 15 Changed 5 years ago by dgollub

Could it be that those *-new-new files were present _already_ before you applied the fix?

comment:15 in reply to: ↑ 14 Changed 5 years ago by chi70

Replying to dgollub:

Could it be that those *-new-new files were present _already_ before you applied the fix?

No. at least the -new-new were definitive not there.

comment:16 in reply to: ↑ 13 Changed 5 years ago by dgollub

  • Summary changed from [NEEDINFO] Conflict between several SAME/SIMILAR mappings (was: Fake conflict) to [PATCH] Conflict between several SAME/SIMILAR mappings (was: Fake conflict)

Replying to chi70:

Replying to dgollub:

Please try attached fix. (Can't commit right now to /trunk, due to bad testsuite-state)

I am pretty unsure if there is an improvment.

Just to be clear - the duplicates have very likely a different root-cause. That they "duplicate silently" now, could be "correct" behavior on the synchronization engine.

To clarify that i would ask for to create a new ticket to avoid mixing-up different issues. There will be no ticket "make OpenSync? perfect" - we have address different issue in separated tickets. This ticket is about "Fake conflict" - which is hopefully solved now.

First run, it doubled every entry. I recovered from Phone backup and tried again. Calanedar does not work at all anylonger.

This doesn't sound like a bug which got introduced by the provided patch. Sounds more like that no capabilities/merger got used and the engine duplicate those entries which doesn't have any mapping ...

Please reproduce the duplication issue in a different ticket and provide trace files (without sensitive information, preferred with a limited number of contacts).

With the recovered phone I did a config change (move the filesync dir for contacts). Now I saw, no conflicts were reported, the entries seems to be silently duplicated:

Put this information as well into the separated bug please. [...]

If there are no objections i will commit the patch during the week, as soon the testsuite got fixed in /trunk.

comment:17 Changed 5 years ago by dgollub

The current patch breaks some testcases of unittest sync (.e.g sync_easy_conflict). The problem is that *mapping_engine get set to NULL with the next iteration and if the next sink_engine is the "own" sink_engine.

But the idea of the patch is basically the same - if the result is just "SIMILAR" keep looking and remember the first mapping_engine which fit at least somehow...

Changed 5 years ago by dgollub

new patch which address duplicates and failing unittest "sync"

Changed 5 years ago by dgollub

New patch - address testsuite issues "sync" and "mapping_engine" at the same time. This tests finally prefers "SAME"-mappings correctly.

comment:18 Changed 5 years ago by dgollub

  • Status changed from assigned to closed
  • Resolution set to fixed

Patch v3 got committed with r4415. Please reopen if the issue is still reproducible.

comment:19 Changed 5 years ago by dgollub

  • Component changed from OpenSync to OpenSync: Engine

comment:20 Changed 5 years ago by dgollub

  • Status changed from closed to reopened
  • Resolution fixed deleted

Fails again on x86_64 - not on x86-32.

comment:21 Changed 5 years ago by dgollub

  • Owner changed from tuju to dgollub
  • Status changed from reopened to new

comment:22 Changed 5 years ago by dgollub

*mapping_engine_same_similar_conflict

  • mapping_engine_same_similar_conflict_multi

Both are failing on Linux x86-64

comment:23 Changed 5 years ago by dgollub

  • Status changed from new to closed
  • Resolution set to fixed

(In [5315]) Final fix of mapping_engine for similiar&same conflicts.

The change/entry_engine of the "similar" entry got overwritten by the "same" entry. So the similar&same was never fixed to 100%

This fixes tescases: mapping_engine_same_similar_conflict mapping_engine_same_similar_conflict_multi

(This was not a $ARCH problem as stated before)

fixes #883

Note: See TracTickets for help on using tickets.