Ticket #992 (closed defect: fixed)

Opened 6 years ago

Last modified 5 years ago

Support for ObjType mixing in OSyncEngine

Reported by: dgollub Owned by: dgollub
Priority: highest Milestone: OpenSync 0.39
Component: OpenSync: Engine Version: 0.38
Severity: blocker Keywords:
Cc: tuju, cstender

Description

OpenSync 0.3x doesn't support ObjType mixing.

This results in Mapping-Table inconsistence and other un-obvious issues. One of them are asserts like:

opensync/engine/opensync_mapping_entry_engine.c:107:E:osync_entry_engine_matches: Assertion "mapping_entry_uid" failed

The ObjType data is actually a "wildcard"-objtype which is supposed to be compatible all ObjTypes.

With the refactoring of !OSyncObjEngine this functionally got broken and never worked for 0.3x. This needs to be fixed for 0.39.

Lots of bug-reports which involve the file-sync plugin and the default configuration of file-sync plus another plugin using different other ObjTypes trigger the lack of ObjType mixing.

Change History

comment:1 Changed 6 years ago by dgollub

Commit r4409 sets file-sync plugin default configuration to ObjType? "contact" to avoid that people use "data". This needs to get reverted to "data" as soon this issue got fixed.

comment:2 Changed 6 years ago by tuju

  • Cc tuju added

comment:3 follow-ups: ↓ 4 ↓ 5 Changed 6 years ago by bricks

My understanding of ObjType data is that it should only be used for backup purposes. Therefore all entries of ObjType "data" aren't converted through the internal xmlformat. Should the converter really support a conversion path of e.g. formatA/"data" -> formatB/"contact" ?

comment:4 in reply to: ↑ 3 ; follow-up: ↓ 6 Changed 6 years ago by tuju

Replying to bricks:

My understanding of ObjType data is that it should only be used for backup purposes. Therefore all entries of ObjType "data" aren't converted through the internal xmlformat.

Well that explains a lot. I never understood the difference before. IMO data is quite bad choice for such descriptor. It sounds like everything else wouldn't be data at all.

How about renaming it to raw or to stress the xmlformat-bybass with direct?

comment:5 in reply to: ↑ 3 Changed 6 years ago by dgollub

Replying to bricks:

My understanding of ObjType data is that it should only be used for backup purposes. Therefore all entries of ObjType "data" aren't converted through the internal xmlformat. Should the converter really support a conversion path of e.g. formatA/"data" -> formatB/"contact" ?

Thats not to 100% correct.

  1. The highly requested "backup"-sync would look like this:
    • group disabled converter (this also disables merger)!
    • Member1: $peer-plugin to backup - ObjTypes/ObjFormat e.g. contact/vcard21,event/vevent10,...
    • Member2: file-sync - ObjTypes e.g. data/file
  2. Whats so special about ObjType "data"?
    • It's a "wildcard" type and should be able to be "compatible" to all other objtypes
  3. Whats so sepcial about ObjFormat "file"?
    • It's a encapsulate format which can store/wrap a completely different format - e.g. vcard21,vcard30

So a "backup" scenario would never get in touch with xmlforamt, since there would be no conversion - as long the plugin doesn't do "internal format conversion" inside get_changes().

All above describe is about: file/data -> formatB/"contact"

There is no format-Wildcard/data - ObjFormats are fixed to one ObjType. AFAIK right now we have plain/data, file/data. There is no -e.g. vcard21/data.

What would be possible and is highly requested for backup purposes: file(vcard21)/data (vcard21 is encaspulated and untouched by the file-ObjFormat)

comment:6 in reply to: ↑ 4 ; follow-up: ↓ 7 Changed 6 years ago by dgollub

Replying to tuju:

Replying to bricks:

My understanding of ObjType data is that it should only be used for backup purposes. Therefore all entries of ObjType "data" aren't converted through the internal xmlformat.

Well that explains a lot. I never understood the difference before. IMO data is quite bad choice for such descriptor. It sounds like everything else wouldn't be data at all.

How about renaming it to raw or to stress the xmlformat-bybass with direct?

No - see 5. "Frontend"-end-users would get never in touch with the terms "data/file". This should be completely hidden by the frontend and only available when creating a "backup group". osynctool is only for developers/testers. So no renaming required!

IIRC, data and file got explained in one of the Whitepapers.

comment:7 in reply to: ↑ 6 Changed 6 years ago by tuju

Replying to dgollub:

No - see 5. "Frontend"-end-users would get never in touch with the terms "data/file". This should be completely hidden by the frontend

I don't see connection with user interface and if something is not descriptive name, no matter how deep it's buried.

comment:8 Changed 6 years ago by dgollub

  • Status changed from new to assigned
  • Milestone changed from 0.40 to 0.39

WIP

comment:9 Changed 6 years ago by dgollub

  • Component changed from OpenSync to OpenSync: Engine

comment:10 follow-up: ↓ 11 Changed 6 years ago by dgollub

Testcases to write:

  1. Slow-Sync with mixed-objtypes
    1. trigger for all ObjEngine? a slow-sync which share Mappings! This could end-up that all ObjEngine? needs to run a Slow-Sync.
      • Group1 with Member1, 2 and 3
        • Member1: file/data
          • contains contacts and events wrapped in file-format
        • Member2: vevent10/event, vcard21/contact
        • Member3: icalendar/calendar
          • contains events and todos wrapped in icalendar-foramt
      • Member2 triggers a slow-sync for ObjTypeSink event
      • Expected result:
        • Member1 get triggered for Slow-Sync for ObjTypeSink data
        • Member2 get triggered for Slow-Sync for ObjTypeSink contact aswell! Due to the contacts which will get reported by the ObjTypeSink data of Member1
        • Member3 get triggered for Slow-Sync for ObjTypeSink event (and todo, but only, if Member1 already has todo entries)

comment:11 in reply to: ↑ 10 Changed 6 years ago by cstender

  • Cc cstender added

The testcase sounds good, but what will happen with the following setup:
plugin 1: vcalender resource (e.g. irmc with a SE K750i which supports both, vevent and vtodo)
plugin 2: vtodo and vevent resource (e.g. kdepim)

How can you disable for example todos without disabling the whole vcalender resource? And who will request the slow-sync for vevent and vtodo if you enable it again?

comment:12 Changed 6 years ago by dgollub

With r5816 r5817 and r5818 one of the last changes to get initial objtype mix syncing are committed. Needs more real-life testing and special testcases.

Ides for testing:

  • calendar - event/todo syncing
  • slow-sync of a mixed-objtype-group - check for dataloss
    • slow-sync only for calendar
    • slow-sync only for todo

There are still some traces of data objtype, which are hardcoded and should get dropped.

comment:13 Changed 6 years ago by dgollub

Hit an issue with handling MappingTable? on MixedObjType? Syncing. To speed-up the development on this and avoid additionally complexity i introduced following limitation on purpose:

  • mixed-objtype syncing is only supported for encapsualted formats
  • this covers for example:
    • data <-> event/contact/data (example encapsulated format: file)
    • calendar <-> event/todo (example encapsulated format: icalendar)

The reason for this:

  • mapping table is today per objtype
  • mixed objtype syncing would require a objtype independent mapping table
  • how to handle slow-sync if a single ObjType? which has mixed-peer-records?

How to handle slow-sync with mixed-objtype-syncing?

  • trigger all target-encapsulator objtypes?
    • e.g. slow-sync:calendar
      • trigger slow-sync for objengine calendar
      • trigger slow-sync for objengine todo/event
        • triggered due to OSyncFormatEnv query of calendar, whcih results in decapsulators for todo/event
    • e.g. slow-sync:data
      • trigger slow-sync for objengine data
      • trigger slow-sync for objengine contact/event (no other objtype in this scenario available)
        • triggered due to OSyncFormatEnv query of data, which results in decapuslator for contact/event

Regarding "how to ignore todo when just syncing event of calendar objtype":

  • resource configuration to disable on purpose todo?
  • advantage: the ignoring is controlled and not done by default
  • disadvantage: it requires additional configuration
    • solution: recommendation for applications/frontends do this kind of configuration during the discovery call?
  • DONT: make this inside the plugin -> the engine should deal with this

comment:14 Changed 6 years ago by dgollub

Mixed objtype syncing should be now completed. Missing slow-sync handling got implemented with: r5222

Next: testcases

comment:15 Changed 6 years ago by dgollub

Implemented logic to handle limitation of old ObjEngine?-intalization code which required at least two member with the same objtype - r5237

comment:16 Changed 5 years ago by dgollub

osync_engine_initalize() needs to handle the case of not supported objtypes in different members. (e.g. no peer !OSyncSinkEngine)

Example:

Member1: mockformat1
Member2: mockformat1, mockformat2, mockformat3

The engine would fail during the osync_engine_sync* call.

comment:17 Changed 5 years ago by ianmartin

Trying to sink:

Evolution2: contact, event, note, todo (ie the default config)

file-sync: contact (also default config)

Sync fails with: ERROR: error while synchronizing: ROOT CAUSE: "Couldn't find Object Type Engine for Object Type "event" while preparing for write phase."

The syncmember.conf file for evo2 also grows with every sync with objformats not supported by the objtypes being added.

See attached files for config and syncmember.conf

comment:18 Changed 5 years ago by ianmartin

<?xml version="1.0"?>
<syncmember version="1.0">
  <pluginname>evo2-sync</pluginname>
  <objtype>
    <enabled>1</enabled>
    <read>1</read>
    <getchanges>1</getchanges>
    <write>1</write>
    <name>contact</name>
    <preferred_format></preferred_format>
    <objformat>
      <name>vcard30</name>
      <config>VCARD_EXTENSION=Evolution</config>
    </objformat>
    <objformat>
      <name>vcard21</name>
      <config>VCARD_EXTENSION=Evolution</config>
    </objformat>
    <objformat>
      <name>vevent20</name>
      <config></config>
    </objformat>
    <objformat>
      <name>vtodo20</name>
      <config></config>
    </objformat>
    <objformat>
      <name>vjournal</name>
      <config></config>
    </objformat>
  </objtype>
  <objtype>
    <enabled>1</enabled>
    <read>1</read>
    <getchanges>1</getchanges>
    <write>1</write>
    <name>event</name>
    <preferred_format></preferred_format>
    <objformat>
      <name>vevent20</name>
      <config></config>
    </objformat>
    <objformat>
      <name>vtodo20</name>
      <config></config>
    </objformat>
    <objformat>
      <name>vjournal</name>
      <config></config>
    </objformat>
  </objtype>
  <objtype>
    <enabled>1</enabled>
    <read>1</read>
    <getchanges>1</getchanges>
    <write>1</write>
    <name>todo</name>
    <preferred_format></preferred_format>
    <objformat>
      <name>vtodo20</name>
      <config></config>
    </objformat>
    <objformat>
      <name>vjournal</name>
      <config></config>
    </objformat>
  </objtype>
  <objtype>
    <enabled>1</enabled>
    <read>1</read>
    <getchanges>1</getchanges>
    <write>1</write>
    <name>note</name>
    <preferred_format></preferred_format>
    <objformat>
      <name>vjournal</name>
      <config></config>
    </objformat>
  </objtype>
</syncmember>

comment:19 Changed 5 years ago by dgollub

(In [5312]) Add additional test description for sync_detect_obj2, also why it is currently failing.

refs #992

comment:20 Changed 5 years ago by dgollub

  • Status changed from assigned to closed
  • Resolution set to fixed

(In [5313]) Fix sync_detect_obj2 and outstanding issue as described in #992 comment:16 and comment:17

fixes #992

Note: See TracTickets for help on using tickets.