Support for ObjType mixing in OSyncEngine

[dgollub]

OpenSync 0.3x doesn't support ObjType mixing.

This results in Mapping-Table inconsistence and other un-obvious issues. One of them are asserts like:

opensync/engine/opensync_mapping_entry_engine.c:107:E:osync_entry_engine_matches: Assertion "mapping_entry_uid" failed

The ObjType data is actually a "wildcard"-objtype which is supposed to be compatible all ObjTypes.

With the refactoring of !OSyncObjEngine this functionally got broken and never worked for 0.3x. This needs to be fixed for 0.39.

Lots of bug-reports which involve the file-sync plugin and the default configuration of file-sync plus another plugin using different other ObjTypes trigger the lack of ObjType mixing.

[dgollub]

Commit r4409 sets file-sync plugin default configuration to ObjType? "contact" to avoid that people use "data". This needs to get reverted to "data" as soon this issue got fixed.

[bricks]

My understanding of ObjType data is that it should only be used for backup purposes. Therefore all entries of ObjType "data" aren't converted through the internal xmlformat. Should the converter really support a conversion path of e.g. formatA/"data" -> formatB/"contact" ?

[tuju]

Replying to bricks:

My understanding of ObjType data is that it should only be used for backup purposes. Therefore all entries of ObjType "data" aren't converted through the internal xmlformat.

Well that explains a lot. I never understood the difference before. IMO data is quite bad choice for such descriptor. It sounds like everything else wouldn't be data at all.

How about renaming it to raw or to stress the xmlformat-bybass with direct?

[dgollub]

Replying to bricks:

My understanding of ObjType data is that it should only be used for backup purposes. Therefore all entries of ObjType "data" aren't converted through the internal xmlformat. Should the converter really support a conversion path of e.g. formatA/"data" -> formatB/"contact" ?

Thats not to 100% correct.

1. The highly requested "backup"-sync would look like this:
   • group disabled converter (this also disables merger)!
   • Member1: $peer-plugin to backup - ObjTypes/ObjFormat e.g. contact/vcard21,event/vevent10,...
   • Member2: file-sync - ObjTypes e.g. data/file
2. Whats so special about ObjType "data"?
   • It's a "wildcard" type and should be able to be "compatible" to all other objtypes
3. Whats so sepcial about ObjFormat "file"?
   • It's a encapsulate format which can store/wrap a completely different format - e.g. vcard21,vcard30

So a "backup" scenario would never get in touch with xmlforamt, since there would be no conversion - as long the plugin doesn't do "internal format conversion" inside get_changes().

All above describe is about: file/data -> formatB/"contact"

There is no format-Wildcard/data - ObjFormats are fixed to one ObjType. AFAIK right now we have plain/data, file/data. There is no -e.g. vcard21/data.

What would be possible and is highly requested for backup purposes: file(vcard21)/data (vcard21 is encaspulated and untouched by the file-ObjFormat)

[dgollub]

Replying to tuju:

Replying to bricks:

My understanding of ObjType data is that it should only be used for backup purposes. Therefore all entries of ObjType "data" aren't converted through the internal xmlformat.

Well that explains a lot. I never understood the difference before. IMO data is quite bad choice for such descriptor. It sounds like everything else wouldn't be data at all.

How about renaming it to raw or to stress the xmlformat-bybass with direct?

No - see 5. "Frontend"-end-users would get never in touch with the terms "data/file". This should be completely hidden by the frontend and only available when creating a "backup group". osynctool is only for developers/testers. So no renaming required!

IIRC, data and file got explained in one of the Whitepapers.

[tuju]

Replying to dgollub:

No - see 5. "Frontend"-end-users would get never in touch with the terms "data/file". This should be completely hidden by the frontend

I don't see connection with user interface and if something is not descriptive name, no matter how deep it's buried.

[dgollub]

WIP

[dgollub]

Testcases to write:

1. Slow-Sync with mixed-objtypes
   1. trigger for all ObjEngine? a slow-sync which share Mappings! This could end-up that all ObjEngine? needs to run a Slow-Sync.
      • Group1 with Member1, 2 and 3
        • Member1: file/data
          • contains contacts and events wrapped in file-format
        • Member2: vevent10/event, vcard21/contact
        • Member3: icalendar/calendar
          • contains events and todos wrapped in icalendar-foramt
      • Member2 triggers a slow-sync for ObjTypeSink event
      • Expected result:
        • Member1 get triggered for Slow-Sync for ObjTypeSink data
        • Member2 get triggered for Slow-Sync for ObjTypeSink contact aswell! Due to the contacts which will get reported by the ObjTypeSink data of Member1
        • Member3 get triggered for Slow-Sync for ObjTypeSink event (and todo, but only, if Member1 already has todo entries)

[cstender]

The testcase sounds good, but what will happen with the following setup:
plugin 1: vcalender resource (e.g. irmc with a SE K750i which supports both, vevent and vtodo)
plugin 2: vtodo and vevent resource (e.g. kdepim)

How can you disable for example todos without disabling the whole vcalender resource? And who will request the slow-sync for vevent and vtodo if you enable it again?

[dgollub]

With r5816 r5817 and r5818 one of the last changes to get initial objtype mix syncing are committed. Needs more real-life testing and special testcases.

Ides for testing:

• calendar - event/todo syncing
• slow-sync of a mixed-objtype-group - check for dataloss
  • slow-sync only for calendar
  • slow-sync only for todo

There are still some traces of data objtype, which are hardcoded and should get dropped.

[dgollub]

Hit an issue with handling MappingTable? on MixedObjType? Syncing. To speed-up the development on this and avoid additionally complexity i introduced following limitation on purpose:

• mixed-objtype syncing is only supported for encapsualted formats
• this covers for example:
  • data <-> event/contact/data (example encapsulated format: file)
  • calendar <-> event/todo (example encapsulated format: icalendar)

The reason for this:

• mapping table is today per objtype
• mixed objtype syncing would require a objtype independent mapping table
• how to handle slow-sync if a single ObjType? which has mixed-peer-records?

How to handle slow-sync with mixed-objtype-syncing?

• trigger all target-encapsulator objtypes?
  • e.g. slow-sync:calendar
    • trigger slow-sync for objengine calendar
    • trigger slow-sync for objengine todo/event
      • triggered due to OSyncFormatEnv query of calendar, whcih results in decapsulators for todo/event
  • e.g. slow-sync:data
    • trigger slow-sync for objengine data
    • trigger slow-sync for objengine contact/event (no other objtype in this scenario available)
      • triggered due to OSyncFormatEnv query of data, which results in decapuslator for contact/event

Regarding "how to ignore todo when just syncing event of calendar objtype":

• resource configuration to disable on purpose todo?
• advantage: the ignoring is controlled and not done by default
• disadvantage: it requires additional configuration
  • solution: recommendation for applications/frontends do this kind of configuration during the discovery call?
• DONT: make this inside the plugin -> the engine should deal with this

[dgollub]

Mixed objtype syncing should be now completed. Missing slow-sync handling got implemented with: r5222

Next: testcases

[dgollub]

Implemented logic to handle limitation of old ObjEngine?-intalization code which required at least two member with the same objtype - r5237

[dgollub]

osync_engine_initalize() needs to handle the case of not supported objtypes in different members. (e.g. no peer !OSyncSinkEngine)

Example:

Member1: mockformat1
Member2: mockformat1, mockformat2, mockformat3

The engine would fail during the osync_engine_sync* call.

[ianmartin]

Trying to sink:

Evolution2: contact, event, note, todo (ie the default config)

file-sync: contact (also default config)

Sync fails with: ERROR: error while synchronizing: ROOT CAUSE: "Couldn't find Object Type Engine for Object Type "event" while preparing for write phase."

The syncmember.conf file for evo2 also grows with every sync with objformats not supported by the objtypes being added.

See attached files for config and syncmember.conf

[ianmartin]

<?xml version="1.0"?>
<syncmember version="1.0">
  <pluginname>evo2-sync</pluginname>
  <objtype>
    <enabled>1</enabled>
    <read>1</read>
    <getchanges>1</getchanges>
    <write>1</write>
    <name>contact</name>
    <preferred_format></preferred_format>
    <objformat>
      <name>vcard30</name>
      <config>VCARD_EXTENSION=Evolution</config>
    </objformat>
    <objformat>
      <name>vcard21</name>
      <config>VCARD_EXTENSION=Evolution</config>
    </objformat>
    <objformat>
      <name>vevent20</name>
      <config></config>
    </objformat>
    <objformat>
      <name>vtodo20</name>
      <config></config>
    </objformat>
    <objformat>
      <name>vjournal</name>
      <config></config>
    </objformat>
  </objtype>
  <objtype>
    <enabled>1</enabled>
    <read>1</read>
    <getchanges>1</getchanges>
    <write>1</write>
    <name>event</name>
    <preferred_format></preferred_format>
    <objformat>
      <name>vevent20</name>
      <config></config>
    </objformat>
    <objformat>
      <name>vtodo20</name>
      <config></config>
    </objformat>
    <objformat>
      <name>vjournal</name>
      <config></config>
    </objformat>
  </objtype>
  <objtype>
    <enabled>1</enabled>
    <read>1</read>
    <getchanges>1</getchanges>
    <write>1</write>
    <name>todo</name>
    <preferred_format></preferred_format>
    <objformat>
      <name>vtodo20</name>
      <config></config>
    </objformat>
    <objformat>
      <name>vjournal</name>
      <config></config>
    </objformat>
  </objtype>
  <objtype>
    <enabled>1</enabled>
    <read>1</read>
    <getchanges>1</getchanges>
    <write>1</write>
    <name>note</name>
    <preferred_format></preferred_format>
    <objformat>
      <name>vjournal</name>
      <config></config>
    </objformat>
  </objtype>
</syncmember>

[dgollub]

(In [5312]) Add additional test description for sync_detect_obj2, also why it is currently failing.

refs #992

[dgollub]

(In [5313]) Fix sync_detect_obj2 and outstanding issue as described in #992 comment:16 and comment:17

fixes #992