There are two issues: distinguishing between name collisions and the same entity in two different federations. It should not be hard to distinguish name collisions with heuristics or distance measurements. Confidence in the same entity in two different federations ?? They may have the same title or same description.

Take subsets of all the entities that the MDQ might know about and create merged data with a merging strategy. It would be good to make that data available, but make it clear that it is merged.

Mikael states they use pyFF as an authoritative source. If they select which entity is published then they will be making decisions for their users. He would like a clearer policy on how the merge is done. Right now, the "replace with the latest-read entity id" rule is hidden in the code. There are provisions in the code for how to merge when there are conflicts. We could invent a way to do the merge. Enrique would like to look into this. Mikael states that whenever pyFF is run as a daemon and the update endpoint is called, the provisions are applied. Mikael is not sure how other people are using pyFF - eduGAIN uses pyFF - how do they do aggregation? Alex says there is an algorithm using a precedence rule, but he is not sure if that is using pyFF functionality or if there is something additional written in Python whcih makes that selection.

Mikael would like a cleaner way to define how auto merging is done, and Enrique would like to have all the data available. Maybe in the select pipeline, there is some way to configure/indicate using different lists. The aggregate is like a hard rule/limit, and the discovery service is more a convenience approach. But If the entity is not part of the discovery flow, it won't be used at all.

Here are Enrique's notes summarizing the discussion, with a potential solution via Alex. Thank you, Enrique! :

Matthew asked where the pyFF documentation is? Mikael says there are documents exported to readthedocs, and an old version of inline documentation in the code that is not picked up by modern tools. Mikael is trying to improve the documentation and test without breaking anything.

Matthew wanted to use pyfFF for a private thiss.js deployment but couldn't quite get it working. For the pipelines, it uses a domain specific language that is not easy to work with. We are stuck with that right now unless there is a new major version. Without a working example its hard to get something going - but there are examples in the unit tests.

instead of pyFF, the UK federation uses the shibboleth MDA framework. EduGAIN uses both pyFF and shibboleth MDA. It's a complicated framework to set up. The spring config files are XML-based. In both cases, the SAML pipelines refresh.