[389-devel] Re: Please have a look at rewriters design

Hello,

I agree that the term generic is not appropriate. I should change it
(design/PR) if it still exist somewhere.

https://pagure.io/389-ds-base/pull-request/50981 is said to extend the
usability of existing interfaces and I think it is what it does.
People needing to map/rewrite/transform (whatever the word) an
attribute/value know what they want to obtain but usually do not care
about the burden of writing/deploying a new plugin.

In my mind only the rewriter is complex and knows when/how it applies
(attribute, scope, crafting values, authentication...) so I wanted to
keep the interface very simple: just load your rewriter and let core
server call it. William raised that it could contain helper function,
for example going through a filter and call rewriter function for each
filter components. I am looking at that at the moment.
I think that a rewriter may also appreciate some configuration area, for
example if a rewriter is generic and apply some transformation rules
specific to a rewriter instance.

I agree that it needs to be documented and plugin guide is a good place.
I would like to use the design to describe the interfaces.

best regards
thierry

On 4/1/20 9:24 AM, Ludwig Krispenz wrote:
> Ok, so Thierry's solution is useful to make using rewriters simpler,
> but I really want to have its use and interface  documented somewhere
> outside the code, PR, or design doc on the 389ds wiki - it needs to go
> to the official doc eg plugin guide.
>
> Regards,
> Ludwig
>
> On 04/01/2020 01:02 AM, William Brown wrote:
>>
>>> On 1 Apr 2020, at 01:04, Ludwig Krispenz <lkrispen@redhat.com> wrote:
>>>
>>> Hi,
>>>
>>> I was away and am late in the discussion, maybe too late.
>>>
>> Not too late, it's not released in production yet ;). There are two
>> PR's that have been discussed here:
>>
>> https://pagure.io/389-ds-base/pull-request/50988
>>
>> https://pagure.io/389-ds-base/pull-request/50981
>>
>>> In my understanding what you mean by "generic" is that for a new
>>> rewriter you do not need a plugin, but to provide some rewrite
>>> functions and specify them in a rewriters config entry. But there is
>>> still the need to write rewriter functions, compile  and deploy
>>> them, and instead of plugins you now have a new interface of
>>> "filterRewriter" and "returendAttrRewriter functions - so far not
>>> documented anywhere.
>>>
>>> Under generic rewriter I would understand an approach where you do
>>> not need to provide own functions, but have a rewriter plugin, which
>>> does rewriting based on rules in rewrite config entries, eg in which
>>> subtree, for which entries (filter to select), how to map a saerch
>>> filter, how to rename attrs on return,....
>> I had the same feeling too, to have these statically in libslapd, and
>> much simpler than resolving symbols and dlopen. However, it's looking
>> more like it will be a plugin style, but without using the current
>> slapi plugin architecture - just a symload at start up. The reason
>> for this that thierry explained is that freeipa plans to link to
>> samba or sssd as part of one of the rewriter classes, and we don't
>> want that to become a dependency of 389-ds.
>>
>> I have argued in the past for a "lib-ipa" that has the needed shared
>> logic between various pieces of the project, but honestly, I forgot
>> if that ever happened. I think these days sssd is libipa in a lot of
>> ways ...
>>
>> Anyway, that's why Thierry want's to have a symload in this case :)
>>
>>> Best regards,
>>> Ludwig
>>>
>>>
>>> On 03/19/2020 01:09 AM, William Brown wrote:
>>>>> On 19 Mar 2020, at 04:08, thierry bordaz <tbordaz@redhat.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 3/18/20 1:51 AM, William Brown wrote:
>>>>>>> On 18 Mar 2020, at 04:08, thierry bordaz <tbordaz@redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi William,
>>>>>>>
>>>>>>> I updated the design according to our offline exchange
>>>>>> Thanks Thierry, I appreciate the conversation and the updates to
>>>>>> the document: it made clear there were extra details up in your
>>>>>> brain but not in words yet :) it's always hard to remember all
>>>>>> the details as we write things, so thanks for the discussion.
>>>>>> Like you said, it's always good to have a team who is really
>>>>>> invested and cares about the work we do!
>>>>>>
>>>>>>
>>>>>> Your design for the core server version looks much better! Thank
>>>>>> you. I still think there are some missing points. The reason to
>>>>>> have a libpath rather than inbuild is to avoid a potential
>>>>>> linking to sssd/samba. I think also that the problem space of the
>>>>>> global catalog here needs to be looked at too. This feature is
>>>>>> not in isolation, it's really a part of that.
>>>>> Okay, I will work on a new PR making core server able to
>>>>> retrieve/registers rewriters.
>>>>>
>>>>> I think the "need" to improve the usability of rewriters is not
>>>>> specific to global catalog. Global Catalog is just an opportunity
>>>>> to implement it. I think parts of slapi-nis, integration of
>>>>> vsphere, GC (and likely others) are also use case for rewriters.
>>>>> They were implemented in different ways because rewriters were not
>>>>> easy to use or simply not known.
>>>> Yes, that's perfectly reasonable, and shouldn't stop your idea from
>>>> being created - what's concerning me is that without a full picture
>>>> you don't know how far to take these rewriters or what direction,
>>>> or what might be needed.
>>>>
>>>>>> This means we have a whole set of deployment cases to look at.
>>>>>>
>>>>>> So the deployment will look like:
>>>>>>
>>>>>> IPA DS --> IPA GC
>>>>>>
>>>>>>
>>>>>> So an ipaAccount from the IPA DS instance will be "copied and
>>>>>> transformed" into the IPA GC. This process is as yet undefined
>>>>>> (it sounds like it may be offline or something else ...). We are
>>>>>> simply not dealing with one instance now, but an out-of-band
>>>>>> replication and transformation process. It's unclear whether the
>>>>>> data transform is during this loading process, or in the IPA GC
>>>>>> somehow.
>>>>>>
>>>>>>  From what I understand, it sounds like a method to take an
>>>>>> ipaAccount and transform it to an AD GC account stub. Then inside
>>>>>> of that IPA GC there are some virtual attributes you wish to add
>>>>>> like objectSid binary vs string representations, objectCategory,
>>>>>> maybe others.
>>>>>>
>>>>>> So from our discussion, we have currently focused on "how do we
>>>>>> transform entries within a single directory server". But that's
>>>>>> not the problem here. We are saying:
>>>>>>
>>>>>> "We take an entry from IPA DS, transform it to an IPA GC stub
>>>>>> entry, and then apply a set of further "in memory" transformations"
>>>>> One of the biggest issue with GC is schema. IPA DS and IPA GC have
>>>>> not compatible schema. They can not be in the same replication
>>>>> topology.
>>>>> So provisioning of IPA GC requires transformations rules to
>>>>> present an other "view" of IPA DS data. Those transformations will
>>>>> be on the write path (i.e. stored in DB/indexed). This
>>>>> transformation work is almost done and is completely independent
>>>>> of 389-ds.
>>>>> All of this is "write" path: provisioning (online or offline) and
>>>>> transformation.
>>>>>
>>>>> The problem for IPA GC is now on the "read" path. AD clients are
>>>>> use to smart shortcuts/control that are supported by IPA GC.
>>>>> This is the IPA GC instance that will register the rewriters to
>>>>> act as GC does.
>>>> Yep, I'm aware :)
>>>>
>>>>>> If that's the process, why not do all the transforms as required
>>>>>> in the DS -> GC load process? You raised a critically key point -
>>>>>> we have a concern about the write path as the transform point due
>>>>>> to IO or time to do the transform, but it sounds like you have to
>>>>>> do this anyway as an element of the DS -> GC process.
>>>>> Some of the transformation rules, on the write path, are quite
>>>>> complex. Looking at slapi-nis config entries gives an idea what is
>>>>> needed. In addition to those transformations, DS to GC online
>>>>> provisioning is not simple at all. Relying on sync-repl, you then
>>>>> need to transform a received entry into an update. At the moment
>>>>> it is an offline provisioning via transformation and import (much
>>>>> simpler).
>>>>>
>>>>> To be honest I am afraid that the transform rules will result in
>>>>> rewriting slapi-nis.
>>>> *puts finger on nose* I do not want to be near that toxic rewrite
>>>> at all.
>>>>
>>>>>> I think everytime I have spoken to you about this, I have kept
>>>>>> learning more and more about this, and the more I see, I have
>>>>>> many concerns about this feature. I think we do not have the full
>>>>>> picture. You have admitted that you don't know the full extend or
>>>>>> ideas here. There is clearly a communication break down here to
>>>>>> our team from the IPA project, and they aren't telling us what
>>>>>> they want. It sounds like they are asking you to just do "a small
>>>>>> piece" but only they know the bigger picture.
>>>>>>
>>>>>> The IPA project has the following designs:
>>>>>>
>>>>>> https://www.freeipa.org/page/V4/Global_Catalog_Support
>>>>>>
>>>>>> https://www.freeipa.org/page/V4/Global_Catalog_HLD
>>>>>>
>>>>>> https://www.freeipa.org/page/V4/Global_Catalog_Access_Control
>>>>>>
>>>>>> https://www.freeipa.org/page/V4/Global_Catalog_Data_Transformation
>>>>>>
>>>>>> This also links to:
>>>>>>
>>>>>> https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2003/cc737410(v=ws.10)?redirectedfrom=MSDN
>>>>>>
>>>>>>
>>>>>> The freeipa design pages are extremely shallow on details. The
>>>>>> entire section on how they plan to get data into the GC is:
>>>>>>
>>>>>> """
>>>>>> Global Catalog provisioning
>>>>>>
>>>>>> The data in Global Catalog is provisioned from the primary LDAP
>>>>>> server instance running on the same FreeIPA master. A SYNCREPL
>>>>>> mechanism is used to retrieve the changes and a modified
>>>>>> slapi-nis module is used to transform FreeIPA original data to a
>>>>>> schema compatible with Global Catalog in Active Directory. Unlike
>>>>>> the original slapi-nis module, the data is stored in a proper
>>>>>> LDAP backend so it is persistent across the directory server
>>>>>> restarts.
>>>>>> """
>>>>> You are right I do not know the big picture. What I know is that
>>>>> parts of GC needs can be solved with rewriters that is by the way
>>>>> a supported 389-ds interface. So storing rewriters in simple
>>>>> shared library rather than in plugins will help both IPA and 389-ds.
>>>> Without the big picture we don't know what they will ask from the
>>>> rewriters, and what we can or cannot deliver.
>>>>
>>>>>> Where is the example config? Proof of concept? Even a conceptual
>>>>>> set of accounts and groups showing the data transformation? How
>>>>>> will they synthesise stable object data points?
>>>>>>
>>>>>> The section of "data transformation" even goes to a blank page.
>>>>>> Is the rewrite you are being asked to do just for objectSid once
>>>>>> all these other transforms are done? Or is there more?
>>>>>>
>>>>>>
>>>>>> Honestly, it's worth reading the "how global catalog works" from
>>>>>> msdn. Just to put it in contrast, that document (when converted
>>>>>> to a pdf) is 61 pages long. Look at the features. Group caching,
>>>>>> GC replication, partialAttribute replication based on schema,
>>>>>> more ...
>>>>>>
>>>>>>
>>>>>> Honestly, Thierry, I trust you as a very smart and capable
>>>>>> engineer, but you do not have the full picture here - none of us
>>>>>> do. This seems like a feature that will explode in complexity and
>>>>>> scale, and if not done *properly* from the start, may end up with
>>>>>> many many half-baked, poorly designed solutions tacked together
>>>>>> to make it look like it works. And that means we'll end up
>>>>>> carrying that burden, just like slapi-nis (which is everyones
>>>>>> favourite plugin ...)
>>>>> Again, rewriters is not new. It has been a supported interface for
>>>>> years. The design is just to make them simpler to develop/deploy.
>>>>> Looking at some plugins I think they are related to a way to give
>>>>> different "views" of the same dataset. Many time, a rewriter,
>>>>> specific to ldap client needs is a good option.
>>>>> If GC can make use of it great. But I am sure that others (like
>>>>> vsphere) will appreciate.
>>>> That's not the problem. You are right that having improved rewriter
>>>> support, probably has some good options for other plugins, or other
>>>> areas. The issue is without the bigger picture, we don't know what
>>>> they need. We don't know what we are on the hook for.
>>>>
>>>> Let's be clear, to me as an external person, a core team of the 389
>>>> project, the information in those design documents is not enough
>>>> for me to make informed engineering decisions about this feature.
>>>>
>>>> There is a pattern and history to this behaviour.
>>>>
>>>> I think what's really concerning isn't the technical issues, but
>>>> the social. I want to make clear - You, Thierry, yourself admitted
>>>> you do not know what is fully expected of you in this feature. How
>>>> are you, as an engineer meant to do your best possible work,
>>>> without the full picture. You are very smart, but not psychic last
>>>> time I checked :)
>>>>
>>>> This does not make me comfortable.
>>>>
>>>> I also know - having been inside Red Hat, and now external to it,
>>>> that the FreeIPA team does a lot of discussion internally and
>>>> privately. It needs to stop. If they want to request features of
>>>> our project, they need to accept that 389-ds has upstream, core
>>>> team members who are not part of Red Hat. They need to be engaging
>>>> on 389-devel, not coming to you internally, and asking for
>>>> features. They need to be designing their features, publicly, and
>>>> clearly, in detail, so that we can make informed engineering
>>>> decisions for our project, that includes our full team and community.
>>>>
>>>>
>>>> I have resisted talking about this publicly for a long time, but I
>>>> think it's time that it's taken to the open - the FreeIPA team has
>>>> communication and social challenges that they need to address.
>>>> While these social issues continue, we will continue to see poor
>>>> quality features being churned out that negatively impact our
>>>> users, and our reputation as a project.
>>>>
>>>>
>>>> At this point, I believe that this rewriter feature can not
>>>> progress until the FreeIPA project puts forward a complete,
>>>> detailed and well constructed design of "what they require" for
>>>> their global catalog feature.
>>>>
>>>> Thanks
>>>>
>>>> —
>>>> Sincerely,
>>>>
>>>> William Brown
>>>>
>>>> Senior Software Engineer, 389 Directory Server
>>>> SUSE Labs
>>>> _______________________________________________
>>>> 389-devel mailing list -- 389-devel@lists.fedoraproject.org
>>>> To unsubscribe send an email to
>>>> 389-devel-leave@lists.fedoraproject.org
>>>> Fedora Code of Conduct:
>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>> List Guidelines:
>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>> List Archives:
>>>> https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
>>> --
>>> Red Hat GmbH, http://www.de.redhat.com/, Sitz: Grasbrunn,
>>> Handelsregister: Amtsgericht München, HRB 153243,
>>> Geschäftsführer: Charles Cachera, Laurie Krebs, Michael O'Neill,
>>> Thomas Savage
>>>
>> —
>> Sincerely,
>>
>> William Brown
>>
>> Senior Software Engineer, 389 Directory Server
>> SUSE Labs
>>
>
_______________________________________________
389-devel mailing list -- 389-devel@lists.fedoraproject.org
To unsubscribe send an email to 389-devel-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org