On Sun, 2018-03-18 at 21:57 -0500, Sergei Gerasenko wrote:
> Thank you for the detailed response, William. That's great info. You
> mentioned FreeIPA in passing and that's actually what I use 389-ds
> for. You mentioned dogtag eating memory. You mean it has a memory
> leak or some other memory mismanagement issue?
Dogtag is Java/Tomcat. It's well known for consuming large volumes of
ram!
I actually wrote the autotuning system specifically for FreeIPA since
it historically did no DS tuning. So I'm glad that you are finding it
useful!
>
> I have 125G of RAM on my systems. So, I can allocate quite a bit of
> RAM to the caches. My plan is to set nsslapd-cache-autosize to 20%
> and set the dncache size to 5G for all three backends. Does that
> sound reasonable to you? These are dedicated FreeIPA machines, so
> it's the main consumer of the memory.
Sure, sounds reasonable to me - I'd want to see your database sizes to
make a complete assesment, but it seems pretty reasonable to me.
>
> Also, can I set the dncachesize once at the cn=config,cn=ldbm
> database,cn=plugins,cn=config level and have all the other backends
> inherit that? Would I have to remove that parameter from the other
> backends for it to become inherited?
Sadly there is no method to inherit cache sizes.
Additionally, check that cn=config,cn=ldbm ... is actually
d'B'cachesize not d'N'cachesize.
Hope that helps!
>
> Thanks again, William!
> Sergei
>
> > On Mar 18, 2018, at 8:06 PM, William Brown <william@blackhats.net.a
> > u> wrote:
> >
> > On Tue, 2018-03-13 at 21:10 -0500, Sergei Gerasenko wrote:
> > > Hi William,
> > >
> > > With autosizing on I configured the changelog backend:
> > >
> > > dn: cn=changelog,cn=ldbm database,cn=plugins,cn=config
> > > cn: changelog
> > > objectClass: top
> > > objectClass: extensibleObject
> > > objectClass: nsBackendInstance
> > > nsslapd-suffix: cn=changelog
> > > nsslapd-cachesize: -1
> > > nsslapd-cachememsize: 8858370048
> > > nsslapd-readonly: off
> > > nsslapd-require-index: off
> > > nsslapd-directory: /var/lib/dirsrv/slapd-CNVR-NET/db/changelog
> > > nsslapd-dncachememsize: 3000000000
> > >
> > > Here's the ldif:
> > >
> > > dn: cn=changelog,cn=ldbm database,cn=plugins,cn=config
> > > changetype: modify
> > > replace: nsslapd-dncachememsize
> > > nsslapd-dncachememsize: 3000000000
> > > -
> > > replace: nsslapd-cachememsize
> > > nsslapd-cachememsize: 3000000000
> > >
> > > The server refused to change nsslapd-cachememsize because of
> > > autosizing but nsslapd-dncachememsize did get changed. So it
> > > seems
> > > that I can still control nsslapd-dncachememsize? When I restart
> > > 389-
> > > ds, the 3G persists as well.
> > >
> > > Does that make sense?
> >
> > Yes it does.
> >
> > >
> > > Thank you for a quick response!
> >
> > At the current point in time, dncachememsize is NOT controlled by
> > autosizing. There is some ideas in my head about how to improve
> > this,
> > but the issue is legacy. We have one control: autosize percentage.
> > But
> > this doesn't really represent a complete picture and story about
> > how we
> > size our backends and data.
> >
> > We have to continue to support the current variables, so I can't
> > easily
> > change this from it's current form.
> >
> > *IF* I was given unlimited power (I never will be ;) ) I would
> > probably
> > make the interface something like:
> >
> >
> >
> > backend-foo:
> > autosize-max-memory: <value in bytes/mb/gb>
> > entrycache-slice: A%
> > dncache-slice: B%
> > dbcache-slice: C%
> > ndncache-slice: D%
> >
> > So how would this work?
> >
> > You have autosize max that is the "total ram" we want to use. Lets
> > say
> > 2GB. Each backend would be given it's own "limit". We'd try to
> > detect
> > some sane values on your system of course, but we can only do so
> > much
> > :) So say ... each backend gets 5-10% of system ram limit. (We have
> > to
> > be super conservative due to dogtag in freeipa which eats ram).
> >
> > Then we slice that up. So entrycache-slice + dncache-slice +
> > dbcache-
> > slice + ndncache-slice = 100%. So by default we might do something
> > like:
> >
> > entrycache-slice: 75
> > dbcache-slice: 10
> > dncache-slice: 10
> > ndncache-slice: 5
> >
> > So this on a 1GB backend translates to:
> >
> > 750MB of entrycache
> > 100MB of dncache
> > 100MB of dbcache
> > 50MB of ndncache
> >
> > of course some tests would be needed to find the right slice values
> > by
> > default.
> >
> >
> > Then it's as simple as "if you want more from your backend" just up
> > the
> > one memlimit value and "everything" increases in a correct
> > proportional
> > rate. Rather than thinking about how to hand tune everything, we
> > make
> > informed design decisions, you just say "here's mem max". You have
> > control to say each backend gets a different value (say FreeIPA you
> > may
> > want 4GB to userRoot, 1GB to CA system backend). you still have
> > lots of
> > flexibility as an admin, but you don't have to worry about so many
> > moving interacting parts.
> >
> >
> > Hope that helps,
> >
> > >
> > > Sergei
> > >
> > >
> > > > On Mar 13, 2018, at 7:40 PM, William Brown <william@blackhats.n
> > > > et.a
> > > > u> wrote:
> > > >
> > > > If autosize > 0, we write a new entrychace/cachememsize every
> > > > start
> > > > up.
> > > >
> > > > So you only need to set autosize to between 1 and 99 for it to
> > > > work.
> > > >
> > > > There is some other logic in there to account for other
> > > > scenarioes
> > > > =
> > > > for example, if you set autosize to 0 AND you set the entry
> > > > cachesize
> > > > to 0, we'll autosize it at start up anyway. BUT if you have
> > > > autosize =
> > > > 0, and entry cachesize > 0, we won't touch it.
> > >
> > > _______________________________________________
> > > 389-users mailing list -- 389-users@lists.fedoraproject.org
> > > To unsubscribe send an email to 389-users-leave@lists.fedoraproje
> > > ct.o
> > > rg
> >
> > --
> > Thanks,
> >
> > William Brown
> > _______________________________________________
> > 389-users mailing list -- 389-users@lists.fedoraproject.org
> > To unsubscribe send an email to 389-users-leave@lists.fedoraproject
> > .org
>
> _______________________________________________
> 389-users mailing list -- 389-users@lists.fedoraproject.org
> To unsubscribe send an email to 389-users-leave@lists.fedoraproject.o
> rg
--
Thanks,
William Brown
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
No comments:
Post a Comment