[389-users] Re: Lock table is out of available lock entries

On 09-08-2021 19:25, Mark Reynolds wrote:
> On 8/9/21 1:16 PM, Kees Bakker wrote:
>> On 09-08-2021 18:43, Mark Reynolds wrote:
>>> On 8/9/21 11:20 AM, Kees Bakker wrote:
>>>> On 09-08-2021 16:00, Mark Reynolds wrote:
>>>>> On 8/9/21 8:09 AM, Kees Bakker wrote:
>>>>>> Hi,
>>>>>>
>>>>>> When my dirsrv was trying to compact the databases I was getting this
>>>>>> error
>>>>>>
>>>>>> [07/Aug/2021:23:59:02.715984489 +0200] - NOTICE - bdb_compact -
>>>>>> Compacting databases ...
>>>>>> [07/Aug/2021:23:59:02.765932397 +0200] - NOTICE - bdb_compact -
>>>>>> Compacting DB start: userRoot
>>>>>> [07/Aug/2021:23:59:03.518175414 +0200] - NOTICE - bdb_compact -
>>>>>> compactdb: compact userRoot - 417 pages freed
>>>>>> [07/Aug/2021:23:59:03.576427786 +0200] - NOTICE - bdb_compact -
>>>>>> Compacting DB start: ipaca
>>>>>> [07/Aug/2021:23:59:03.659941533 +0200] - NOTICE - bdb_compact -
>>>>>> compactdb: compact ipaca - 419 pages freed
>>>>>> [07/Aug/2021:23:59:03.718445310 +0200] - NOTICE - bdb_compact -
>>>>>> Compacting DB start: changelog
>>>>>> [08/Aug/2021:00:00:40.807571334 +0200] - NOTICE -
>>>>>> NSMMReplicationPlugin - changelog program - cl5CompactDBs -
>>>>>> compacting
>>>>>> replication changelogs...
>>>>>> [08/Aug/2021:00:00:54.309357211 +0200] - ERR - libdb - BDB2055 Lock
>>>>>> table is out of available lock entries
>>>>>> [08/Aug/2021:00:00:54.726504736 +0200] - ERR - bdb_compact -
>>>>>> compactdb: failed to compact changelog; db error - 12 Cannot allocate
>>>>>> memory
>>>>>> [08/Aug/2021:00:00:54.801571421 +0200] - ERR - libdb - BDB2055 Lock
>>>>>> table is out of available lock entries
>>>>>> [08/Aug/2021:00:00:54.876618702 +0200] - ERR -
>>>>>> NSMMReplicationPlugin -
>>>>>> changelog program - cl5CompactDBs - Failed to compact
>>>>>> a797bb0b-be1d11eb-88c0b677-613aa2ad; db error - 12 Cannot allocate
>>>>>> memory
>>>>>> [08/Aug/2021:00:00:57.253006449 +0200] - NOTICE - bdb_compact -
>>>>>> Compacting databases finished.
>>>>>>
>>>>>> There are about 402k entries in cn=changelog.
>>>>>>
>>>>>> I have a few questions
>>>>>> 1) is it normal to have so many entries in cn=changelog? On another
>>>>>> replica I have almost 3M entries Isn't this cleaned up?
>>>>>> 2) the number of locks is 50000 (there are two config items).
>>>>>> Should I
>>>>>> increase that number? If so, increase to what?
>>>>>> 3) is there maybe something else going on, causing the exhaustion of
>>>>>> the locks?
>>>>>
>>>>> Ok, so by default there is no changelog trimming enabled. So the
>>>>> changelog will grow without bounds, which is bad.
>>>>
>>>> How much of this [1] applies? Indeed it says "By default the
>>>> Replication Changelog does not use any trimming by default."
>>>> So, how is the trimming actually enabled? I have these entries
>>>>
>>>> dn: cn=changelog5,cn=config
>>>> cn: changelog5
>>>> nsslapd-changelogdir: /var/lib/dirsrv/slapd-EXAMPLE-COM/cldb
>>>> nsslapd-changelogmaxage: 30d
>>>> objectClass: top
>>>> objectClass: extensibleobject
>>>
>>> So trimming is set, but it's set to 30 days (30d), that could/should be
>>> shortened to "7d".
>>
>> Somehow I doubt that this will do anything.
>
> Ahhh, I thought this was the Replication Changelog, but it is the
> "Retro" Changelog.  Ok so check this out:
>
> https://access.redhat.com/documentation/en-us/red_hat_directory_server/11/html/administration_guide/managing_replication-using_the_retro_changelog_plug_in#Using_the_Retro_Changelog_Plug_in-Trimming_the_Retro_Changelog
>
> Or update config directly using ldapmodify:
>
> # ldapmodify -D ...
> dn: cn=Retro Changelog Plugin,cn=plugins,cn=config
> changetype: modify
> replace: nsslapd-changelogmaxage: 7d
> nsslapd-changelogmaxage: 7d

(( Typo? The replace line is without the value ))

Hmm, that maxage was already at 2 days

dn: cn=Retro Changelog Plugin,cn=plugins,cn=config
cn: Retro Changelog Plugin
nsslapd-attribute: nsuniqueid:targetUniqueId
nsslapd-changelogmaxage: 2d
nsslapd-include-suffix: cn=dns,dc=example,dc=com
nsslapd-plugin-depends-on-named: Class of Service
nsslapd-plugin-depends-on-type: database
nsslapd-pluginDescription: Retrocl Plugin
nsslapd-pluginEnabled: on
nsslapd-pluginId: retrocl
nsslapd-pluginInitfunc: retrocl_plugin_init
nsslapd-pluginPath: libretrocl-plugin
nsslapd-pluginType: object
nsslapd-pluginVendor: 389 Project
nsslapd-pluginVersion: 1.4.3.231
nsslapd-pluginbetxn: on
nsslapd-pluginprecedence: 25
objectClass: top
objectClass: nsSlapdPlugin
objectClass: extensibleObject

>
>
> HTH,
>
> Mark
>
>>
>> The first entry in cn=changelog is more than two months old. (May 26th
>> is about
>> the time I installed this server as a FreeIPA replica.)
>>
>> dn: changenumber=1,cn=changelog
>> objectClass: top
>> objectClass: changelogentry
>> objectClass: extensibleObject
>> targetuniqueid: 1e89ebcb-e20111e8-8820f96e-fc0c60a4
>> changeNumber: 1
>> targetDn:
>> idnsname=_ldap._tcp,idnsname=example.com.,cn=dns,dc=example,dc=com
>> changeTime: 20210526122901Z
>> changeType: modify
>> changes:: YWRkOiBzUlZSZWNvc...gplbnRyeXVzbjogMTEyCi0KAA==
>>
>> Is there a way to see trimming in action? I'm afraid it is not doing
>> anything.

What diagnostic can I enable to see the trimming in action?

>>
>>>
>>>>
>>>>
>>>>>
>>>>> I recommend setting up the changelog max age to 7 days:
>>>>>
>>>>> https://access.redhat.com/documentation/en-us/red_hat_directory_server/11/html/administration_guide/trimming_the_replication_changelog
>>>>>
>>>>>
>>>>>
>>>>> Once that the trimming cleans up the changelog the database compaction
>>>>> should work without issue.  You can also increase the database
>>>>> locks to
>>>>> 1 million (that does require a restart of the server to take effect).
>>>>
>>>> Let's hope that 1 million is enough :-)
>>>
>>> It might not be, you can keep bumping it up if needed.  There is no
>>> limit, or negative impact on db performance.
>>>
>>> Mark
>>>
>>>>
>>>>>
>>>>> HTH,
>>>>>
>>>>> Mark
>>>>>
>>>> [1]
>>>> https://directory.fedoraproject.org/docs/389ds/FAQ/changelog-trimming.html
>>>>
>>>
>>> --
>>> Directory Server Development Team
>>>
>> _______________________________________________
>> 389-users mailing list -- 389-users@lists.fedoraproject.org
>> To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
>> Fedora Code of Conduct:
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org
>> Do not reply to spam on the list, report it:
>> https://pagure.io/fedora-infrastructure
>
> --
> Directory Server Development Team
>
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure