Tuesday, September 12, 2023

[389-users] Re: 389-ds freezes with deadlock

Hi,

there are two threads that are at 100% CPU utilisation. I did not start
any admin task myself, maybe it is some built-in task that is doing
this? Or could an unindexed search on the changelog be causing this?

I have noticed this message:
NOTICE - ldbm_back_search - Unindexed search: search base="cn=changelog"
scope=1 filter="(changeNumber>=1)" conn=35871 op=1

There is an external server that is reading the changelog and syncing
some stuff depending on that. I don't know why they are starting at
changeNumber>=1, they probably should start way higher. If it is
possible that this is the cause I will kick them to stop that ;)

I am running version 2.3.1 on Debian 12, installed from the Debian
repositories.

Kind regards
Julian

Am 08.09.23 um 13:23 schrieb Thierry Bordaz:
> Hi Julian,
>
> It looks that an update (Thread 62) is either eating CPU either is
> blocked while update the changelog.
> When it occurs could you run 'top -H -p <pid>' to see if some thread are
> eating CPU.
> Else (no cpu consumption), you may take a pstack and dump DB lock info
> (db_stat -N -C A -h /var/lib/dirsrv/<inst>db)
>
> Did you run admin task (import/export/index...) before it occurred ?
> What version are you running ?
>
> best regards
> Thierry
>
> On 9/8/23 09:28, Julian Kippels wrote:
>> Hi,
>>
>> it happened again and now I ran the gdb-command like Mark suggested.
>> The Stacktrace is attached. Again I got this error message:
>>
>> [07/Sep/2023:15:22:43.410333038 +0200] - ERR - ldbm_back_seq -
>> deadlock retry BAD 1601, err=0 Unexpected dbimpl error code
>>
>> and the remote program that called also stopped working at that time.
>>
>> Thanks
>> Julian Kippels
>>
>> Am 28.08.23 um 14:28 schrieb Thierry Bordaz:
>>> Hi Julian,
>>>
>>> I agree with Mark suggestion. If new connections are failing a pstack
>>> + error logged msg would be helpful.
>>>
>>> Regarding the error logged. LDAP server relies on a database that,
>>> under pressure by multiple threads, may end into a db_lock deadlock.
>>> In such situation the DB, selects one deadlocking thread, returns a
>>> DB_Deadlock error to that thread while the others threads continue to
>>> proceed. This is very normal error that is caught by the server that
>>> simply retries to access the DB. If the same thread fails to many
>>> time, it stops retry and return a fatal error to the request.
>>>
>>> In your case it reports code 1601 that is transient deadlock with
>>> retry. So the impacted request just retried and likely succeeded.
>>>
>>> best regards
>>> thierry
>>>
>>> On 8/24/23 14:46, Mark Reynolds wrote:
>>>> Hi Julian,
>>>>
>>>> It would be helpful to get a pstack/stacktrace so we can see where
>>>> DS is stuck:
>>>>
>>>> https://www.port389.org/docs/389ds/FAQ/faq.html#sts=Debugging%C2%A0Hangs
>>>>
>>>> Thanks,
>>>> Mark
>>>>
>>>> On 8/24/23 4:13 AM, Julian Kippels wrote:
>>>>> Hi,
>>>>>
>>>>> I am using 389-ds Version 2.3.1 and have encountered the same error
>>>>> twice in three days now. There are some MOD operations and then I
>>>>> get a line like this in the errors-log:
>>>>>
>>>>> [23/Aug/2023:13:27:17.971884067 +0200] - ERR - ldbm_back_seq -
>>>>> deadlock retry BAD 1601, err=0 Unexpected dbimpl error code
>>>>>
>>>>> After this the server keeps running, systemctl status says
>>>>> everything is fine, but new incoming connections are failing with
>>>>> timeouts.
>>>>>
>>>>> Any advice would be welcome.
>>>>>
>>>>> Thanks in advance
>>>>> Julian Kippels
>>>>>
>>>>> _______________________________________________
>>>>> 389-users mailing list -- 389-users@lists.fedoraproject.org
>>>>> To unsubscribe send an email to
>>>>> 389-users-leave@lists.fedoraproject.org
>>>>> Fedora Code of Conduct:
>>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>> List Guidelines:
>>>>> https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>> List Archives:
>>>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org
>>>>> Do not reply to spam, report it:
>>>>> https://pagure.io/fedora-infrastructure/new_issue
>>>>
>>> _______________________________________________
>>> 389-users mailing list -- 389-users@lists.fedoraproject.org
>>> To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
>>> Fedora Code of Conduct:
>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives:
>>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org
>>> Do not reply to spam, report it:
>>> https://pagure.io/fedora-infrastructure/new_issue
>>
>>
>> _______________________________________________
>> 389-users mailing list -- 389-users@lists.fedoraproject.org
>> To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
>> Fedora Code of Conduct:
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org
>> Do not reply to spam, report it:
>> https://pagure.io/fedora-infrastructure/new_issue
> _______________________________________________
> 389-users mailing list -- 389-users@lists.fedoraproject.org
> To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

--
---------------------------------------------------------
| | Julian Kippels
| | M.Sc. Informatik
| |
| | Zentrum für Informations- und Medientechnologie
| | Heinrich-Heine-Universität Düsseldorf
| | Universitätsstr. 1
| | Raum 25.41.O1.32
| | 40225 Düsseldorf / Germany
| |
| | Tel: +49-211-81-14920
| | mail: kippels@hhu.de
---------------------------------------------------------

No comments:

Post a Comment