[389-users] Re: ldapsearch performance problem

On 06/15/2018 05:41 PM, Jan Kowalsky wrote:
> Hi Marc,
>
> thanks for help.
>
> Am 15.06.2018 um 22:50 schrieb Mark Reynolds:
>> You did not run logconv.pl the way I requested, can you please run it
>> again this way:
>>
>> logconv.pl -ulatn  <access logs>
> I omitted the detailed searches because there are user-data in it...
>
> but this here does't look like this is a problem:
>
> ----- Top 20 Most Frequent etimes -----
>
> 64902 etime=0
> 157 etime=1
> 34 etime=3
> 27 etime=2
> 18 etime=4
>
>> I think the next step, besides gathering the right logconv output, would
>> be to get several pstacks of the ns-slapd process at the same time the
>> "hang" occurs.  Then we can at least see what all the threads are doing
>> and hopefully get better information.  I would try and get the
>> 389-ds-base-debuginfo package installed first though (this should give
>> us more readable pstacks).
> hm, I installed 389-ds-base-dbg and 389-ds-base-libs-dbg. What else do I
> have to do?
>
> pstack gives me:
>
> pstack 13463
>
> 13463: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-ldap0 -i
> /var/run/dirsrv/slapd-ldap0.pid -w /var/run/dirsrv/slapd-ldap0.startpid
> (No symbols found)
> 0x7f808a7717bc: ???? (7f803c03cf20, 0, 7f803c0421f0, 0, 7f808c0dc5d6,
> 5615f1d4bc80) + ffffd695834bb170
> 0x300000157: ???? (0, 200000, 0, 31, 3, 0) + ffffa9ea0c848bd3
> crawl: Input/output error
> Error tracing through process 13463
>
> Does this help:
Are you sure the "debuginfo" packages are the same version as
389-ds-base/libs?

Run:  rpm -qa | grep 389-ds-base

The other option is attach gdb to the process and get a stack trace that
way:

# gdb -p PID
(gdb) thread apply all bt
(gdb) quit


>
> If everything runs fast:
>
> root@ldap0:~# cat /proc/13463/stack
> [<ffffffff810eaa9a>] hrtimer_start_range_ns+0x1aa/0x3d0
> [<ffffffff810ea660>] hrtimer_init+0x110/0x110
> [<ffffffff8121ce26>] poll_schedule_timeout+0x46/0x70
> [<ffffffff8121e50f>] do_sys_poll+0x44f/0x560
> [<ffffffff814aab95>] loopback_xmit+0x65/0xa0
> [<ffffffff815785e5>] tcp_rate_gen+0x105/0x180
> [<ffffffff81565fc7>] tcp_ack+0xd97/0x16e0
> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
> [<ffffffff81209bcd>] __fput+0x17d/0x220
> [<ffffffff8102f530>] recalibrate_cpu_khz+0x10/0x10
> [<ffffffff810efbf6>] ktime_get_ts64+0x46/0xf0
> [<ffffffff8121e707>] SyS_poll+0x67/0x120
> [<ffffffff81003bd1>] do_syscall_64+0x91/0x1a0
> [<ffffffff816137ce>] entry_SYSCALL_64_after_swapgs+0x58/0xc6
> [<ffffffffffffffff>] 0xffffffffffffffff
>
>
> When it hangs:
>
> root@ldap0:~# cat /proc/13463/stack
> [<ffffffff810fa433>] futex_wait_queue_me+0xd3/0x140
> [<ffffffff810fb2cc>] futex_wait+0xfc/0x260
> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
> [<ffffffff810fce17>] do_futex+0x2b7/0xb40
> [<ffffffff810e81e7>] lock_timer_base+0x67/0x80
> [<ffffffff810e839a>] internal_add_timer+0x1a/0x70
> [<ffffffff81202d3b>] __check_object_size+0x10b/0x1dc
> [<ffffffff814eeece>] move_addr_to_user+0xbe/0xe0
> [<ffffffff814ef160>] SYSC_getsockname+0x80/0xd0
> [<ffffffff810fd723>] SyS_futex+0x83/0x180
> [<ffffffff814efb17>] SyS_setsockopt+0xb7/0xf0
> [<ffffffff81003bd1>] do_syscall_64+0x91/0x1a0
> [<ffffffff816137ce>] entry_SYSCALL_64_after_swapgs+0x58/0xc6
> [<ffffffffffffffff>] 0xffffffffffffffff
Its hard to say for sure if this relates or not to the issue.  This
looks to be more of OS issue (polling) than a DS issue, but I can't say
be sure.  Lets try and get some useful stack traces using gdb and go
from there.

Thanks,
Mark
>
> This is reproducable.
>
> Regards
> Jan
> _______________________________________________
> 389-users mailing list -- 389-users@lists.fedoraproject.org
> To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/AH4EBT7XK6T6F6GHOIW4V46IVOAWXTRN/
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/HPUTRKEP7OUVUEBDEUVTUU2YPKGABJA3/