[389-users] Re: ldapsearch performance problem

On 06/15/2018 11:50 PM, Mark Reynolds wrote:
>
>
> On 06/15/2018 05:41 PM, Jan Kowalsky wrote:
>> Hi Marc,
>>
>> thanks for help.
>>
>> Am 15.06.2018 um 22:50 schrieb Mark Reynolds:
>>> You did not run logconv.pl the way I requested, can you please run it
>>> again this way:
>>>
>>> logconv.pl -ulatn <access logs>
>> I omitted the detailed searches because there are user-data in it...
>>
>> but this here does't look like this is a problem:
>>
>> ----- Top 20 Most Frequent etimes -----
>>
>> 64902 etime=0
>> 157 etime=1
>> 34 etime=3
>> 27 etime=2
>> 18 etime=4
>>
>>> I think the next step, besides gathering the right logconv output,
>>> would
>>> be to get several pstacks of the ns-slapd process at the same time the
>>> "hang" occurs. Then we can at least see what all the threads are doing
>>> and hopefully get better information. I would try and get the
>>> 389-ds-base-debuginfo package installed first though (this should give
>>> us more readable pstacks).
>> hm, I installed 389-ds-base-dbg and 389-ds-base-libs-dbg. What else do I
>> have to do?
>>
>> pstack gives me:
>>
>> pstack 13463
>>
>> 13463: /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-ldap0 -i
>> /var/run/dirsrv/slapd-ldap0.pid -w /var/run/dirsrv/slapd-ldap0.startpid
>> (No symbols found)
>> 0x7f808a7717bc: ???? (7f803c03cf20, 0, 7f803c0421f0, 0, 7f808c0dc5d6,
>> 5615f1d4bc80) + ffffd695834bb170
>> 0x300000157: ???? (0, 200000, 0, 31, 3, 0) + ffffa9ea0c848bd3
>> crawl: Input/output error
>> Error tracing through process 13463
>>
>> Does this help:
> Are you sure the "debuginfo" packages are the same version as
> 389-ds-base/libs?
>
> Run: rpm -qa | grep 389-ds-base
>
> The other option is attach gdb to the process and get a stack trace
> that way:
>
> # gdb -p PID
> (gdb) thread apply all bt
> (gdb) quit
>
>
>>
>> If everything runs fast:
>>
>> root@ldap0:~# cat /proc/13463/stack
>> [<ffffffff810eaa9a>] hrtimer_start_range_ns+0x1aa/0x3d0
>> [<ffffffff810ea660>] hrtimer_init+0x110/0x110
>> [<ffffffff8121ce26>] poll_schedule_timeout+0x46/0x70
>> [<ffffffff8121e50f>] do_sys_poll+0x44f/0x560
>> [<ffffffff814aab95>] loopback_xmit+0x65/0xa0
>> [<ffffffff815785e5>] tcp_rate_gen+0x105/0x180
>> [<ffffffff81565fc7>] tcp_ack+0xd97/0x16e0
>> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
>> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
>> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
>> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
>> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
>> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
>> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
>> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
>> [<ffffffff81209bcd>] __fput+0x17d/0x220
>> [<ffffffff8102f530>] recalibrate_cpu_khz+0x10/0x10
>> [<ffffffff810efbf6>] ktime_get_ts64+0x46/0xf0
>> [<ffffffff8121e707>] SyS_poll+0x67/0x120
>> [<ffffffff81003bd1>] do_syscall_64+0x91/0x1a0
>> [<ffffffff816137ce>] entry_SYSCALL_64_after_swapgs+0x58/0xc6
>> [<ffffffffffffffff>] 0xffffffffffffffff
>>
>>
>> When it hangs:
>>
>> root@ldap0:~# cat /proc/13463/stack
>> [<ffffffff810fa433>] futex_wait_queue_me+0xd3/0x140
>> [<ffffffff810fb2cc>] futex_wait+0xfc/0x260
>> [<ffffffff8121cff0>] poll_select_copy_remaining+0x150/0x150
>> [<ffffffff810fce17>] do_futex+0x2b7/0xb40
>> [<ffffffff810e81e7>] lock_timer_base+0x67/0x80
>> [<ffffffff810e839a>] internal_add_timer+0x1a/0x70
>> [<ffffffff81202d3b>] __check_object_size+0x10b/0x1dc
>> [<ffffffff814eeece>] move_addr_to_user+0xbe/0xe0
>> [<ffffffff814ef160>] SYSC_getsockname+0x80/0xd0
>> [<ffffffff810fd723>] SyS_futex+0x83/0x180
>> [<ffffffff814efb17>] SyS_setsockopt+0xb7/0xf0
>> [<ffffffff81003bd1>] do_syscall_64+0x91/0x1a0
>> [<ffffffff816137ce>] entry_SYSCALL_64_after_swapgs+0x58/0xc6
>> [<ffffffffffffffff>] 0xffffffffffffffff
> Its hard to say for sure if this relates or not to the issue. This
> looks to be more of OS issue (polling) than a DS issue, but I can't
> say be sure. Lets try and get some useful stack traces using gdb and
> go from there.
these stacks don't look like slapd stacks, also for ns-slapd process
there should be many threads.
Are you sure you run pstack on the right process ?
>
> Thanks,
> Mark
>>
>> This is reproducable.
>>
>> Regards
>> Jan
>> _______________________________________________
>> 389-users mailing list -- 389-users@lists.fedoraproject.org
>> To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>> https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/AH4EBT7XK6T6F6GHOIW4V46IVOAWXTRN/
> _______________________________________________
> 389-users mailing list -- 389-users@lists.fedoraproject.org
> To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/HPUTRKEP7OUVUEBDEUVTUU2YPKGABJA3/

--
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org/message/USROEUTDCMWYYS5CY32GQO5KLZ2AHQSP/