text:
Flake: ASAN memory leak for ProxyEnd2end/ProxyEnd2endTest.RpcDeadlineExpires test

ProxyEnd2end/ProxyEnd2endTest.RpcDeadlineExpires/2 test sometimes fails with the following ASAN failure.
Repro:  Do the following at grpc root:
tools/run_tests/run_tests.py -lc++ -casan -r ".*RpcDeadlineExpires/2" --force_use_pollers epollex -n inf -S
After a few minutes (somtimes takes as many as 3000 runs or so, it fails with the following ASAN failure)
=================================================================
==165659==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 320 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x8a87b8 in grpc_raw_compressed_byte_buffer_create /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/byte_buffer.cc:36:26
    #3 0x8a863f in grpc_raw_byte_buffer_create /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/byte_buffer.cc:27:10
    #4 0x8c0a70 in process_data_after_md(batch_control*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/call.cc:1388:33
    #5 0x8be463 in receiving_stream_ready(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/call.cc:1412:5
    #6 0x8bb917 in receiving_stream_ready_in_call_combiner(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/call.cc:1424:3
    #7 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #8 0x721550 in grpc_closure_run(char const*, int, grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/./src/core/lib/iomgr/closure.h:258:5
    #9 0xd97234 in recv_message_ready(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/message_size/message_size_filter.cc:135:3
    #10 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #11 0x721550 in grpc_closure_run(char const*, int, grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/./src/core/lib/iomgr/closure.h:258:5
    #12 0xd2a071 in hs_recv_message_ready(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/http/server/http_server_filter.cc:305:5
    #13 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #14 0x711a34 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:125:9
    #15 0x786165 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1011:7
    #16 0x70e5e5 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:247:21
    #17 0x8d325b in cq_next(grpc_completion_queue*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:925:23
    #18 0x8cfdf0 in grpc_completion_queue_next /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1001:10
    #19 0x802775 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) /usr/local/google/home/sreek/workspace/grpc/src/cpp/common/completion_queue_cc.cc:56:15
    #20 0x849787 in grpc::CompletionQueue::NextStatus grpc::CompletionQueue::AsyncNext<gpr_timespec>(void**, bool*, gpr_timespec const&) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/completion_queue.h:188:12
    #21 0x84cdf7 in grpc::Server::SyncRequestThreadManager::PollForWork(void**, bool*) /usr/local/google/home/sreek/workspace/grpc/src/cpp/server/server_cc.cc:276:13
    #22 0x86831c in grpc::ThreadManager::MainWorkLoop() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:122:30
    #23 0x8680f5 in grpc::ThreadManager::WorkerThread::Run() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:38:3
    #24 0x86d564 in void std::__invoke_impl<void, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(std::__invoke_memfun_deref, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227:14
    #25 0x86d3a4 in std::result_of<void (grpc::ThreadManager::WorkerThread::* const&(grpc::ThreadManager::WorkerThread*&&))()>::type std::__invoke<void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250:14
    #26 0x86d292 in _ZNKSt12_Mem_fn_baseIMN4grpc13ThreadManager12WorkerThreadEFvvELb1EEclIJPS2_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOS8_ /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604:11
    #27 0x86d1cf in void std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390:18
    #28 0x86d14f in std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380:16
    #29 0x86cbb6 in std::thread::_State_impl<std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)> >::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197:13

Objects leaked above:
0x612000047d40 (320 bytes)

Indirect leak of 17015 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x875c26 in gpr_malloc_aligned /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:92:13
    #3 0x9c05a4 in zalloc_aligned(unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/arena.cc:48:15
    #4 0x9c099b in gpr_arena_alloc(gpr_arena*, unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/arena.cc:82:23
    #5 0x9c0c4d in gpr_arena_alloc(gpr_arena*, unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/arena.cc:94:12
    #6 0x8ab326 in grpc_call_create(grpc_call_create_args const*, grpc_call**) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/call.cc:333:22
    #7 0x8e4c20 in accept_stream(void*, grpc_transport*, void const*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/server.cc:780:23
    #8 0xc93b4e in grpc_chttp2_parsing_accept_stream(grpc_chttp2_transport*, unsigned int) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:789:3
    #9 0xd0a9b2 in init_header_frame_parser(grpc_chttp2_transport*, int) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/parsing.cc:593:9
    #10 0xd08682 in init_frame_parser(grpc_chttp2_transport*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/parsing.cc:289:14
    #11 0xd0695e in grpc_chttp2_perform_read(grpc_chttp2_transport*, grpc_slice) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/parsing.cc:174:13
    #12 0xcb6b3d in read_action_locked(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:2416:19
    #13 0x7716fe in grpc_combiner_continue_exec_ctx() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/combiner.cc:257:5
    #14 0x711a97 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:128:17
    #15 0x786165 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1011:7
    #16 0x70e5e5 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:247:21
    #17 0x8d325b in cq_next(grpc_completion_queue*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:925:23
    #18 0x8cfdf0 in grpc_completion_queue_next /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1001:10
    #19 0x802775 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) /usr/local/google/home/sreek/workspace/grpc/src/cpp/common/completion_queue_cc.cc:56:15
    #20 0x849787 in grpc::CompletionQueue::NextStatus grpc::CompletionQueue::AsyncNext<gpr_timespec>(void**, bool*, gpr_timespec const&) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/completion_queue.h:188:12
    #21 0x84cdf7 in grpc::Server::SyncRequestThreadManager::PollForWork(void**, bool*) /usr/local/google/home/sreek/workspace/grpc/src/cpp/server/server_cc.cc:276:13
    #22 0x86831c in grpc::ThreadManager::MainWorkLoop() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:122:30
    #23 0x8680f5 in grpc::ThreadManager::WorkerThread::Run() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:38:3
    #24 0x86d564 in void std::__invoke_impl<void, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(std::__invoke_memfun_deref, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227:14
    #25 0x86d3a4 in std::result_of<void (grpc::ThreadManager::WorkerThread::* const&(grpc::ThreadManager::WorkerThread*&&))()>::type std::__invoke<void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250:14
    #26 0x86d292 in _ZNKSt12_Mem_fn_baseIMN4grpc13ThreadManager12WorkerThreadEFvvELb1EEclIJPS2_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOS8_ /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604:11
    #27 0x86d1cf in void std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390:18
    #28 0x86d14f in std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380:16
    #29 0x86cbb6 in std::thread::_State_impl<std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)> >::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197:13

Objects leaked above:
0x62900000f200 (17015 bytes)

Indirect leak of 16520 byte(s) in 1 object(s) allocated from:
    #0 0x4cdf10 in calloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdf10)
    #1 0x875d56 in zalloc_with_calloc(unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:27:53
    #2 0x875812 in gpr_zalloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:67:7
    #3 0xca9d0f in grpc_create_chttp2_transport(grpc_channel_args const*, grpc_endpoint*, bool) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:3100:31
    #4 0x95f17a in on_handshake_done(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/server/chttp2_server.cc:130:11
    #5 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #6 0x711a34 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:125:9
    #7 0x786165 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1011:7
    #8 0x70e5e5 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:247:21
    #9 0x8d325b in cq_next(grpc_completion_queue*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:925:23
    #10 0x8cfdf0 in grpc_completion_queue_next /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1001:10
    #11 0x802775 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) /usr/local/google/home/sreek/workspace/grpc/src/cpp/common/completion_queue_cc.cc:56:15
    #12 0x849787 in grpc::CompletionQueue::NextStatus grpc::CompletionQueue::AsyncNext<gpr_timespec>(void**, bool*, gpr_timespec const&) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/completion_queue.h:188:12
    #13 0x84cdf7 in grpc::Server::SyncRequestThreadManager::PollForWork(void**, bool*) /usr/local/google/home/sreek/workspace/grpc/src/cpp/server/server_cc.cc:276:13
    #14 0x86831c in grpc::ThreadManager::MainWorkLoop() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:122:30
    #15 0x8680f5 in grpc::ThreadManager::WorkerThread::Run() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:38:3
    #16 0x86d564 in void std::__invoke_impl<void, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(std::__invoke_memfun_deref, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227:14
    #17 0x86d3a4 in std::result_of<void (grpc::ThreadManager::WorkerThread::* const&(grpc::ThreadManager::WorkerThread*&&))()>::type std::__invoke<void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250:14
    #18 0x86d292 in _ZNKSt12_Mem_fn_baseIMN4grpc13ThreadManager12WorkerThreadEFvvELb1EEclIJPS2_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOS8_ /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604:11
    #19 0x86d1cf in void std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390:18
    #20 0x86d14f in std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380:16
    #21 0x86cbb6 in std::thread::_State_impl<std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)> >::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197:13
    #22 0x7f46d961be6e  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb9e6e)

...    #26 0x7f46d961be6e  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb9e6e)

Objects leaked above:
0x61100004ff00 (224 bytes)

Indirect leak of 168 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x702785 in grpc_error_create(char const*, int, grpc_slice, grpc_error**, unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/error.cc:321:34
    #3 0xcb1da2 in write_action_end_locked(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:1028:14
    #4 0x7716fe in grpc_combiner_continue_exec_ctx() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/combiner.cc:257:5
    #5 0x711a97 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:128:17
    #6 0x7ba573 in run_closures(grpc_closure_list) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/executor.cc:80:5
    #7 0x7b9e72 in executor_thread(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/executor.cc:177:22
    #8 0x8814f6 in thread_body(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/thd_posix.cc:67:3
    #9 0x7f46d9b11493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

Objects leaked above:
0x60f000006fd0 (168 bytes)

Indirect leak of 168 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x784708 in fd_create(int, char const*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:341:24
    #3 0x70dbb1 in grpc_fd_create(int, char const*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:193:10
    #4 0xc7630a in on_read(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/tcp_server_posix.cc:247:22
    #5 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #6 0x711a34 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:125:9
    #7 0x786165 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1011:7
    #8 0x70e5e5 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:247:21
    #9 0x8d325b in cq_next(grpc_completion_queue*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:925:23
    #10 0x8cfdf0 in grpc_completion_queue_next /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1001:10
    #11 0x802775 in grpc::CompletionQueue::AsyncNextInternal(void**, bool*, gpr_timespec) /usr/local/google/home/sreek/workspace/grpc/src/cpp/common/completion_queue_cc.cc:56:15
    #12 0x849787 in grpc::CompletionQueue::NextStatus grpc::CompletionQueue::AsyncNext<gpr_timespec>(void**, bool*, gpr_timespec const&) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/completion_queue.h:188:12
    #13 0x84cdf7 in grpc::Server::SyncRequestThreadManager::PollForWork(void**, bool*) /usr/local/google/home/sreek/workspace/grpc/src/cpp/server/server_cc.cc:276:13
    #14 0x86831c in grpc::ThreadManager::MainWorkLoop() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:122:30
    #15 0x8680f5 in grpc::ThreadManager::WorkerThread::Run() /usr/local/google/home/sreek/workspace/grpc/src/cpp/thread_manager/thread_manager.cc:38:3
    #16 0x86d564 in void std::__invoke_impl<void, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(std::__invoke_memfun_deref, void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:227:14
    #17 0x86d3a4 in std::result_of<void (grpc::ThreadManager::WorkerThread::* const&(grpc::ThreadManager::WorkerThread*&&))()>::type std::__invoke<void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*>(void (grpc::ThreadManager::WorkerThread::* const&)(), grpc::ThreadManager::WorkerThread*&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:250:14
    #18 0x86d292 in _ZNKSt12_Mem_fn_baseIMN4grpc13ThreadManager12WorkerThreadEFvvELb1EEclIJPS2_EEEDTclsr3stdE8__invokedtdefpT6_M_pmfspclsr3stdE7forwardIT_Efp_EEEDpOS8_ /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:604:11
    #19 0x86d1cf in void std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1390:18
    #20 0x86d14f in std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)>::operator()() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/functional:1380:16
    #21 0x86cbb6 in std::thread::_State_impl<std::_Bind_simple<std::_Mem_fn<void (grpc::ThreadManager::WorkerThread::*)()> (grpc::ThreadManager::WorkerThread*)> >::_M_run() /usr/bin/../lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/thread:197:13
    #22 0x7f46d961be6e  (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xb9e6e)

Objects leaked above:
0x60f00001df50 (168 bytes)

Indirect leak of 168 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x702785 in grpc_error_create(char const*, int, grpc_slice, grpc_error**, unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/error.cc:321:34
    #3 0x99230c in timer_callback(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/deadline/deadline_filter.cc:65:9
    #4 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #5 0x711a34 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:125:9
    #6 0x7403e1 in run_some_timers() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/timer_manager.cc:128:3
    #7 0x73fcba in timer_main_loop() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/timer_manager.cc:230:9
    #8 0x73fa34 in timer_thread(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/timer_manager.cc:277:3
    #9 0x8814f6 in thread_body(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/thd_posix.cc:67:3
    #10 0x7f46d9b11493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

Objects leaked above:
0x60f000016570 (168 bytes)

Indirect leak of 168 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x702785 in grpc_error_create(char const*, int, grpc_slice, grpc_error**, unsigned long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/error.cc:321:34
    #3 0xc991da in close_transport_locked(grpc_chttp2_transport*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:594:13
    #4 0xcb1db2 in write_action_end_locked(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:1027:7
    #5 0x7716fe in grpc_combiner_continue_exec_ctx() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/combiner.cc:257:5
    #6 0x711a97 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:128:17
    #7 0x7ba573 in run_closures(grpc_closure_list) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/executor.cc:80:5
    #8 0x7b9e72 in executor_thread(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/executor.cc:177:22
    #9 0x8814f6 in thread_body(void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/thd_posix.cc:67:3
    #10 0x7f46d9b11493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493)

Objects leaked above:
0x60f000006ee0 (168 bytes)

Indirect leak of 137 byte(s) in 1 object(s) allocated from:
    #0 0x4cdd38 in __interceptor_malloc (/usr/local/google/home/sreek/workspace/grpc/bins/asan/end2end_test+0x4cdd38)
    #1 0x8756c2 in gpr_malloc /usr/local/google/home/sreek/workspace/grpc/src/core/lib/gpr/alloc.cc:56:7
    #2 0x753aa1 in grpc_slice_intern /usr/local/google/home/sreek/workspace/grpc/src/core/lib/slice/slice_intern.cc:240:33
    #3 0xff1ba3 in user_agent_from_args(grpc_channel_args const*, char const*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/http/client/http_client_filter.cc:513:12
    #4 0xfebce2 in init_channel_elem(grpc_channel_element*, grpc_channel_element_args*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/http/client/http_client_filter.cc:530:7
    #5 0xc6829e in grpc_channel_stack_init(int, void (*)(void*, grpc_error*), void*, grpc_channel_filter const**, unsigned long, grpc_channel_args const*, grpc_transport*, char const*, grpc_channel_stack*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/channel/channel_stack.cc:120:25
    #6 0x8a31fb in grpc_channel_stack_builder_finish(grpc_channel_stack_builder*, unsigned long, int, void (*)(void*, grpc_error*), void*, void**) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/channel/channel_stack_builder.cc:283:23
    #7 0x985e3a in publish_transport_locked(grpc_subchannel*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/client_channel/subchannel.cc:573:23
    #8 0x9809a6 in on_subchannel_connected(void*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/ext/filters/client_channel/subchannel.cc:628:7
    #9 0x71200d in exec_ctx_run(grpc_closure*, grpc_error*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:38:3
    #10 0x711a34 in grpc_core::ExecCtx::Flush() /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/exec_ctx.cc:125:9
    #11 0x786165 in pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1011:7
    #12 0x70e5e5 in grpc_pollset_work(grpc_pollset*, grpc_pollset_worker**, long) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/iomgr/ev_posix.cc:247:21
    #13 0x8d6855 in cq_pluck(grpc_completion_queue*, void*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1163:9
    #14 0x8d022b in grpc_completion_queue_pluck /usr/local/google/home/sreek/workspace/grpc/src/core/lib/surface/completion_queue.cc:1190:10
    #15 0x8038c1 in grpc::CoreCodegen::grpc_completion_queue_pluck(grpc_completion_queue*, void*, gpr_timespec, void*) /usr/local/google/home/sreek/workspace/grpc/src/cpp/common/core_codegen.cc:70:10
    #16 0x602eed in grpc::CompletionQueue::Pluck(grpc::internal::CompletionQueueTag*) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/completion_queue.h:296:15
    #17 0x69442d in grpc::internal::BlockingUnaryCallImpl<grpc::testing::EchoRequest, grpc::testing::EchoResponse>::BlockingUnaryCallImpl(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, grpc::testing::EchoRequest const&, grpc::testing::EchoResponse*) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/client_unary_call.h:72:9
    #18 0x689e1b in grpc::Status grpc::internal::BlockingUnaryCall<grpc::testing::EchoRequest, grpc::testing::EchoResponse>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, grpc::testing::EchoRequest const&, grpc::testing::EchoResponse*) /usr/local/google/home/sreek/workspace/grpc/include/grpc++/impl/codegen/client_unary_call.h:41:10
    #19 0x684347 in grpc::testing::EchoTestService::Stub::Echo(grpc::ClientContext*, grpc::testing::EchoRequest const&, grpc::testing::EchoResponse*) /usr/local/google/home/sreek/workspace/grpc/gens/src/proto/grpc/testing/echo.grpc.pb.cc:42:10
    #20 0x5826b0 in grpc::testing::(anonymous namespace)::ProxyEnd2endTest_RpcDeadlineExpires_Test::TestBody() /usr/local/google/home/sreek/workspace/grpc/test/cpp/end2end/end2end_test.cc:1299:14
    #21 0x160296b in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2402:10
    #22 0x15d1000 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2438:14
    #23 0x159b609 in testing::Test::Run() /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2474:5
    #24 0x159d511 in testing::TestInfo::Run() /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2656:5
    #25 0x159e6a1 in testing::TestCase::Run() /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2774:5
    #26 0x15ae2cc in testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:4649:11
    #27 0x160bd46 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2402:10
    #28 0x15d6620 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:2438:14
    #29 0x15ad8c1 in testing::UnitTest::Run() /usr/local/google/home/sreek/workspace/grpc/third_party/googletest/googletest/src/gtest.cc:4257:10
...

SUMMARY: AddressSanitizer: 67327 byte(s) leaked in 111 allocation(s).
FAILED: bins/asan/end2end_test --gtest_filter=ProxyEnd2end/ProxyEnd2endTest.RpcDeadlineExpires/2  GRPC_POLL_STRATEGY=epollex [ret=1, pid=165659, time=13.2sec]
FLAKE: bins/asan/end2end_test --gtest_filter=ProxyEnd2end

Seen on master:
https://sponge.corp.google.com/invocation?id=77c0694f-57ad-494f-8a66-aa3577ec92ad

@AspirinSJL , we occasionally see this on master and  it can be reproduced locally. Can you take a look?

Sure, I'm checking it.

I'm having problem building with asan after upgrading my OS. After ignoring all the make warnings and switching to clang-5.0, I'm still having numerous undefined reference to __sanitizer_cov_module_init'` issues. Trying to fix this first.

Fixed the asan build issue. Checking the original problem now.

@sreecha Actually, when I am trying to reproduce this issue, I find that the test failure is overwhelmed by #12917. The flakiness is around 1/400. Maybe we need to fix that one first?

Recent failures on master:
https://sponge.corp.google.com/invocation?id=7d34afc2-21ef-4a26-be86-5d20f3b81d90
https://sponge.corp.google.com/invocation?id=e2e774bd-ac0d-4922-9fa2-27e4679e82cd
https://sponge.corp.google.com/invocation?id=77c0694f-57ad-494f-8a66-aa3577ec92ad
https://sponge.corp.google.com/invocation?id=b067b883-94af-4ec4-ad5f-fe92d766ab2d
https://sponge.corp.google.com/invocation?id=ec0f05db-89f5-4b51-b1cf-222f6d8174ec

There seems to be two distinct failures:
First one is sync.cc:93: void gpr_ref_non_zero(gpr_refcount *): Assertion prior > 0' failed.`:
D0302 21:26:27.169451562   30611 end2end_test.cc:217]        TestScenario{use_proxy=true, inproc=false, credentials='INSECURE_CREDENTIALS'}
D0302 21:26:27.169780577   30611 ev_posix.cc:142]            Using polling engine: poll
D0302 21:26:27.169837144   30611 dns_resolver.cc:331]        Using native dns resolver
D0302 21:26:27.172992463   30611 ev_posix.cc:142]            Using polling engine: poll
D0302 21:26:27.173081934   30611 dns_resolver.cc:331]        Using native dns resolver
I0302 21:26:27.173439099   30611 server_builder.cc:269]      Synchronous server. Num CQs: 4, Min pollers: 1, Max Pollers: 2, CQ timeout (msec): 10
I0302 21:26:27.179491964   30611 server_builder.cc:269]      Synchronous server. Num CQs: 4, Min pollers: 1, Max Pollers: 2, CQ timeout (msec): 10
I0302 21:26:27.182784893   30611 subchannel.cc:607]          New connected subchannel at 0x603000023bf0 for subchannel 0x61600008d980
end2end_test: src/core/lib/gpr/sync.cc:93: void gpr_ref_non_zero(gpr_refcount *): Assertion `prior > 0' failed.

and the other one is a memory leak (or a heap-use-after-free).

When trying to reproduce locally, all I'm getting is
I0309 10:15:46.813531448  142565 subchannel.cc:607]          New connected subchannel at 0x60300006d9c0 for subchannel 0x61600005e280
test/cpp/end2end/end2end_test.cc:1300: Failure
      Expected: StatusCode::DEADLINE_EXCEEDED
      Which is: 4
To be equal to: s.error_code()
      Which is: 0

not sure if that's because  the 40ms server sleep in the test case is not sufficient or if there's some newly introduce change has made the test more prone to failure.
https://github.com/grpc/grpc/blob/59ea0ae3ebcca0aef5a15c5aa5b4d27b7f3fc9c4/test/cpp/end2end/end2end_test.cc#L1293

One of the leaks seems to be caused by this:
D0309 14:00:02.889189881   73534 iomgr.cc:96]                Waiting for 1 iomgr objects to be destroyed
D0309 14:00:03.890059685   73534 iomgr.cc:96]                Waiting for 1 iomgr objects to be destroyed
D0309 14:00:04.891072058   73534 iomgr.cc:96]                Waiting for 1 iomgr objects to be destroyed
D0309 14:00:05.891947877   73534 iomgr.cc:96]                Waiting for 1 iomgr objects to be destroyed
D0309 14:00:06.892810588   73534 iomgr.cc:96]                Waiting for 1 iomgr objects to be destroyed
D0309 14:00:07.893724424   73534 iomgr.cc:96]                Waiting for 1 iomgr objects to be destroyed
D0309 14:00:08.894589311   73534 iomgr.cc:96]                Waiting for 1 iomgr objects to be destroyed
D0309 14:00:09.895448848   73534 iomgr.cc:96]                Waiting for 1 iomgr objects to be destroyed
D0309 14:00:10.896345572   73534 iomgr.cc:96]                Waiting for 1 iomgr objects to be destroyed
D0309 14:00:11.897251412   73534 iomgr.cc:127]               Failed to free 1 iomgr objects before shutdown deadline: memory leaks are likely
D0309 14:00:11.897303182   73534 iomgr.cc:75]                LEAKED OBJECT: tcp-server-connection:ipv4:127.0.0.1:54766 fd=15 0x60d0000137e8
D0309 14:00:11.897362021   73534 metadata.cc:135]            WARNING: 1 metadata elements were leaked
D0309 14:00:11.897387537   73534 metadata.cc:135]            WARNING: 1 metadata elements were leaked
D0309 14:00:11.897400514   73534 metadata.cc:135]            WARNING: 1 metadata elements were leaked
D0309 14:00:11.897410692   73534 metadata.cc:135]            WARNING: 1 metadata elements were leaked
D0309 14:00:11.897427074   73534 slice_intern.cc:315]        WARNING: 2 metadata strings were leaked
D0309 14:00:11.897452034   73534 slice_intern.cc:322]        LEAKED: 66 6f 6f 2e 74 65 73 74 2e 79 6f 75 74 75 62 65 2e 63 6f 6d 'foo.test.youtube.com'
D0309 14:00:11.897470103   73534 slice_intern.cc:322]        LEAKED: 31 32 37 2e 30 2e 30 2e 31 3a 31 34 39 35 33 '127.0.0.1:14953'
D0309 14:00:11.897484861   73534 slice_intern.cc:315]        WARNING: 1 metadata strings were leaked
D0309 14:00:11.897507697   73534 slice_intern.cc:322]        LEAKED: 2f 67 72 70 63 2e 74 65 73 74 69 6e 67 2e 64 75 70 6c 69 63 61 74 65 2e 45 63 68 6f 54 65 73 74 53 65 72 76 69 63 65 2f 52 65 73 70 6f 6e 73 65 53 74 72 65 61 6d '/grpc.testing.duplicate.EchoTestService/ResponseStream'
D0309 14:00:11.897519274   73534 slice_intern.cc:315]        WARNING: 1 metadata strings were leaked
D0309 14:00:11.897539794   73534 slice_intern.cc:322]        LEAKED: 2f 67 72 70 63 2e 74 65 73 74 69 6e 67 2e 45 63 68 6f 54 65 73 74 53 65 72 76 69 63 65 2f 52 65 71 75 65 73 74 53 74 72 65 61 6d '/grpc.testing.EchoTestService/RequestStream'
D0309 14:00:11.897550324   73534 slice_intern.cc:315]        WARNING: 1 metadata strings were leaked
D0309 14:00:11.897566487   73534 slice_intern.cc:322]        LEAKED: 2f 67 72 70 63 2e 74 65 73 74 69 6e 67 2e 45 63 68 6f 54 65 73 74 53 65 72 76 69 63 65 2f 45 63 68 6f '/grpc.testing.EchoTestService/Echo'
D0309 14:00:11.897577480   73534 slice_intern.cc:315]        WARNING: 2 metadata strings were leaked
D0309 14:00:11.897599402   73534 slice_intern.cc:322]        LEAKED: 2f 67 72 70 63 2e 74 65 73 74 69 6e 67 2e 45 63 68 6f 54 65 73 74 53 65 72 76 69 63 65 2f 55 6e 69 6d 70 6c 65 6d 65 6e 74 65 64 '/grpc.testing.EchoTestService/Unimplemented'
D0309 14:00:11.897627497   73534 slice_intern.cc:322]        LEAKED: 67 72 70 63 2d 63 2b 2b 2f 31 2e 31 31 2e 30 2d 64 65 76 20 67 72 70 63 2d 63 2f 36 2e 30 2e 30 2d 64 65 76 20 28 6c 69 6e 75 78 3b 20 63 68 74 74 70 32 3b 20 67 6f 72 67 65 6f 75 73 29 20 65 6e 64 32 65 6e 64 5f 74 65 73 74 'grpc-c++/1.11.0-dev grpc-c/6.0.0-dev (linux; chttp2; gorgeous) end2end_test'
D0309 14:00:11.897638269   73534 slice_intern.cc:315]        WARNING: 1 metadata strings were leaked
D0309 14:00:11.897654518   73534 slice_intern.cc:322]        LEAKED: 2f 67 72 70 63 2e 74 65 73 74 69 6e 67 2e 64 75 70 6c 69 63 61 74 65 2e 45 63 68 6f 54 65 73 74 53 65 72 76 69 63 65 2f 45 63 68 6f '/grpc.testing.duplicate.EchoTestService/Echo'
D0309 14:00:11.897666824   73534 slice_intern.cc:315]        WARNING: 2 metadata strings were leaked
D0309 14:00:11.897684496   73534 slice_intern.cc:322]        LEAKED: 2f 67 72 70 63 2e 74 65 73 74 69 6e 67 2e 45 63 68 6f 54 65 73 74 53 65 72 76 69 63 65 2f 52 65 73 70 6f 6e 73 65 53 74 72 65 61 6d '/grpc.testing.EchoTestService/ResponseStream'
D0309 14:00:11.897695797   73534 slice_intern.cc:322]        LEAKED: 31 6d '1m'
D0309 14:00:11.897706118   73534 slice_intern.cc:315]        WARNING: 1 metadata strings were leaked
D0309 14:00:11.897722876   73534 slice_intern.cc:322]        LEAKED: 2f 67 72 70 63 2e 74 65 73 74 69 6e 67 2e 45 63 68 6f 54 65 73 74 53 65 72 76 69 63 65 2f 42 69 64 69 53 74 72 65 61 6d '/grpc.testing.EchoTestService/BidiStream'
[       OK ] ProxyEnd2end/ProxyEnd2endTest.RpcDeadlineExpires/2 (10188 ms)

Here's a stacktrace for the sync.cc:93: void gpr_ref_non_zero(gpr_refcount *): Assertion 'prior > 0' failed.  issue. (from a local reproduction).
#0  0x00007f844b7b5fcf in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f844b7b73fa in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f844b7aee37 in __assert_fail_base () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f844b7aeee2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x000055e815e20e16 in gpr_ref_non_zero (r=0x7f842c00d7e0) at src/core/lib/gpr/sync.cc:93
#5  0x000055e815e3c76f in grpc_stream_ref (refcount=0x7f842c00d7e0, reason=0x55e815fe09a9 "propagate_cancel") at src/core/lib/transport/transport.cc:50
#6  0x000055e815e2a9de in grpc_call_internal_ref (c=0x7f842c00ce20, reason=0x55e815fe09a9 "propagate_cancel") at src/core/lib/surface/call.cc:500
#7  0x000055e815e2cbca in post_batch_completion (bctl=0x7f842c007500) at src/core/lib/surface/call.cc:1230
#8  0x000055e815e2cdbc in finish_batch_step (bctl=0x7f842c007500) at src/core/lib/surface/call.cc:1276
#9  0x000055e815e2d9cd in finish_batch (bctlp=0x7f842c007500, error=0x0) at src/core/lib/surface/call.cc:1543
#10 0x000055e815dd0e11 in exec_ctx_run (closure=0x7f842c007580, error=0x0) at src/core/lib/iomgr/exec_ctx.cc:40
#11 0x000055e815dd4b9d in grpc_closure_run (file=0x55e815fe9c28 "src/core/ext/filters/deadline/deadline_filter.cc", line=137, c=0x7f842c007580, error=0x0)
    at ./src/core/lib/iomgr/closure.h:258
#12 0x000055e815e6255a in on_complete (arg=0x7f842c005a70, error=0x0) at src/core/ext/filters/deadline/deadline_filter.cc:137
#13 0x000055e815dd0e11 in exec_ctx_run (closure=0x7f842c005b08, error=0x0) at src/core/lib/iomgr/exec_ctx.cc:40
#14 0x000055e815dd121d in grpc_core::ExecCtx::Flush (this=0x7f84407f7d40) at src/core/lib/iomgr/exec_ctx.cc:128
#15 0x000055e815ddbcf6 in run_some_timers () at src/core/lib/iomgr/timer_manager.cc:121
#16 0x000055e815ddbfb6 in timer_main_loop () at src/core/lib/iomgr/timer_manager.cc:223
#17 0x000055e815ddc0e3 in timer_thread (completed_thread_ptr=0x55e817e2e980) at src/core/lib/iomgr/timer_manager.cc:270
#18 0x000055e815e2220a in grpc_core::(anonymous namespace)::ThreadInternalsPosix::<lambda(void*)>::operator()(void *) const (__closure=0x0, v=0x55e817e2d850)
    at src/core/lib/gprpp/thd_posix.cc:105
#19 0x000055e815e22233 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::<lambda(void*)>::_FUN(void *) () at src/core/lib/gprpp/thd_posix.cc:108
#20 0x00007f844cf02494 in start_thread (arg=0x7f84407f8700) at pthread_create.c:333

I was poking into this a few nights ago. As the logs above show, the crash comes from this line:
https://github.com/grpc/grpc/blob/59ea0ae3ebcca0aef5a15c5aa5b4d27b7f3fc9c4/src/core/lib/surface/call.cc#L1230
So it looks like some race condition in which the server thinks it has child calls but doesn't.

@ncteisen I think there is a child call (there's a proxy in place).  https://github.com/grpc/grpc/blob/59ea0ae3ebcca0aef5a15c5aa5b4d27b7f3fc9c4/test/cpp/end2end/end2end_test.cc#L177  (please doublecheck, you know the code better than I do)
But I agree that there's some kind of race - we're probably hitting some edge case in the parent - child propagation.

the last line  (null) 0x7f79a000d9d0:(nil)   REF 0->1 propagate_cancel   is pretty odd.   It looks like the refcount object wasn't yet initialized or it was already destroyed. See the end of this comment for explanation why I think the refcount is not fully initialized yet.
A failing test run:
[ RUN      ] ProxyEnd2end/ProxyEnd2endTest.RpcDeadlineExpires/1
D0309 17:03:31.003112374  165243 end2end_test.cc:216]        TestScenario{use_proxy=true, inproc=false, credentials='ssl'}
D0309 17:03:31.003189494  165243 ev_posix.cc:142]            Using polling engine: epollex
D0309 17:03:31.003212709  165243 dns_resolver.cc:331]        Using native dns resolver
D0309 17:03:31.005022120  165243 ev_posix.cc:142]            Using polling engine: epollex
D0309 17:03:31.005051490  165243 dns_resolver.cc:331]        Using native dns resolver
I0309 17:03:31.005113580  165243 server_builder.cc:269]      Synchronous server. Num CQs: 4, Min pollers: 1, Max Pollers: 2, CQ timeout (msec): 10
I0309 17:03:31.008327986  165243 server_builder.cc:269]      Synchronous server. Num CQs: 4, Min pollers: 1, Max Pollers: 2, CQ timeout (msec): 10
D0309 17:03:31.008730035  165243 transport.cc:43]            CLIENT_CHANNEL 0x561273ba09f0:0x561273ba0990   REF 1->2 call
D0309 17:03:31.008752892  165243 transport.cc:43]            CALL_STACK 0x561273ba2ec0:0x561273ba2500   REF 1->2 deadline_timer
D0309 17:03:31.008799705  165243 transport.cc:43]            CALL_STACK 0x561273ba2ec0:0x561273ba2500   REF 2->3 completion
D0309 17:03:31.008814976  165243 transport.cc:43]            CLIENT_CHANNEL 0x561273ba09f0:0x561273ba0990   REF 2->3 resolver
D0309 17:03:31.008978360  165888 subchannel_index.cc:189]    SUBCHANNEL: 0x7f79b8007f40     WEAK_REF 0x10000 -> 0x10001 [index_register]
D0309 17:03:31.009013640  165888 subchannel_index.cc:189]    SUBCHANNEL: 0x7f79b80089e0     WEAK_REF 0x10000 -> 0x10001 [index_register]
D0309 17:03:31.009026926  165888 subchannel_index.cc:111]    SUBCHANNEL: 0x7f79b8007f40     WEAK_REF 0x10001 -> 0x10002 [subchannel_index]
D0309 17:03:31.009035505  165888 subchannel_index.cc:107]    SUBCHANNEL: 0x7f79b8007f40   WEAK_UNREF 0x10002 -> 0x10001 [subchannel_index]
D0309 17:03:31.009044235  165888 transport.cc:43]            CLIENT_CHANNEL 0x561273ba09f0:0x561273ba0990   REF 3->4 re-resolution
D0309 17:03:31.009055957  165888 transport.cc:43]            CLIENT_CHANNEL 0x561273ba09f0:0x561273ba0990   REF 4->5 watch_lb_policy
D0309 17:03:31.009064693  165888 transport.cc:43]            CALL_STACK 0x561273ba2ec0:0x561273ba2500   REF 3->4 pick_callback
D0309 17:03:31.009072719  165888 subchannel.cc:502]          SUBCHANNEL: 0x7f79b8007f40     WEAK_REF 0x10001 -> 0x10002 [external_state_watcher]
D0309 17:03:31.009083865  165888 subchannel.cc:456]          SUBCHANNEL: 0x7f79b8007f40     WEAK_REF 0x10002 -> 0x10003 [connecting]
D0309 17:03:31.009168348  165888 transport.cc:43]            CALL_STACK 0x561273ba2ec0:0x561273ba2500   REF 4->5 pick_callback_cancel
D0309 17:03:31.009182202  165888 subchannel.cc:408]          SUBCHANNEL: 0x7f79b8007f40   WEAK_UNREF 0x10003 -> 0x10002 [external_state_watcher]
D0309 17:03:31.009190239  165888 subchannel.cc:502]          SUBCHANNEL: 0x7f79b8007f40     WEAK_REF 0x10002 -> 0x10003 [external_state_watcher]
D0309 17:03:31.009197438  165888 transport.cc:43]            CLIENT_CHANNEL 0x561273ba09f0:0x561273ba0990   REF 5->6 watch_lb_policy
D0309 17:03:31.009203685  165888 transport.cc:57]            CLIENT_CHANNEL 0x561273ba09f0:0x561273ba0990 UNREF 6->5 watch_lb_policy
D0309 17:03:31.009251853  165243 subchannel.cc:628]          SUBCHANNEL: 0x7f79b8007f40     WEAK_REF 0x10003 -> 0x10004 [on_subchannel_connected]
I0309 17:03:31.009281043  165243 subchannel.cc:607]          New connected subchannel at 0x561273b9aaf0 for subchannel 0x7f79b8007f40
D0309 17:03:31.009290701  165243 subchannel.cc:612]          SUBCHANNEL: 0x7f79b8007f40     WEAK_REF 0x10004 -> 0x10005 [state_watcher]
D0309 17:03:31.009287717  165913 transport.cc:43]            SERVER_CHANNEL 0x7f79a8006300:0x7f79a80062a0   REF 1->2 connectivity
D0309 17:03:31.009299984  165243 subchannel.cc:613]          SUBCHANNEL: 0x7f79b8007f40   WEAK_UNREF 0x10005 -> 0x10004 [connecting]
D0309 17:03:31.009323832  165243 subchannel.cc:651]          SUBCHANNEL: 0x7f79b8007f40   WEAK_UNREF 0x10004 -> 0x10003 [connected]
D0309 17:03:31.009332238  165243 subchannel.cc:408]          SUBCHANNEL: 0x7f79b8007f40   WEAK_UNREF 0x10003 -> 0x10002 [external_state_watcher]
D0309 17:03:31.009376851  165243 subchannel_list.cc:45]      SUBCHANNEL: 0x7f79b80089e0 STRONG_UNREF 0x10001 -> 0x2 [selected_different_subchannel]
D0309 17:03:31.009389565  165243 subchannel_index.cc:111]    SUBCHANNEL: 0x7f79b8007f40     WEAK_REF 0x10002 -> 0x10003 [subchannel_index]
D0309 17:03:31.009399301  165243 subchannel_index.cc:107]    SUBCHANNEL: 0x7f79b8007f40   WEAK_UNREF 0x10003 -> 0x10002 [subchannel_index]
D0309 17:03:31.009406419  165243 subchannel_index.cc:107]    SUBCHANNEL: 0x7f79b80089e0   WEAK_UNREF 0x2 -> 0x1 [subchannel_index]
D0309 17:03:31.009414078  165243 subchannel.cc:252]          SUBCHANNEL: 0x7f79b80089e0   WEAK_UNREF 0x1 -> 0x0 [strong-unref]
D0309 17:03:31.009421705  165243 subchannel.cc:502]          SUBCHANNEL: 0x7f79b8007f40     WEAK_REF 0x10002 -> 0x10003 [external_state_watcher]
D0309 17:03:31.009432073  165243 transport.cc:43]            CLIENT_CHANNEL 0x561273ba09f0:0x561273ba0990   REF 5->6 watch_lb_policy
D0309 17:03:31.009453753  165243 transport.cc:57]            CLIENT_CHANNEL 0x561273ba09f0:0x561273ba0990 UNREF 6->5 watch_lb_policy
D0309 17:03:31.009468026  165243 transport.cc:57]            CALL_STACK 0x561273ba2ec0:0x561273ba2500 UNREF 5->4 pick_callback
D0309 17:03:31.009488583  165243 ref_counted.h:103]          stream_refcount:0x561273b9aaf0 src/core/ext/filters/client_channel/subchannel.cc:790 ref 3 -> 4 subchannel_call
D0309 17:03:31.009497954  165243 transport.cc:43]            CALL_STACK 0x561273bab120:0x561273bab110   REF 1->2 chttp2
D0309 17:03:31.009515624  165243 transport.cc:43]            CALL_STACK 0x561273bab120:0x561273bab110   REF 2->3 perform_stream_op
D0309 17:03:31.009526831  165243 transport.cc:43]            CALL_STACK 0x561273bab120:0x561273bab110   REF 3->4 chttp2_writing:become
D0309 17:03:31.009538809  165243 transport.cc:57]            CALL_STACK 0x561273bab120:0x561273bab110 UNREF 4->3 perform_stream_op
D0309 17:03:31.009590968  165243 transport.cc:57]            CALL_STACK 0x561273bab120:0x561273bab110 UNREF 3->2 chttp2_writing:end
D0309 17:03:31.009618524  165910 transport.cc:43]            SERVER_CHANNEL 0x7f79a8006300:0x7f79a80062a0   REF 2->3 call
D0309 17:03:31.009643435  165910 transport.cc:43]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70   REF 1->2 chttp2
D0309 17:03:31.009666318  165910 transport.cc:43]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70   REF 2->3 completion
D0309 17:03:31.009714592  165910 transport.cc:43]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70   REF 3->4 perform_stream_op
D0309 17:03:31.009750824  165910 transport.cc:57]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70 UNREF 4->3 perform_stream_op
D0309 17:03:31.009766600  165910 transport.cc:43]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70   REF 3->4 deadline_timer
D0309 17:03:31.009791220  165910 transport.cc:43]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70   REF 4->5 completion
D0309 17:03:31.009806645  165910 transport.cc:57]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70 UNREF 5->4 completion
D0309 17:03:31.009818246  165910 transport.cc:43]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70   REF 4->5 perform_stream_op
D0309 17:03:31.009836497  165910 transport.cc:57]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70 UNREF 5->4 perform_stream_op
D0309 17:03:31.009910269  165910 transport.cc:57]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70 UNREF 4->3 completion
D0309 17:03:31.009977098  165910 transport.cc:43]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70   REF 3->4 completion
D0309 17:03:31.009994851  165910 transport.cc:43]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70   REF 4->5 perform_stream_op
D0309 17:03:31.010031125  165910 transport.cc:57]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70 UNREF 5->4 perform_stream_op
D0309 17:03:31.010065163  165889 transport.cc:43]            CALL_STACK 0x561273bab120:0x561273bab110   REF 2->3 perform_stream_op
D0309 17:03:31.010085597  165910 transport.cc:43]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70   REF 4->5 child
D0309 17:03:31.010088826  165889 transport.cc:43]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70   REF 4->5 perform_stream_op
D0309 17:03:31.010115723  165889 transport.cc:57]            CALL_STACK 0x561273ba2ec0:0x561273ba2500 UNREF 4->3 pick_callback_cancel
D0309 17:03:31.010132611  165889 transport.cc:57]            CALL_STACK 0x561273bab120:0x561273bab110 UNREF 3->2 chttp2
D0309 17:03:31.010140858  165889 transport.cc:57]            CALL_STACK 0x561273bab120:0x561273bab110 UNREF 2->1 perform_stream_op
D0309 17:03:31.010149034  165889 transport.cc:57]            CALL_STACK 0x561273ba2ec0:0x561273ba2500 UNREF 3->2 deadline_timer
D0309 17:03:31.010186242  165889 transport.cc:57]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70 UNREF 6->5 chttp2
D0309 17:03:31.010202353  165889 transport.cc:57]            CALL_STACK 0x7f79a0007730:0x7f79a0006d70 UNREF 5->4 perform_stream_op
D0309 17:03:31.010216771  165889 transport.cc:43]            (null) 0x7f79a000d9d0:(nil)   REF 0->1 propagate_cancel
end2end_test: src/core/lib/gpr/sync.cc:93: void gpr_ref_non_zero(gpr_refcount*): Assertion `prior > 0' failed.

Interestingly, in one of the successful test run, I saw this log where it looks like the refcount starts as not fully initialized but in the following log lines, it is already fully populated.
...
D0309 17:14:02.802507683  168525 transport.cc:43]            CLIENT_CHANNEL 0x559e58f06cb0:0x559e58f06c50   REF 1->2 call
D0309 17:14:02.802517948  168519 transport.cc:43]            (null) 0x7f91c000d730:(nil)   REF 0->1 propagate_cancel
D0309 17:14:02.802531707  168525 transport.cc:43]            CALL_STACK 0x7f91c000d730:0x7f91c000cd70   REF 1->2 deadline_timer
D0309 17:14:02.802534986  168519 transport.cc:43]            CALL_STACK 0x7f91c000d730:0x7f91c000cd70   REF 2->3 termination
D0309 17:14:02.802548018  168525 transport.cc:57]            CALL_STACK 0x7f91c000d730:0x7f91c000cd70 UNREF 3->2 deadline_timer
D0309 17:14:02.802550357  168519 transport.cc:57]            CALL_STACK 0x7f91c000d730:0x7f91c000cd70 UNREF 4->3 propagate_cancel
...

Ok, I'm now 99% sure that the stream_refcount is getting initialized by another thread concurrently with the refcount being updated.
Thread 160601  calls  grpc_stream_ref_init  but thread 160560 is accessing it before it's fully populated.
D0309 18:18:22.534668087  160601 transport.cc:43]            CLIENT_CHANNEL 0x5629f305f500:0x5629f305f4a0   REF 1->2 call
D0309 18:18:22.534677981  160560 transport.cc:43]            (null) 0x7f62a000d790:(nil)   REF 0->1 propagate_cancel
I0309 18:18:22.534682411  160601 transport.cc:127]           grpc_stream_ref_init 0x7f62a000d790
end2end_test: src/core/lib/gpr/sync.cc:93: void gpr_ref_non_zero(gpr_refcount*): Assertion `prior > 0' failed.

Good point about the proxy... definitely will be child call.
The refcount modification is protected by child_list_mu. I bet the initialization code should be taking that lock, but is not

I think the rootcause is in grpc_call_create.

While the new call is being initialized, if there's a parent call, the call is published in the parent's call child list (that's part is protected child_list_mu)
https://github.com/grpc/grpc/blob/59ea0ae3ebcca0aef5a15c5aa5b4d27b7f3fc9c4/src/core/lib/surface/call.cc#L381

But the call published is actually not fully initialized yet  - the next thing that's done is  actually initializing the call's  refcount  (which boils down to grpc_stream_ref_init).
https://github.com/grpc/grpc/blob/59ea0ae3ebcca0aef5a15c5aa5b4d27b7f3fc9c4/src/core/lib/surface/call.cc#L447

Another thread takes hold of the parent's child list (which contained pointer to the ininitialized call) and if the timing is right and the parent call is just finalizing, it also accesses  the new call's  refcount - which is not fully initialized yet (there's a race).

Here's a log with partial proof:
D0309 19:07:33.600651637  191558 transport.cc:57]            CALL_STACK 0x7fa59800b770:0x7fa59800adb0 UNREF 4->3 chttp2
I0309 19:07:33.600663594  191527 call.cc:448]                calling grpc_call_stack_init from grpc_call_create. call: 0x7fa5a8002440, parent_call: 0x7fa59800d720
D0309 19:07:33.600690298  190776 transport.cc:57]            CALL_STACK 0x55f8ed15f280:0x55f8ed15e8c0 UNREF 1->0 destroy
D0309 19:07:33.600693093  191558 transport.cc:43]            (null) 0x7fa5a8002e00:(nil)   REF 0->1 propagate_cancel
D0309 19:07:33.600714623  190776 transport.cc:57]            CALL_STACK 0x55f8ed166f00:0x55f8ed166ef0 UNREF 1->0 client_channel_destroy_call
I0309 19:07:33.600702508  191527 transport.cc:127]           grpc_stream_ref_init 0x7fa5a8002e00 (initial_refs: 1)
D0309 19:07:33.600725890  190776 ref_counted.h:124]          stream_refcount:0x55f8ed1604a0 src/core/ext/filters/client_channel/subchannel.cc:665 unref 3 -> 2 subchannel_call
D0309 19:07:33.600736519  191527 transport.cc:43]            CALL_STACK 0x7fa5a8002e00:0x7fa5a8002440   REF 1->2 deadline_timer
D0309 19:07:33.600743309  190776 transport.cc:57]            CLIENT_CHANNEL 0x55f8ed15cea0:0x55f8ed15ce40 UNREF 5->4 call
D0309 19:07:33.600753604  191527 transport.cc:57]            CALL_STACK 0x7fa5a8002e00:0x7fa5a8002440 UNREF 2->1 deadline_timer
D0309 19:07:33.600775007  191527 transport.cc:43]            CALL_STACK 0x7fa5a8002e00:0x7fa5a8002440   REF 1->2 completion
D0309 19:07:33.600786062  191527 transport.cc:43]            CALL_STACK 0x7fa5a8002e00:0x7fa5a8002440   REF 2->3 termination
D0309 19:07:33.600796804  191527 transport.cc:43]            CALL_STACK 0x7fa5a8002e00:0x7fa5a8002440   REF 3->4 termination
D0309 19:07:33.600815730  191527 transport.cc:57]            CALL_STACK 0x7fa5a8002e00:0x7fa5a8002440 UNREF 4->3 termination
D0309 19:07:33.600822001  191527 transport.cc:57]            CALL_STACK 0x7fa5a8002e00:0x7fa5a8002440 UNREF 3->2 termination
D0309 19:07:33.600828055  191527 transport.cc:57]            CALL_STACK 0x7fa5a8002e00:0x7fa5a8002440 UNREF 2->1 completion
end2end_test: src/core/lib/gpr/sync.cc:93: void gpr_ref_non_zero(gpr_refcount*): Assertion `prior > 0' failed.

@ncteisen @AspirinSJL  feel free to take over from here. I think I rootcaused the problem correctly but the fix might need some reoganizing of grpc_call_create and I don't understand the performance consequences of that doing that very well - so I'll better leave this to domain experts.

I think that analysis is correct. It certainly matches up with the data. I can take a look at this for the rest of the day, but am OOO all next week, so if the fix is not trivial, I will have to pass it off

FTR, this is the command I've been using for local debugging:
GRPC_TRACE=stream_refcount GRPC_ABORT_ON_LEAKS=true tools/run_tests/run_tests.py -l c++ -cdbg -r ".*RpcDeadlineExpires" -j 30 -n inf -S (it takes several minutes to reproduce)

I'm looking into it. But since I'm new to this part of code, I need to take some time to get familiar with the code first, so please expect the fixing time to be a little longer.

The use-before-init problem can be solved by extending the time that the call creation thread holds the mutex.
The flakiness significantly drops, but still exists for another refcounting issue. It seems that somewhere else we have a use-after-destroy problem. Trying to figure out who is reffing.
D0312 14:41:34.781092091  102549 transport.cc:43]            CALL_STACK 0x55ba9388bc80:0x55ba9388b2c0   REF 3->4 termination
D0312 14:41:34.781134497  102549 transport.cc:57]            CALL_STACK 0x55ba9388bc80:0x55ba9388b2c0 UNREF 4->3 termination
D0312 14:41:34.781141133  102549 transport.cc:57]            CALL_STACK 0x55ba9388bc80:0x55ba9388b2c0 UNREF 3->2 termination
D0312 14:41:34.781156489   79159 transport.cc:57]            CALL_STACK 0x55ba9388bc80:0x55ba9388b2c0 UNREF 2->1 completion
D0312 14:41:34.781177536   79159 transport.cc:57]            CALL_STACK 0x55ba9388bc80:0x55ba9388b2c0 UNREF 1->0 destroy
D0312 14:41:34.781187997   79159 transport.cc:57]            CLIENT_CHANNEL 0x55ba93889190:0x55ba93889130 UNREF 5->4 call
end2end_test: src/core/lib/gpr/sync.cc:93: void gpr_ref_non_zero(gpr_refcount*): Assertion `prior > 0' failed.

Actually, that might still be a use-before-init issue, if we are reffing a new call.

We are reffing something that isn't any existing channel or call, at least. But I don't know what it is.
D0312 16:58:45.280036763   34625 end2end_test.cc:216]        TestScenario{use_proxy=false, inproc=false, credentials='ssl'}
D0312 16:58:45.280099293   34625 ev_posix.cc:142]            Using polling engine: poll-cv
D0312 16:58:45.280114820   34625 dns_resolver.cc:331]        Using native dns resolver
D0312 16:58:46.287528265   34625 ev_posix.cc:142]            Using polling engine: poll-cv
D0312 16:58:46.287554799   34625 dns_resolver.cc:331]        Using native dns resolver
I0312 16:58:46.287609758   34625 server_builder.cc:269]      Synchronous server. Num CQs: 4, Min pollers: 1, Max Pollers: 2, CQ timeout (msec): 10
D0312 16:58:46.291138127   34625 transport.cc:43]            CLIENT_CHANNEL 0x559a15b9c200:0x559a15b9c1a0   REF 1->2 call
D0312 16:58:46.291155856   34625 transport.cc:43]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370   REF 1->2 deadline_timer
D0312 16:58:46.291186219   34625 transport.cc:43]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370   REF 2->3 completion
D0312 16:58:46.291196511   34625 transport.cc:43]            CLIENT_CHANNEL 0x559a15b9c200:0x559a15b9c1a0   REF 2->3 resolver
D0312 16:58:46.291734585   55995 subchannel_index.cc:189]    SUBCHANNEL: 0x7fd66c0047f0     WEAK_REF 0x10000 -> 0x10001 [index_register]
D0312 16:58:46.291757979   55995 transport.cc:43]            CLIENT_CHANNEL 0x559a15b9c200:0x559a15b9c1a0   REF 3->4 re-resolution
D0312 16:58:46.291774029   55995 transport.cc:43]            CLIENT_CHANNEL 0x559a15b9c200:0x559a15b9c1a0   REF 4->5 watch_lb_policy
D0312 16:58:46.291782289   55995 transport.cc:43]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370   REF 3->4 pick_callback
D0312 16:58:46.291788812   55995 subchannel.cc:502]          SUBCHANNEL: 0x7fd66c0047f0     WEAK_REF 0x10001 -> 0x10002 [external_state_watcher]
D0312 16:58:46.291794034   55995 subchannel.cc:456]          SUBCHANNEL: 0x7fd66c0047f0     WEAK_REF 0x10002 -> 0x10003 [connecting]
D0312 16:58:46.291901559   55995 transport.cc:43]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370   REF 4->5 pick_callback_cancel
D0312 16:58:46.291913109   55995 subchannel.cc:408]          SUBCHANNEL: 0x7fd66c0047f0   WEAK_UNREF 0x10003 -> 0x10002 [external_state_watcher]
D0312 16:58:46.291920874   55995 subchannel.cc:502]          SUBCHANNEL: 0x7fd66c0047f0     WEAK_REF 0x10002 -> 0x10003 [external_state_watcher]
D0312 16:58:46.291933533   55995 transport.cc:43]            CLIENT_CHANNEL 0x559a15b9c200:0x559a15b9c1a0   REF 5->6 watch_lb_policy
D0312 16:58:46.291939002   55995 transport.cc:57]            CLIENT_CHANNEL 0x559a15b9c200:0x559a15b9c1a0 UNREF 6->5 watch_lb_policy
D0312 16:58:46.292700116   55998 transport.cc:57]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370 UNREF 5->4 pick_callback_cancel
D0312 16:58:46.292720048   55998 transport.cc:57]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370 UNREF 4->3 pick_callback
D0312 16:58:46.292731376   55998 transport.cc:43]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370   REF 3->4 termination
D0312 16:58:46.292744646   55998 transport.cc:57]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370 UNREF 4->3 deadline_timer
D0312 16:58:46.292751117   55998 transport.cc:43]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370   REF 3->4 termination
D0312 16:58:46.292792714   55998 transport.cc:57]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370 UNREF 4->3 termination
D0312 16:58:46.292800074   55998 transport.cc:57]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370 UNREF 3->2 termination
D0312 16:58:46.292945554   34625 transport.cc:57]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370 UNREF 2->1 completion
D0312 16:58:46.292976307   34625 transport.cc:57]            CALL_STACK 0x559a15b9ed30:0x559a15b9e370 UNREF 1->0 destroy
D0312 16:58:46.292989579   34625 transport.cc:57]            CLIENT_CHANNEL 0x559a15b9c200:0x559a15b9c1a0 UNREF 5->4 call
I0312 16:58:46.297141862   56117 sync.cc:94]                 PRIOR <= 0 (0x559a15b91590)
end2end_test: src/core/lib/gpr/sync.cc:96: void gpr_ref_non_zero(gpr_refcount*): Assertion `prior > 0' failed.

@AspirinSJL  do we have a stacktrace?

It's like
D0313 09:36:55.268856651  211407 transport.cc:57]            CALL_STACK 0x5636c80e82d0:0x5636c80e7910 UNREF 2->1 completion
D0313 09:36:55.268875389  211407 transport.cc:57]            CALL_STACK 0x5636c80e82d0:0x5636c80e7910 UNREF 1->0 destroy
D0313 09:36:55.268886054  211407 transport.cc:57]            CALL_STACK 0x5636c80eff50:0x5636c80eff40 UNREF 1->0 client_channel_destroy_call
D0313 09:36:55.268892952  211407 ref_counted.h:124]          stream_refcount:0x5636c80e9440 src/core/ext/filters/client_channel/subchannel.cc:665 unref 3 -> 2 subchannel_call
D0313 09:36:55.268899794  211407 transport.cc:57]            CLIENT_CHANNEL 0x5636c80e5cf0:0x5636c80e5c90 UNREF 5->4 call
end2end_test: src/core/lib/gpr/sync.cc:93: void gpr_ref_non_zero(gpr_refcount*): Assertion `prior > 0' failed.

*******************************
Caught signal SIGABRT
bins/dbg/end2end_test(+0x1b62f5)[0x5636c79942f5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7fa17420f0c0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7fa1726a5fcf]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fa1726a73fa]
/lib/x86_64-linux-gnu/libc.so.6(+0x2be37)[0x7fa17269ee37]
/lib/x86_64-linux-gnu/libc.so.6(+0x2bee2)[0x7fa17269eee2]
bins/dbg/end2end_test(+0x10ea1a)[0x5636c78eca1a]
bins/dbg/end2end_test(+0x160b26)[0x5636c793eb26]
bins/dbg/end2end_test(+0x13e2cd)[0x5636c791c2cd]
bins/dbg/end2end_test(+0x15fd49)[0x5636c793dd49]
bins/dbg/end2end_test(+0xbe0ad)[0x5636c789c0ad]
bins/dbg/end2end_test(+0xbe4a9)[0x5636c789c4a9]
bins/dbg/end2end_test(+0xdf3af)[0x5636c78bd3af]
bins/dbg/end2end_test(+0xbda49)[0x5636c789ba49]
bins/dbg/end2end_test(+0x1204d2)[0x5636c78fe4d2]
bins/dbg/end2end_test(+0x120917)[0x5636c78fe917]
bins/dbg/end2end_test(+0xf39b8)[0x5636c78d19b8]
bins/dbg/end2end_test(+0x1047c4)[0x5636c78e27c4]
bins/dbg/end2end_test(+0x103adb)[0x5636c78e1adb]
bins/dbg/end2end_test(+0x10a6ce)[0x5636c78e86ce]
bins/dbg/end2end_test(+0x10a1a9)[0x5636c78e81a9]
bins/dbg/end2end_test(+0x10a0a2)[0x5636c78e80a2]
bins/dbg/end2end_test(+0x10a0c2)[0x5636c78e80c2]
bins/dbg/end2end_test(+0x10fe06)[0x5636c78ede06]
bins/dbg/end2end_test(+0x10fe2f)[0x5636c78ede2f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)[0x7fa174205494]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fa17275ba8f]

Is there any way to find the code mapped from the addresses? I tried addr2line, but got
grpc git:(head) ✗ addr2line 0x5636c79942f5
addr2line: 'a.out': No such file

I'm trying to debug in gdb, but the stacktrace in the coredump doesn't match the failure log. The log shows end2end_test: src/core/lib/gpr/sync.cc:93: void gpr_ref_non_zero(gpr_refcount*): Assertion prior > 0' failed, but the stack shows that the abortion is due to gpr_mu_lock()` here. Any idea?
(gdb) info threads
  Id   Target Id         Frame
  25   Thread 0x7f8d537fe700 (LWP 26496) __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  24   Thread 0x7f8d99ee9700 (LWP 26498) __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  23   Thread 0x7f8d98cd5700 (LWP 26499) 0x000055ea8a0df941 in pollset_work (pollset=0x0, worker_hdl=0x0, deadline=0)
    at src/core/lib/iomgr/ev_poll_posix.cc:877
  22   Thread 0x7f8d73fff700 (LWP 17787) __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  21   Thread 0x7f8d717fa700 (LWP 17808) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  20   Thread 0x7f8d6a7fc700 (LWP 17815) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  19   Thread 0x7f8d797fa700 (LWP 17784) __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  18   Thread 0x7f8d71ffb700 (LWP 17803) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  17   Thread 0x7f8d72ffd700 (LWP 17797) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  16   Thread 0x7f8d737fe700 (LWP 17795) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  15   Thread 0x7f8d69ffb700 (LWP 17826) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  14   Thread 0x7f8d6affd700 (LWP 17811) __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  13   Thread 0x7f8d6b7fe700 (LWP 17810) 0x00007f8d99fc963d in poll () from /lib/x86_64-linux-gnu/libc.so.6
  12   Thread 0x7f8d727fc700 (LWP 17801) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  11   Thread 0x7f8d8eff5700 (LWP 17546) pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  10   Thread 0x7f8d78ff9700 (LWP 17786) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  9    Thread 0x7f8d79ffb700 (LWP 17595) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  8    Thread 0x7f8d8f7f6700 (LWP 17544) pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  7    Thread 0x7f8d6bfff700 (LWP 17791) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  6    Thread 0x7f8d7b7fe700 (LWP 17587) pthread_cond_signal@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S:87
  5    Thread 0x7f8d70ff9700 (LWP 17809) __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  4    Thread 0x7f8d7affd700 (LWP 17590) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  3    Thread 0x7f8d7bfff700 (LWP 17586) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
  2    Thread 0x7f8d7a7fc700 (LWP 17591) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
* 1    Thread 0x7f8d9be87840 (LWP 256854) raise (sig=3) at ../sysdeps/unix/sysv/linux/raise.c:51
(gdb) t 1
[Switching to thread 1 (Thread 0x7f8d9be87840 (LWP 256854))]
#0  raise (sig=3) at ../sysdeps/unix/sysv/linux/raise.c:51
51	in ../sysdeps/unix/sysv/linux/raise.c
(gdb) bt
#0  raise (sig=3) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x000055ea8a1b7345 in crash_handler (signum=3, info=0x55ea8a6045b0 <g_alt_stack+64528>, data=0x55ea8a604480 <g_alt_stack+64224>)
    at test/core/util/test_config.cc:255
#2  <signal handler called>
#3  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#4  0x00007f8d9ba7eb95 in __GI___pthread_mutex_lock (mutex=0x55ea8a5f2680 <g_cvfds>) at ../nptl/pthread_mutex_lock.c:80
#5  0x000055ea8a10fbbd in gpr_mu_lock (mu=0x55ea8a5f2680 <g_cvfds>) at src/core/lib/gpr/sync_posix.cc:47
#6  0x000055ea8a0e4711 in cv_fd_wakeup (fd_info=0x7f8d740017b0) at src/core/lib/iomgr/wakeup_fd_cv.cc:71
#7  0x000055ea8a0cacc5 in grpc_wakeup_fd_wakeup (fd_info=0x7f8d740017b0) at src/core/lib/iomgr/wakeup_fd_posix.cc:74
#8  0x000055ea8a0df27a in pollset_kick_ext (p=0x55ea8a7ed350, specific_worker=0x7f8d6affc9a0, flags=2) at src/core/lib/iomgr/ev_poll_posix.cc:745
#9  0x000055ea8a0de3d2 in pollset_kick_locked (watcher=0x7f8d6affb7b8) at src/core/lib/iomgr/ev_poll_posix.cc:377
#10 0x000055ea8a0de4cc in wake_all_watchers_locked (fd=0x55ea8a7ef840) at src/core/lib/iomgr/ev_poll_posix.cc:399
#11 0x000055ea8a0de6ae in fd_orphan (fd=0x55ea8a7ef840, on_done=0x55ea8a7ef7d0, release_fd=0x0, already_closed=false,
    reason=0x55ea8a2dba9f "tcp_listener_shutdown") at src/core/lib/iomgr/ev_poll_posix.cc:442
#12 0x000055ea8a0be79c in grpc_fd_orphan (fd=0x55ea8a7ef840, on_done=0x55ea8a7ef7d0, release_fd=0x0, already_closed=false,
    reason=0x55ea8a2dba9f "tcp_listener_shutdown") at src/core/lib/iomgr/ev_posix.cc:207
#13 0x000055ea8a160963 in deactivated_all_ports (s=0x55ea8a7ef5d0) at src/core/lib/iomgr/tcp_server_posix.cc:168
#14 0x000055ea8a160de2 in on_read (arg=0x7f8d80001bb0, err=0x55ea8a7f5040) at src/core/lib/iomgr/tcp_server_posix.cc:275
#15 0x000055ea8a0bf0ad in exec_ctx_run (closure=0x7f8d80001c60, error=0x55ea8a7f5040) at src/core/lib/iomgr/exec_ctx.cc:40
#16 0x000055ea8a0bf4a9 in grpc_core::ExecCtx::Flush (this=0x7ffe2b59b2a0) at src/core/lib/iomgr/exec_ctx.cc:128
#17 0x000055ea8a0b92a7 in grpc_core::ExecCtx::~ExecCtx (this=0x7ffe2b59b2a0, __in_chrg=<optimized out>) at ./src/core/lib/iomgr/exec_ctx.h:87
#18 0x000055ea8a126d0e in grpc_server_shutdown_and_notify (server=0x55ea8a7ede60, cq=0x55ea8a7dd9b0, tag=0x7ffe2b59b3b8)
    at src/core/lib/surface/server.cc:1223
#19 0x000055ea8a10288c in grpc::Server::ShutdownInternal (this=0x55ea8a7ed3f0, deadline=...) at src/cpp/server/server_cc.cc:565
#20 0x000055ea8a07e656 in grpc::ServerInterface::Shutdown (this=0x55ea8a7ed3f0) at include/grpcpp/impl/codegen/server_interface.h:71
#21 0x000055ea8a0421ac in grpc::testing::(anonymous namespace)::End2endTest::TearDown (this=0x55ea8a78ebc0) at test/cpp/end2end/end2end_test.cc:232
#22 0x000055ea8a294e1f in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x55ea8a78ebc0,
    method=&virtual testing::Test::TearDown(), location=0x55ea8a2f9e01 "TearDown()") at third_party/googletest/googletest/src/gtest.cc:2402
#23 0x000055ea8a28fca2 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x55ea8a78ebc0,
    method=&virtual testing::Test::TearDown(), location=0x55ea8a2f9e01 "TearDown()") at third_party/googletest/googletest/src/gtest.cc:2438
#24 0x000055ea8a2760d7 in testing::Test::Run (this=0x55ea8a78ebc0) at third_party/googletest/googletest/src/gtest.cc:2482
#25 0x000055ea8a276910 in testing::TestInfo::Run (this=0x55ea8a7af090) at third_party/googletest/googletest/src/gtest.cc:2656
#26 0x000055ea8a276f67 in testing::TestCase::Run (this=0x55ea8a7ac9c0) at third_party/googletest/googletest/src/gtest.cc:2774
#27 0x000055ea8a27dadb in testing::internal::UnitTestImpl::RunAllTests (this=0x55ea8a77bad0) at third_party/googletest/googletest/src/gtest.cc:4649
#28 0x000055ea8a295d63 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x55ea8a77bad0,
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x55ea8a27d818 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x55ea8a2fa630 "auxiliary test code (environments or event listeners)") at third_party/googletest/googletest/src/gtest.cc:2402
#29 0x000055ea8a2909e0 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x55ea8a77bad0,
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x55ea8a27d818 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x55ea8a2fa630 "auxiliary test code (environments or event listeners)") at third_party/googletest/googletest/src/gtest.cc:2438
#30 0x000055ea8a27c7b2 in testing::UnitTest::Run (this=0x55ea8a605d20 <testing::UnitTest::GetInstance()::instance>)
    at third_party/googletest/googletest/src/gtest.cc:4257
#31 0x000055ea8a080f42 in RUN_ALL_TESTS () at third_party/googletest/googletest/include/gtest/gtest.h:2233
#32 0x000055ea8a061339 in main (argc=1, argv=0x7ffe2b59b8e8) at test/cpp/end2end/end2end_test.cc:1836
(gdb)

(gdb) t 1

You are explicitly telling GDB to switch to another thread (away from the one that just hit the assertion).
The thread you happen to switch to is blocked on a mutex, but isn't relevant to the failure.

Yeah, t 1 is the default thread when I started gdb.
Another coredump shows that the crash may happen in gpr_cv_wait(). I'm lost. The common thing between these two stacktrace is that they happen during shutdown.
(gdb) bt
#0  raise (sig=3) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x0000556df554a345 in crash_handler (signum=3, info=0x556df59975b0 <g_alt_stack+64528>, data=0x556df5997480 <g_alt_stack+64224>)
    at test/core/util/test_config.cc:255
#2  <signal handler called>
#3  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#4  0x0000556df54a2e85 in gpr_cv_wait (cv=0x556df59856b0 <g_cvfds+48>, mu=0x556df5985680 <g_cvfds>, abs_deadline=...) at src/core/lib/gpr/sync_posix.cc:89
#5  0x0000556df547579a in global_cv_fd_table_shutdown () at src/core/lib/iomgr/ev_poll_posix.cc:1700
#6  0x0000556df5475867 in shutdown_engine () at src/core/lib/iomgr/ev_poll_posix.cc:1721
#7  0x0000556df5451661 in grpc_event_engine_shutdown () at src/core/lib/iomgr/ev_posix.cc:189
#8  0x0000556df5452c32 in grpc_iomgr_platform_shutdown () at src/core/lib/iomgr/iomgr_posix.cc:38
#9  0x0000556df5452aa0 in grpc_iomgr_shutdown () at src/core/lib/iomgr/iomgr.cc:148
#10 0x0000556df54a92c5 in grpc_shutdown () at src/core/lib/surface/init.cc:173
#11 0x0000556df547974d in grpc::internal::GrpcLibrary::shutdown (this=0x556df5976a60 <grpc::internal::g_gli>) at include/grpcpp/impl/grpc_library.h:35
#12 0x0000556df54111d1 in grpc::GrpcLibraryCodegen::~GrpcLibraryCodegen (this=0x556df5f27620, __in_chrg=<optimized out>)
    at include/grpcpp/impl/codegen/grpc_library.h:54
#13 0x0000556df5482157 in grpc::Channel::~Channel (this=0x556df5f27600, __in_chrg=<optimized out>) at src/cpp/client/channel_cc.cc:56
#14 0x0000556df54821b0 in grpc::Channel::~Channel (this=0x556df5f27600, __in_chrg=<optimized out>) at src/cpp/client/channel_cc.cc:56
#15 0x0000556df548477b in std::_Sp_counted_ptr<grpc::Channel*, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x556df5f270e0)
    at /usr/include/c++/7/bits/shared_ptr_base.h:376
#16 0x0000556df5417be4 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x556df5f270e0)
    at /usr/include/c++/7/bits/shared_ptr_base.h:154
#17 0x0000556df5414839 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x556df5ec1108, __in_chrg=<optimized out>)
    at /usr/include/c++/7/bits/shared_ptr_base.h:684
#18 0x0000556df54140b6 in std::__shared_ptr<grpc::Channel, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x556df5ec1100, __in_chrg=<optimized out>)
    at /usr/include/c++/7/bits/shared_ptr_base.h:1123
#19 0x0000556df54140d2 in std::shared_ptr<grpc::Channel>::~shared_ptr (this=0x556df5ec1100, __in_chrg=<optimized out>)
    at /usr/include/c++/7/bits/shared_ptr.h:93
#20 0x0000556df53d8709 in grpc::testing::(anonymous namespace)::End2endTest::~End2endTest (this=0x556df5ec10e0, __in_chrg=<optimized out>)
    at test/cpp/end2end/end2end_test.cc:219
#21 0x0000556df53e7cff in grpc::testing::(anonymous namespace)::ProxyEnd2endTest::~ProxyEnd2endTest (this=0x556df5ec10e0, __in_chrg=<optimized out>)
    at test/cpp/end2end/end2end_test.cc:1248
#22 0x0000556df540205d in grpc::testing::(anonymous namespace)::ProxyEnd2endTest_RpcDeadlineExpires_Test::~ProxyEnd2endTest_RpcDeadlineExpires_Test (
    this=0x556df5ec10e0, __in_chrg=<optimized out>) at test/cpp/end2end/end2end_test.cc:1279
#23 0x0000556df540207e in grpc::testing::(anonymous namespace)::ProxyEnd2endTest_RpcDeadlineExpires_Test::~ProxyEnd2endTest_RpcDeadlineExpires_Test (
    this=0x556df5ec10e0, __in_chrg=<optimized out>) at test/cpp/end2end/end2end_test.cc:1279
#24 0x0000556df561ce92 in testing::Test::DeleteSelf_ (this=0x556df5ec10e0) at third_party/googletest/googletest/include/gtest/gtest.h:453
#25 0x0000556df5627e1f in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x556df5ec10e0,
    method=(void (testing::Test::*)(testing::Test * const)) 0x556df561ce68 <testing::Test::DeleteSelf_()>,
    location=0x556df568cf7f "the test fixture's destructor") at third_party/googletest/googletest/src/gtest.cc:2402
#26 0x0000556df5622ca2 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x556df5ec10e0,
    method=(void (testing::Test::*)(testing::Test * const)) 0x556df561ce68 <testing::Test::DeleteSelf_()>,
    location=0x556df568cf7f "the test fixture's destructor") at third_party/googletest/googletest/src/gtest.cc:2438
#27 0x0000556df560995d in testing::TestInfo::Run (this=0x556df5ee14d0) at third_party/googletest/googletest/src/gtest.cc:2661
#28 0x0000556df5609f67 in testing::TestCase::Run (this=0x556df5eb8d10) at third_party/googletest/googletest/src/gtest.cc:2774
#29 0x0000556df5610adb in testing::internal::UnitTestImpl::RunAllTests (this=0x556df5eadad0) at third_party/googletest/googletest/src/gtest.cc:4649
#30 0x0000556df5628d63 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x556df5eadad0,
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x556df5610818 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x556df568d630 "auxiliary test code (environments or event listeners)") at third_party/googletest/googletest/src/gtest.cc:2402
#31 0x0000556df56239e0 in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x556df5eadad0,
    method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x556df5610818 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x556df568d630 "auxiliary test code (environments or event listeners)") at third_party/googletest/googletest/src/gtest.cc:2438
#32 0x0000556df560f7b2 in testing::UnitTest::Run (this=0x556df5998d20 <testing::UnitTest::GetInstance()::instance>)
    at third_party/googletest/googletest/src/gtest.cc:4257
#33 0x0000556df5413f42 in RUN_ALL_TESTS () at third_party/googletest/googletest/include/gtest/gtest.h:2233
#34 0x0000556df53f4339 in main (argc=1, argv=0x7ffd09848708) at test/cpp/end2end/end2end_test.cc:1836

@sreecha helped me find out that it's caused by multiple coredumps overwriting the previous ones. After changing the coredump filename pattern, I'm able to find the matching stacktrace now. Thanks Sree!
It turns out to be another refcount issue (in grpc_tcp_server_ref()). So I've filed it in #14703.

authors: AspirinSJL jtattermusch kpayson64 ncteisen sreecha

repo: grpc/grpc


text:
grpc.io website not accessible via HTTPS

If I visit https://www.grpc.io/, my browser prevents me viewing it because the presented certificate is for www.github.com. gRPC's website should be HTTPS accessible.

I'll take a look. It's running using Github Pages and Jekyll, which should support HTTPS access.

Hrmm, seems it's only enabled by default for *.github.io domains. Will investigate further.

@mugurm as I think he set up the domain stuff.....

Though if America can do it then so can we: https://theunitedstates.io/

@mugurm, friendly ping.

This is still not fixed, and this issue doesn't have an explanation for why. If it's not possible with github pages it's trivial to do with netlify. Perhaps consider switching? It's frankly embarassing that a high profile google project like gRPC doesn't serve it's documentation of HTTPS.

Github Pages is the problem - it unfortunately doesn't support HTTPS for sites with a custom domain (which grpc.io uses). I know at least one other site that has used a workaround for it but IIRC it was pretty kludgey.
I have been looking into Netlify for other reasons so will add that to our "reasons to investigate".

This issue has been resolved. https://grpc.io works now.

source: gharchive/issue

authors: LisaFC dsymonds johanbrandhorst jtattermusch mehrdada

license_type: permissive

license_source: github-api


text:
c_linux_dbg_native_x64_gcc8_3.bins/dbg/httpscli_test failing at 100% on master

From the portability master suite: https://fusion.corp.google.com/projectanalysis/summary/KOKORO/prod%3Agrpc%2Fcore%2Fmaster%2Flinux%2Fgrpc_portability
Log snippet:
D0117 01:20:16.009991164    2053 ev_posix.cc:174]            Using polling engine: poll
D0117 01:20:16.010214358    2053 dns_resolver_ares.cc:516]   Using ares dns resolver
D0117 01:20:16.011004707    2053 tcp_posix.cc:1811]          cannot set inq fd=3 errno=92
/var/local/git/grpc/src/core/tsi/test_creds/server1.pem
server running on port 11967
Traceback (most recent call last):
  File "bins/dbg/../../test/core/http/test_server.py", line 64, in <module>
    server_side=True)
  File "/usr/lib/python2.7/ssl.py", line 931, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 542, in __init__
    self._context.load_cert_chain(certfile, keyfile)
ssl.SSLError: [SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:2815)
D0117 01:20:21.012465286    2053 test_config.cc:385]         test slowdown factor: sanitizer=1, fixture=1, poller=1, total=1
D0117 01:20:21.012739203    2053 ev_posix.cc:174]            Using polling engine: poll
D0117 01:20:21.012834684    2053 dns_resolver_ares.cc:516]   Using ares dns resolver
I0117 01:20:21.012898913    2053 httpscli_test.cc:72]        test_get
I0117 01:20:21.012905891    2053 httpscli_test.cc:75]        requesting from localhost:11967
I0117 01:20:22.015425653    2053 httpscli_test.cc:53]        response status=0 error={"created":"@1579224022.015352031","description":"Failed HTTP requests to all targets","file":"src/core/lib/http/httpcli.cc","file_line":202,"referenced_errors":[{"created":"@1579224022.015307820","description":"Failed HTTP/1 client request","file":"src/core/lib/http/httpcli.cc","file_line":113,"referenced_errors":[{"created":"@1579224022.015299152","description":"Failed to connect to remote host: Connection refused","errno":111,"file":"src/core/lib/iomgr/tcp_client_posix.cc","file_line":200,"os_error":"Connection refused","syscall":"connect","target_address":"ipv6:[::1]:11967"},{"created":"@1579224022.015344404","description":"Failed to connect to remote host: Connection refused","errno":111,"file":"src/core/lib/iomgr/tcp_client_posix.cc","file_line":200,"os_error":"Connection refused","syscall":"connect","target_address":"ipv4:127.0.0.1:11967"}]}]}
E0117 01:20:22.015445341    2053 httpscli_test.cc:55]        assertion failed: response->status == 200

*******************************
Caught signal SIGABRT
bins/dbg/httpscli_test(+0x36880)[0x556a3292a880]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fb63b47a730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7fb63afbb7bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7fb63afa6535]
bins/dbg/httpscli_test(+0x341fc)[0x556a329281fc]
bins/dbg/httpscli_test(+0x35041)[0x556a32929041]
bins/dbg/httpscli_test(+0x3c351)[0x556a32930351]
bins/dbg/httpscli_test(+0x3c881)[0x556a32930881]
bins/dbg/httpscli_test(+0x6988f)[0x556a3295d88f]
bins/dbg/httpscli_test(+0x5bc70)[0x556a3294fc70]
bins/dbg/httpscli_test(+0x3d89c)[0x556a3293189c]
bins/dbg/httpscli_test(+0x345ae)[0x556a329285ae]
bins/dbg/httpscli_test(+0x34db9)[0x556a32928db9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7fb63afa809b]
bins/dbg/httpscli_test(+0x3402a)[0x556a3292802a]
Command terminated by signal 6
real 6.14
user 0.00
sys 0.00

Is it still happening?

Doesn't seem like this has been seen for a long time, so I'm closing it.

authors: apolcyn markdroth nicolasnoble


text:
zlib's CVE-2018-25032 in gprcio

What version of gRPC and what language are you using?
grpcio 1.44.0
What operating system (Linux, Windows,...) and version?
Linux
What runtime / compiler are you using (e.g. python version or version of gcc)
python 3.10
What did you do?
install grpcio by pip install
What did you expect to see?
pip install grpcio will not install a zlib component with CVES.
What did you see instead?
As my scanning, i found that cygrpc.cpython-310-x86_64-linux-gnu.so in the wheel contains a 1.2.11 zlib which has CVE-2018-25032.
You should update zlib in the https://github.com/grpc/grpc/tree/v1.44.0/third_party to 1.2.12 to fix this CVE or make sure you are not affected by this.

cc @veblush This might require a zlib upgrade for all wrapper languages.

For this is a High severity CVE, i hope you can fix it as soon as possible.

Thanks for the report. Yeah, I'm going to upgrade it to the 1.2.12.

Fixed by https://github.com/grpc/grpc/pull/29531

authors: lidizheng veblush wqh17101


text:
Maybe find a better place for the grpcio-tools directory

Hanging out in tools/distrib/python is a holdover from me not wanting to think very hard about what the 'right' home for a grab-bag of code vendored from the repository root directory should be. I have no idea if folks think it's confusing or if it's just 'how things are', but if the past has anything to teach us it's that 'how things are' isn't necessarily 'how things should be' (... for some philosophy of 'should').

(To be clear: this is still "something we'd rather happen than not"; we just don't need an open issue for it.)

authors: nathanielmanistaatgoogle soltanmm


text:
c#: add remaining missing channel options

Heya,
ChannelOptions currently doesn't implement the following options that are available in the native library:
- grpc.loadreporting
- grpc.http2.lookahead_bytes
- grpc.http2.hpack_table_size.decoder
- grpc.http2.hpack_table_size.encoder
- grpc.max_reconnect_backoff_ms
- grpc.max_metadata_size
- grpc.so_reuseport

I care most about grpc.max_reconnect_backoff_ms to reduce the reconnect backoff delay, which currently defaults to a maximum of 120s. I'm unsure how important the other options are, or if they were previously left out for a reaso, but some people might have a need to change them away from their defaults.

You should actually be able to currently use "grpc.max_reconnect_backoff_ms" channel arg, among others, by just passing the argument as a raw string.
E.g.,
new ChannelOption("grpc.max_reconnect_backoff_ms", 100)

The existing ChannelOption constants are just for convenience for using the strings like above AFAIK.
cc @jtattermusch, should the ChannelOption constants be kept up to date to match core?

Oh, can't believe I missed that. Thanks so much.

@apolcyn , yes we can add constants for channel options that C# users are likely to use. In general, the channel options are intented to be something that 99% of users should need to touch and gRPC should come with reasonable presets.

authors: apolcyn imiuka jtattermusch


text:
Decouple new connection timeout from time of next retry

Context: #8382
In the current implementation the channel connection deadline and time of the next backoff attempt are the same value. That causes issues because a short initial backoff interval (which seems desirable) leads to a very short timeout for the first connection attempt.
For secure connections, this is even worse, because the entire SSL handshake is counted towards the deadline and if not finished on time, the connection gets closed (even though it is likely that the secure connection would be established correctly once we have the TCP/IP connection ready).
On windows, we are definitely seeing issues when creating many secure connections in parallel, but the issue itself is OS agnostic.
Suggestion: differentiate between "connection attempt timeout/deadline" and "time of next attempt if current attempt fails".

CC @apolcyn
@yang-g  can you please assign this to the right person in C core team?

I talked with Craig about this before and yes we need to have a fix here.

https://github.com/grpc/grpc/blob/24e5f3756742e4f65cfefc6c88d10395f1ac92be/src/core/ext/client_channel/subchannel.c#L378

I think this is a duplicate of #12867.

authors: jtattermusch markdroth yang-g


text:
Makefile: Fix libgrpc++.so link

The library is built as libgrpc++.so.6, but installed as libgrpc++.so.1.
Signed-off-by: Bernhard Walle bernhard@bwalle.de

Thank you for your pull request. Before we can look at your contribution, we need to ensure all contributors are covered by a Contributor License Agreement.
After the following items are addressed, please respond with a new comment here, and the automated system will re-verify.

User @bwalle isn't covered by a CLA. They will need to complete the form at https://identity.linuxfoundation.org/projects/cncf

Regards,
CLA GitHub bot

Something missing here?

Is there a corresponding issue for this?

Makefile is generated, you need to update the corresponding template too.
I have a feeling that this was fixed already by another PR - can you check if the problem still exists on the latest HEAD?

Seems fixed here? https://github.com/grpc/grpc/pull/16832

Yes, it was fixed. The pull request is more than 6 month old!

source: gharchive/pull-request

authors: bwalle jtattermusch thelinuxfoundation yashykt


text:
Improve BUILDING.md

Add a note about installation, as discussed in https://github.com/grpc/grpc/pull/16409.

Objective-C binary sizes
*****************STATIC******************
  New size                      Old size
 1,952,962      Total (=)      1,952,962

 No significant differences in binary sizes

***************FRAMEWORKS****************
  New size                      Old size
10,664,803      Total (>)     10,664,800

 No significant differences in binary sizes

authors: grpc-testing jtattermusch


text:
Introduced grpc_error_handle

Replacing grpc_error* with grpc_error_handle which is an alias of grpc_error* so this change shouldn't change anything but this will be the bridge between grpc_error to absl::Status. This grpc_error_handle will be absl::Status later.
Mostly mechanical change replacing grpc_error* with grpc_error_handle in the first commit. The second commit simply defines grpc_error_handle in both error.h and iomgr.pxd.pxi

Based on this PR, I am assuming that the migration plan here is something like the following:

Change all code to use grpc_error_handle, which is defined as grpc_error*.
Change the definition of grpc_error_handle to absl::Status, and change functions like grpc_error_get_status() to work on the new type.
Change all callers to directly use the new absl::Status API.
Remove grpc_error* API.

Is that right?  If that's right, then this PR looks good.
It would probably be a good idea to present at the weekly design review meeting about this, both in terms of what the new API will look like and what the migration plan will be.

@markdroth Yes, you're right about the migration plan. There will be an extra step between 1 and 2, having the state where the old and new API coexist, mostly in the macro form, allowing the code to be built with the either API. Also I have a plan to have a weekly discussion on this.

authors: markdroth veblush


text:
Add min/max tls version to tls creds options core and C++ APIs.

Plumb the APIs to core and C++.

@ZhenLian How may I assign this to you for a first look? Thanks.

Interop ALTS Tests #18111
Portability Tests Windows #31401
Neither of these seems related to this PR.

As @ZhenLian will not have enough bandwidth to further review my PRs in a while, @markdroth Could you PTAL when you get a chance (or please assign a different reviewer if you think they are better suited)? Thank you!
The APIs in this PR should be fairly simple. As mentioned in the comment above, such a test would invoke low-level TSI errors and result in test timeout before grpc could successfully shut down. Therefore I am currently unsure whether it is still worth adding such e2e tests in core to test wrong TLS versions. Please let me know what you think.

@erm-g Could you approve the PR if it looks good to you? You can do so by going to the "Files changed" section and the option is under the "Review Changes" green button. Thank you!

Going to create a new one.


text:
Cache encodings accepted by peer on mdelem

Built on #3754

LGTM, thanks for the example.
I'll wait for the merge of #3754

Hi all, just checking in on the status here.

authors: ctiller dgquintas vjpai


text:
Expand gtest suites into individual run_tests tests

This will allow tracking individual test status in some of the large suites that we have in C++ land currently.

Thank you. We really need this.

I'm wondering if that could also be somewhat useful for #5357 ?

authors: ctiller nicolasnoble vjpai


text:
Ruby docker files and config

Note: Currently, the ruby stress tests fail due to https://github.com/grpc/grpc/issues/6405 (thanks to @murgatroid99 and @nicolasnoble for helping me in identifying the issue)
It will still be good to merge the changes in this PR. I will add a jenkins job to run the tests once the above issue is fixed.

LGTM, but let's first have green performance smoke test.

authors: jtattermusch sreecha


text:
Move deadline functionality into filters.

I still need to do some clean-up work here (e.g., removing a bunch of logging that I've added for debugging), but the core functionality should be done.  However, as I mentioned earlier today, I think there's some sort of subtle timing bug here, because the tests are failing sporadically.  I'd appreciate any pointers you can offer as to where the problem might be.  Thanks!

retest this please

There are probably still some test failures here that I'll need to look into, but I think this is finally in a good state to start reviewing.

Note that this is part of addressing #7818.

I'm yet to do a detailed review, but architecture LGTM.

retest this please

retest this please

authors: ctiller markdroth


text:
EKS nuke should take into account fargate profiles

Since you can't destroy an EKS cluster with fargate profiles, the nuking fails. This is a request to also delete fargate profiles on the EKS cluster prior to destroying the cluster.

Looking into this.

Looking into this.

Fixed in https://github.com/gruntwork-io/cloud-nuke/releases/tag/v0.7.0

repo: gruntwork-io/cloud-nuke

license: MIT


text:
Log accurate number of AMIs deleted

Description
#161
Log the actual count of AMIs deleted.
Tests
I tried running the tests, but my AWS educate account doesn't have enough permissions to run them properly.
Its a simple change though.

Code changes look good, I'm kicking off tests now!

Tests passed, merging!
Thanks for the contribution!

https://github.com/gruntwork-io/cloud-nuke/releases/tag/v0.1.27

authors: StandB infraredgirl


text:
Go modules

While I was trying to help upgrade terrafrom to v0.12.12, I have found the module dependency has been changed to use Subpackage github.com/hashicorp/hcl/v2.
Based on the Version-Selection doc, to be able to use hcl/v2, it requires to do the go modules upgrade.
Furthermore, with go v1.14, go modules would be the default toolchain, so I think it would be nice that we can upgrade the build toolchain to use go modules.
Kind of relates to https://github.com/Homebrew/homebrew-core/pull/45521

This is on our roadmap, but I'm not sure if we'll be able to get to it soon. In the meantime, a PR to switch to Go modules is very welcome!

Done in #1105

repo: gruntwork-io/terragrunt

license: mit

license_source: bigquery


text:
Dynamically set module version in terratest

Hi Team,
Is there a way we can dynamically set a "version" field  in the module available in terraform configuration file similar to
terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
	TerraformDir: "../fixtures/cloud_storage",
	// Reconfigure:  true,
	Vars: map[string]interface{}{
		"project_id": projectId,
		"name": fmt.Sprintf("bucket-asd-"+environment+"-encryption-testauto"),
		"location":         location,
		"storage_class":    "STANDARD",
		"force_destroy": true,
		"additional_labels": labels["ap1"],
		"uniform_bucket_level_access": true,
		"keynames": kmsKeyId,
	},
	VarFiles: []string{
		"common_cloud_storage.tfvars",
	},

})

I usually use https://github.com/minamijoyo/hcledit for things like this.

repo: gruntwork-io/terratest


text:
MigrateState not working with Workspace

When using MigrateState Terraform option with a Workspace(terraform.WorkspaceSelectOrNew) I see the following failure:
Error: Error asking for state migration action: Error asking for confirmation: EOF
Presumably, there are extra prompts when using workspaces...
If I run the same tests without the workspace it works as expected.

Hm, yea, looks like a bug. We probably need some sort of -input=false or other "non-interactive" style flag here. PRs with a fix are very welcome.