Use after free during transaction shutdown with attach_server_session_to_client enabled

[shinrich]

I am seeing this on the current master. But I have only started running ASAN in our environment. I see nothing in the 9.1.3 code we were running previously that would have protected us from this case.

I am labeling this as a crash, although I'm only seeing this in ASAN at the moment. Getting lucky so far I guess.

==4114384==ERROR: AddressSanitizer: heap-use-after-free on address 0x619000072680 at pc 0x5640be2ee369 bp 0x7fdf0a8d90f0 sp 0x7fdf0a8d90e0
READ of size 8 at 0x619000072680 thread T3
#0 0x5640be2ee368 in Http1ClientSession::do_io_close(int) /home/ubuntu/ats10/proxy/http/Http1ClientSession.cc:233
#1 0x5640be2ef3ac in Http1ClientSession::state_keep_alive(int, void*) /home/ubuntu/ats10/proxy/http/Http1ClientSession.cc:396
#2 0x5640bec56535 in Continuation::handleEvent(int, void*) /home/ubuntu/ats10/iocore/eventsystem/I_Continuation.h:228
#3 0x5640bec56535 in Continuation::handleEvent(int, void*) /home/ubuntu/ats10/iocore/eventsystem/I_Continuation.h:224
#4 0x5640bec56535 in read_signal_and_update /home/ubuntu/ats10/iocore/net/UnixNetVConnection.cc:82
#5 0x5640bec70e7d in read_signal_done /home/ubuntu/ats10/iocore/net/UnixNetVConnection.cc:143
#6 0x5640bec70e7d in read_from_net /home/ubuntu/ats10/iocore/net/UnixNetVConnection.cc:285
#7 0x5640bed0d678 in NetHandler::process_ready_list() /home/ubuntu/ats10/iocore/net/NetHandler.cc:252
#8 0x5640bed0e054 in NetHandler::waitForActivity(long) /home/ubuntu/ats10/iocore/net/NetHandler.cc:340
#9 0x5640beddb02b in EThread::execute_regular() /home/ubuntu/ats10/iocore/eventsystem/UnixEThread.cc:285
#10 0x5640beddb5f2 in EThread::execute() /home/ubuntu/ats10/iocore/eventsystem/UnixEThread.cc:334
#11 0x5640beddb5f2 in EThread::execute() /home/ubuntu/ats10/iocore/eventsystem/UnixEThread.cc:312
#12 0x5640bedd4cef in spawn_thread_internal /home/ubuntu/ats10/iocore/eventsystem/Thread.cc:78
#13 0x7fdf10e21b42 in start_thread nptl/pthread_create.c:442
#14 0x7fdf10eb39ff (/lib/x86_64-linux-gnu/libc.so.6+0x1269ff)

0x619000072680 is located 0 bytes inside of 1072-byte region [0x619000072680,0x619000072ab0)
freed by thread T3 here:
#0 0x5640be13bde7 in __interceptor_free (/opt/ats/ats_10.0.0/bin/traffic_server+0x51dde7)
#1 0x5640bee5e7ec in ink_freelist_free(_InkFreeList*, void*) /home/ubuntu/ats10/src/tscore/ink_queue.cc:299
#2 0x5640be40ff44 in Http1ServerSession::do_io_close(int) /home/ubuntu/ats10/proxy/http/Http1ServerSession.cc:134
#3 0x5640be522232 in Http1ServerTransaction::transaction_done() /home/ubuntu/ats10/proxy/http/Http1ServerTransaction.cc:51
#4 0x5640be3dfc98 in HttpSM::kill_this() /home/ubuntu/ats10/proxy/http/HttpSM.cc:7486
#5 0x5640be3e3517 in HttpSM::main_handler(int, void*) /home/ubuntu/ats10/proxy/http/HttpSM.cc:2682
#6 0x5640be4f9adc in Continuation::handleEvent(int, void*) /home/ubuntu/ats10/iocore/eventsystem/I_Continuation.h:228
#7 0x5640be4f9adc in Continuation::handleEvent(int, void*) /home/ubuntu/ats10/iocore/eventsystem/I_Continuation.h:224
#8 0x5640be4f9adc in HttpTunnel::main_handler(int, void*) /home/ubuntu/ats10/proxy/http/HttpTunnel.cc:1698
#9 0x5640be4f9adc in HttpTunnel::main_handler(int, void*) /home/ubuntu/ats10/proxy/http/HttpTunnel.cc:1658
#10 0x5640bec55ef5 in Continuation::handleEvent(int, void*) /home/ubuntu/ats10/iocore/eventsystem/I_Continuation.h:228
#11 0x5640bec55ef5 in Continuation::handleEvent(int, void*) /home/ubuntu/ats10/iocore/eventsystem/I_Continuation.h:224
#12 0x5640bec55ef5 in write_signal_and_update /home/ubuntu/ats10/iocore/net/UnixNetVConnection.cc:113
#13 0x5640bec6a362 in write_signal_done /home/ubuntu/ats10/iocore/net/UnixNetVConnection.cc:155
#14 0x5640bec6a362 in write_to_net_io(NetHandler*, UnixNetVConnection*, EThread*) /home/ubuntu/ats10/iocore/net/UnixNetVConnection.cc:514
#15 0x5640bed0d9f5 in NetHandler::process_ready_list() /home/ubuntu/ats10/iocore/net/NetHandler.cc:262
#16 0x5640bed0e054 in NetHandler::waitForActivity(long) /home/ubuntu/ats10/iocore/net/NetHandler.cc:340
#17 0x5640beddb02b in EThread::execute_regular() /home/ubuntu/ats10/iocore/eventsystem/UnixEThread.cc:285
#18 0x5640beddb5f2 in EThread::execute() /home/ubuntu/ats10/iocore/eventsystem/UnixEThread.cc:334
#19 0x5640beddb5f2 in EThread::execute() /home/ubuntu/ats10/iocore/eventsystem/UnixEThread.cc:312
#20 0x5640bedd4cef in spawn_thread_internal /home/ubuntu/ats10/iocore/eventsystem/Thread.cc:78
#21 0x7fdf10e21b42 in start_thread nptl/pthread_create.c:442

previously allocated by thread T3 here:
#0 0x5640be13ce2c in posix_memalign (/opt/ats/ats_10.0.0/bin/traffic_server+0x51ee2c)
#1 0x5640bee5c4f4 in ats_memalign(unsigned long, unsigned long) /home/ubuntu/ats10/src/tscore/ink_memory.cc:108
#2 0x5640bee5e753 in ink_freelist_new(_InkFreeList*) /home/ubuntu/ats10/src/tscore/ink_queue.cc:202
#3 0x5640be411e0d in FreelistAllocator::alloc_void() ../../include/tscore/Allocator.h:63
#4 0x5640be411e0d in Http1ServerSession* ClassAllocator<Http1ServerSession, true, FreelistAllocator>::alloc<>() ../../include/tscore/Allocator.h:245
#5 0x5640be411e0d in create_h1_server_session::{lambda()#1}::operator()() const /home/ubuntu/ats10/proxy/http/Http1ServerSession.cc:260
#6 0x5640be411e0d in PoolableSession* std::__invoke_impl<PoolableSession*, create_h1_server_session::{lambda()#1}&>(std::__invoke_other, create_h1_server_session::{lambda()#1}&) /usr/include/c++/11/bits/invoke.h:61
#7 0x5640be411e0d in std::enable_if<is_invocable_r_v<PoolableSession*, create_h1_server_session::{lambda()#1}&>, std::enable_if>::type std::__invoke_r<PoolableSession*, create_h1_server_session::{lambda()#1}&>(PoolableSession*&&, (create_h1_server_session::{lambda()#1}&)...) /usr/include/c++/11/bits/invoke.h:114
#8 0x5640be411e0d in std::_Function_handler<PoolableSession* (), create_h1_server_session::{lambda()#1}>::_M_invoke(std::_Any_data const&) /usr/include/c++/11/bits/std_function.h:290
#9 0x5640beab0dcf in std::function<PoolableSession* ()>::operator()() const /usr/include/c++/11/bits/std_function.h:590
#10 0x5640beab0dcf in ProxySession::create_outbound_session(int) /home/ubuntu/ats10/proxy/ProxySession.cc:333

[cmcfarlen]

Could this be related to #10116 ?