Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deadlock in System.Data.SqlClient.SNI.SNIMarsHandle, SNIMarsConnection, TdsParserStateObjectManaged #383

Closed
czd890 opened this issue Jan 9, 2020 · 7 comments
Closed

Deadlock in System.Data.SqlClient.SNI.SNIMarsHandle, SNIMarsConnection, TdsParserStateObjectManaged #383

czd890 opened this issue Jan 9, 2020 · 7 comments

Comments

@czd890
Copy link

czd890 commented Jan 9, 2020

System vestion

$ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.1.100
 Commit:    cd82f021f4

Runtime Environment:
 OS Name:     centos
 OS Version:  7
 OS Platform: Linux
 RID:         centos.7-x64
 Base Path:   /usr/share/dotnet/sdk/3.1.100/

Host (useful for support):
  Version: 3.1.0
  Commit:  157910edee

.NET Core SDKs installed:
  2.2.402 [/usr/share/dotnet/sdk]
  3.0.100 [/usr/share/dotnet/sdk]
  3.1.100 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

System.Data.SqlClient.dll version

> exiftool System.Data.SqlClient.dll
ExifTool Version Number         : 11.82
File Name                       : System.Data.SqlClient.dll
Directory                       : /xignite/his1383
File Size                       : 934 kB
File Modification Date/Time     : 2019:11:15 00:56:02+00:00
File Access Date/Time           : 2020:01:09 04:24:34+00:00
File Inode Change Date/Time     : 2020:01:06 09:52:18+00:00
File Permissions                : rw-rw-r--
File Type                       : Win32 DLL
File Type Extension             : dll
MIME Type                       : application/octet-stream
Machine Type                    : Intel 386 or later, and compatibles
Time Stamp                      : 2085:08:25 18:35:15+00:00
Image File Characteristics      : Executable, Large address aware, DLL
PE Type                         : PE32
Linker Version                  : 48.0
Code Size                       : 944128
Initialized Data Size           : 2048
Uninitialized Data Size         : 0
Entry Point                     : 0xe74e2
OS Version                      : 4.0
Image Version                   : 0.0
Subsystem Version               : 4.0
Subsystem                       : Windows command line
File Version Number             : 4.700.19.56404
Product Version Number          : 3.1.0.0
File Flags Mask                 : 0x003f
File Flags                      : (none)
File OS                         : Win32
Object File Type                : Dynamic link library
File Subtype                    : 0
Language Code                   : Neutral
Character Set                   : Unicode
Comments                        : System.Data.SqlClient
Company Name                    : Microsoft Corporation
File Description                : System.Data.SqlClient
File Version                    : 4.700.19.56404
Internal Name                   : System.Data.SqlClient.dll
Legal Copyright                 : © Microsoft Corporation. All rights reserved.
Original File Name              : System.Data.SqlClient.dll
Product Name                    : Microsoft® .NET Core
Product Version                 : 3.1.0+0f7f38c4fd323b26da10cce95f857f77f0f09b48
Assembly Version                : 4.6.1.0

List clrthreads and syncblk

> clrthreads
ThreadCount:      50
UnstartedThread:  0
BackgroundThread: 18
PendingThread:    0
DeadThread:       31
Hosted Runtime:   no
                                                                                                        Lock
 DBG   ID OSID ThreadOBJ           State GC Mode     GC Alloc Context                  Domain           Count Apt Exception
   0    1  1a8 0000000001AF5930  2020020 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn
   6    2  1ae 0000000001B50520    21220 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Finalizer)
   8    3  1b0 00007F53C80009F0  1020220 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
   9    6  1b4 00007F53CC0346A0    21220 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn
  10    8  1b6 0000000002883B20  2021220 Preemptive  00007F53D94174A0:00007F53D94179C0 0000000001AF3B70 0     Ukn
  11    9  1b7 00007F53C40130F0  2021220 Preemptive  00007F53D96BFAD0:00007F53D96C0670 0000000001AF3B70 0     Ukn
  12   10  1b8 00007F53C404E760  2021220 Preemptive  00007F53D9659980:00007F53D965A670 0000000001AF3B70 0     Ukn
  13   11  1b9 00007F53C4002BB0  2021220 Preemptive  00007F53D9723B90:00007F53D9724670 0000000001AF3B70 0     Ukn
  14   12  1ba 00007F53C40524D0  2021220 Preemptive  00007F53D97CEB40:00007F53D97D0670 0000000001AF3B70 0     Ukn
  15   13  1bb 00007F53C4054000  2021220 Preemptive  00007F53D97534B0:00007F53D9754670 0000000001AF3B70 0     Ukn
  16   14  1bc 00007F53C4055D10  2021220 Preemptive  00007F53D974E400:00007F53D974E670 0000000001AF3B70 0     Ukn
  17   15  1bd 00007F53C4057A20  2021220 Preemptive  00007F53D9789120:00007F53D978A670 0000000001AF3B70 0     Ukn
  18   16  1be 00007F53C4059ED0  2021220 Preemptive  00007F53D977A4C0:00007F53D977A670 0000000001AF3B70 0     Ukn
  19   17  1bf 00007F53C405BE00  2021220 Preemptive  00007F53D97DC7B0:00007F53D97DE670 0000000001AF3B70 0     Ukn
  20   18  1c0 00007F53C405DC80  2021220 Preemptive  00007F53D974F968:00007F53D9750670 0000000001AF3B70 0     Ukn
XXXX   43    0 00007F53F4002CE0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX    7    0 00007F53F00261D0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
  21   26  320 0000000002886100  3021220 Preemptive  00007F53D9418228:00007F53D941A1F0 0000000001AF3B70 1     Ukn (Threadpool Worker)
  22   24  33b 00007F53B00170B0  3021220 Preemptive  00007F53D9420AE0:00007F53D94221F0 0000000001AF3B70 2     Ukn (Threadpool Worker)
XXXX   23    0 0000000001DE9670  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   37    0 00007F53C0027910  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   27    0 00007F53B002BDE0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   30    0 00007F53B002DF70  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   25    0 00007F53C001A860  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   20    0 00007F53F40171F0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   21    0 00007F53CC06BEC0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   31    0 00007F53F40025A0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   44    0 00007F546C00EA00  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   36    0 00007F53CC000C10  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   22    0 00007F547006C8D0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   28    0 00007F547006FFA0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   47    0 00007F547006E6A0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   33    0 00007F53B4011D20  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   32    0 00007F53BC0253E0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   34    0 00007F53BC0297E0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   35    0 00007F53B4013D40  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   42    0 00007F53B80142C0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   45    0 00007F53CC0638D0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   46    0 00007F53B8014F70  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   29    0 00007F53B8015E00  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   19    0 00007F53F4001840  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX    4    0 00007F53F4015250  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX    5    0 00007F53F40162C0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   38    0 00007F546C000D50  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   39    0 00007F5468006B40  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   40    0 00007F53F4018980  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
  23   41  ae1 00007F53F4018F70  1021220 Preemptive  00007F53D97D9DA0:00007F53D97DA670 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   48    0 00007F53B40151F0  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
  25   49  b98 00007F53B4016F10  1021220 Preemptive  00007F53D97DE6B8:00007F53D97E0670 0000000001AF3B70 0     Ukn (Threadpool Worker)
XXXX   50    0 00007F53B4030480  1031820 Preemptive  0000000000000000:0000000000000000 0000000001AF3B70 0     Ukn (Threadpool Worker)
> syncblk
Index         SyncBlock MonitorHeld Recursion Owning Thread Info          SyncBlock Owner
   11 00007F53F8004E38            3         2 00007F53B00170B0 33b  22   00007f53d8286a58 System.Data.SqlClient.SNI.SNIMarsHandle
  105 00007F53F8004DA8            3         2 0000000002886100 320  21   00007f53d827a410 System.Data.SqlClient.SNI.SNIMarsConnection
  109 00007F546C012308            1         1 00007F53B00170B0 33b  22   00007f53d8282928 System.Data.SqlClient.SNI.TdsParserStateObjectManaged
-----------------------------
Total           180
Free            0

List stack of thread 22

> setthread 22
> clrstack
OS Thread Id: 0x33b (22)
        Child SP               IP Call Site
00007F53AA7F1E40 00007f547add69f5 [GCFrame: 00007f53aa7f1e40]
00007F53AA7F1F30 00007f547add69f5 [GCFrame: 00007f53aa7f1f30]
00007F53AA7F1F90 00007f547add69f5 [HelperMethodFrame_1OBJ: 00007f53aa7f1f90] System.Threading.Monitor.ReliableEnter(System.Object, Boolean ByRef)
00007F53AA7F20E0 00007F5403A8FECE System.Data.SqlClient.SNI.SNIMarsConnection.SendAsync(System.Data.SqlClient.SNI.SNIPacket, System.Data.SqlClient.SNI.SNIAsyncCallback) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SNI/SNIMarsConnection.cs @ 99]
00007F53AA7F2120 00007F5403A81976 System.Data.SqlClient.SNI.SNIMarsHandle.InternalSendAsync(System.Data.SqlClient.SNI.SNIPacket, System.Data.SqlClient.SNI.SNIAsyncCallback) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SNI/SNIMarsHandle.cs @ 172]
00007F53AA7F2170 00007F5403A0CDE6 System.Data.SqlClient.SNI.SNIMarsHandle.SendPendingPackets() [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SNI/SNIMarsHandle.cs @ 193]
00007F53AA7F21A0 00007F5403A82782 System.Data.SqlClient.SNI.SNIMarsHandle.SendAsync(System.Data.SqlClient.SNI.SNIPacket, Boolean, System.Data.SqlClient.SNI.SNIAsyncCallback) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SNI/SNIMarsHandle.cs @ 228]
00007F53AA7F21F0 00007F5403A8FBE2 System.Data.SqlClient.SNI.SNIProxy.WritePacket(System.Data.SqlClient.SNI.SNIHandle, System.Data.SqlClient.SNI.SNIPacket, Boolean) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SNI/SNIProxy.cs @ 234]
00007F53AA7F2210 00007F5403A8F913 System.Data.SqlClient.TdsParserStateObject.SNIWritePacket(System.Data.SqlClient.PacketHandle, UInt32 ByRef, Boolean, Boolean) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/TdsParserStateObject.cs @ 3336]
00007F53AA7F22B0 00007F5403A8F385 System.Data.SqlClient.TdsParserStateObject.WriteSni(Boolean) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/TdsParserStateObject.cs @ 3473]
00007F53AA7F2320 00007F5403A8EA48 System.Data.SqlClient.TdsParserStateObject.WritePacket(Byte, Boolean) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/TdsParserStateObject.cs @ 3114]
00007F53AA7F2360 00007F5403A82579 System.Data.SqlClient.TdsParserStateObject.ExecuteFlush() [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/TdsParserStateObject.cs @ 970]
00007F53AA7F23A0 00007F5403A08F1B System.Data.SqlClient.TdsParser.TdsExecuteRPC(System.Data.SqlClient._SqlRPC[], Int32, Boolean, System.Data.Sql.SqlNotificationRequest, System.Data.SqlClient.TdsParserStateObject, Boolean, Boolean, System.Threading.Tasks.TaskCompletionSource`1<System.Object>, Int32, Int32) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/TdsParser.cs @ 7251]
00007F53AA7F2480 00007F5403A8E328 System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, Boolean, Int32, System.Threading.Tasks.Task ByRef, Boolean, System.Data.SqlClient.SqlDataReader) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SqlCommand.cs @ 2710]
00007F53AA7F2540 00007F5403A97F68 System.Data.SqlClient.SqlCommand.BeginExecuteReader(System.AsyncCallback, System.Object, System.Data.CommandBehavior) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SqlCommand.cs @ 1598]
00007F53AA7F25B0 00007F5403A97B8F System.Threading.Tasks.TaskFactory`1[[System.__Canon, System.Private.CoreLib]].FromAsyncImpl[[System.Data.CommandBehavior, System.Data.Common]](System.Func`4<System.Data.CommandBehavior,System.AsyncCallback,System.Object,System.IAsyncResult>, System.Func`2<System.IAsyncResult,System.__Canon>, System.Action`1<System.IAsyncResult>, System.Data.CommandBehavior, System.Object, System.Threading.Tasks.TaskCreationOptions)
00007F53AA7F2620 00007F5403A8B318 System.Data.SqlClient.SqlCommand.ExecuteReaderAsync(System.Data.CommandBehavior, System.Threading.CancellationToken) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SqlCommand.cs @ 1771]
00007F53AA7F26B0 00007F5403A975DE System.Data.SqlClient.SqlCommand.ExecuteDbDataReaderAsync(System.Data.CommandBehavior, System.Threading.CancellationToken)
00007F53AA7F26D0 00007F54039F14CB Dapper.SqlMapper+<QueryAsync>d__33`1[[System.__Canon, System.Private.CoreLib]].MoveNext() [C:\projects\dapper\Dapper\SqlMapper.Async.cs @ 419]
00007F53AA7F2890 00007F5403A8AF4F System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Dapper.SqlMapper+<QueryAsync>d__33`1[[System.__Canon, System.Private.CoreLib]], Dapper]](<QueryAsync>d__33`1<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 1036]
00007F53AA7F28E0 00007F5403A8AE15 Dapper.SqlMapper.QueryAsync[[System.__Canon, System.Private.CoreLib]](System.Data.IDbConnection, System.Type, Dapper.CommandDefinition)
00007F53AA7F29E0 00007F5403A96D78 Dapper.SqlMapper.QueryAsync[[System.__Canon, System.Private.CoreLib]](System.Data.IDbConnection, System.String, System.Object, System.Data.IDbTransaction, System.Nullable`1<Int32>, System.Nullable`1<System.Data.CommandType>)
00007F53AA7F2AB0 00007F5403A64C22 xPriceHistory.Core.Repository.Impl.DatabaseRepositoryBase+<>c__DisplayClass15_1`1+<<FetchAsync>b__1>d[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F53AA7F2B40 00007F5403A7E7CF System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[xPriceHistory.Core.Repository.Impl.DatabaseRepositoryBase+<>c__DisplayClass15_1`1+<<FetchAsync>b__1>d[[System.__Canon, System.Private.CoreLib]], xPriceHistory.Core.Repository.Impl]](<<FetchAsync>b__1>d<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 1036]
00007F53AA7F2B90 00007F5403A7E6FF xPriceHistory.Core.Repository.Impl.DatabaseRepositoryBase+<>c__DisplayClass15_1`1[[System.__Canon, System.Private.CoreLib]].<FetchAsync>b__1()
00007F53AA7F2C00 00007F5403A4ACE7 Survey.HIS1383.MyAWSXRayRecorderService+<TraceExecuteAsync>d__3`1[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F53AA7F2C60 00007F5403A5A4CF System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Survey.HIS1383.MyAWSXRayRecorderService+<TraceExecuteAsync>d__3`1[[System.__Canon, System.Private.CoreLib]], Survey.HIS1383]](<TraceExecuteAsync>d__3`1<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 1036]
00007F53AA7F2CB0 00007F5403A5A394 Survey.HIS1383.MyAWSXRayRecorderService.TraceExecuteAsync[[System.__Canon, System.Private.CoreLib]](System.String, System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>, System.String, System.Object, System.Func`2<System.__Canon,System.Object>)
00007F53AA7F2D20 00007F540358B6D9 ILStubClass.IL_STUB_InstantiatingStub
00007F53AA7F2D40 00007F5403A68B51 xPriceHistory.Core.Repository.Impl.DatabaseRepositoryBase+<>c__DisplayClass15_0`1+<<FetchAsync>b__0>d[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F53AA7F2DB0 00007F5403A80D0F System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[xPriceHistory.Core.Repository.Impl.DatabaseRepositoryBase+<>c__DisplayClass15_0`1+<<FetchAsync>b__0>d[[System.__Canon, System.Private.CoreLib]], xPriceHistory.Core.Repository.Impl]](<<FetchAsync>b__0>d<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 1036]
00007F53AA7F2E00 00007F5403A7DF0F xPriceHistory.Core.Repository.Impl.DatabaseRepositoryBase+<>c__DisplayClass15_0`1[[System.__Canon, System.Private.CoreLib]].<FetchAsync>b__0()
00007F53AA7F2E70 00007F5403A7F970 Polly.Policy+<>c__DisplayClass188_0`1[[System.__Canon, System.Private.CoreLib]].<ExecuteAsync>b__0(Polly.Context, System.Threading.CancellationToken)
00007F53AA7F2E80 00007F5403A68FB5 Polly.Policy+<>c__DisplayClass208_0`1+<<ExecuteAsyncInternal>b__0>d[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F53AA7F2F00 00007F5403A7F88F System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Polly.Policy+<>c__DisplayClass208_0`1+<<ExecuteAsyncInternal>b__0>d[[System.__Canon, System.Private.CoreLib]], Polly]](<<ExecuteAsyncInternal>b__0>d<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 1036]
00007F53AA7F2F50 00007F5403A7F770 Polly.Policy+<>c__DisplayClass208_0`1[[System.__Canon, System.Private.CoreLib]].<ExecuteAsyncInternal>b__0(Polly.Context, System.Threading.CancellationToken)
00007F53AA7F2FC0 00007F5403A691EF Polly.RetrySyntaxAsync+<>c__DisplayClass21_1+<<WaitAndRetryAsync>b__1>d.MoveNext()
00007F53AA7F3050 00007F5403A7F634 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Polly.RetrySyntaxAsync+<>c__DisplayClass21_1+<<WaitAndRetryAsync>b__1>d, Polly]](<<WaitAndRetryAsync>b__1>d ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 1036]
00007F53AA7F3090 00007F5403A7DE54 Polly.RetrySyntaxAsync+<>c__DisplayClass21_1.<WaitAndRetryAsync>b__1(Polly.Context, System.Threading.CancellationToken)
00007F53AA7F30E0 00007F54035818CF Polly.Retry.RetryEngine+<ImplementationAsync>d__1`1[[Polly.Utilities.EmptyStruct, Polly]].MoveNext()
00007F53AA7F31F0 00007F5403A80B74 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Polly.Retry.RetryEngine+<ImplementationAsync>d__1`1[[Polly.Utilities.EmptyStruct, Polly]], Polly]](<ImplementationAsync>d__1`1<Polly.Utilities.EmptyStruct> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 1036]
00007F53AA7F3230 00007F5403A7DC48 Polly.Retry.RetryEngine.ImplementationAsync[[Polly.Utilities.EmptyStruct, Polly]](System.Func`3<Polly.Context,System.Threading.CancellationToken,System.Threading.Tasks.Task`1<Polly.Utilities.EmptyStruct>>, Polly.Context, System.Threading.CancellationToken, System.Collections.Generic.IEnumerable`1<Polly.ExceptionPredicate>, System.Collections.Generic.IEnumerable`1<Polly.ResultPredicate`1<Polly.Utilities.EmptyStruct>>, System.Func`1<Polly.Retry.IRetryPolicyState`1<Polly.Utilities.EmptyStruct>>, Boolean)
00007F53AA7F32E0 00007F5403A80ACC Polly.RetrySyntaxAsync+<>c__DisplayClass21_0.<WaitAndRetryAsync>b__0(System.Func`3<Polly.Context,System.Threading.CancellationToken,System.Threading.Tasks.Task>, Polly.Context, System.Threading.CancellationToken, Boolean)
00007F53AA7F3340 00007F5403A69D74 Polly.Policy+<ExecuteAsyncInternal>d__208`1[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F53AA7F33C0 00007F5403A808DF System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Polly.Policy+<ExecuteAsyncInternal>d__208`1[[System.__Canon, System.Private.CoreLib]], Polly]](<ExecuteAsyncInternal>d__208`1<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 1036]
00007F53AA7F3410 00007F5403A807A6 Polly.Policy.ExecuteAsyncInternal[[System.__Canon, System.Private.CoreLib]](System.Func`3<Polly.Context,System.Threading.CancellationToken,System.Threading.Tasks.Task`1<System.__Canon>>, Polly.Context, System.Threading.CancellationToken, Boolean)
00007F53AA7F34A0 00007F540357FAED ILStubClass.IL_STUB_InstantiatingStub
00007F53AA7F34B0 00007F5403A805B5 Polly.Policy.ExecuteAsync[[System.__Canon, System.Private.CoreLib]](System.Func`3<Polly.Context,System.Threading.CancellationToken,System.Threading.Tasks.Task`1<System.__Canon>>, Polly.Context, System.Threading.CancellationToken, Boolean)
00007F53AA7F3500 00007F5403A804BA Polly.Policy.ExecuteAsync[[System.__Canon, System.Private.CoreLib]](System.Func`1<System.Threading.Tasks.Task`1<System.__Canon>>)
00007F53AA7F3540 00007F5403A56CE7 xPriceHistory.Core.Repository.Impl.DatabaseRepositoryBase+<FetchAsync>d__15`1[[System.__Canon, System.Private.CoreLib]].MoveNext()
00007F53AA7F3590 00007F5403A81EAF System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[xPriceHistory.Core.Repository.Impl.DatabaseRepositoryBase+<FetchAsync>d__15`1[[System.__Canon, System.Private.CoreLib]], xPriceHistory.Core.Repository.Impl]](<FetchAsync>d__15`1<System.__Canon> ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 1036]
00007F53AA7F35E0 00007F5403A8038C xPriceHistory.Core.Repository.Impl.DatabaseRepositoryBase.FetchAsync[[System.__Canon, System.Private.CoreLib]](System.String, System.String, System.Object, System.Data.CommandType)
00007F53AA7F3660 00007F5403A56FC9 xPriceHistory.Core.Repository.Impl.OneDayTicksDataBaseRepositoryService+<GetAsync>d__15.MoveNext()
00007F53AA7F36D0 00007F5403A964F4 System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[xPriceHistory.Core.Repository.Impl.OneDayTicksDataBaseRepositoryService+<GetAsync>d__15, xPriceHistory.Core.Repository.Impl]](<GetAsync>d__15 ByRef) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 1036]
00007F53AA7F3710 00007F5403A81033 xPriceHistory.Core.Repository.Impl.OneDayTicksDataBaseRepositoryService.GetAsync(System.String, System.DateTime, System.DateTime)
00007F53AA7F3770 00007F5402C83A84 Survey.HIS1383.Program+<CheckAsync>d__4.MoveNext() [D:\sourcecode\Xignite\Services\Price\xPriceHistory\Survey.HIS1383\Program.cs @ 204]
00007F53AA7F3890 00007F5403A38AA1 System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 315]
00007F53AA7F38D0 00007F5403A4B7D5 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[Survey.HIS1383.Program+<CheckAsync>d__4, Survey.HIS1383]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 654]
00007F53AA7F3900 00007F5403A3CEB9 System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/System.Private.CoreLib/shared/System/Threading/ThreadPool.cs @ 699]
00007F53AA7F3D10 00007f54795e9d1f [DebuggerU2MCatchHandlerFrame: 00007f53aa7f3d10]
>

List stack of thread 21

> setthread 21
> clrstack
OS Thread Id: 0x320 (21)
        Child SP               IP Call Site
00007F53AE7FB2B0 00007f547add69f5 [GCFrame: 00007f53ae7fb2b0]
00007F53AE7FB3A0 00007f547add69f5 [GCFrame: 00007f53ae7fb3a0]
00007F53AE7FB400 00007f547add69f5 [HelperMethodFrame_1OBJ: 00007f53ae7fb400] System.Threading.Monitor.ReliableEnter(System.Object, Boolean ByRef)
00007F53AE7FB550 00007F5403A8517E System.Data.SqlClient.SNI.SNIMarsHandle.HandleReceiveComplete(System.Data.SqlClient.SNI.SNIPacket, System.Data.SqlClient.SNI.SNISMUXHeader) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SNI/SNIMarsHandle.cs @ 351]
00007F53AE7FB5D0 00007F5403A0EDC4 System.Data.SqlClient.SNI.SNIMarsConnection.HandleReceiveComplete(System.Data.SqlClient.SNI.SNIPacket, UInt32) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SNI/SNIMarsConnection.cs @ 259]
00007F53AE7FB630 00007F5403A9A051 System.Data.SqlClient.SNI.SNIPacket.ReadFromStreamAsync(System.IO.Stream, System.Data.SqlClient.SNI.SNIAsyncCallback) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SNI/SNIPacket.NetCoreApp.cs @ 65]
00007F53AE7FB6B0 00007F5403A99D9B System.Data.SqlClient.SNI.SNITCPHandle.ReceiveAsync(System.Data.SqlClient.SNI.SNIPacket ByRef) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SNI/SNITcpHandle.cs @ 554]
00007F53AE7FB6F0 00007F5403A99B57 System.Data.SqlClient.SNI.SNIMarsConnection.ReceiveAsync(System.Data.SqlClient.SNI.SNIPacket ByRef) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SNI/SNIMarsConnection.cs @ 117]
00007F53AE7FB730 00007F5403A0EE12 System.Data.SqlClient.SNI.SNIMarsConnection.HandleReceiveComplete(System.Data.SqlClient.SNI.SNIPacket, UInt32) [/_/src/System.Data.SqlClient/src/System/Data/SqlClient/SNI/SNIMarsConnection.cs @ 275]
00007F53AE7FB790 00007F5403A6EDA1 System.Data.SqlClient.SNI.SNIPacket+<<ReadFromStreamAsync>g__ReadFromStreamAsync|27_0>d.MoveNext()
00007F53AE7FB840 00007F5403A2F75D System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 201]
00007F53AE7FB890 00007F5403A98D7A System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib],[System.Data.SqlClient.SNI.SNIPacket+<<ReadFromStreamAsync>g__ReadFromStreamAsync|27_0>d, System.Data.SqlClient]].MoveNext(System.Threading.Thread) [/_/src/System.Private.CoreLib/shared/System/Runtime/CompilerServices/AsyncMethodBuilder.cs @ 654]
00007F53AE7FB8C0 00007F5403A93028 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.InvokeContinuation(System.Action`1<System.Object>, System.Object, Boolean, Boolean) [/_/src/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs @ 1019]
00007F53AE7FB900 00007F5403A3CED6 System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/System.Private.CoreLib/shared/System/Threading/ThreadPool.cs @ 699]
00007F53AE7FBD10 00007f54795e9d1f [DebuggerU2MCatchHandlerFrame: 00007f53ae7fbd10]
>
@danmoseley danmoseley transferred this issue from dotnet/corefx Jan 14, 2020
@Wraith2
Copy link
Contributor

Wraith2 commented Jan 15, 2020 •
edited
Loading

cc @cheenamalhotra @saurabh500

I had a peek through this. I think what is happening is

thread 21: SNIMarsConnection.HandleReceiveComplete calls SNITCPHandle.ReceiveAsync which locks itself which calls back into SNIMarsConnection.HandleReceiveComplete later on. The SNIMarsConnection.HandleReceiveComplete is careful to not call the lower handle while locking itself but re-entrance breaks this and causes both locks to be taken in the "down" order

thread 22: SNIMarsHandle.InternalSendAsync locks self and calls into SNIMarsConnection.SendAsync which then locks itself. Both locks taken in "up" order.

Classic two locks in a different order deadlock situation.

@cheenamalhotra cheenamalhotra transferred this issue from dotnet/runtime Jan 15, 2020
@cheenamalhotra
Copy link
Member

@czd890

Do you have a repro app we can try? Also if you can test the same with Microsoft.Data.SqlClient 1.1.0, let us know if issue exists.

@Wraith2 looks like that from the logs, it would be good if we can know the impacted APIs and if the issue still exists with Microsoft.Data.SqlClient.

@czd890
Copy link
Author

czd890 commented Jan 15, 2020

@cheenamalhotra
The deadlocked dump comes from an application on our product, and difficult to reproduce this problem in a test environment.

@Wraith2
Copy link
Contributor

Wraith2 commented Jan 15, 2020

Those parts are unchanged in the current versions. The reproduction will require some very precise timing on sends and receives on the same connection.

@cheenamalhotra
Copy link
Member

Hi @czd890

I noticed your application uses S.D.S driver v4.7.0 (File Version Number: 4.700.19.56404), which was buggy with a similar deadlock issue as reported in #262 .

Could you try downgrading to v4.6.1 and verify if that works fine in a case if this is an outcome of the same issue?

@cheenamalhotra
Copy link
Member

Hi @czd890

Did you get a chance to investigate the problem with v4.6.1?

@cheenamalhotra
Copy link
Member

Closing as inactive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@czd890 @Wraith2 @cheenamalhotra