File upload failed even after retry

[jeffschwMSFT]

Build

https://dev.azure.com/dnceng/internal/_build/results?buildId=2365835

Error:

Fail to upload '/mnt/vss/_work/1/a/IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Ref.9.0.0-preview.2.24080.4.symbols.nupkg' due to 'Bad Request'.
Microsoft.VisualStudio.Services.WebApi.VssServiceResponseException: Bad Request
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Byte[] contentId, Int64 fileLength, Boolean isGzipped, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Int32 chunkRetryTimes, Boolean uploadFirstChunk, Object userState)
   at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Boolean uploadFirstChunk, Object userState, Boolean compressStream)
   at Microsoft.VisualStudio.Services.Agent.Worker.Build.FileContainerServer.UploadAsync(IAsyncCommandContext context, Int32 uploaderId, CancellationToken token) in /home/vsts/work/1/s/src/Agent.Worker/Build/FileContainerServer.cs:line 250
Detail upload trace for file that fail to upload: IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Ref.9.0.0-preview.2.24080.4.symbols.nupkg

Chunk '2' attempt '1' of file 'IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Ref.9.0.0-preview.2.24080.4.symbols.nupkg' fail to send request to server. Error: Microsoft.VisualStudio.Services.Common.VssServiceException: TF203009: The size of the file that you uploaded does not match the size that the client computer provided.
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.HandleResponseAsync(HttpResponseMessage response, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.UploadFileAsync(Int64 containerId, String itemPath, Stream fileStream, Byte[] contentId, Int64 fileLength, Boolean isGzipped, Guid scopeIdentifier, CancellationToken cancellationToken, Int32 chunkSize, Int32 chunkRetryTimes, Boolean uploadFirstChunk, Object userState).

##[error]File upload failed even after retry.

Build leg reported

No response

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "##[error]File upload failed even after retry",
    "BuildRetry": false,
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Report

Build	Definition	Step Name	Console log	Pull Request
1335577	dotnet/xharness	Build and run tests	Log	dotnet/xharness#1563
1334980	dotnet/maui	publish artifacts	Log
1334245	dotnet/maui	publish artifacts	Log	dotnet/maui#34474
1333629	dotnet/maui	publish artifacts	Log	dotnet/maui#34016
1333631	dotnet/maui	publish artifacts	Log	dotnet/maui#34424
1333738	dotnet/xharness	Build and run tests	Log	dotnet/xharness#1560
1332048	dotnet/xharness	Build and run tests	Log
2924544	dotnet-maui	publish artifacts	Log
1330262	dotnet/maui	publish artifacts	Log
1330272	dotnet/maui	publish artifacts	Log
2924167	dotnet-runtime	🔒 Publish intermediate artifacts	Log
1330229	dotnet/arcade	Unix Build / Publish	Log	dotnet/arcade#16578
1330086	dotnet/xharness	Build and run tests	Log	dotnet/xharness#1544
1329795	dotnet/maui	publish artifacts	Log	dotnet/maui#34331
1329412	dotnet/maui	publish artifacts	Log	dotnet/maui#28151
1328146	dotnet/maui	publish artifacts	Log
1328119	dotnet/maui	publish artifacts	Log
2923025	dotnet-maui	publish artifacts	Log
1328044	dotnet/maui	publish artifacts	Log	dotnet/maui#34382
1327314	dotnet/maui	publish artifacts	Log
1327279	dotnet/maui	publish artifacts	Log
1327514	dotnet/xharness	Build and run tests	Log
1326542	dotnet/maui	publish artifacts	Log	dotnet/maui#32333
1326214	dotnet/xharness	Build and run tests	Log	dotnet/xharness#1551
1325941	dotnet/xharness	Build and run tests	Log	dotnet/xharness#1532
1325883	dotnet/xharness	Build and run tests	Log	dotnet/xharness#1530
1325704	dotnet/maui	publish artifacts	Log
1325132	dotnet/maui	publish artifacts	Log
1325061	dotnet/maui	publish artifacts	Log	dotnet/maui#27093
1322901	dotnet/xharness	Build and run tests	Log	dotnet/xharness#1550
1323556	dotnet/maui	publish artifacts	Log
1322218	dotnet/maui	publish artifacts	Log
1321878	dotnet/maui	publish artifacts	Log
1321856	dotnet/maui	publish artifacts	Log
1321648	dotnet/maui	publish artifacts	Log
1321818	dotnet/maui	publish artifacts	Log
1321157	dotnet/maui	publish artifacts	Log
1321136	dotnet/maui	publish artifacts	Log
1321024	dotnet/maui	publish artifacts	Log
1321014	dotnet/maui	publish artifacts	Log
1320362	dotnet/maui	publish artifacts	Log
1320317	dotnet/maui	publish artifacts	Log	dotnet/maui#34311
2918236	dotnet-runtime	🔒 Publish intermediate artifacts	Log
1319050	dotnet/maui	publish artifacts	Log
1319128	dotnet/maui	publish artifacts	Log
1315785	dotnet/maui	publish artifacts	Log
2916113	dotnet-runtime	Publish Logs	Log	#58446
1312677	dotnet/xharness	Build and run tests	Log	dotnet/xharness#1545
1311737	dotnet/maui	publish artifacts	Log
1311147	dotnet/maui	publish artifacts	Log
1310815	dotnet/maui	publish artifacts	Log
1310882	dotnet/maui	publish artifacts	Log	dotnet/maui#33991
1309268	dotnet/maui	publish artifacts	Log
1308173	dotnet/maui	publish artifacts	Log
1307605	dotnet/maui	publish artifacts	Log
1307268	dotnet/maui	publish artifacts	Log	dotnet/maui#27847
1307246	dotnet/maui	publish artifacts	Log
1307198	dotnet/maui	publish artifacts	Log
1306774	dotnet/maui	publish artifacts	Log
1306534	dotnet/maui	publish artifacts	Log
2911021	dotnet-maui	publish artifacts	Log
1306187	dotnet/maui	publish artifacts	Log
1305716	dotnet/maui	publish artifacts	Log	dotnet/maui#34023
1305547	dotnet/maui	publish artifacts	Log
1305055	dotnet/xharness	Build and run tests	Log	dotnet/xharness#1532
1304085	dotnet/maui	publish artifacts	Log
1303563	dotnet/maui	publish artifacts	Log	dotnet/maui#33246
1300870	dotnet/maui	publish artifacts	Log
1298629	dotnet/maui	publish artifacts	Log	dotnet/maui#34085
1295444	dotnet/maui	publish artifacts	Log	dotnet/maui#30133

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	27	70

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2365835
Error message validated: [##[error]File upload failed even after retry]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 2:03:27 AM UTC

[lewing]

This is often an from multiple lanes attempting to upload the same artifact

[markwilkie]

@lewing is right - there's usually not a whole lot dnceng can do to help.

[missymessa]

@jeffschwMSFT @lewing is this still an issue?

[lewing]

@jeffschwMSFT @lewing is this still an issue?

It still happens but it is typically a (mostly harmless) repository issue. Perhaps this issue should be updated to explain the cause and suggest fixes. It is clearly a recurring problem.