Add MNTR aggregated failure logs to artifacts #4100

IgorFedchenko · 2019-12-16T16:25:02Z

Related to #4059

The goal of this PR is to add separate artifact containing only aggregated timeline logs for only failed specs. So that when MNTR tests are failing on CI, we could just download single small zip file and get all we need to know.

In draft state since need to debug Azure Pipelines templates update for artifacts publishing.

…' into add-mntr-aggregated-to-artifacts

Aaronontheweb · 2019-12-16T19:06:31Z

@IgorFedchenko well look at that... finally looks like Azure DevOps is running properly.

IgorFedchenko · 2019-12-16T19:18:54Z

@IgorFedchenko well look at that... finally looks like Azure DevOps is running properly.

@Aaronontheweb Indeed - so you did not change anything?

IgorFedchenko · 2019-12-16T19:21:25Z

@Aaronontheweb Also, please check out artifacts for failed MNTR here. Does it look good, or any ideas about how to make it even better?
Artifacts with failure logs are containing "FAILED" suffix.

Aaronontheweb · 2019-12-16T19:35:00Z

@Aaronontheweb Indeed - so you did not change anything?

Nope, I did not - just filed a bug report and a support ticket with Azure DevOps.

IgorFedchenko · 2019-12-17T18:32:01Z

@Aaronontheweb Now MNTR output per each node and aggregated per spec can be found on Attachments tab in test view:

Do you have any ideas to make it even better, or this looks good in current state?

IgorFedchenko · 2019-12-17T18:33:30Z

build-system/pr-validation.yaml

      artifactName: 'net_core_tests_linux-$(Build.BuildId)'

-  - template: azure-pipeline.template.yaml
+  - template: azure-pipeline.mntr-template.yaml


Have to use different template here to add build artifacts that are not used for non-mntr tests. There is some conditional execution support, but it does not seem to be flexible enough

Makes sense

IgorFedchenko · 2019-12-17T18:34:05Z

build-system/pr-validation.yaml

      outputDirectory: 'TestResults'
      artifactName: 'net_core_mntr_windows-$(Build.BuildId)'
+      mntrFailuresDir: 'TestResults\\multinode\\FAILED_SPECS_LOGS'
+      mntrFailuresArtifactName: 'net_core_mntr_FAILED_windows-$(Build.BuildId)'


So here is where all failed spec aggregated logs can be downloaded at once

Aaronontheweb

Couple of changes (un-breaking the test mostly) needed.

I also noticed this issue on a different PR but it's showing up here as well:

Looks like it might be a race condition - this bug isn't related to this PR as it's already happening with the current code that we have in-place for reporting MNTR results.

Aaronontheweb · 2019-12-18T08:17:17Z

build-system/pr-validation.yaml

      artifactName: 'net_core_tests_linux-$(Build.BuildId)'

-  - template: azure-pipeline.template.yaml
+  - template: azure-pipeline.mntr-template.yaml


Makes sense

...contrib/cluster/Akka.Cluster.Sharding.Tests.MultiNode/ClusterShardingGracefulShutdownSpec.cs

src/core/Akka.MultiNodeTestRunner/Program.cs

Aaronontheweb · 2019-12-18T08:45:52Z

Since I didn't mention this anywhere: the MNTR output looks great. Much easier to be able to access the node-specific logs from the Azure DevOps UI - that should help improve contributor productivity. Nicely done @IgorFedchenko

Aaronontheweb · 2019-12-18T16:40:02Z

Hah, one small issue before I approve this - the build step fails if there are no failed MNTR specs

Starting: Copy MNTR Failure Logs
==============================================================================
Task         : Copy files
Description  : Copy files from a source folder to a target folder using patterns matching file paths (not folder paths)
Version      : 2.158.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/copy-files
==============================================================================
##[error]Unhandled: Not found SourceFolder: D:\a\1\s\TestResults\multinode\FAILED_SPECS_LOGS
Finishing: Copy MNTR Failure Logs

IgorFedchenko · 2019-12-20T10:37:28Z

Hah, one small issue before I approve this - the build step fails if there are no failed MNTR specs

Yup, seems like CopyFiles Azure Pipelines task can not handle case when source folder does not exist at all... Updated the pipeline.
Also, I could not find simple way to make this FAILED_SPECS_LOGS artifact not to be generated when no failed specs are found. But most likely it will always be generated - will see once CI will finish validation.

IgorFedchenko · 2019-12-20T10:39:33Z

All right, now did not failed on that step.

Aaronontheweb · 2019-12-20T15:05:34Z

Thanks @IgorFedchenko! Looking forward to using this in the rest of our build pipeline.

IgorFedchenko added 8 commits December 16, 2019 19:22

Added MNTR failure

ae0baa9

Added failures aggregation and publishing

0f8c8d5

Merge branch 'dev' into add-mntr-aggregated-to-artifacts

d80293f

Moved MNTR tasks into separate template

47de8ed

Merge remote-tracking branch 'origin/add-mntr-aggregated-to-artifacts…

be6e55f

…' into add-mntr-aggregated-to-artifacts

Updated parameter names

b476eba

Fixed indent

022f36d

Fixed failed specs folder path in config

240c31b

Aaronontheweb added multi node spec logging labels Dec 16, 2019

Added test logs to .trx test output

84f54f8

IgorFedchenko marked this pull request as ready for review December 17, 2019 18:30

IgorFedchenko commented Dec 17, 2019

View reviewed changes

Aaronontheweb requested changes Dec 18, 2019

View reviewed changes

Remove first node failure exception

eadcb1d

IgorFedchenko mentioned this pull request Dec 18, 2019

Document MNTR failed logs directory generation #4107

Closed

Merge branch 'dev' into add-mntr-aggregated-to-artifacts

dc9902a

IgorFedchenko mentioned this pull request Dec 19, 2019

MNTR CI test results generation issues #4109

Closed

IgorFedchenko added 2 commits December 20, 2019 13:33

CopyFiles AzurePipeline task fix

d564a97

Merge branch 'dev' into add-mntr-aggregated-to-artifacts

961b71f

Aaronontheweb approved these changes Dec 20, 2019

View reviewed changes

Aaronontheweb merged commit 63472ea into akkadotnet:dev Dec 20, 2019

IgorFedchenko deleted the add-mntr-aggregated-to-artifacts branch December 20, 2019 18:52

Add MNTR aggregated failure logs to artifacts #4100

Add MNTR aggregated failure logs to artifacts #4100

Uh oh!

Conversation

IgorFedchenko commented Dec 16, 2019

Uh oh!

Aaronontheweb commented Dec 16, 2019

Uh oh!

IgorFedchenko commented Dec 16, 2019

Uh oh!

IgorFedchenko commented Dec 16, 2019

Uh oh!

Aaronontheweb commented Dec 16, 2019

Uh oh!

IgorFedchenko commented Dec 17, 2019

Uh oh!

IgorFedchenko Dec 17, 2019

Choose a reason for hiding this comment

Uh oh!

Aaronontheweb Dec 18, 2019

Choose a reason for hiding this comment

Uh oh!

IgorFedchenko Dec 17, 2019

Choose a reason for hiding this comment

Uh oh!

Aaronontheweb left a comment

Choose a reason for hiding this comment

Uh oh!

Aaronontheweb Dec 18, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Aaronontheweb commented Dec 18, 2019

Uh oh!

Aaronontheweb commented Dec 18, 2019

Uh oh!

IgorFedchenko commented Dec 20, 2019

Uh oh!

IgorFedchenko commented Dec 20, 2019

Uh oh!

Aaronontheweb commented Dec 20, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants