Convert consolidated pipeline to a YAML based pipeline and enable nightly builds

[aishwaryabh]

Issue describing the changes in this PR

resolves #4270 #4271
This PR converts the consolidated pipeline which was initially based off of azure classic pipelines UX to a yaml based pipeline.

I have updated the ArtifactAssembler within core tools to be able to process each artifact type one at a time. For example, if I were to pass in Azure.Functions.Cli.win-x64 when running the artifact assembler, the artifact assembler would only assemble artifacts for that win-x64. This enables us to use parallelization when running the consolidated pipeline, which significantly improves the run time overall.

The consolidated pipeline also publishes the artifacts to the pipeline directly, rather than publishing to a storage account. Please look at this run for an example of what this would look like.

This pipeline also runs significantly faster than the azure classic based pipeline, as this pipeline takes on average 35-37 min, whereas previous runs would easily take over an hour. This runtime will increase even more once the E2E tests are updated to run in a more efficient and parallel manner outside of the consolidated pipeline.

Finally, nightly builds are enabled for the consolidated artifact pipeline to run every night, after the in-proc and main artifacts have been created from their nightly builds. This PR also adds a tag called nightly-build to those builds for the in-proc and main branch to distinguish scheduled builds from manually triggered ones. That way the consolidated pipeline knows which builds to pick for the nightly builds.

Pull request checklist

• My changes do not require documentation changes
  • Otherwise: Documentation issue linked to PR
• My changes do not need to be backported to a previous version
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• I have added all required tests (Unit tests, E2E tests)

[liliankasem]

Last thing, have you evaluated the artifacts/drops that are built? Did you try running those cli builds to test they work as expected, and are also signed?

[aishwaryabh]

Last thing, have you evaluated the artifacts/drops that are built? Did you try running those cli builds to test they work as expected, and are also signed?

Yes! I downloaded the artifacts to cross check that they are produced correctly and ensured that the MSI files that are produced for win-x64 and win-x86 are signed. The rest of the artifacts are signed since they are just taken from the original core-tools.official build pipeline.

As for testing, we have E2E tests that run here to make sure that everything still works for win-x64 and win-x86 for the CLI. We also have tests for core tools via VS here.

Also this pipeline won't be used immediately yet, since we would have to update the release pipeline to consume the artifacts produced from this pipeline. This is just the first step to getting there :)