feat: Add Azure EventHubs module #1183

WakaToa · 2024-05-22T12:04:27Z

What does this PR do?

Microsoft released an emulator for EventHubs yesterday, see Azure/azure-service-bus#223 (comment)

This PR included a module for the emulator.

Info

You need to provide an Azurite (emulator) instance/endpoint to start the EventHub emulator. This can be done using the EventHubsConfiguration.

You need to specify the following configuration file when starting the emulator.

{
  "UserConfig": {
    "NamespaceConfig": [
      {
        "Type": "EventHub",
        "Name": "emulatorNs1",
        "Entities": [
          {
            "Name": "eh1",
            "PartitionCount": "2",
            "ConsumerGroups": [
              {
                "Name": "cg1"
              }
            ]
          }
        ]
      }
    ], 
    "LoggingConfig": {
      "Type": "File"
    }
  }
}

It will be dynamically mapped and i have built a floating builder so that the user can set the configuration as easily as possible.
The namespace "emulatorNs1" is mandatory and it is the only one currently supported by the emulator.

var configurationBuilder = ConfigurationBuilder
    .Create()
    .WithEventHub(EventHubName, "2", new[] { EventHubConsumerGroupName });

var builder = new EventHubsBuilder()
    .WithNetwork(_network)
    .WithConfigurationBuilder(configurationBuilder)
    .WithAzuriteBlobEndpoint(AzuriteNetworkAlias)
    .WithAzuriteTableEndpoint(AzuriteNetworkAlias);

_eventHubsContainer = builder.Build();

await _eventHubsContainer.StartAsync();

netlify · 2024-05-22T12:04:42Z

✅ Deploy Preview for testcontainers-dotnet ready!

Name	Link
🔨 Latest commit	`b264e42`
🔍 Latest deploy log	https://app.netlify.com/sites/testcontainers-dotnet/deploys/664de1efb02bc800082444d3
😎 Deploy Preview	https://deploy-preview-1183--testcontainers-dotnet.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

eddumelendez · 2024-05-31T10:37:18Z

tests/Testcontainers.EventHubs.Tests/EventHubsContainerTest.cs

+    private readonly AzuriteContainer _azuriteContainer;
+
+    private EventHubsContainer _eventHubsContainer;
+
+    private readonly INetwork _network = new NetworkBuilder().WithName(NetworkName).Build();
+
+    private const string NetworkName = "eh-network";
+    private const string AzuriteNetworkAlias = "azurite";
+
+    private const string EventHubName = "testeventhub";
+    private const string EventHubConsumerGroupName = "testconsumergroup";
+
+    private EventHubsContainerTest()
+    {
+        _azuriteContainer = new AzuriteBuilder()
+            .WithNetwork(_network)
+            .WithNetworkAliases(AzuriteNetworkAlias)
+            .Build();
+    }


This is an idea still but due to EventHubs emulator will not work without Azurite then we can provide two ways to run EventHub Container implementation.

Embed Azurite, so, users don't worry about it.

Allow external Azurite in case users need a specific configuration

/cc @kiview WDYT?

Does a pattern exist already ?

recently, service bus was added with the same pattern. See https://github.com/testcontainers/testcontainers-dotnet/blob/develop/src/Testcontainers.ServiceBus/ServiceBusBuilder.cs#L114

I think it makes sense to align with the pattern Service Bus is using and then follow up with something that allows us to use an existing Azurite instance. Please also see this issue: #1323. This is something we need to figure out and support in general.

eddumelendez · 2024-06-25T21:25:40Z

Also, please add documentation for the module.

Saglodha · 2025-01-06T06:55:28Z

Hi @WakaToa / @eddumelendez / @kiview ,

I am from Azure Event Hubs Product team and was looking at this PR. I do realize that there were some plausible solutions offered above to get this integration setup.

Could you please let us know if we could extend any help from our side to integrate Event Hubs emulator with testcontainers framework? We'll be happy to provide any product related assistance here.

HofmeisterAn

Thanks for the PR. There are a few things we need to address before we can merge the PR.

HofmeisterAn · 2025-01-08T05:39:15Z

src/Testcontainers.EventHubs/EventHubsBuilder.cs

+    {
+        Validate();
+
+        var waitStrategy = Wait.ForUnixContainer().UntilMessageIsLogged("Emulator Service is Successfully Up!");


The wait strategy should be moved to Init().

HofmeisterAn · 2025-01-08T05:44:37Z

src/Testcontainers.EventHubs/Configuration/ConfigurationBuilder.cs

@@ -0,0 +1,53 @@
+namespace Testcontainers.EventHubs.Configuration


Please change all to file scoped namespaces.

HofmeisterAn · 2025-01-08T05:52:30Z

src/Testcontainers.EventHubs/EventHubsBuilder.cs

+    public EventHubsBuilder WithConfigurationBuilder(ConfigurationBuilder configurationBuilder)
+    {
+        var configBytes = Encoding.UTF8.GetBytes(configurationBuilder.Build());
+
+        return Merge(DockerResourceConfiguration, new EventHubsConfiguration(configurationBuilder: configurationBuilder))
+            .WithResourceMapping(configBytes, "Eventhubs_Emulator/ConfigFiles/Config.json");
+    }


Does this builder cover all Event Hubs configuration? Are we responsible for it? Is there a default configuration? The EventHubsBuilder will fail if the user does not call WithConfigurationBuilder(ConfigurationBuilder) explicitly. Until now, no other module requires an additional explicit call to a builder method (expect license agreements). _ = new FooBuilder.Build() always returns a working configuration. This is something I favor.

With configuration builder is needed, because we don't have a default eventhub assigned to the default namespace. We can eventually introduce something, but I don't advise to do so as usually we want to have something custom. For an example, we have azure function:

public static class ProcessSalesData { [FunctionName("ProcessSalesData")] public static void Run( [EventHubTrigger("SalesDataHub", Connection = "EventHubConnectionAppSetting", ConsumerGroup = "RealTimeAnalytics")] string[] events, ILogger log) { foreach (var eventData in events) { log.LogInformation($"C# Event Hub trigger function processed an event: {eventData}"); } } }

Our eventhub is SalesDataHub and usually in integration tests we won't enforce to change it to the ones that is created by the testcontainer.eventhubs. We'd rather to replicate solution that we have in our infra with custom event hub names and just automatically subscribe to this when new message came up.

On a second thought we can introduce eh1 by default. Always we will be able to expand that even further, but for most of the time maybe that will be enought (for most scenarios).

HofmeisterAn · 2025-01-08T05:54:39Z

src/Testcontainers.EventHubs/EventHubsBuilder.cs

+    {
+        return base.Init()
+            .WithImage(EventHubsImage)
+            .WithEnvironment("ACCEPT_EULA", "Y")


Please do not hide license agreements. Right now, we use the following pattern.

HofmeisterAn · 2025-01-08T05:57:48Z

src/Testcontainers.EventHubs/EventHubsConfiguration.cs

+    /// <summary>
+    /// Gets the configuration builder
+    /// </summary>
+    public ConfigurationBuilder ConfigurationBuilder { get; }


It looks like we do not use the ConfigurationBuilder sometime later. In that case, we do not need to store it in the EventHubsConfiguration.

HofmeisterAn · 2025-01-08T06:06:06Z

src/Testcontainers.EventHubs/EventHubsBuilder.cs

@@ -0,0 +1,127 @@
+using DotNet.Testcontainers;


Please use the global usings file.

HofmeisterAn · 2025-01-08T06:08:42Z

tests/Testcontainers.EventHubs.Tests/EventHubsContainerTest.cs

+
+    private EventHubsContainer _eventHubsContainer;
+
+    private readonly INetwork _network = new NetworkBuilder().WithName(NetworkName).Build();


Please do not use static network names. This may clash with already existing resources and may prevent tests from running in parallel. See our best practices.

HofmeisterAn · 2025-01-08T06:11:05Z

tests/Testcontainers.EventHubs.Tests/EventHubsContainerTest.cs

+    private readonly AzuriteContainer _azuriteContainer;
+
+    private EventHubsContainer _eventHubsContainer;
+
+    private readonly INetwork _network = new NetworkBuilder().WithName(NetworkName).Build();
+
+    private const string NetworkName = "eh-network";
+    private const string AzuriteNetworkAlias = "azurite";
+
+    private const string EventHubName = "testeventhub";
+    private const string EventHubConsumerGroupName = "testconsumergroup";
+
+    private EventHubsContainerTest()
+    {
+        _azuriteContainer = new AzuriteBuilder()
+            .WithNetwork(_network)
+            .WithNetworkAliases(AzuriteNetworkAlias)
+            .Build();
+    }


I think it makes sense to align with the pattern Service Bus is using and then follow up with something that allows us to use an existing Azurite instance. Please also see this issue: #1323. This is something we need to figure out and support in general.

HofmeisterAn · 2025-01-08T06:11:42Z

tests/Testcontainers.EventHubs.Tests/Testcontainers.EventHubs.Tests.csproj

@@ -0,0 +1,22 @@
+<Project Sdk="Microsoft.NET.Sdk">
+    <PropertyGroup>
+        <TargetFrameworks>net8.0</TargetFrameworks>


Suggested change

<TargetFrameworks>net8.0</TargetFrameworks>

<TargetFrameworks>net9.0</TargetFrameworks>

HofmeisterAn · 2025-01-08T06:12:36Z

tests/Testcontainers.EventHubs.Tests/Testcontainers.EventHubs.Tests.csproj

+    <ItemGroup>
+      <PackageVersion Update="Azure.Messaging.EventHubs" Version="5.11.3" />
+    </ItemGroup>


Suggested change

<ItemGroup>

<PackageVersion Update="Azure.Messaging.EventHubs" Version="5.11.3" />

</ItemGroup>

Already included above.

rafek1241 · 2025-01-18T18:15:30Z

@HofmeisterAn

I see that @WakaToa is not so responsive so I've made my fork on a top of his changes and implement everything you suggested & aligned to the latest changes. I will create new PR for that (as I dont' know how to merge it) and we can abandon this, I think.

PR: #1342

WakaToa added 2 commits May 22, 2024 13:34

Support Azure EventHubs

8fe0070

Adjust EventHubsConfiguration

3d5c226

WakaToa added 2 commits May 22, 2024 14:14

Add CPM for Azure.Messaging.EventHubs

ebccc8c

better ordering of package version Azure.Messaging.EventHubs

b264e42

HofmeisterAn added enhancement New feature or request module An official Testcontainers module labels May 27, 2024

eddumelendez reviewed May 31, 2024

View reviewed changes

HofmeisterAn requested changes Jan 8, 2025

View reviewed changes

rafek1241 mentioned this pull request Jan 18, 2025

feat: Add Azure Event Hubs module #1342

Closed

HofmeisterAn mentioned this pull request Feb 16, 2025

feat: Add EventHubs module #1373

Merged

HofmeisterAn closed this in #1373 Feb 16, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Azure EventHubs module #1183

feat: Add Azure EventHubs module #1183

WakaToa commented May 22, 2024

netlify bot commented May 22, 2024 •

edited

Loading

eddumelendez May 31, 2024

scrocquesel-ml150 Jul 31, 2024

eddumelendez Jan 7, 2025

HofmeisterAn Jan 8, 2025

eddumelendez commented Jun 25, 2024

Saglodha commented Jan 6, 2025

HofmeisterAn left a comment

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

rafek1241 Jan 18, 2025

rafek1241 Jan 18, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

HofmeisterAn Jan 8, 2025

rafek1241 commented Jan 18, 2025 •

edited

Loading

		@@ -0,0 +1,53 @@
		namespace Testcontainers.EventHubs.Configuration


		private EventHubsContainer _eventHubsContainer;

		private readonly INetwork _network = new NetworkBuilder().WithName(NetworkName).Build();

	<TargetFrameworks>net8.0</TargetFrameworks>
	<TargetFrameworks>net9.0</TargetFrameworks>

	<ItemGroup>
	<PackageVersion Update="Azure.Messaging.EventHubs" Version="5.11.3" />
	</ItemGroup>

feat: Add Azure EventHubs module #1183

feat: Add Azure EventHubs module #1183

Conversation

WakaToa commented May 22, 2024

What does this PR do?

Info

netlify bot commented May 22, 2024 • edited Loading

✅ Deploy Preview for testcontainers-dotnet ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eddumelendez commented Jun 25, 2024

Saglodha commented Jan 6, 2025

HofmeisterAn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rafek1241 commented Jan 18, 2025 • edited Loading

netlify bot commented May 22, 2024 •

edited

Loading

rafek1241 commented Jan 18, 2025 •

edited

Loading