Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Service Bus: Bulk publish does not apply metadata properties to events #1421

Open
bely300 opened this issue Dec 10, 2024 · 1 comment
Open
Assignees
Labels
area/client/pubsub kind/bug Something isn't working
Milestone

Comments

@bely300
Copy link

bely300 commented Dec 10, 2024

Expected Behavior

When publishing events to Azure Service Bus Topic via the Dapr BulkPublishEventAsync method, the metadata properties and values should be applied to all events in the batch of events.

Actual Behavior

None of the metadata properties and values are applied to the events in the call to the BulkPublishEventAsync method.

Steps to Reproduce the Problem

using System;
using System.Collections.Generic;
using Dapr.Client;

const string PubsubName = "my-pubsub-name";
const string TopicName = "topic-a";
IReadOnlyList<object> BulkPublishData = new List<object>() {
    new { Id = "17", Amount = 10m },
    new { Id = "18", Amount = 20m },
    new { Id = "19", Amount = 30m }
};

using var client = new DaprClientBuilder().Build();

var metadata = new Dictionary<string, string>() { { "PartitionId", "RandomIDValue" } };
var res = await client.BulkPublishEventAsync(PubsubName, TopicName, BulkPublishData, metadata);

Reading events back via Dapr or viewing them in the Service Bus topic/Subscription in the Azure Portal, the events will not contain the expected metadata properties.

The issue is related to a mis-match between the way the Dapr .Net SDK handles the metadata versus how the Dapr runtime processes the passed in metadata.

The Dapr .Net SDK only packages up a single metadata map in the BulkPublishRequest object, so expects the Dapr runtime will apply the metadata to each event before writing to the Azure Service Bus Topic. See DaprClientGrpc.MakeBulkPublishRequest(....)

The logic on the runtime side, implemented in the 'components-contrib' repo ignores the metadata in the bulkPublishRequest object. Instead the logic walks through each event and processes any metadata object associated with the individual event. Refer to the 'PublishPubSubBulk' method in 'components-contrib/common/component/azure/servicebus/publisher.go'.

Release Note

RELEASE NOTE:

@bely300 bely300 added the kind/bug Something isn't working label Dec 10, 2024
@WhitWaldo WhitWaldo added this to the v1.15 milestone Dec 10, 2024
@WhitWaldo
Copy link
Contributor

I'll see if I can't squeeze this into the 1.15 release. Thanks for spotting this @bely300 !

@WhitWaldo WhitWaldo self-assigned this Dec 10, 2024
@WhitWaldo WhitWaldo modified the milestones: v1.15, Future Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/client/pubsub kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants