Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added a metric to expose consumer batch size #1296

Merged
merged 4 commits into from
Mar 13, 2019
Merged

added a metric to expose consumer batch size #1296

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4ecd1e5
added a metric to expose consumer batch size
Feb 28, 2019
af26fcd
Merge branch 'master' into add-consumer-batchsize-metric
Mar 13, 2019
eb519d6
updated the documentation
Mar 13, 2019
d1c48e3
Small docs indentation fix
bai Mar 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions consumer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ import (
"sync"
"sync/atomic"
"time"

"github.com/rcrowley/go-metrics"
)

// ConsumerMessage encapsulates a Kafka message returned by the consumer.
Expand Down Expand Up @@ -550,6 +552,15 @@ func (child *partitionConsumer) parseRecords(batch *RecordBatch) ([]*ConsumerMes
}

func (child *partitionConsumer) parseResponse(response *FetchResponse) ([]*ConsumerMessage, error) {
var (
metricRegistry = child.conf.MetricRegistry
consumerBatchSizeMetric metrics.Histogram
)

if metricRegistry != nil {
consumerBatchSizeMetric = getOrRegisterHistogram("consumer-batch-size", metricRegistry)
}

block := response.GetBlock(child.topic, child.partition)
if block == nil {
return nil, ErrIncompleteResponse
Expand All @@ -563,6 +574,9 @@ func (child *partitionConsumer) parseResponse(response *FetchResponse) ([]*Consu
if err != nil {
return nil, err
}

consumerBatchSizeMetric.Update(int64(nRecs))

if nRecs == 0 {
partialTrailingMessage, err := block.isPartial()
if err != nil {
Expand Down
8 changes: 8 additions & 0 deletions sarama.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,14 @@ Producer related metrics:
| compression-ratio-for-topic-<topic> | histogram | Distribution of the compression ratio times 100 of record batches for a given topic |
+-------------------------------------------+------------+--------------------------------------------------------------------------------------+

Consumer related metrics:

+-------------------------------------------+------------+--------------------------------------------------------------------------------------+
| Name | Type | Description |
+-------------------------------------------+------------+--------------------------------------------------------------------------------------+
| consumer-batch-size | histogram | Distribution of the number of messages in a batch |
+-------------------------------------------+------------+--------------------------------------------------------------------------------------+

*/
package sarama

Expand Down