Implement baseline cloudwatch metrics #154
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,124 @@ | ||
| // @flow | ||
|
|
||
| type Metric = { | ||
| send: () => void, | ||
| }; | ||
|
|
||
| process.env.AWS_PROFILE = 'frontend'; | ||
|
|
||
| const AWS = require('aws-sdk'); | ||
|
|
||
| AWS.config.update({ region: 'eu-west-1' }); | ||
|
|
||
| // how frequently we send metrics to aws in ms | ||
| const METRICS_TIME_RESOLUTION = 60 * 1000; | ||
|
|
||
| const sendMetric = (m: Array) => { | ||
| if (m.length === 0) { | ||
| return; | ||
| } | ||
|
|
||
| const cloudWatchClient = new AWS.CloudWatch(); | ||
|
|
||
| const params = { | ||
| MetricData: m, | ||
| Namespace: 'Application', | ||
| }; | ||
|
|
||
| cloudWatchClient.putMetricData(params, err => { | ||
| if (err) { | ||
| console.error(err, err.stack); | ||
| } | ||
| }); | ||
| }; | ||
|
|
||
| // handles sending matrics to AWS | ||
|
|
||
| const collectAndSendAWSMetrics = function(...metrics: Metric) { | ||
| setInterval(() => { | ||
| console.log('Collecting metrics'); | ||
| metrics.forEach(m => m.send()); | ||
| }, METRICS_TIME_RESOLUTION); | ||
| }; | ||
|
|
||
| // to record things like latency | ||
|
|
||
| const TimingMetric = function TimingMetric( | ||
| app: String, | ||
| stage: String, | ||
| metricName: String, | ||
| ) { | ||
| const values = []; | ||
|
|
||
| return { | ||
| recordDuration: (n: Number) => { | ||
| values.push(n); | ||
| }, | ||
|
|
||
| send: () => { | ||
| sendMetric( | ||
| values.map(v => ({ | ||
| Dimensions: [ | ||
| { | ||
| Name: 'ApplicationName', | ||
| Value: app, | ||
| }, | ||
| { | ||
| Name: 'Stage', | ||
| Value: stage, | ||
| }, | ||
| ], | ||
| MetricName: metricName, | ||
| Unit: 'Milliseconds', | ||
| Value: v, | ||
| })), | ||
| ); | ||
|
|
||
| values.length = 0; | ||
|
There was a problem hiding this comment. I did not know you could truncate an array like this! JS is full of surprises. |
||
| }, | ||
| }; | ||
| }; | ||
|
|
||
| // to record memory or file sizes | ||
|
|
||
| const BytesMetric = function BytesMetric( | ||
|
There was a problem hiding this comment. I think since this and That said, they are serving as factory functions. Maybe we need a convention around these? What's better:
? |
||
| app: String, | ||
| stage: String, | ||
| metricName: String, | ||
| ) { | ||
| const values = []; | ||
|
|
||
| return { | ||
| record: (n: Number) => { | ||
| values.push(n); | ||
| }, | ||
|
|
||
| send: () => { | ||
| sendMetric( | ||
| values.map(v => ({ | ||
| Dimensions: [ | ||
| { | ||
| Name: 'ApplicationName', | ||
| Value: app, | ||
| }, | ||
| { | ||
| Name: 'Stage', | ||
| Value: stage, | ||
| }, | ||
| ], | ||
| MetricName: metricName, | ||
| Unit: 'Bytes', | ||
| Value: v, | ||
| })), | ||
| ); | ||
|
|
||
| values.length = 0; | ||
| }, | ||
| }; | ||
| }; | ||
|
|
||
| module.exports = { | ||
| collectAndSendAWSMetrics, | ||
| BytesMetric, | ||
| TimingMetric, | ||
| }; | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| // @flow | ||
|
|
||
| import os from 'os'; | ||
| import disk from 'diskusage'; | ||
| import { BytesMetric, collectAndSendAWSMetrics } from './aws-metrics'; | ||
|
|
||
| const maxHeapMemory = BytesMetric('rendering', 'PROD', 'max-heap-memory'); | ||
| const freeDiskSpace = BytesMetric('rendering', 'PROD', 'free-disk-memory'); | ||
| const usedHeapMemory = BytesMetric('rendering', 'PROD', 'used-heap-memory'); | ||
| const freePhysicalMemory = BytesMetric( | ||
| 'rendering', | ||
| 'PROD', | ||
| 'free-physical-memory', | ||
| ); | ||
| const totalPhysicalMemory = BytesMetric( | ||
| 'rendering', | ||
| 'PROD', | ||
| 'total-physical-memory', | ||
| ); | ||
|
|
||
| // transmits metrics to AWS | ||
|
|
||
| collectAndSendAWSMetrics( | ||
| maxHeapMemory, | ||
| usedHeapMemory, | ||
| totalPhysicalMemory, | ||
| freePhysicalMemory, | ||
| freeDiskSpace, | ||
| ); | ||
|
|
||
| // records system metrics | ||
|
|
||
| const recordBaselineCloudWatchMetrics = function recordBaselineCloudWatchMetrics() { | ||
| disk.check('/', (err, diskinfo) => { | ||
| if (err) { | ||
| console.error(err); | ||
| } else { | ||
| maxHeapMemory.record(process.memoryUsage().heapTotal); | ||
| usedHeapMemory.record(process.memoryUsage().heapUsed); | ||
| totalPhysicalMemory.record(os.totalmem()); | ||
| freePhysicalMemory.record(os.freemem()); | ||
| freeDiskSpace.record(diskinfo.free); | ||
| } | ||
| }); | ||
| }; | ||
|
|
||
| export default recordBaselineCloudWatchMetrics; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this used for? Note that environment variables set on
process.envare not available outside of the Node runtimeThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes the node aws client library use the appropriate local aws credentials if they're available, so that it works locally on your machine if you're testing the production server.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay. Do we always want to set this, or just when we're running the app locally?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should be ok to always set this, as aws credentials work essentially as a series of fallbacks - it should try your local aws profile first, then look for credentials in other standard places. That's how it works in scala too (if it doesn't work in production it should be a quick tweak to force it to look explicitly).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, thanks for explaining, sounds good 👍