Adds support for uploading threat intelligence in Custom Format … #1493
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…JSON (opensearch-project#1455) * revert common utils dep change Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * adds jsonpath deps Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * adds custom JsonSchema request model Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * adds parsing iocs via new tif source type - custom schema ioc upload Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * change Ioc Type variable from enum to string to support custom ioc types Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove ioc type check to allow custom types Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add name and id field parsing via json path annotation Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * adds custom schema json parsing codec that parses based on JsonPath notations Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix stix ioc parsing with null checks on each text field Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove ioc type lower case conversion in ioc scan service Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * compute ioc types from iocs list instead of fetching from request Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * compute ioc types from parsed iocs in S3 threat intel source download Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add null check Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * rewrite amazon s3 connector to compute correct input codec Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * revert if else flip for create connector Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * add logging to trace threat intel monitor execution Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove source type CUSTOM_SCHEMA_IOC_UPLOAD and merge the new source object into IOC_UPlOAD Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * validate that threat intel ioc type and schema that json is valid and also a legal string Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove iskey Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * update java docs Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * handle numbers in ioc value column Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix review comments Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix doc level monitor input constructor Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * udpate jar Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * revert build.gradle change Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * upgrade json smart to 2.5.2 to deal with CVE-2024-57699 Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
eirsep
requested review from
amsiglan,
AWSHurneyt,
getsaurabh02,
lezzago,
praveensameneni,
sbcd90,
jowg-amazon,
engechas,
goyamegh and
riysaxen-amzn
as code owners
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
AWSHurneyt
reviewed
Mar 14, 2025
| @@ -204,26 +199,27 @@ public static STIX2IOC parse(XContentParser xcp, String id, Long version) throws | |||
|
|
|||
| switch (fieldName) { | |||
| case NAME_FIELD: | |||
| if (xcp.currentToken() == XContentParser.Token.VALUE_NULL) { | |||
There was a problem hiding this comment.
Just curious. I noticed that the STIX2IOCDto class is using a getString helper function when parsing these fields. Could you clarify why we don't use something similar here?
AWSHurneyt
reviewed
Mar 14, 2025
| private InputCodec getInputCodec(FeedConfiguration feedConfiguration, SATIFSourceConfig satifSourceConfig) { | ||
| final InputCodec inputCodec; | ||
| if (satifSourceConfig != null && satifSourceConfig.getIocSchema() != null) { | ||
| logger.info("Parsing custom schema JSON from S3 for threat intel source [{}]", satifSourceConfig.getName()); |
There was a problem hiding this comment.
Nitpick: Would using the source config ID make more sense for this log? Or do we have validation in place to prevent customers from creating configs with the same name?
AWSHurneyt
approved these changes
Mar 14, 2025
|
Approved as PR to 2.x was reviewed and merged. |
|
The security test workflow should be fixed now. You'll need to rebase this commit |
riysaxen-amzn
approved these changes
Mar 14, 2025
66 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-picks from 2.x (#1455) to main
Closed original PR #1450 to main due to dep conflicts in CIs