Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds support for uploading threat intelligence in Custom Format … #1493

Merged
merged 2 commits into from
Mar 14, 2025

Conversation

eirsep
Copy link
Member

@eirsep eirsep commented Mar 3, 2025

Cherry-picks from 2.x (#1455) to main

Closed original PR #1450 to main due to dep conflicts in CIs

…JSON (opensearch-project#1455)

* revert common utils dep change

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* adds jsonpath deps

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* adds custom JsonSchema request model

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* adds parsing iocs via new tif source type - custom schema ioc upload

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* change Ioc Type variable from enum to string to support custom ioc types

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* remove ioc type check to allow custom types

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add name and id field parsing via json path annotation

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* adds custom schema json parsing codec that parses based on JsonPath notations

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix stix ioc parsing with null checks on each text field

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* remove ioc type lower case conversion in ioc scan service

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* compute ioc types from iocs list instead of fetching from request

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* compute ioc types from parsed iocs in S3 threat intel source download

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add null check

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* rewrite amazon s3 connector to compute correct input codec

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* revert if else flip for create connector

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* add logging to trace threat intel monitor execution

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* remove source type CUSTOM_SCHEMA_IOC_UPLOAD and merge the new source object into IOC_UPlOAD

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* validate that threat intel ioc type and schema that json is valid and also a legal string

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* remove iskey

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* update java docs

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* handle numbers in ioc value column

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix review comments

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* fix doc level monitor input constructor

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* udpate jar

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* revert build.gradle change

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

* upgrade json smart to 2.5.2 to deal with CVE-2024-57699

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>

---------

Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
Signed-off-by: Surya Sashank Nistala <snistala@amazon.com>
@@ -204,26 +199,27 @@ public static STIX2IOC parse(XContentParser xcp, String id, Long version) throws

switch (fieldName) {
case NAME_FIELD:
if (xcp.currentToken() == XContentParser.Token.VALUE_NULL) {
Copy link
Collaborator

@AWSHurneyt AWSHurneyt Mar 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious. I noticed that the STIX2IOCDto class is using a getString helper function when parsing these fields. Could you clarify why we don't use something similar here?

private InputCodec getInputCodec(FeedConfiguration feedConfiguration, SATIFSourceConfig satifSourceConfig) {
final InputCodec inputCodec;
if (satifSourceConfig != null && satifSourceConfig.getIocSchema() != null) {
logger.info("Parsing custom schema JSON from S3 for threat intel source [{}]", satifSourceConfig.getName());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: Would using the source config ID make more sense for this log? Or do we have validation in place to prevent customers from creating configs with the same name?

@AWSHurneyt
Copy link
Collaborator

Approved as PR to 2.x was reviewed and merged.

@AWSHurneyt
Copy link
Collaborator

The security test workflow should be fixed now. You'll need to rebase this commit
f198ac2

@AWSHurneyt AWSHurneyt merged commit f5b7b5f into opensearch-project:main Mar 14, 2025
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
v3.0.0 Issues related to 3.0.0 release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants