Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: create jupyter-lab deployment #18

Open
wants to merge 53 commits into
base: rh-main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
5a060a7
chore(deps): add tekton and kustomize resources
ruromero Nov 15, 2024
76ab83f
feat: add id and timestamps to output file
ruromero Nov 14, 2024
7ef4977
feat: add http output plugin
ruromero Nov 14, 2024
f8acd5f
feat: log input file only for non http requests
ruromero Nov 14, 2024
c5b34f6
chore: deploy latest tag
ruromero Nov 18, 2024
f02499b
chore(deps): add nginx cache
ruromero Nov 18, 2024
c87a41f
fix: depsdev url encoding in the golang packages
ruromero Nov 18, 2024
da3ea4d
feat: use pydantic default factory
ruromero Nov 21, 2024
17cdaf2
fix: dns was not properly being resolved
ruromero Nov 21, 2024
5e93255
chore(deps): add nim-llm and nim-embed deployments
ruromero Nov 21, 2024
6fabe33
feat!: support zip artifact resources
ruromero Nov 21, 2024
ac54749
fix: configure local Llama 3-1-8b LLM to work with nginx
zvigrinberg Nov 24, 2024
ba4b0f8
refactor: parameterize openai_upstream from environment
zvigrinberg Nov 25, 2024
2d26749
fix: address required changes
zvigrinberg Nov 25, 2024
84e14ab
docs: add instruction to deploy helm chart of nim-meta/llama3.1-8b
zvigrinberg Nov 25, 2024
6802712
docs: add instructions to deploy llama-3.1-8b-instruct LLM
zvigrinberg Nov 26, 2024
90b1b42
feat: remove unnecessary stage
ruromero Nov 27, 2024
024b641
feat: make VulnerableDependencyChecker facility pluggable
zvigrinberg Nov 21, 2024
d28307b
refactor: rename to use_dependency_checker
zvigrinberg Nov 27, 2024
e02c491
docs: new general configuration setting - use_dependency_checker
zvigrinberg Nov 27, 2024
f73f200
Update configs/schemas/config.schema.json
zvigrinberg Nov 27, 2024
c3a37a4
fix: vulnerable_dependencies are expected in the dataFrame
ruromero Nov 29, 2024
64f15b2
fix: make broken 'openai' model type to work with the agent
zvigrinberg Nov 28, 2024
2b31095
chore(deps): use shared nim and refactor kustomize resources
ruromero Nov 28, 2024
cf04d4d
fix: lock httpx dependency version and fix incompatibility with other…
ruromero Dec 2, 2024
a0f0565
feat: implement plugin based intel retrieval
ruromero Nov 26, 2024
162f664
fix: a little typo
zvigrinberg Dec 4, 2024
d6d1314
feat: add new overlay of llama-3.1-70b-4bit
zvigrinberg Dec 4, 2024
f5f8ad6
fix: only create openai_llm for openai model service type
zvigrinberg Dec 9, 2024
f3681de
docs: point to agent-morpheus-models module to deploy self hosted LLMs
zvigrinberg Dec 9, 2024
a1b7fb9
docs: add documentation to install helm chart of self hosted models
zvigrinberg Dec 9, 2024
c4efce9
chore: change to remote submodule
zvigrinberg Dec 9, 2024
b78a865
docs: fix link for installing LLMs and models helm chart
zvigrinberg Dec 10, 2024
5c0fb62
docs: refine the instruction to install chart
zvigrinberg Dec 10, 2024
45effa4
fix: pointer of git submodule to latest one
zvigrinberg Dec 10, 2024
68ea029
chore: fix LLM service name for upstream OPENAI
zvigrinberg Dec 10, 2024
8c2ddb6
docs: add instruction to force update to latest version of git submodule
zvigrinberg Dec 10, 2024
3026da2
refactor: use a single variable for both types of llm service objects
zvigrinberg Dec 10, 2024
4bed0c6
fix: ensure agent checks for vulnerable functions/methods if mentione…
IlonaShishov Dec 9, 2024
0f7e5d6
fix: agent crash when startuping pipeline
zvigrinberg Dec 10, 2024
80906d9
fix: agent loop not working
zvigrinberg Dec 10, 2024
808cc83
feat: remove local nim configuration in favor of the 7b config
ruromero Dec 10, 2024
0a6e9cc
chore: update deployment instructions
ruromero Dec 12, 2024
f227161
chore: update deployment instructions
ruromero Dec 12, 2024
55dd0b1
feat: do not retry for client exceptions
ruromero Dec 13, 2024
e136c12
fix: escape backslashes in intel descriptions
ruromero Dec 13, 2024
9548cd0
Merge branch '4xx-retry-upstream' into rh-main
ruromero Dec 16, 2024
bc19755
Merge branch 'parse-intel-upstream' into rh-main
ruromero Dec 16, 2024
52f1490
fix: add ES module support to JavaScript parser
vbelouso Dec 17, 2024
efd6f2a
chore: increase nginx cache pvc
ruromero Dec 18, 2024
37e461f
Merge pull request #17 from vbelouso/js-fix
ruromero Dec 18, 2024
6a69faa
feat: make retry_on_client_errors configurable
ruromero Dec 19, 2024
fe0eda9
chore: create jupyter-lab deployment
ruromero Dec 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
200 changes: 200 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
###### Place new entries directly below this line! ######

# Ignore anything in the ./.tmp directory
.tmp/

# Ignore user-generated fiass databases
vdb/

# Explicitly ignore .vscode/. Shared settings should go in morpheus.code-workspace
# and user settings will go in .vscode/
.vscode/

# Ignore .sqlite cache files
*.sqlite


##### Do not alter the items below this line! #####
########## They are managed by a script! ##########

# Created by https://www.gitignore.io/api/vim,c++,cmake,python,synology

### C++ ###
# Prerequisites
*.d

# Compiled Object files
*.slo
*.lo
*.o
*.obj

# Precompiled Headers
*.gch
*.pch

# Compiled Dynamic libraries
*.so
*.dylib
*.dll

# Fortran module files
*.mod
*.smod

# Compiled Static libraries
*.lai
*.la
*.a
*.lib

# Executables
*.exe
*.out
*.app

### CMake ###
CMakeCache.txt
CMakeFiles
CMakeScripts
Testing
cmake_install.cmake
install_manifest.txt
compile_commands.json
CTestTestfile.cmake

### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# pyenv
.python-version

# celery beat schedule file
celerybeat-schedule

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Env files
*.env

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/

### Synology ###
# Thumbnails
@eaDir
# Recycle bin
\#recycle

### Vim ###
# Swap
[._]*.s[a-v][a-z]
[._]*.sw[a-p]
[._]s[a-rt-v][a-z]
[._]ss[a-gi-z]
[._]sw[a-p]

# Session
Session.vim

# Temporary
.netrwhist
*~
# Auto-generated tag files
tags
# Persistent undo
[._]*.un~


# End of https://www.gitignore.io/api/vim,c++,cmake,python,synology
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,9 @@ ENV/
env.bak/
venv.bak/

# Env files
*.env

# Spyder project settings
.spyderproject
.spyproject
Expand Down
4 changes: 4 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[submodule "agent-morpheus-models"]
path = agent-morpheus-models
url = ../agent-morpheus-models
branch= main
3 changes: 3 additions & 0 deletions .pytest.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# pytest.ini
[pytest]
asyncio_default_fixture_loop_scope = function
116 changes: 116 additions & 0 deletions .tekton/on-pull-request.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: vulnerability-analysis-on-pr
annotations:
# The event we are targeting as seen from the webhook payload
# this can be an array too, i.e: [pull_request, push]
pipelinesascode.tekton.dev/on-event: "[pull_request]"

# The branch or tag we are targeting (ie: main, refs/tags/*)
pipelinesascode.tekton.dev/on-target-branch: "[main, rh-main]"

# Fetch the git-clone task from hub, we are able to reference later on it
# with taskRef and it will automatically be embedded into our pipeline.
pipelinesascode.tekton.dev/task: "git-clone"

# How many runs we want to keep.
pipelinesascode.tekton.dev/max-keep-runs: "5"
spec:
params:
# The variable with brackets are special to Pipelines as Code
# They will automatically be expanded with the events from Github.
- name: repo_url
value: "{{ repo_url }}"
- name: revision
value: "{{ revision }}"
- name: image-expires-after
value: 5d
- name: output-image
value: quay.io/ecosystem-appeng/agent-morpheus-rh:on-pr-{{revision}}
- name: path-context
value: .
- name: dockerfile
value: ./Dockerfile
pipelineSpec:
params:
- name: repo_url
- name: revision
- name: output-image
description: Fully Qualified Output Image
type: string
- name: path-context
default: .
description: Path to the source code of an application's component from where to build image.
type: string
- name: dockerfile
default: Dockerfile
description: Path to the Dockerfile inside the context specified by parameter path-context
type: string
- name: image-expires-after
default: ""
description: Image tag expiration time, time values could be something like 1h, 2d, 3w for hours, days, and weeks, respectively.
workspaces:
- name: source
- name: basic-auth
tasks:
- name: fetch-repository
taskRef:
name: git-clone
workspaces:
- name: output
workspace: source
- name: basic-auth
workspace: basic-auth
params:
- name: url
value: $(params.repo_url)
- name: revision
value: $(params.revision)
- name: buildah
runAfter:
- fetch-repository
params:
- name: IMAGE
value: $(params.output-image)
- name: DOCKERFILE
value: $(params.dockerfile)
- name: CONTEXT
value: $(params.path-context)
- name: BUILDER_IMAGE
value: >-
registry.redhat.io/rhel8/buildah@sha256:aac6629389db17e99894c5bee0da01d4c8065d11d8c6f6e1602f9484290baa70
- name: STORAGE_DRIVER
value: vfs
- name: FORMAT
value: docker
- name: BUILD_EXTRA_ARGS
value: >-
--target base
--label=quay.expires-after=$(params.image-expires-after)
taskRef:
kind: ClusterTask
name: buildah
workspaces:
- name: source
workspace: source
- name: dockerconfig
workspace: dockerconfig-ws
workspaces:
- name: source
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
# This workspace will inject secret to help the git-clone task to be able to
# checkout the private repositories
- name: basic-auth
secret:
secretName: "{{ git_auth_secret }}"
- name: dockerconfig-ws
secret:
secretName: ecosystem-appeng-morpheus-quay
Loading