Appimage automated build #1136

mssalvatore · 2021-04-30T14:20:31Z

What does this PR do?

Modifies the AppImage build script so that AppImage packages can be easily build locally by developers or by Jenkins.

Closes #1103

PR Checklist

Have you added an explanation of what your changes do and why you'd like to include them?
Is the TravisCI build passing?
Was the CHANGELOG.md updated to reflect the changes?
Was the documentation framework updated to reflect the changes?

Testing Checklist

Added relevant unit tests?
Have you successfully tested your changes locally? Elaborate:

Tested by building locally and running on my local network. Then I built a package with Jenkins and ran blackbox tests.
If applicable, add screenshots or log transcripts of the feature working

The node modules do not need to be deliverer with the appimage. Removing them from the AppDir saves 50MB.

Jenkins sets a $WORKSPACE environment variable. We'll use this if it's been set. Otherwise, use $HOME.

…clude

codecov · 2021-04-30T14:28:17Z

Codecov Report

Merging #1136 (f475df7) into develop (78ca2c2) will decrease coverage by 0.33%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #1136      +/-   ##
===========================================
- Coverage    28.73%   28.40%   -0.34%     
===========================================
  Files          410      410              
  Lines        12869    13104     +235     
===========================================
+ Hits          3698     3722      +24     
- Misses        9171     9382     +211

Impacted Files	Coverage Δ
monkey/infection_monkey/utils/linux/users.py	`90.32% <0.00%> (-1.68%)`	⬇️
monkey/infection_monkey/utils/windows/users.py	`24.35% <0.00%> (-0.30%)`	⬇️
monkey/infection_monkey/dropper.py	`0.00% <0.00%> (ø)`
monkey/infection_monkey/exploit/hadoop.py	`0.00% <0.00%> (ø)`
monkey/infection_monkey/windows_upgrader.py	`0.00% <0.00%> (ø)`
monkey/infection_monkey/exploit/shellshock.py	`0.00% <0.00%> (ø)`
...ction_monkey/system_info/windows_info_collector.py	`0.00% <0.00%> (ø)`
...n_monkey/post_breach/actions/use_signed_scripts.py	`0.00% <0.00%> (ø)`
monkey/infection_monkey/control.py	`23.15% <0.00%> (+1.33%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78ca2c2...f475df7. Read the comment docs.

VakarisZ · 2021-04-30T14:37:09Z

Do we have OS compatibility for app-image documented somewhere?

mssalvatore · 2021-04-30T14:38:35Z

Do we have OS compatibility for app-image documented somewhere?

Not yet. We have to decide on an official standard for how we test compatibility. It should work on "any modern Linux" as long as FUSE is available. There will, of course, be exceptions.

VakarisZ · 2021-04-30T14:59:43Z

appimage/build_appimage.sh

-  log_message "downloading configuration"
-  curl -L -s -o "$tmpfile" "$CONFIG_URL"
+is_valid_git_repo() {
+  pushd "$1" 2>/dev/null || return 1


Wait, what's the $1 argument?

A local directory that potentially contains a valid git repo.

VakarisZ · 2021-04-30T15:09:49Z

appimage/build_appimage.sh

+case "$1" in
+  --agent-binary-dir)
+    if [ -n "$2" ] && [ "${2:0:1}" != "-" ]; then
+      agent_binary_dir=$2


A lot of these flags are parsed with basically the same code, except the argument is assigned to a different variable, maybe there's an opportunity to remove duplication

Bash functions are a little strange, so I can't get it quite as clean as I would like, but I've made a modest improvement.

VakarisZ · 2021-04-30T15:12:00Z

appimage/build_appimage.sh

+      missing_argument "$1"
+    fi
+    ;;
+  -*)


why do we need the dash?

Since we don't have any positional arguments, we don't. Good catch.

VakarisZ · 2021-04-30T15:20:01Z

appimage/build_appimage.sh

+remove_node_modules() {
+  # Node has served its purpose. We don't need to deliver the node modules with
+  # the AppImage.
+  rm -rf "$ISLAND_PATH"/cc/ui/node_modules


This will probably be slow. can we use npx remove-node-modules?

I ran a single test. The rm ran in 1.7 seconds. the npx remode-node-modules ran in 5.9 seconds.

VakarisZ

Looks good, but adress my questions before merging

mssalvatore added 22 commits April 30, 2021 10:17

appimage: remove node_modules from appdir

408a0de

The node modules do not need to be deliverer with the appimage. Removing them from the AppDir saves 50MB.

appimage: Upgrade from Python 3.7.9 -> 3.7.10

c33f9cf

appimage: Remove unnecessary recursive chmod in clone_monkey_repo()

7276760

appimage: Relocate squashfs-root/ to same dir as build_appimage.sh

c47f5a5

appimage: Use $WORKSPACE variable if present

1f4af68

Jenkins sets a $WORKSPACE environment variable. We'll use this if it's been set. Otherwise, use $HOME.

appimage: Only load binary config if downloading agent binaries

84c875c

appimage: Make spacing consistent

68759e3

appimage: Add cli parameter for specifying the package version

8cb47ce

appimage: Add cli parameter for specifying which agent binaries to in…

2856b85

…clude

appimage: Use npm-ci instead of npm-install

e58281b

appimage: Add a cli parameter for for specifying a git branch

9c11f23

appimage: Put cli parameters in alphabetical order

280b1e2

appimage: Add cli parameter to specify local monkey repo dir

fe25cd2

appimage: Exit with error if unsupported parameters are passed

f91a52f

appimage: Add --help

d53ba0e

appimage: Perform root checks after parsing args

febad65

appimage: Add --as-root flag

ebd2fdd

appimage: Replace apt with apt-get

7ab7e56

appimage: Remove references to deployment scripts in logging

9f150bb

appimage: Add a few log messages

f87514c

appimage: Address minor shellcheck findings

c61b551

appimage: Replace some tabs with spaces

0192e08

mssalvatore requested a review from VakarisZ April 30, 2021 14:23

Add PR #1136 to the changelog

de9d750

VakarisZ reviewed Apr 30, 2021

View reviewed changes

VakarisZ approved these changes Apr 30, 2021

View reviewed changes

mssalvatore added 3 commits May 3, 2021 07:45

appimage: Match any unknown parameters (not just flags)

684a6f8

appimage: reduce some duplication in argument parsing

998a14d

appimage: fix spacing in argument parsing

f475df7

mssalvatore merged commit 7f06ec4 into develop May 3, 2021

mssalvatore deleted the appimage-automated-build branch May 13, 2021 16:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Appimage automated build #1136

Appimage automated build #1136

mssalvatore commented Apr 30, 2021 •

edited

Loading

codecov bot commented Apr 30, 2021 •

edited

Loading

VakarisZ commented Apr 30, 2021

mssalvatore commented Apr 30, 2021

VakarisZ Apr 30, 2021

mssalvatore May 3, 2021

VakarisZ Apr 30, 2021

mssalvatore May 3, 2021

VakarisZ Apr 30, 2021

mssalvatore May 3, 2021

VakarisZ Apr 30, 2021

mssalvatore May 3, 2021

VakarisZ left a comment

Appimage automated build #1136

Appimage automated build #1136

Conversation

mssalvatore commented Apr 30, 2021 • edited Loading

What does this PR do?

PR Checklist

Testing Checklist

codecov bot commented Apr 30, 2021 • edited Loading

Codecov Report

VakarisZ commented Apr 30, 2021

mssalvatore commented Apr 30, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VakarisZ left a comment

Choose a reason for hiding this comment

mssalvatore commented Apr 30, 2021 •

edited

Loading

codecov bot commented Apr 30, 2021 •

edited

Loading