2817 replace agent singleton #3068

cakekoa · 2023-03-07T19:32:53Z

What does this PR do?

Fixes #2817.

PR Checklist

Have you added an explanation of what your changes do and why you'd like to include them?
Is the TravisCI build passing?
Was the CHANGELOG.md updated to reflect the changes?
Was the documentation framework updated to reflect the changes?
Have you checked that you haven't introduced any duplicate code?

Testing Checklist

Added relevant unit tests?
Have you successfully tested your changes locally? Elaborate:

Tested by {Running the Monkey locally with relevant config/running Island/...}
If applicable, add screenshots or log transcripts of the feature working
Output of for i in {1..3}; do python ./infection_monkey.py m0nk3y -s 172.17.0.1:5000 > monkey-${i}.log 2>&1 & done:

monkey-3.log
monkey-2.log
monkey-1.log

ilija-lazoroski

Approved, if tested. GJ!

mssalvatore

Has this been tested with multiple agents running on the same machine? We should probably also run ETE tests.

mssalvatore

When I test:

2023-03-07 20:34:01,680 [12620:MainThread:INFO] main.main.46: writing log file to /home/ubuntu/tmp/infection-monkey-agent-2023-03-07-20-34-01-92ie8i_d.log
2023-03-07 20:34:01,680 [12620:MainThread:INFO] main._run_agent.137: >>>>>>>>>> Initializing the Infection Monkey Agent: PID 12620 <<<<<<<<<<
2023-03-07 20:34:01,680 [12620:MainThread:INFO] main._run_agent.141: version: 2.0.0+dev
2023-03-07 20:34:01,680 [12620:MainThread:INFO] monkey.__init__.102: Agent is initializing...
2023-03-07 20:34:01,680 [12620:MainThread:INFO] monkey.__init__.106: Agent ID: 7a3ee3ba-427d-4f91-aec0-c05913f08190
2023-03-07 20:34:01,680 [12620:MainThread:INFO] monkey.__init__.107: Process ID: 12620
/home/ubuntu/venvs/monkey/src/paramiko/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
2023-03-07 20:34:02,195 [12620:MainThread:INFO] monkey._log_arguments.214: Agent started with arguments: parent: None, servers: [SocketAddress(ip=IPv4Address('10.27.225.28'), port=5000)], depth: 0
2023-03-07 20:34:02,195 [12620:MainThread:DEBUG] monkey._connect_to_island_api.168: Trying to wake up with servers: 10.27.225.28:5000
2023-03-07 20:34:02,196 [12620:FindIslandServer-01:DEBUG] utils._check_if_island_server.62: Trying to connect to server: 10.27.225.28:5000
2023-03-07 20:34:02,253 [12620:FindIslandServer-01:DEBUG] utils._check_if_island_server.68: Successfully connected to the Island via 10.27.225.28:5000
2023-03-07 20:34:02,253 [12620:MainThread:INFO] monkey._connect_to_island_api.176: Using 10.27.225.28:5000 to communicate with the Island
2023-03-07 20:34:02,256 [12620:MainThread:DEBUG] http_client._send_request.132: POST https://10.27.225.28:5000/api/agents, timeout=2.5
2023-03-07 20:34:02,303 [12620:MainThread:INFO] heart.start.21: Starting the Agent's heart
2023-03-07 20:34:02,303 [12620:MainThread:DEBUG] code_utils.start.134: Starting AgentHeart
2023-03-07 20:34:02,303 [12620:AgentHeart:DEBUG] code_utils.run.144: Successfully started AgentHeart
2023-03-07 20:34:02,303 [12620:AgentHeart:DEBUG] http_client._send_request.132: POST https://10.27.225.28:5000/api/agent/7a3ee3ba-427d-4f91-aec0-c05913f08190/heartbeat, timeout=5
2023-03-07 20:34:02,305 [12620:MainThread:INFO] main._run_agent.152: Starting InfectionMonkey
2023-03-07 20:34:02,305 [12620:MainThread:DEBUG] http_client._send_request.132: GET https://10.27.225.28:5000/api/agent-signals/7a3ee3ba-427d-4f91-aec0-c05913f08190, timeout=2.5
2023-03-07 20:34:02,349 [12620:MainThread:INFO] monkey.start.219: The Monkey Island has instructed this agent to stop
2023-03-07 20:34:02,349 [12620:MainThread:INFO] monkey.cleanup.431: Agent cleanup started
2023-03-07 20:34:02,349 [12620:MainThread:INFO] monkey._self_delete.510: Cleaning up the Agent's artifacts
2023-03-07 20:34:02,349 [12620:MainThread:INFO] monkey._send_log.488: Sending agent logs to the Island
2023-03-07 20:34:02,350 [12620:MainThread:DEBUG] http_client._send_request.132: PUT https://10.27.225.28:5000/api/agent-logs/7a3ee3ba-427d-4f91-aec0-c05913f08190, timeout=5
2023-03-07 20:34:02,352 [12620:MainThread:DEBUG] pypubsub_agent_event_queue._publish_event.55: Publishing a AgentShutdownEvent event to all_events_topic
2023-03-07 20:34:02,352 [12620:MainThread:DEBUG] pypubsub_agent_event_queue._publish_event.55: Publishing a AgentShutdownEvent event to AgentShutdownEvent-type
2023-03-07 20:34:02,352 [12620:MainThread:ERROR] monkey.cleanup.458: An error occurred while cleaning up the monkey agent: 'NoneType' object has no attribute 'flush'
Traceback (most recent call last):
  File "/home/ubuntu/git/monkey/monkey/infection_monkey/monkey.py", line 451, in cleanup
    self._agent_event_forwarder.flush()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'flush'
2023-03-07 20:34:02,353 [12620:MainThread:INFO] plugin_event_forwarder.stop.66: Stopping plugin event forwarder
2023-03-07 20:34:02,353 [12620:MainThread:ERROR] main._run_agent.160: Exception thrown from monkey's cleanup function: More info: cannot join thread before it is started
Traceback (most recent call last):
  File "/home/ubuntu/git/monkey/monkey/infection_monkey/main.py", line 158, in _run_agent
    monkey.cleanup()
  File "/home/ubuntu/git/monkey/monkey/infection_monkey/monkey.py", line 462, in cleanup
    self._plugin_event_forwarder.stop()
  File "/home/ubuntu/git/monkey/monkey/infection_monkey/plugin_event_forwarder.py", line 68, in stop
    self._thread.join(timeout)
  File "/usr/lib/python3.11/threading.py", line 1107, in join
    raise RuntimeError("cannot join thread before it is started")
RuntimeError: cannot join thread before it is started
2023-03-07 20:34:02,456 [12620:MainThread:DEBUG] main.main.53: Stopping the queue listener

codecov · 2023-03-08T14:21:38Z

Codecov Report

Patch coverage has no change and project coverage change: +0.35 🎉

Comparison is base (a1978bd) 67.14% compared to head (bc8f0f2) 67.50%.

❗ Current head bc8f0f2 differs from pull request most recent head 889fc66. Consider uploading reports for the commit 889fc66 to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3068      +/-   ##
===========================================
+ Coverage    67.14%   67.50%   +0.35%     
===========================================
  Files          442      441       -1     
  Lines        12751    12685      -66     
===========================================
+ Hits          8562     8563       +1     
+ Misses        4189     4122      -67

Impacted Files	Coverage Δ
monkey/monkey/infection_monkey/monkey.py	`0.00% <0.00%> (ø)`
.../monkey/infection_monkey/plugin_event_forwarder.py	`100.00% <0.00%> (ø)`
monkey/monkey/infection_monkey/system_singleton.py

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

The island now determines when duplicate agents are running, and sets the terminate signal when an agent should close. The agent checks whether or not it should stop on startup, as well as periodically.

ilija-lazoroski approved these changes Mar 7, 2023

View reviewed changes

mssalvatore reviewed Mar 7, 2023

View reviewed changes

mssalvatore force-pushed the 2817-replace-agent-singleton branch from 54c2947 to df282b0 Compare March 7, 2023 20:25

mssalvatore requested changes Mar 7, 2023

View reviewed changes

cakekoa force-pushed the 2817-replace-agent-singleton branch from 7a856d6 to fe6b06a Compare March 8, 2023 14:09

Agent: Remove SystemSingleton

c17781e

The island now determines when duplicate agents are running, and sets the terminate signal when an agent should close. The agent checks whether or not it should stop on startup, as well as periodically.

mssalvatore force-pushed the 2817-replace-agent-singleton branch from 44fad63 to ee630bb Compare March 8, 2023 17:12

cakekoa and others added 3 commits March 8, 2023 12:53

Agent: Add a comment about startup/stop order

a53e362

Changelog: Add an entry for #2817

98f836f

Agent: Fix cleanup when stop signal is received during startup

889fc66

mssalvatore force-pushed the 2817-replace-agent-singleton branch from bc8f0f2 to 889fc66 Compare March 8, 2023 17:54

mssalvatore approved these changes Mar 8, 2023

View reviewed changes

mssalvatore merged commit ee39637 into develop Mar 8, 2023

mssalvatore deleted the 2817-replace-agent-singleton branch March 8, 2023 17:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2817 replace agent singleton #3068

2817 replace agent singleton #3068

cakekoa commented Mar 7, 2023 •

edited

Loading

ilija-lazoroski left a comment

mssalvatore left a comment

mssalvatore left a comment

codecov bot commented Mar 8, 2023 •

edited

Loading

2817 replace agent singleton #3068

2817 replace agent singleton #3068

Conversation

cakekoa commented Mar 7, 2023 • edited Loading

What does this PR do?

PR Checklist

Testing Checklist

ilija-lazoroski left a comment

Choose a reason for hiding this comment

mssalvatore left a comment

Choose a reason for hiding this comment

mssalvatore left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 8, 2023 • edited Loading

Codecov Report

cakekoa commented Mar 7, 2023 •

edited

Loading

codecov bot commented Mar 8, 2023 •

edited

Loading