Rewards

Reward functions are important, especially for ML implementations. Reward values can be set in a Scenarios agent settings, by giving rewards to the DefenderSettings or AttackerSettings.

Scenario file rewards example

agents:
  'Attacker1':
    type: attacker
    entry_points:
    - 'OS App:fullAccess'

    # Rewards for each attack step (by asset name, can also be given by asset type)
    rewards:
      by_asset_name:
        Identity:8:
          assume: 50

  'Defender1':
    type: defender
    # Rewards for each attack step (actually work as penalty)
    rewards:
      by_asset_name:
        OS App:
          notPresent: 2
          supplyChainAuditing: 7
        Program 1:
          notPresent: 3
          supplyChainAuditing: 7
        SoftwareVulnerability:4:
          notPresent: 4
        Data:5:
          notPresent: 1
        Credentials:6:
          notPhishable: 7
        Identity:11:
          notPresent: 3.5

RewardMode can be either CUMULATIVE or ONE_OFF, this is set through the MalSimulatorSettings given to MalSimulator (sim_settings). Default is CUMULATIVE.

CUMULATIVE reward

One off rewards means that the reward and agent receives for a specific step is calculated only based on all nodes performed so far in the simulation.

ONE_OFF reward

One off rewards means that the reward and agent receives for a specific step is calculated only based on the nodes performed that step.

Default behavior

• Attacker is rewarded for compromised nodes
• Defender is penalized for compromised nodes and enabled defenses

Custom behavior

If a user wants to implement their own custom reward function, the current recommendation is to:

• Create a class that inherits MalSimulator
• Override methods:
  • _attacker_step_reward
  • _defender_step_reward

These methods are run to generate a reward every step for each agent (MalSimAgentState), and they are defined like this:

    def _attacker_step_reward(
        self,
        attacker_state: MalSimAttackerState,
        reward_mode: RewardMode,
    ) -> float:

    def _defender_step_reward(
        self,
        defender_state: MalSimDefenderState,
        reward_mode: RewardMode
    ) -> float: