MEME vs. PPO: Analyzing Evasion Rates and Binary Modifications in Malware Detection

Table of Links

Abstract & Introduction

Threat Model

Background and Related Work

Methodology

Experiments Setup

Results

Discussion

Conclusion, Acknowledgments, and References

Appendix

6 Results

6.1 Malware Evasion

To evaluate MEME, we tested its evasion capabilities compared to four other algorithms. Figure 3 shows the evasion rates of all algorithms against four different targets. Generally, we can see that some targets are more challenging than others. For example, the Sorel-FFNN target seems the hardest to evade, while Ember seems the easiest. MEME outperforms the PPO algorithm in all four targets, with a difference in mean evasion rate between 5-10% depending on the target. This result indicates that creating a surrogate model and running additional training steps is beneficial and produces a better policy. MEME also outperforms MAB in three out of four targets and GAMMA in all four, showing stable overall performance. GAMMA is the only method affected by the time constraint and, in most experiments, manages to process less than 100 binaries. However, it performs almost as well as MEME on the Sorel-FFNN. This shows that the RL methods, including MAB, did not learn the simple strategy of section injection that GAMMA uses, which is part of their action set.

Table 2. Average number of binary modifications to evade the target. The numbers for MAB correspond to the minimal samples created. Random, PPO, and MEME numbers correspond to the mean episode length of the final evaluation on the test set.

The AV evasion rate was over 50% for both PPO and MEME, with MEME achieving an average evasion rate of 59%. This demonstrates that even though AV is more complex than a single classifier, it can still be bypassed.

In terms of required modifications for a malicious binary to appear benign (Table 2), MEME has the lowest average changes against Sorel-FFNN and Microsoft Defender. MAB performs better with lower average changes against Ember and Sorel-LGB, as it aims to minimize the required actions. However, if MAB struggles with evasion, the average modifications increase. Unfortunately, it was not possible to get the number of modifications for each binary from the GAMMA attack since the framework that implements the attack does not provide this metric.

Table 3. Surrogate agreement metrics with the test set for each target model.

6.2 Surrogate Evaluation

Table 3 shows the agreement scores of the surrogate models with their respective targets. The label agreement scores were higher than 97% for all targets. In the feature agreement metrics, 9 out of 10 top features, according to SHAP feature importances were the same for the Ember target and the respective surrogates. The percentage is slightly lower for Sorel-LGB, with 7.2 out of the top 10 feature agreement and 16.2 out of the top 20. However, the feature agreement metrics get significantly lower regarding the Sorel-FFNN target, even though the label agreement is higher than 98%. This target is also the harder one to evade for the RL-based methods. This high level of label agreement was reached with only 2,048 queries, significantly lower than reported in prior work (25k) [32].