Experimenting and assessing machine learning tools for detecting and analyzing malicious behaviors in complex environments