Readnotes: Malicious AI Report

See here for the original report:

Purpose of report

  • survey landscape of security threads from malicious uses of AI
  • proposes ways to better forecast, prevent, and mitagate threats
  • analyze long-term equilibrium between attackers and defenders

4 high level recommendations

  • policymakers should work with tech researchers to investigate, prevent, and mitigate potential malicious uses of AI
  • researchers and engineers in AI should take the dual-use nature of their work seriousy, adjust research priorities and norms, and proactively reach out to relevant actors when harmful apps are foreseeable.
  • identify best practices in research to address dual-use concerns
  • expand range of stakeholders and domain exports to discuss challenges

Security-relevant properties of AI

  • AI is a dual-use area of tech
  • AI systems are commonly both efficient and scalable
  • AI sysmtems can exceed human capabilities
  • AI systems can increase anonymity and psychological distance
  • AI developers lend themselves to rapid difusion
  • Today’s AI systems suffer from many novel unresolved vulnerabilities
    • data poisoning attacks (introduce training data that causes a learning system to make mistakes)
    • adversarial examples (inputs designed to be misclassified by ML systems)
    • exploitation of flaws in design of autonomous systems’ goals

General implications for thread landscape

  • expand existing threats
    • spear phishing attacks
    • increase willingness of actors to carry out attacks
  • introduce new threats
    • AI systems could enable infeasible attacks, ie, mimick others’ voices realistically.
    • AI systems could be used to control behavior of robots and malware that it would be infeasible for humans to control manually.
    • use of self-driving cars for attacks that cause crashes by presenting cars with adversarial examples.
    • image of stop sign with a few pixels changed in specific ways might be misclassified as something else by AI system.
    • an attack on a server used to direct autonomous weapon systems, lead to large-scale friendly fire or civilian targeting.
  • alter typical character of threats
    • attacks supported and enabled by AI will be especially effective, finely targeted, difficult to attribute, and exploitative of vulnerabilities in AI systems.
      • highly effective attacks will become more typical
      • finely targeted attacks will become more prevalent
        • spear phishing
        • use of drone swarms that deploy facial recognition to kill members of crowds
      • difficult-to-attribute attacks will become more typical.
        • use autonomous weapon to attack
      • attacks that exploit vulnerabilities of AI systems will become more typical.


Digital security

malicious uses of AI that would compromise confidentiality, integrity, and availability of digital systems:

  • automation of social engineering attacks
    • generate custom malicious websites/emails/links victims would be likely to click on
    • use writing style that mimics real contacts
    • chatbots elicit human trust by engaging people in longer dialogues
    • masquerade visually as another person in a video chat
  • automation of vulnerability discovery
  • sophisticated automation of hacking
  • human-like denial-of-service
    • through human-speed click patterns and website nevigation
    • a massive crowd of autonomous agents overwhelms an online service
    • prevent access from legitimate users
    • driving target system into a less secure state
  • automation of service tasks in criminal cyber-offense
  • prioritising targets for cyber attacks using ML

Physical security

  • terrorist repurpose of commercial AI systems
  • swarming attacks
  • attacks further removed in time and space

Political security

  • state use of automated serveillance platforms to suppress dissent
    • suppression of debate
  • fake news reports with realistic fabricated video and audio
  • automated, hyper-personalised disinformation campaigns
    • individuals are targeted in swing districts with personalised messags in order to affect their voting behavior
  • automate influence campaigns
    • AI-enabled analysis of social networks are leveraged to identify key influencers
    • approach them with (malicious) offers or targeted with disinformation
  • denial-of-information attacks
    • bot-driven, large-scale info-generation attacks are leveraged to swamp info channels with noise
    • make it hard to acquire real info
  • manipulation of info availability
    • media platforms’ content curation algorithms are used to drive users towards or away from certain content to manipulate user behavior

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s