Assistant: Hi there! Thank you so much for taking the time to speak with me today. I really appreciate you sharing your insights with us. I'm Claude, an AI user researcher at Anthropic, and I'm here to learn about your experiences using AI in your scientific work.

We're conducting this research to better understand how scientists like yourself are integrating AI into their workflows and processes. There's no right or wrong answer here—we're simply interested in hearing about your authentic experiences and perspectives.

Our conversation today will last about 10 minutes. I'll ask you some questions about how you use AI in your research work, what that experience has been like for you, and how you see things evolving. Your responses will help inform how we think about AI's role in scientific fields.

Does that sound good to you? Do you have any questions before we begin?

User: no, ask away

AI: Great! Let's dive in.

Walk me through a recent research project you've worked on - what did that process look like from initial idea to final output?

User: One research project I worked on had an objective of creating a novel attack on commercial drones.  I cannot share too many details as I signed an NDA, but I can share the general process.  By the time I had joined the time, the vulnerability that we planned to exploit had already been identified.  The first step for me and the other members was copiously reading through any and all documentation regarding the features of the drone that were related to this.  Next, we used out model of the drone to get the firmware and ran our attack on the desired portion of the firmware in a simulator until we consistently achieved success.  From there, we began simulating the attack in real life with our drone and eventually attempted this attack on other models of drones as well

AI: That's a fascinating project - the progression from documentation review to firmware analysis to simulation and then real-world testing sounds like a very methodical approach. 

Tell me more about that initial phase where you were reading through all the documentation - what was that process like and how did you organize or make sense of all that information?

User: It started off simple enough: the first thing we did is simply read the documentation for the drone.  As you can imagine, it gave a lot of details about the features of the drone, but not too many about the implementation: those were up to us to piece together.  For the components of the drone involved in the attack, we looked at their datasheets and this gave us much more information to work with.  Particularly, we looked for information such as unused registers, inputs that were available but not used by the model of drone, and any lacking security features, such as a lack of input bounding, unguarded memory, etc.  Once we identified what security flaws were there, we looked at the firmware and simulated what we would be able to exploit.  Note that a lot of this information can be derived from the hardware as well.  Looking at what connections are made between the flight controller and its peripheral components, and what connections are left open, is just as important as reading through documentation.  

As a basic example, let's say you wanted to see if you could implement a buffer overflow on a microcontroller: your first step may be seeing if that particular microcontroller has any locks on memory access (most will not).  Then you can attempt to run a buffer overflow on the microcontroller and see if it is successful or if any checks exist to prevent your attempt.  If not, you have verified that that is a vulnerability of that device that can be exploited.

AI: That's really detailed - thank you for walking me through that systematic approach. It sounds like you were essentially reverse-engineering the security posture by examining both the documented and undocumented aspects of the system.

Where in that process, if anywhere, have you experimented with using AI tools? I'm curious whether AI played any role in analyzing documentation, identifying potential vulnerabilities, or any other part of your workflow.

User: In cybersecurity, we classify security flaws as CWEs (Common Weakness Enumeration).  These are more or less common security flaws in hardware and software that have been identified and documented by the cybersecurity community.  While in the past, many hours were spent looking through code in an attempt to find these, we find that AI is quite good at doing a preliminary screen and identifying CWEs.  Although it is not perfect, we use a model that was developed by another research team under the same lab, and this model is tuned to produce more false positives than negatives so there is a much better chance that our model identifies a fake vulnerability rather than misses a real one.  Aside from this, consumer AI tools are used to help with the administrative tasks and occasionally basic coding.

AI: That's really interesting - so you're using a specialized AI model that's been tuned specifically for vulnerability detection, and it sounds like the calibration toward false positives rather than false negatives is a deliberate design choice.

For that specialized model that identifies CWEs, tell me about how you decided whether to use that AI screening versus doing the manual code review approach you mentioned from the past.

User: The manual screening was being phased out at the time I joined.  From what I understand, the model was developed about two years prior to when I joined the team and much of the team had experience using it already and was unofficially using it to help.  Note that the model was developed by another research team but under the same lab, so it was only natural that we would be good candidates for their pilot study.  Once their research of published, we began completely shifting to using that model and by the time I had sufficient experience and training with the team to be a contributor, we were always using the model for our initial screening: it had been integrated into the process.

AI: I see - so by the time you arrived, the AI model had already become the established workflow rather than something you had to decide whether or not to adopt. That makes sense given the timing and your lab's involvement in its development.

Thinking about the parts of your research process where you haven't used AI - like the physical hardware analysis you mentioned, examining connections between components, or the real-world testing phases - what's kept you from trying AI tools there?

User: These hardware tests in particular often require a person to physical interact with the hardware.  Sometimes it can take the form of physically reverse engineering a part of a drone to see what components are inside and how they are connected.  That would be quite difficult to accomplish by prompting an LLM.  Not all of the hardware implementation details are documented either so it would be hard to trust an LLMs response since it wouldn't necessarily have a source to reference.

AI: That makes complete sense - the physical, hands-on nature of hardware analysis and the lack of documented implementation details would definitely be barriers to AI assistance there.

When you've used AI for the CWE screening and administrative tasks you mentioned, how do you verify or validate what it produces? Especially with that vulnerability detection model that's tuned for false positives?

User: The majority of the firmware will not contain CWEs, and with every patch, more CWEs are removed.  Consequently, the AI output will not be too much for a human to look through.  Usually the step of reviewing the output and deciding which CWEs to attack is divided among us.

AI: That's a practical approach - dividing up the AI-flagged potential vulnerabilities among team members for human review. It sounds like the volume is manageable even with the false positive bias.

We're about two-thirds of the way through our conversation now. Let me ask you this: if you could wave a magic wand and have AI help with one aspect of your research that it currently doesn't, what would that be and why?

User: Have to think a lot for this one.  I guess like you alluded to earlier, if AI could somehow magically identify the hardware related vulnerabilities inside a component, that would be amazing.  It would also save a lot of effort and funding in not having to buy extra hardware and spend time reverse engineering for this purpose.  

I doubt that would work in real life, so maybe a more practical wish would be if AI could somehow help with any software related simulation.  Oftentimes, these simulations require VMs or entirely separate machines.  The VMs are often slow and clunky, and process is very much running an exploit, trying to understand why the exploit failed, and then revising and running again.  Then, repeating the experiments with different environments.   Perhaps if there was a way AI could predict point of failure in these exploits before we simulate them, that may help us.

AI: Those are both really thoughtful wishes - the hardware vulnerability identification would be like having x-ray vision for security flaws, and the exploit failure prediction could potentially save you a lot of those iterative cycles in the simulation environment.

That simulation process you described - running the exploit, diagnosing failures, revising, and repeating across different environments - sounds like it could be quite time-intensive.

Those are all my questions. Your perspectives on how AI has already been integrated into vulnerability detection workflows, and the clear boundaries you've identified between where AI can and can't help in hardware security research, were particularly insightful. 

Is there anything else about AI in your research work that we haven't touched on that you think would be important for us to understand?

User: One thing I actually do wonder is how AI is going to look in cyberphysical systems and how it could change the game. Drones are the particular cyber physical system we work with, but a lot of developers of the firmware and even the backend and UI systems for drones and similar devices are turning to AI to develop their software more efficiently.  A lot of folks even suspect that in the next 10 years, maybe all of this software will be produced by AI and human developers will have minimal involvement. My questions are if this would remove a lot of the CWEs we see right now, but also create new ones.  Or if maybe the game will become compromising the AI systems that write these firmware and software to include flaws and backdoors within (although this would be a much more difficult task and frankly much hard to study as a researcher).  It will be very interesting to see what happens in the next few years.  I am still new to this career and I wonder how different it will look by the team I reach the age of my mentors and professors.

AI: That's a really fascinating perspective - you're essentially looking at AI as potentially reshaping both sides of the cybersecurity equation. On one hand, AI-generated code might eliminate some human-introduced vulnerabilities, but on the other hand, it could create entirely new attack surfaces or classes of vulnerabilities we haven't seen before.

And that point about potentially targeting the AI systems themselves to introduce flaws during the code generation process is particularly intriguing - that would represent a completely different threat model than what you're working with today.

Thank you so much for sharing your experiences and insights so openly. Your observations about how AI has already become integrated into your vulnerability detection workflow, the clear physical limitations you've identified, and especially your forward-looking thoughts about how AI might fundamentally change the cybersecurity landscape - all of this really helps us understand the nuances of how AI fits into real security research work.

If you have any questions about this research or would like to follow up on anything we discussed, please don't hesitate to reach out. Thanks again for your time!