Prioritizing Technical Governance Investments: Verification
Summary
Robert Trager argues that as transformative AI develops, rigorous verification—both internal and external—is essential to prevent actors from compromising safety standards for performance gains.
SESSION Transcript
I want to talk about verification and the critical need, I think, for verification. I think it's more and more accepted today by more and more people that maybe we're on the cusp of really transformative AI maybe pretty soon, and that that might have huge impacts on geopolitics.
But I think maybe a lot of people's response to that is great, let's figure out how to make it safe and secure. I feel really grateful that a lot of people are responding in that way. But I also think it's important that we recognize that probably there are some other things, in fact, a long list of other things that we also have to do from a technical perspective and of course from a social perspective in order to get to a flourishing AI future.
I want to talk about verification, which in fact, two forms of verification, which I think may be close to the top of that list of other things that we need to be working on. So what is verification from the point of view of computer scientists? There's a whole field of verification that has that as its name. And it's sort of interrogating a system that you have control over.
It's like saying, what were you doing, system? What were you doing last night? Do you have a set of properties that I want you to have? It's very closely related to saying, did I build the system correctly?
The interesting thing is that there's this other way that the term verification is used, which is closely related, but different. That is, you're still asking about the properties of a system and whether there are guarantees that the system has a set of properties, but it's not a system that you have complete control over.
Somebody else has control of that system and maybe that somebody else is trying to signal to you as the verifier that in fact that system has a set of properties. And that opens up a whole set of technical challenges in addition to the technical challenges of the first sort of verification. So I think that both sorts of verification are very important for different reasons. I just want to give a sense of one reason why I think they're important, and particularly the second one that relates to this thing that you might call a system security system performance trade-off.
This exists across many technology races. Sometimes these things go together, but often they don't go together, or sometimes they don't go together. You might think about the rush to get a system to market. How long are you going to test a system?
Well, if you're taking a long time for safety, then that's lowering performance because the system isn't yet deployed. In fact, we see, for a variety of reasons, even more fundamental and more interesting trade-offs between system security and system performance. The thing that I think most people would agree there, but something that they don't quite realize. What are the implications of that?
If we really think through implications, we can have a situation like what you see on this graph where you see these two schedules. You might be at that first star on the blue line there. Maybe that's the equilibrium where you're choosing this relationship. There are a bunch of competing actors choosing this relationship between performance and system security.
Then somebody comes along and they say, okay, we can show you, we have great safety insight. We can show you how to do things safer at each level of performance. But then you just end up sliding down that new trade-off curve to the new star on the green line. Verification is sort of the way that maybe we can avoid doing that, avoid sliding down there if we can be sure that the other actors are doing something similar.
There are some complicated strategic considerations here. My last 20 seconds. I just want to say that I really think we need both forms of verification. Both home system verification where compute providers can be intermediaries, regulatory intermediates, the way that financial institutions are in those spaces, and also remote system verification so that we can, when we need them, have the sort of agreements that people were talking about earlier today in terms of agreements on the SP curve.
So I hope that some of you will work on those topics. Thank you very much.
But I think maybe a lot of people's response to that is great, let's figure out how to make it safe and secure. I feel really grateful that a lot of people are responding in that way. But I also think it's important that we recognize that probably there are some other things, in fact, a long list of other things that we also have to do from a technical perspective and of course from a social perspective in order to get to a flourishing AI future.
I want to talk about verification, which in fact, two forms of verification, which I think may be close to the top of that list of other things that we need to be working on. So what is verification from the point of view of computer scientists? There's a whole field of verification that has that as its name. And it's sort of interrogating a system that you have control over.
It's like saying, what were you doing, system? What were you doing last night? Do you have a set of properties that I want you to have? It's very closely related to saying, did I build the system correctly?
The interesting thing is that there's this other way that the term verification is used, which is closely related, but different. That is, you're still asking about the properties of a system and whether there are guarantees that the system has a set of properties, but it's not a system that you have complete control over.
Somebody else has control of that system and maybe that somebody else is trying to signal to you as the verifier that in fact that system has a set of properties. And that opens up a whole set of technical challenges in addition to the technical challenges of the first sort of verification. So I think that both sorts of verification are very important for different reasons. I just want to give a sense of one reason why I think they're important, and particularly the second one that relates to this thing that you might call a system security system performance trade-off.
This exists across many technology races. Sometimes these things go together, but often they don't go together, or sometimes they don't go together. You might think about the rush to get a system to market. How long are you going to test a system?
Well, if you're taking a long time for safety, then that's lowering performance because the system isn't yet deployed. In fact, we see, for a variety of reasons, even more fundamental and more interesting trade-offs between system security and system performance. The thing that I think most people would agree there, but something that they don't quite realize. What are the implications of that?
If we really think through implications, we can have a situation like what you see on this graph where you see these two schedules. You might be at that first star on the blue line there. Maybe that's the equilibrium where you're choosing this relationship. There are a bunch of competing actors choosing this relationship between performance and system security.
Then somebody comes along and they say, okay, we can show you, we have great safety insight. We can show you how to do things safer at each level of performance. But then you just end up sliding down that new trade-off curve to the new star on the green line. Verification is sort of the way that maybe we can avoid doing that, avoid sliding down there if we can be sure that the other actors are doing something similar.
There are some complicated strategic considerations here. My last 20 seconds. I just want to say that I really think we need both forms of verification. Both home system verification where compute providers can be intermediaries, regulatory intermediates, the way that financial institutions are in those spaces, and also remote system verification so that we can, when we need them, have the sort of agreements that people were talking about earlier today in terms of agreements on the SP curve.
So I hope that some of you will work on those topics. Thank you very much.
