[Fireside Chat] Control & Computer Security
Summary
Former FBI special agent Steve Kelly explains how national security’s approach to insider threats—including rigorous vetting, continuous monitoring, and incident response planning—offers valuable lessons for AI control.
SESSION Transcript
Buck Shlegeris: Okay, great, let's get started. Hello, everyone, this is Steve Kelly. Steve was—yeah, my intro of you—so you were at the FBI for a long time?
Steve Kelly: Yeah, so retired FBI special agent.
Buck Shlegeris: Yeah, you did—you reported to the NSC for the FBI. Sorry, the American National Security Council for the FBI, where the FBI is the American Federal Bureau of Investigation, which among other things is in charge of counterintelligence on American soil.
Steve Kelly: That's right.
Buck Shlegeris: And has had a bunch of brutal insider threat problems over the years. So, the most famous of which—I guess there's a bunch of famous ones, but Robert Hanson was a famous mole inside the FBI. You were around slightly after \—slightly after him.
Steve Kelly: Yeah, so I joined the FBI at the end of 2001, and had ended up working in cybersecurity and cybercrime investigations, cyber national security investigations for the last 22 years. I retired a little less than two years ago, but my foray through cyber security ended up leading to headquarters jobs and then interacting with national policymaking around cybersecurity, and that led me to emerging technology, quantum and AI.
And I now work at a think tank called the Institute for Security Technology, which is a US-based think tank working on these sorts of problems. So, thrilled to be here, Buck. But yes, the Robert Hanson era was a little bit before I showed up, but that is one of the more notorious insider threat cases which led to massive losses of various sensitive information.
Buck Shlegeris: Yeah. So, I guess one question I'm interested in starting with. So, I feel like America has—the American national security establishment has had a bunch of real brutal insider threat failures over the years. There was the Robert Hanson one in 2001, there was Snowden in 2008, a couple other ones of these. So, something that I've found a little tricky to figure out as just a random person who's not really a professional in this space.
Is it just a really hard problem? FBI has a million—there's millions of people with security clearances, or does the American just—the American National security establishment particularly have trouble mitigating insider threats? How much would you say?
Steve Kelly: It's a legitimate question. And so, not to put all this on the FBI, the FBI has had its share of incidents. Some of these other ones, Aldrich Ames and Chelsea Manning, which is the WikiLeaks case, Edward Snowden, which was an NSA issue. There have been examples of security breaches throughout different parts of the U.S. government. But I think, and the purpose of this talk is to figure out what can we learn from U.S. government security programs, insider threat programs in particular, and how does that apply to AI control?
And so, is the U.S. government good at this? Better than this than most countries? Worse than most countries? How does it compare to corporate security programs? That's a great question. I don't think anybody actually knows the answer. The U.S. government does have four million clearance holders. So, it is a massive number of people that have access to sensitive classified information. And there's a risk-based approach, where you've got information that's at the normal for official use only level that is protected, kind of.
And then through confidential, secret, top secret, and then compartments above that. And so, more protections are put in as you get to the higher and higher levels, and because the gravity of a breach would be more and more consequential, so the investment in keeping that information secret is ratcheted up. I can't say, I don't think anybody knows as to whether the U.S. government is the best at this or middling or average compared to other countries. But there aren't that many incidents that show up in the news.
There's one recently where some military service members have been charged with leaking information that they took from systems and pictures they took on a base to the Chinese government. But that happens periodically, and typically those do make news, because there's a prosecution that's associated with that, and so who knows, in terms of—because what's the measuring stick? But I certainly do want to talk about what can we learn from those incidents and their security approach in general.
Buck Shlegeris: Yes. So, maybe can you say something about like, how does the American National Security Establishment try to handle insider threat in general?
Steve Kelly: Yeah. Okay, so I'll make one other caveat since this is being recorded and we are here kind of in a non-secure environment. I will speak about kind of security programs that's informed by public information that the U.S. government's put out about how it does business. And also, this is applicable to how companies operate their security programs because it's very, very similar. Everybody's got some sort of intellectual property or trade secrets that they're trying to keep safe.
So, I won't go beyond the things that I'm not allowed to talk about. But in general, there's kind of the before, during, and after types of activities. And starting with before—and then I'll try to frame this with how does this relate to AI control? The before is in employee vetting. So, we want to hire somebody to do a job that's quite sensitive, I need a person that's competent, I need a person who's enthusiastic about the mission, and doesn't have, you know, a substance abuse problem, doesn't have a criminal record, seems to be an upstanding citizen, and so that is all that sort of vetting up front. Most companies do that, check their social media, etc.
So that we can make sure we're aligned. And once we're done with that, we onboard the employee, and then we'll see, we could be right, we could be wrong. It could be somebody that, a competitor, or a foreign intelligence service has kind of insinuated into our, into our hiring process. And then we get into, well, what are then the controls that you put on to make sure that information stays secret, or that bad things don't happen, like sabotage and that sort of thing.
And so, then you get into some of the traditional security domains. We already touched on kind of personnel security, that's that employee vetting, that doesn't stop, that's a continuous process. And then information security controls, and implementing concepts like the need-to-know, and the least privilege principle, making sure that an individual that is working on a particular model, or a particular project doesn't need to know and have access to all the HR data or all the financial data. So, accounts have those proper protections in there, and then you certainly want to audit that and make sure things are working.
And then on top of that, physical security as well. The person that's working on project X doesn't also need to be able to get into the necessarily the offsite server farm, or the airplane hangar, unless they have a role in that. And so, all of these controls help to kind of manage the risk, manage physical assets, manage intellectual assets.
And then on top of that, if we have kind of a high risk or high impact mission, we got to assume that our phase one activity may have failed. And so, if you have a large organization, let's assume that an insider is in there. If somebody wants to do harm, either because they weren't aligned in the first place, or maybe their alignment shifted. They started to have relationship problems, they started to have financial problems, they became kind of disgruntled because they see what it's like on the inside of the organization, and they're like, “Damn these people,” and “I'm going to get them.”
And next thing you know, they're doing things, they're stealing information. So, we have to assume they're there, so we have to start monitoring for that. And that's where U.S. government has created insider threat programs, this is public information. It was after the —actually, it was after the Chelsea Manning WikiLeaks activity, there was a whole reform effort that was going, to increase security controls around classified information. And in the midst of trying to implement that, Snowden happened. So, it's like, for the love of Pete. It's like that times 10, because one of them involved secret information off of a secret enclave, and then the next one involved sensitive compartmented information, it was a massively damaging breach.
And so, at the time, lots of agencies had lots of programs to try to ferret this out, but there was not uniformity, and there are some smaller agencies that are consumers of classified information and analysis, but weren't necessarily the ones with the mission to collect it. And so, it was making sure that all agencies, to a level of standards, had an insider threat program, so that there's somebody responsible for looking for the threat that, let's assume, is there. Because otherwise, you're waiting for something to happen, and in terms of how you run your security program, from the physical security to the information system security, to the personnel security.
Some behaviour might not trip. And so, we need some—we need a group of people that are focused on looking for it, and looking for the anomalies. And so, that all exists now. It didn't always exist, at least uniformly, across the government.
Buck Shlegeris: Okay, so one question here, which is very analogous to our question of how much effort we should put into AI alignment versus AI control. How much of the effort at mitigating insider threat goes into the vetting of people on the way in, versus the control measures that you have during their employment?
Steve Kelly: Yeah, well, I mean, in terms of the way that the terms are used, when you talk about a U.S. government's insider threat program, it's not the initial vetting, you know, that is elsewhere in the organization, that's within security division. Interestingly, in terms of reporting chains, it's a good idea to have an insider threat program not be run necessarily by your security apparatus, but maybe a slightly independent team, in the same way that your risk and audit committees in a corporate structure would be distinct, because you don't want self-dealing, and you don't want conflicts of interest, you need a little bit of independence.
And so, the personnel vetting, that's being handled by a security program, or security officer, or an FBI, it's a security division runs that. And the insider threat program…
Buck Shlegeris: In terms of the overall cost?
Steve Kelly: Oh, the overall.
Buck Shlegeris: In terms of the effort, like that the establishment overall has?
Steve Kelly: I can't say for sure, but I know that probably the security program more generally is many times the magnitude of what an insider threat program is, because that is a smaller, more boutique team of people that have access to all the feeds and information and data sources, so they can chase down the activity that they're sensing, and that they can conduct the investigation into that. But it's a fraction of the investment, because in the government, doing background invest, just the clearance investigation, going back to, sometimes the age of 18 and knocking on doors, talking to neighbours-
Buck Shlegeris: What is the list of every foreigner you have ever been friends with?
Steve Kelly: It's brutal.
Buck Shlegeris: Every address you've ever had.
Steve Kelly: It's so convenient when a new hire has never left the country, it makes it so easy.
Buck Shlegeris: Yeah. Okay, cool. So, it sounds like you're saying—yeah, so sorry, what's the overall distribution there? It sounds like-
Steve Kelly: I don't have—I can't give you a number-
Buck Shlegeris: -kind of similar-Ish. It's not clear that one's bigger than the other.
Steve Kelly: Yeah.
Buck Shlegeris: Okay.
Steve Kelly: I couldn't say whether the vetting or whether the insider threat monitoring topic are equally dispersed or not.
Buck Shlegeris: Cool.
Steve Kelly: I don't know the answer to that question.
Buck Shlegeris: Okay, so here's another question. Something which seems like a big issue to me, in general, is you've got a constant trade-off when you're implementing access controls between your employees being productive and having a convenient and also pleasant time doing their work, and preventing insider threats. So, one of the things which went wrong in the Snowden case was that Snowden had asked a bunch of his other NSA people for their passwords, to systems that they had access to, that he didn't have access to, claiming that he needed those passwords in order to do his job correctly.
And you can see why it's hard to stop this if you run an organization, because it's really good to have, inside an organization, people trying to do their jobs well. And it's like... There's just constant pressure to have more permissions for people, so that they can do their jobs better. And then maybe this is like one reason to try to separate out the insider threat program. So, yeah, how do people try to handle the fact that people are constantly going to be pushing for their sub-org to have less access controls?
Steve Kelly: Right. Well, it's absolutely true that security is one over convenience. It is. And the controls that are put in organizations that have highly sensitive IP, or in the government that is processing classified information at a high level, it's absolutely brutal. I mean, in some cases, there's zero work from home. I did not leave the office once during COVID. Every day in the office, working in a secure area with access to secure systems, brutally inconvenient. You can't take your phone into the office. So, your wife can text you all day long, I'm not seeing it, it's out there. It's very inconvenient. And it's just a cost of doing business for the government.
I know that in the private sector, especially in startups, you might be thinking, this idea, we really need to protect this, we don't have any money for a security program. Well, that's just a cost borne by the taxpayer when it comes to national security information. And a lot of security mistakes are made because people are trying to do shortcuts, because it's inconvenient, and they're trying to figure out a way to get information from here to there, and they end up taking information from one context to another context, the next thing you know, you have a spill.
So, there's lots of more minor violations that can be kind of dealt with, and most of those are just people trying to get the job done, not kind of malicious insider threat types of intentional stuff.
Buck Shlegeris: But presumably, they make it harder to detect the insider threat. Like, if people are constantly pushing against the security boundaries in order to do their job, it's a lot harder to tell when something's malicious activity.
Steve Kelly: Well, folks are not supposed to be pushing against the controls. [Laughter]
[Audience laughter]
And of course, some of the, like you had mentioned, and I'm not going to kind of get into the facts and circumstance of the Snowden disclosures, but certainly nowadays, it would not be possible just to share a password, because there's multi-factor authentication on every enclave. And so, this is just a fact of life. And I think there's plenty of circumstances in the tech sector where more focus needs to be put on protecting the technology that's being developed, and I just don't think the maturity of the organizations has yet caught up. And then the business model pressures of how much can we invest in that? That's a real barrier to protecting some of this emerging tech that is not yet dealt with.
Buck Shlegeris: Yeah. I think another thing is that engineers are just spoiled, and they just don't like it when things are annoying, and I think that's an underrated difficulty for, in tech.
Steve Kelly: [Laughter]
Buck Shlegeris: Whereas you poor people are just used to like-
Steve Kelly: [Laughter]
Buck Shlegeris: Of course, I'm not able to use computers in my job, I have to-
Steve Kelly: I think there's a bunch of folks at this conference that probably wouldn't do well in the J. Edgar Hoover era of the FBI, where if you wore other than a white shirt, you got in trouble. [Laughter]
Buck Shlegeris: Yeah.
Steve Kelly: I don't know if that haircut would have made it. [Laughter]
[Audience Laughter]
Buck Shlegeris: Yeah. Okay, so here's a question. So, a thing that we talk a lot about in AI control is kind of adversarial evaluations of security procedures, where you try to decide whether security procedures are good by having some kind of adversarial auditing.
Steve Kelly: Sure.
Buck Shlegeris: So, to what extent is adversarial auditing of processes? You try to check that your processes are good by paying someone to try to break them. To what extent is this type of adversarial auditing process, to what extent is that used to assess security measures?
Steve Kelly: Yeah, there's lots of examples of where that is done, and good examples are in the context of airport security and passenger screening in the U.S., the Transportation Security Administration is doing all the work. And so, the objective being that firearms and knives of any length are not going to pass across this line. And so, therefore, you create a security program that seeks to accomplish that. And so, magnetometers and x-rays and that sort of thing.
And then, once in a while, TSA will test their employees by running stuff through, and the question is, did they find it? And so, that's a red team evaluation, it's really easy to do, because what they're looking for is very specific and finite. Yeah.
Buck Shlegeris: So, who? Is it a sub-team of the TSA that does the red teaming, or…?
Steve Kelly: I don't know TSA intimately, so I won’t—I can't speak to that. But I assume that that's a separate group that is part of their kind of audit function.
Buck Shlegeris: Yeah, I was very surprised when you first told me about the TSA doing this red teaming. It seems the TSA seems so arbitrary in their procedures. And they really seem like an organization that is obeying a set of processes, rather than being allowed to choose whatever processes they like, subject to the constraints that the red teamers can't get guns through or whatever. Yeah, I don't know, I was surprised by this.
Steve Kelly: Yeah. And typically, the red teaming topic, like the U.S. government does a fair amount of red teaming to evaluate their cyber security posture. And that's well established, that makes sense. It's harder to red team a human. And so, we had talked a little bit about in our prep, do you red team a person? Do you do a…
Buck Shlegeris: Sting operation.
Steve Kelly: A sting operation. And it's like, uh. So, certainly in the context of criminal invest, and national security investigations, sometimes you do a sting. Like if a law enforcement agency's investigating a drug organization, you could insert an undercover employee or operate some sort of a source, and get involved and do transactions. And then you are part of it, you're paying for X, or you're selling contraband or whatever. And there's an example, ATF, there was a scandal around automatic firearms going across the Southwest border.
And so, those are sting operations, but highly costly, and there are some risks involved, and also, there's the concern around entrapment, are you inducing somebody to do something that they weren't otherwise predisposed to do. So normally, not just normally, I think every time, there needs to be sufficient predication to open up a full investigation, to get to the point where you're doing that kind of a sting.
And so, the question is, in an insider threat scenario, where you're trying to find the insider threat, do you choose an employee at random, and say, we're going to run a trial against them, we're going to see if we can get them to sell us, or to give us classified information. That would be highly unusual, I think, and generally speaking, it would need to be well predicated, that there's reasonable suspicion that that person is having a problem. And so, I think that's harder. The computer system you're red teaming, or the AI model you're red teaming doesn't have constitutional rights, the employee does.
Buck Shlegeris: Yeah, so there's these two points, where one is that the sting operations are really expensive, and the other is that you're not allowed to entrap people.
Steve Kelly: Right.
Buck Shlegeris: Because of the constitution.
Steve Kelly: Right. And I don't think that—that would leave a bad taste in my mouth, if all of a sudden, these scenarios, like a honey trap, all of a sudden, attractive ladies are coming up to me, well, that doesn't ever happen, so why is this happening?
[Audience Laughter]
Steve Kelly: Yeah, not good.
Buck Shlegeris: Yeah, when we were talking about this when prepping, I just actually hadn't thought of the fact that you're not allowed to just entrap random humans. [Laughter]
Steve Kelly: [Laughter]
Buck Shlegeris: I'm always thinking in the regime of these AIs, where you can kind of do what you want. Yeah, that is very interesting. So, wouldn't you—I would have thought that you are allowed, so you're allowed to do phishing, like phishing tests on your employees?
Steve Kelly: Oh yeah, yeah, that happens all the time, sure.
Buck Shlegeris: And presumably you can fire—presumably, it's not a problem to fire people based on failing sting operations.
Steve Kelly: I have never heard, but I've actually, they get pretty clever, I fell for one a few years ago. I was like, “God, that was a good one.” Yeah, not good, sorry, maybe that's an admission I shouldn't make. But that's part of security training, and to the extent that people bite on it, it gives the organization information as to where they need to close gaps in terms of employee awareness. And it certainly alerts the employee to, you failed, maybe you should take this little web-based training and get better at this. But I've never heard of anyone getting fired because they bit on a phishing case.
Buck Shlegeris: Yeah, yeah, it is interesting, I guess something else I learned about-
Steve Kelly: It's embarrassing, but not fireable.
Buck Shlegeris: Yeah, something I learned is generally, or a lot of the time, insiders inside the FBI or whatever, trying to sell information to foreign governments try pretty hard to keep their identities secret. Like it would actually be a lot—I was surprised by how much effort it would be to do sting operations on people. Like it's very rare that, like the FBI insiders are typically not people who just get approached by some Russian in a bar in D.C. and then agree to sell secrets.
Steve Kelly: Yeah, there are plenty of books written on the topic. Yeah, the cycle of recruiting a source or an asset, and how long that takes. And there's real skill to it; folks that do human intelligence work, folks like CIA and FBI domestically, that's a skill.
Buck Shlegeris: Yeah, and you can’t just-
Steve Kelly: And it's not, it’s not quick.
Buck Shlegeris: Yeah, which makes sense. There's kind of proof of work. And one of the core differences between the humans and the AIs, of course, is that we get to amortize that work, because you get like so much work out of the AIs, the same AI system, it's like effectively an employee who lives for a million years, and if your employees lived for a million years, you really could afford to spend 1,000 person years of sting operation effort on them.
Steve Kelly: Sure, sure, you could replicate them and do more. If you can afford the compute, go for it. So, the thing we haven't talked about is, I think— I can't remember if I said it, but I think there's a third phase after kind of the what's happening now. And that is, I've heard left of boom, right of boom happen, being discussed here, or maybe that was at my cybersecurity conference earlier in the week. But the third phase, which is something happens. And so, inevitably, you're not going to solve all alignment problems, inevitably, your controls are going to fail at some point, and you're going to be in a position where some bad act occurred.
And so, in any organization, whether that's a ransomware attack or some other, you know, a hurricane hits your facility, you've got to have business continuity, disaster recovery and crisis management plans to deal with that. In the same way, I'm kind of wondering coming out of this conference in the AI-control topic, what happens and what are we going to do, or what plans are we going to put in place if there's a loss-control incident? And so, all of that also exists within organizations and the U.S. government.
Sometimes it's aligned with your security program, but some of that stuff is actually not security specifically, but those are important plans to have in place, and you've got to be thinking about that next phase.
Buck Shlegeris: Yeah, so I have a blog post: “Catching AI's Red-handed”, which is about what you should do if you catch your AI's red handed. And in a podcast, an 80,000 Hours podcast that is coming out in the next few days, I talk about the plan for after the AI escapes, how you try to hunt it down and stuff, which you might be interested in on this topic.
Steve Kelly: That would be fantastic.
Buck Shlegeris: It obviously seems like a kind of rough position to end up in. Obviously, a big part of what you do with your employees when you're suspicious of them, my understanding is you remove their privileges, or Hanson, the mole inside the FBI was moved to a IT supervisor role for a while or whatever before they arrested him, where he didn't have access to sensitive information anymore. And in the AI case, of course, it's more like you become suspicious of all of your employees, and you simultaneously—it's hard to simultaneously fire everybody.
So yeah, there's some interesting disanalogies there. So, maybe it's not that different to becoming convinced that some big suborg is deeply compromised. And that's the thing, which I guess the national security establishment is familiar with.
Steve Kelly: Yeah, I couldn't say. But that would be a horrible situation, it's bad enough to have one well-placed employee go rogue, to have an entire chunk of an organization, or a large AI capability that's running out of control, that would be a problem.
Buck Shlegeris: Cool.
Steve Kelly: Yeah.
Buck Shlegeris: Well, we're out of time.
Steve Kelly: Okay.
Buck Shlegeris: Thanks very much.
Steve Kelly: You're welcome, happy to be here.
Buck Shlegeris: Yeah.
[Audience Applause]
Steve Kelly: Yeah, so retired FBI special agent.
Buck Shlegeris: Yeah, you did—you reported to the NSC for the FBI. Sorry, the American National Security Council for the FBI, where the FBI is the American Federal Bureau of Investigation, which among other things is in charge of counterintelligence on American soil.
Steve Kelly: That's right.
Buck Shlegeris: And has had a bunch of brutal insider threat problems over the years. So, the most famous of which—I guess there's a bunch of famous ones, but Robert Hanson was a famous mole inside the FBI. You were around slightly after \—slightly after him.
Steve Kelly: Yeah, so I joined the FBI at the end of 2001, and had ended up working in cybersecurity and cybercrime investigations, cyber national security investigations for the last 22 years. I retired a little less than two years ago, but my foray through cyber security ended up leading to headquarters jobs and then interacting with national policymaking around cybersecurity, and that led me to emerging technology, quantum and AI.
And I now work at a think tank called the Institute for Security Technology, which is a US-based think tank working on these sorts of problems. So, thrilled to be here, Buck. But yes, the Robert Hanson era was a little bit before I showed up, but that is one of the more notorious insider threat cases which led to massive losses of various sensitive information.
Buck Shlegeris: Yeah. So, I guess one question I'm interested in starting with. So, I feel like America has—the American national security establishment has had a bunch of real brutal insider threat failures over the years. There was the Robert Hanson one in 2001, there was Snowden in 2008, a couple other ones of these. So, something that I've found a little tricky to figure out as just a random person who's not really a professional in this space.
Is it just a really hard problem? FBI has a million—there's millions of people with security clearances, or does the American just—the American National security establishment particularly have trouble mitigating insider threats? How much would you say?
Steve Kelly: It's a legitimate question. And so, not to put all this on the FBI, the FBI has had its share of incidents. Some of these other ones, Aldrich Ames and Chelsea Manning, which is the WikiLeaks case, Edward Snowden, which was an NSA issue. There have been examples of security breaches throughout different parts of the U.S. government. But I think, and the purpose of this talk is to figure out what can we learn from U.S. government security programs, insider threat programs in particular, and how does that apply to AI control?
And so, is the U.S. government good at this? Better than this than most countries? Worse than most countries? How does it compare to corporate security programs? That's a great question. I don't think anybody actually knows the answer. The U.S. government does have four million clearance holders. So, it is a massive number of people that have access to sensitive classified information. And there's a risk-based approach, where you've got information that's at the normal for official use only level that is protected, kind of.
And then through confidential, secret, top secret, and then compartments above that. And so, more protections are put in as you get to the higher and higher levels, and because the gravity of a breach would be more and more consequential, so the investment in keeping that information secret is ratcheted up. I can't say, I don't think anybody knows as to whether the U.S. government is the best at this or middling or average compared to other countries. But there aren't that many incidents that show up in the news.
There's one recently where some military service members have been charged with leaking information that they took from systems and pictures they took on a base to the Chinese government. But that happens periodically, and typically those do make news, because there's a prosecution that's associated with that, and so who knows, in terms of—because what's the measuring stick? But I certainly do want to talk about what can we learn from those incidents and their security approach in general.
Buck Shlegeris: Yes. So, maybe can you say something about like, how does the American National Security Establishment try to handle insider threat in general?
Steve Kelly: Yeah. Okay, so I'll make one other caveat since this is being recorded and we are here kind of in a non-secure environment. I will speak about kind of security programs that's informed by public information that the U.S. government's put out about how it does business. And also, this is applicable to how companies operate their security programs because it's very, very similar. Everybody's got some sort of intellectual property or trade secrets that they're trying to keep safe.
So, I won't go beyond the things that I'm not allowed to talk about. But in general, there's kind of the before, during, and after types of activities. And starting with before—and then I'll try to frame this with how does this relate to AI control? The before is in employee vetting. So, we want to hire somebody to do a job that's quite sensitive, I need a person that's competent, I need a person who's enthusiastic about the mission, and doesn't have, you know, a substance abuse problem, doesn't have a criminal record, seems to be an upstanding citizen, and so that is all that sort of vetting up front. Most companies do that, check their social media, etc.
So that we can make sure we're aligned. And once we're done with that, we onboard the employee, and then we'll see, we could be right, we could be wrong. It could be somebody that, a competitor, or a foreign intelligence service has kind of insinuated into our, into our hiring process. And then we get into, well, what are then the controls that you put on to make sure that information stays secret, or that bad things don't happen, like sabotage and that sort of thing.
And so, then you get into some of the traditional security domains. We already touched on kind of personnel security, that's that employee vetting, that doesn't stop, that's a continuous process. And then information security controls, and implementing concepts like the need-to-know, and the least privilege principle, making sure that an individual that is working on a particular model, or a particular project doesn't need to know and have access to all the HR data or all the financial data. So, accounts have those proper protections in there, and then you certainly want to audit that and make sure things are working.
And then on top of that, physical security as well. The person that's working on project X doesn't also need to be able to get into the necessarily the offsite server farm, or the airplane hangar, unless they have a role in that. And so, all of these controls help to kind of manage the risk, manage physical assets, manage intellectual assets.
And then on top of that, if we have kind of a high risk or high impact mission, we got to assume that our phase one activity may have failed. And so, if you have a large organization, let's assume that an insider is in there. If somebody wants to do harm, either because they weren't aligned in the first place, or maybe their alignment shifted. They started to have relationship problems, they started to have financial problems, they became kind of disgruntled because they see what it's like on the inside of the organization, and they're like, “Damn these people,” and “I'm going to get them.”
And next thing you know, they're doing things, they're stealing information. So, we have to assume they're there, so we have to start monitoring for that. And that's where U.S. government has created insider threat programs, this is public information. It was after the —actually, it was after the Chelsea Manning WikiLeaks activity, there was a whole reform effort that was going, to increase security controls around classified information. And in the midst of trying to implement that, Snowden happened. So, it's like, for the love of Pete. It's like that times 10, because one of them involved secret information off of a secret enclave, and then the next one involved sensitive compartmented information, it was a massively damaging breach.
And so, at the time, lots of agencies had lots of programs to try to ferret this out, but there was not uniformity, and there are some smaller agencies that are consumers of classified information and analysis, but weren't necessarily the ones with the mission to collect it. And so, it was making sure that all agencies, to a level of standards, had an insider threat program, so that there's somebody responsible for looking for the threat that, let's assume, is there. Because otherwise, you're waiting for something to happen, and in terms of how you run your security program, from the physical security to the information system security, to the personnel security.
Some behaviour might not trip. And so, we need some—we need a group of people that are focused on looking for it, and looking for the anomalies. And so, that all exists now. It didn't always exist, at least uniformly, across the government.
Buck Shlegeris: Okay, so one question here, which is very analogous to our question of how much effort we should put into AI alignment versus AI control. How much of the effort at mitigating insider threat goes into the vetting of people on the way in, versus the control measures that you have during their employment?
Steve Kelly: Yeah, well, I mean, in terms of the way that the terms are used, when you talk about a U.S. government's insider threat program, it's not the initial vetting, you know, that is elsewhere in the organization, that's within security division. Interestingly, in terms of reporting chains, it's a good idea to have an insider threat program not be run necessarily by your security apparatus, but maybe a slightly independent team, in the same way that your risk and audit committees in a corporate structure would be distinct, because you don't want self-dealing, and you don't want conflicts of interest, you need a little bit of independence.
And so, the personnel vetting, that's being handled by a security program, or security officer, or an FBI, it's a security division runs that. And the insider threat program…
Buck Shlegeris: In terms of the overall cost?
Steve Kelly: Oh, the overall.
Buck Shlegeris: In terms of the effort, like that the establishment overall has?
Steve Kelly: I can't say for sure, but I know that probably the security program more generally is many times the magnitude of what an insider threat program is, because that is a smaller, more boutique team of people that have access to all the feeds and information and data sources, so they can chase down the activity that they're sensing, and that they can conduct the investigation into that. But it's a fraction of the investment, because in the government, doing background invest, just the clearance investigation, going back to, sometimes the age of 18 and knocking on doors, talking to neighbours-
Buck Shlegeris: What is the list of every foreigner you have ever been friends with?
Steve Kelly: It's brutal.
Buck Shlegeris: Every address you've ever had.
Steve Kelly: It's so convenient when a new hire has never left the country, it makes it so easy.
Buck Shlegeris: Yeah. Okay, cool. So, it sounds like you're saying—yeah, so sorry, what's the overall distribution there? It sounds like-
Steve Kelly: I don't have—I can't give you a number-
Buck Shlegeris: -kind of similar-Ish. It's not clear that one's bigger than the other.
Steve Kelly: Yeah.
Buck Shlegeris: Okay.
Steve Kelly: I couldn't say whether the vetting or whether the insider threat monitoring topic are equally dispersed or not.
Buck Shlegeris: Cool.
Steve Kelly: I don't know the answer to that question.
Buck Shlegeris: Okay, so here's another question. Something which seems like a big issue to me, in general, is you've got a constant trade-off when you're implementing access controls between your employees being productive and having a convenient and also pleasant time doing their work, and preventing insider threats. So, one of the things which went wrong in the Snowden case was that Snowden had asked a bunch of his other NSA people for their passwords, to systems that they had access to, that he didn't have access to, claiming that he needed those passwords in order to do his job correctly.
And you can see why it's hard to stop this if you run an organization, because it's really good to have, inside an organization, people trying to do their jobs well. And it's like... There's just constant pressure to have more permissions for people, so that they can do their jobs better. And then maybe this is like one reason to try to separate out the insider threat program. So, yeah, how do people try to handle the fact that people are constantly going to be pushing for their sub-org to have less access controls?
Steve Kelly: Right. Well, it's absolutely true that security is one over convenience. It is. And the controls that are put in organizations that have highly sensitive IP, or in the government that is processing classified information at a high level, it's absolutely brutal. I mean, in some cases, there's zero work from home. I did not leave the office once during COVID. Every day in the office, working in a secure area with access to secure systems, brutally inconvenient. You can't take your phone into the office. So, your wife can text you all day long, I'm not seeing it, it's out there. It's very inconvenient. And it's just a cost of doing business for the government.
I know that in the private sector, especially in startups, you might be thinking, this idea, we really need to protect this, we don't have any money for a security program. Well, that's just a cost borne by the taxpayer when it comes to national security information. And a lot of security mistakes are made because people are trying to do shortcuts, because it's inconvenient, and they're trying to figure out a way to get information from here to there, and they end up taking information from one context to another context, the next thing you know, you have a spill.
So, there's lots of more minor violations that can be kind of dealt with, and most of those are just people trying to get the job done, not kind of malicious insider threat types of intentional stuff.
Buck Shlegeris: But presumably, they make it harder to detect the insider threat. Like, if people are constantly pushing against the security boundaries in order to do their job, it's a lot harder to tell when something's malicious activity.
Steve Kelly: Well, folks are not supposed to be pushing against the controls. [Laughter]
[Audience laughter]
And of course, some of the, like you had mentioned, and I'm not going to kind of get into the facts and circumstance of the Snowden disclosures, but certainly nowadays, it would not be possible just to share a password, because there's multi-factor authentication on every enclave. And so, this is just a fact of life. And I think there's plenty of circumstances in the tech sector where more focus needs to be put on protecting the technology that's being developed, and I just don't think the maturity of the organizations has yet caught up. And then the business model pressures of how much can we invest in that? That's a real barrier to protecting some of this emerging tech that is not yet dealt with.
Buck Shlegeris: Yeah. I think another thing is that engineers are just spoiled, and they just don't like it when things are annoying, and I think that's an underrated difficulty for, in tech.
Steve Kelly: [Laughter]
Buck Shlegeris: Whereas you poor people are just used to like-
Steve Kelly: [Laughter]
Buck Shlegeris: Of course, I'm not able to use computers in my job, I have to-
Steve Kelly: I think there's a bunch of folks at this conference that probably wouldn't do well in the J. Edgar Hoover era of the FBI, where if you wore other than a white shirt, you got in trouble. [Laughter]
Buck Shlegeris: Yeah.
Steve Kelly: I don't know if that haircut would have made it. [Laughter]
[Audience Laughter]
Buck Shlegeris: Yeah. Okay, so here's a question. So, a thing that we talk a lot about in AI control is kind of adversarial evaluations of security procedures, where you try to decide whether security procedures are good by having some kind of adversarial auditing.
Steve Kelly: Sure.
Buck Shlegeris: So, to what extent is adversarial auditing of processes? You try to check that your processes are good by paying someone to try to break them. To what extent is this type of adversarial auditing process, to what extent is that used to assess security measures?
Steve Kelly: Yeah, there's lots of examples of where that is done, and good examples are in the context of airport security and passenger screening in the U.S., the Transportation Security Administration is doing all the work. And so, the objective being that firearms and knives of any length are not going to pass across this line. And so, therefore, you create a security program that seeks to accomplish that. And so, magnetometers and x-rays and that sort of thing.
And then, once in a while, TSA will test their employees by running stuff through, and the question is, did they find it? And so, that's a red team evaluation, it's really easy to do, because what they're looking for is very specific and finite. Yeah.
Buck Shlegeris: So, who? Is it a sub-team of the TSA that does the red teaming, or…?
Steve Kelly: I don't know TSA intimately, so I won’t—I can't speak to that. But I assume that that's a separate group that is part of their kind of audit function.
Buck Shlegeris: Yeah, I was very surprised when you first told me about the TSA doing this red teaming. It seems the TSA seems so arbitrary in their procedures. And they really seem like an organization that is obeying a set of processes, rather than being allowed to choose whatever processes they like, subject to the constraints that the red teamers can't get guns through or whatever. Yeah, I don't know, I was surprised by this.
Steve Kelly: Yeah. And typically, the red teaming topic, like the U.S. government does a fair amount of red teaming to evaluate their cyber security posture. And that's well established, that makes sense. It's harder to red team a human. And so, we had talked a little bit about in our prep, do you red team a person? Do you do a…
Buck Shlegeris: Sting operation.
Steve Kelly: A sting operation. And it's like, uh. So, certainly in the context of criminal invest, and national security investigations, sometimes you do a sting. Like if a law enforcement agency's investigating a drug organization, you could insert an undercover employee or operate some sort of a source, and get involved and do transactions. And then you are part of it, you're paying for X, or you're selling contraband or whatever. And there's an example, ATF, there was a scandal around automatic firearms going across the Southwest border.
And so, those are sting operations, but highly costly, and there are some risks involved, and also, there's the concern around entrapment, are you inducing somebody to do something that they weren't otherwise predisposed to do. So normally, not just normally, I think every time, there needs to be sufficient predication to open up a full investigation, to get to the point where you're doing that kind of a sting.
And so, the question is, in an insider threat scenario, where you're trying to find the insider threat, do you choose an employee at random, and say, we're going to run a trial against them, we're going to see if we can get them to sell us, or to give us classified information. That would be highly unusual, I think, and generally speaking, it would need to be well predicated, that there's reasonable suspicion that that person is having a problem. And so, I think that's harder. The computer system you're red teaming, or the AI model you're red teaming doesn't have constitutional rights, the employee does.
Buck Shlegeris: Yeah, so there's these two points, where one is that the sting operations are really expensive, and the other is that you're not allowed to entrap people.
Steve Kelly: Right.
Buck Shlegeris: Because of the constitution.
Steve Kelly: Right. And I don't think that—that would leave a bad taste in my mouth, if all of a sudden, these scenarios, like a honey trap, all of a sudden, attractive ladies are coming up to me, well, that doesn't ever happen, so why is this happening?
[Audience Laughter]
Steve Kelly: Yeah, not good.
Buck Shlegeris: Yeah, when we were talking about this when prepping, I just actually hadn't thought of the fact that you're not allowed to just entrap random humans. [Laughter]
Steve Kelly: [Laughter]
Buck Shlegeris: I'm always thinking in the regime of these AIs, where you can kind of do what you want. Yeah, that is very interesting. So, wouldn't you—I would have thought that you are allowed, so you're allowed to do phishing, like phishing tests on your employees?
Steve Kelly: Oh yeah, yeah, that happens all the time, sure.
Buck Shlegeris: And presumably you can fire—presumably, it's not a problem to fire people based on failing sting operations.
Steve Kelly: I have never heard, but I've actually, they get pretty clever, I fell for one a few years ago. I was like, “God, that was a good one.” Yeah, not good, sorry, maybe that's an admission I shouldn't make. But that's part of security training, and to the extent that people bite on it, it gives the organization information as to where they need to close gaps in terms of employee awareness. And it certainly alerts the employee to, you failed, maybe you should take this little web-based training and get better at this. But I've never heard of anyone getting fired because they bit on a phishing case.
Buck Shlegeris: Yeah, yeah, it is interesting, I guess something else I learned about-
Steve Kelly: It's embarrassing, but not fireable.
Buck Shlegeris: Yeah, something I learned is generally, or a lot of the time, insiders inside the FBI or whatever, trying to sell information to foreign governments try pretty hard to keep their identities secret. Like it would actually be a lot—I was surprised by how much effort it would be to do sting operations on people. Like it's very rare that, like the FBI insiders are typically not people who just get approached by some Russian in a bar in D.C. and then agree to sell secrets.
Steve Kelly: Yeah, there are plenty of books written on the topic. Yeah, the cycle of recruiting a source or an asset, and how long that takes. And there's real skill to it; folks that do human intelligence work, folks like CIA and FBI domestically, that's a skill.
Buck Shlegeris: Yeah, and you can’t just-
Steve Kelly: And it's not, it’s not quick.
Buck Shlegeris: Yeah, which makes sense. There's kind of proof of work. And one of the core differences between the humans and the AIs, of course, is that we get to amortize that work, because you get like so much work out of the AIs, the same AI system, it's like effectively an employee who lives for a million years, and if your employees lived for a million years, you really could afford to spend 1,000 person years of sting operation effort on them.
Steve Kelly: Sure, sure, you could replicate them and do more. If you can afford the compute, go for it. So, the thing we haven't talked about is, I think— I can't remember if I said it, but I think there's a third phase after kind of the what's happening now. And that is, I've heard left of boom, right of boom happen, being discussed here, or maybe that was at my cybersecurity conference earlier in the week. But the third phase, which is something happens. And so, inevitably, you're not going to solve all alignment problems, inevitably, your controls are going to fail at some point, and you're going to be in a position where some bad act occurred.
And so, in any organization, whether that's a ransomware attack or some other, you know, a hurricane hits your facility, you've got to have business continuity, disaster recovery and crisis management plans to deal with that. In the same way, I'm kind of wondering coming out of this conference in the AI-control topic, what happens and what are we going to do, or what plans are we going to put in place if there's a loss-control incident? And so, all of that also exists within organizations and the U.S. government.
Sometimes it's aligned with your security program, but some of that stuff is actually not security specifically, but those are important plans to have in place, and you've got to be thinking about that next phase.
Buck Shlegeris: Yeah, so I have a blog post: “Catching AI's Red-handed”, which is about what you should do if you catch your AI's red handed. And in a podcast, an 80,000 Hours podcast that is coming out in the next few days, I talk about the plan for after the AI escapes, how you try to hunt it down and stuff, which you might be interested in on this topic.
Steve Kelly: That would be fantastic.
Buck Shlegeris: It obviously seems like a kind of rough position to end up in. Obviously, a big part of what you do with your employees when you're suspicious of them, my understanding is you remove their privileges, or Hanson, the mole inside the FBI was moved to a IT supervisor role for a while or whatever before they arrested him, where he didn't have access to sensitive information anymore. And in the AI case, of course, it's more like you become suspicious of all of your employees, and you simultaneously—it's hard to simultaneously fire everybody.
So yeah, there's some interesting disanalogies there. So, maybe it's not that different to becoming convinced that some big suborg is deeply compromised. And that's the thing, which I guess the national security establishment is familiar with.
Steve Kelly: Yeah, I couldn't say. But that would be a horrible situation, it's bad enough to have one well-placed employee go rogue, to have an entire chunk of an organization, or a large AI capability that's running out of control, that would be a problem.
Buck Shlegeris: Cool.
Steve Kelly: Yeah.
Buck Shlegeris: Well, we're out of time.
Steve Kelly: Okay.
Buck Shlegeris: Thanks very much.
Steve Kelly: You're welcome, happy to be here.
Buck Shlegeris: Yeah.
[Audience Applause]
