Technology, Power & Security Watch Now: The Drone Age

Across the world, drones are coming into the hands of new actors—so it is even more important to understand how they might change the world. Foreign governments, law enforcement, terrorist organizations, humanitarian organizations and even UN peacekeepers—all of these groups now have access to drones.

Michael Boyle’s new book, The Drone Age: How Drone Technology Will Change War and Peace, explores how the unique features of drone technology alter the strategic choices of governments and non-state actors alike by transforming their risk calculations and expanding their goals on and off the battlefield.

What impact will drone technology have on the patterns of war and peace in the next century? Will drones produce a more peaceful world, or will the prospect of remote warfare lead governments to engage in more conflicts? In this conversation, Michael Boyle and our Director Michael Horowitz discuss how drones are quietly altering the dynamics of wars, humanitarian crises, and peacekeeping missions while generating new risks to security and to privacy.

Transcript

Michael Horowitz Hey, everybody. Thank you for joining us for this special program at Perry World House, the University of Pennsylvania's global affairs think tank. I'm Michael Horowitz, Richard Perry Professor and Director of Perry World House. I will do my formal introduction in a moment. But before I begin, a few logistical notes: one, following the conversation, we will open the discussion up for questions. Two, to ask a question, please use the Q&A button at the bottom of the Zoom interface in the middle. Three, you will also see a chat button. Please try to save the chat for technical or other logistical issues. But monitor the chat because we're gonna share resources in there throughout this event that might give you more information or things you want to explore on this topic. Finally, please keep this clean and safe for work, which will help us have a successful event.

Welcome to today's event on The Drone Age, a new book by Michael Boyle. We are excited that Professor Boyle is joining us here today to tell us a little bit more about his new book and the underlying issues he's examining. Many of the questions raised by his research sit at the heart of our research themes here at Perry World House. In particular, our future of the global order, power, technology, and governance theme, which makes this particularly important from Perry World House's perspective. Michael Boyle is an Associate Professor of Political Science at Rutgers-Camden and a senior fellow at the Foreign Policy Research Institute. He's written on the legal and ethical implications of drone warfare, non Western responses to terrorism, and explaining instability and violence in post-conflict states. Also, I would like to think it's the most important thing, he's a former visiting scholar at Perry World House, having joined us in what I believe was the first semester we were open. So he's been part of the community here from the start. Michael and I were also post-doctoral fellows together at Harvard, like way back in the day, back before families and children and whatnot. And so it is especially fun that we've been connected all these years and to have you here today. So welcome and thanks for joining.

Michael Boyle Thank you for having me, Mike, it's a great pleasure to be here with you today. And to be here with everybody at Perry World House, I'm enormously grateful to Perry World House. It's where I actually started working on this book four years ago when Perry World House had just opened. And being in a wonderful academic community like that really helped this book. So I'm enormously grateful to everybody in the Penn community for welcoming me by starting the project. I'm going to talk a little bit about what I aimed to originally do with the book itself. The book itself is written to provide a general introduction to drones for people who say, you know, I hear there's this technology out there and I don't know an enormous amount about it, but also to advance an argument. And that's what I'm going to talk a little bit about today, about the impact that drones have on decision making and how, in fact, actually they alter the way that we think about things. So with that, I will jump in and let me also say I want to apologize in advance. I have a dog and a kid that likes to join my calls. So there's very well possible that may happen. And let me apologize in advance if they show up.

So as I said, the title of the book is The Drone Age: How Drone Technology Will Change War and Peace. And it's really an attempt to think a little bit differently about the way the drones affect decision making. Why do I think this is so important? I think if we could have a step back and take an analysis of this, you know, the diffusion of drone technology is probably the great story of 21st century military affairs. If we go back in 2000, only one country, the United States, had an armed drone. By the time we hit 2020, 38 countries have armed drones. Remarkable diffusion of armed technology. But even beyond that, drones have become an important part of the militaries of a large number of nations. The current estimate is that at least 90 countries have unarmed drones and many more have boutique drone capabilities. The spread of drones is driven by a large number of factors, not just including technological capability, but also questions of prestige.

It's increasingly becoming a question where if you are a major player in defense affairs, if you're a major player military affairs, you have at the very minimum a token drone's capacity. And so this spread of drone technology has been extraordinarily rapid to militaries and it's something that we're just starting to kind of get our hands on and kind of understand what in fact, actually this means. But it's important to emphasize that it's going far beyond just governments themselves. At least a dozen terrorist groups have drones. Very hard to get a good count on this. We can only look at some cases that have drifted out in the public, but that we know, terrorist organizations as wide ranging as the FARC in Colombia to Hezbollah, to the Islamic State, have some drone capacity and in some cases quite sophisticated capacity. Hezbollah, for example, has a very well-developed the drones program on the Islamic State—you can see a picture of an Islamic State drone at the bottom—has pioneered what I would describe as kind of industrial production of drones, where you can pack them with explosives and actually use them in ways to be able to attack troops on the ground.

So in a very real sense, what we've started to see is the drone technology has democratized the airspace, allowing not just powerful militaries to get to the air, but non-state actors who 10 to 15 to 20 years ago couldn't have gotten into the air are now, in fact, able to do it. The spread of the technology is even wider than that. Drones are now more widely used by the U.N., by humanitarian NGOs, and human rights groups. And you may notice that the subtitle of the book is "How Drones Technology Will Change War and Peace." And the book is broadly organized in that way. I'm not just interested in the question about how it changes warfare, but also how in the sense it changes the behavior of other actors, whether that would be the U.N., humanitarian NGOs, and human rights groups, all of which are discussed in the book itself. We also know that in the domestic space, drones are expanding at an incredible rapid pace. The FAA's recent estimate is that there were one million drones in American skies as of 2018. What makes that number so incredibly striking is that not just a few years before that, the FAA had produced an estimate to say that they expected by around 2020 that there would be about 30,000 drones in American skies.

And you can see quite obviously that they got it wrong. Now, the vast majority of these drones are used for relatively routine purposes, whether it will be commerce, real estate, or university research. But the very explosion of the technology is suggesting that even in the domestic U.S. context, we're struggling to get a handle from a regulatory vantage point, from a legal vantage point, about what it means. We start to open up the airspace to so many drones. It's also producing enormous commercial consequences. The global drone market is estimated to be about 5.8 billion dollars in 2020 and by some estimates will grow even larger to that. There have been estimates that say by 2027, it could be worth up to 22 billion dollars. And counter drone technology is now worth about 4.6 billion by estimates by about 2026. So from a commercial vantage point, drone technology is exploding. From the vantage point of international action by governments, non-state actors, international organizations, and NGOs, what we've seen is the rapid diffusion of the technology itself, that technology is moving around the world at a pace.

And I think understanding the scope of that change is extraordinarily important. When we think about the history of this, one of the things that I tried to do a little bit in the book is to kind of say, well, how did we ever get to this point where all of a sudden we have unmanned technology that's moving into a number of different domains and changing things relatively quickly? You know, the story itself starts back in 1898. Nikola Tesla was the first person to file a patent for unmanned vehicles. And Tesla himself—I think this is really interesting, though—he wasn't able to actually build the vehicle himself. He had a view as to what it would mean for world war and peace. And he himself argued that once we had unmanned technology, once it was possible for so many people to get into the sky, war would be virtually eliminated because the consequences of that much explosive potential and lethal potential in the air would mean that war was so terrible it would no longer happen. But in fact, the opposite has actually happened. Because in the military itself, which has driven the spread of drone technology—in fact, as I argue in the book, you can't really understand the development of drone technology without understanding that it's really the military and intelligence arms of developed governments that have really produced and technology.

Here we have one of the very first reusable drones that was created by the British military. This is called the Queen Bee drone. And you can see Winston Churchill next to it. And it's worth thinking that when we started originally, think about how drones were thought about in the first part of the 20th century. In the First World War, there were relatively crude attempts to turn drones into cruise missiles; in other words, to pack airplanes full of explosives and fly them, much like we fly like a cruise missile for a one way voyage into a target. But by the time we got to the Second World War, the idea was very different. Is there some way that we can use this technology in a way that's returnable and reusable? And this drone itself, the Queen Bee, becomes an important sort of technological development for a couple of different reasons. One, it's a returnable, reusable drone that is used for target practice. It gets store high level endorsement by the British military. And it also arguably gives us the name "drone." When we figure out why unmanned technology is called the drone, many people point back to the fact that this was the Queen Bee as the origin of the term itself. In the U.S. you have a very similar development. In the Second World War, the U.S. Navy was in fact actually working on its own target drones, in part to improve the accuracy of gunners on ships.

And those of you who probably are looking at this picture and saying this person looks vaguely familiar. You know, the story of this picture is, I think, one of the more interesting little vignettes around drones. The U.S. military/U.S. Navy investment in drones in the 1940s was taken up by an actor called Reginald Denny, who created Reginald Denny's airplanes in a factory outside California. And the Army, looking for a good story about technological development in the Second World War, sent out a young photographer named David Conover to the factory to take pictures. Reginald Denny asked a young woman working in the factory to take the pictures. That was Norma Jean Dockerty and later known as Marilyn Monroe. As an interesting side note, the person who sent David Conover to take this picture was an Army propaganda officer named Ronald Reagan. So the secret story and the secret history of drones, it goes all the way back to Tesla, but stretches all the way through the Second World War and touches on the lives of a lot of people, even though most of us think it's a relatively recent technology. Perhaps the most important drone of the Cold War era is up here is the Ryan Firebee drone. It was a jet powered drone also developed for target practice. But what makes it so interesting is that it proved that drone technology was fundamentally adaptable. And the Ryan Firebee drone was adopted not just for target practice, but also for various purposes like reconnaissance. It was converted into what was later known as a lightning bug drone and used in the Vietnam War. And in fact, it was actually even used to draw propaganda in the Vietnam War. So we start to move to an area where the technology is reusable and it's adaptable, and that begins to be one of most important drivers for its spread. And finally, when we think about the history of the drone, the development of the Predator really accelerated around the time that the hunt for Osama bin Laden took off in 1999, 2000, 2001 becomes extraordinarily important. This is where the technology really takes off in the popular imagination that we think of drones as essentially being used for the purposes of removing terrorists. And that's largely what the Predator drone, at least in its original iteration, was about.

So the technology is very old. The diffusion is something that we're seeing more recently. But one of the things that I wanted to try and do in the book is to think a little bit about what implications this has for decision making. And when we think about the implications for decision making, the debate over drones and in fact—I think much of the literature over drones—really was tied into debates over targeted killing. And you can see the targeted killing as a practice both in places like Pakistan, Yemen and Somalia. This is data from the New American Foundation has varied over time, but reached its peak in years, 2010, 2011, and 2012. And what this had led to was a debate over whether the drones themselves were enabling targeted killing and whether in fact new legal and ethical standards were emerging as a result of this practice. And I myself have done some work on this myself. But when I thought about the issue of targeted killing, though, I think it's important for us to understand that this practice continues and actually the battlefields have changed under the Trump administration. It seems to be a conflation to think just about targeted killings when we think about drones. It's important for us to widen the frame and say, well, maybe this isn't that drones themselves are just enabling a practice like targeted killing, maybe there's other things going on and maybe they're enabling different kind of practices themselves. Early on in this debate, if you think about the question of what impact do drones have on decision making, and that's really the central question of the book. The early on the arguments were that what's happening is the drones themselves desensitize you to the effects of violence. U.N. Special Rapporteur Philip Alston argued in 2010 that because drone operators are based thousands of miles away from the battlefield and they undertake the operations entirely through a computer screen, that there's a PlayStation mentality to killing. And much of the early debate around drones took this argument that the pilots themselves removed from the battlefield themselves would become more risk taking or be indifferent to casualties and indifferent to the destruction associated with it. That's essentially the core part of the argument.

And I think what I tried to do in this book is to say this is an important question, but it's not the only question. The focus unduly on pilots lets off a whole bunch of people, politicians and government leaders, whose decision calculations may also be changing. And so we need to move away from the question about just assuming this is all about reckless pilots being desensitized to violence and ask other questions about the impact of this on decision making. And I wanted also to push the book beyond targeted killing, although I deal with that in the third chapter of the book. I provide a kind of overview of targeted killing. I thought it was important to go beyond it. And particularly because when you do a little bit of research, the psychology on the distance between yourself and the target and how that affects your propensity use violence is in fact actually more indeterminate. That we don't actually know that people are that removed. For example, the ground control station like this are going to be more violent. In fact, they may not be. And in fact, there are reasons to suggest, certainly when you interview pilots, that that's not true.

One of the great pleasures I had in this book was to do a lot of interviews with drone pilots, and they made the argument that it was very not much not the case that you are essentially indifferent to civilian casualties. And I think that's a kind of crude version of an attack on the technology itself. Moreover, there are other arguments. Some arguments suggest the drones themselves by providing more information, more rich sense of what is happening on the ground, can actually improve your decision making, which can make warfare more precise. And that argument itself is a serious argument and there is evidence behind it. So to simply say drones can't be accepted because they're responsible for enabling targeted killing, they produce this kind of an indifference to casualties. Seems to me to also be wrong. It's also worth pointing out, and I got this a lot from pilots. The drones themselves may have no impact on decision making. They may be entirely neutral and largely because if you think about how they're being used by militaries, you're often looking at very rule governed environments where the decision to, for example, launch a missile is hardly discretionary. It's governed by a series of rules, dense bureaucratic procedures, as well as legal and ethical checks, that suggests that it's not simply that.

So what I wanted to do is take a step back and say, all right, if that's the case, how do we think about drones in decision making and how can we really grapple with the question about whether this, in fact, actually just changes decisions across a number of frameworks? And what I did in the book is I've argued the following, that drones can actually alter decision making in really two dimensions. The first is by risk calculation, by changing the calculations of the things that you're willing to do. Changing a risk threshold in some cases, making them more willing to roll the dice. And secondly, by goal displacement, very similar to what the military means by mission creep and suggesting that you can have a series of organizational pressures as a result of having the technology that changes what you can do. A second argument that I make in the book that I'll talk a little bit less about here, but I'm happy to answer any questions, is that it also changes what you see in the language that you use. So drawing from John I argued a little bit about how the technology itself produces it's own language or technique or mode of seeing things. And it also changes your field  of vision. It changes how you see the battlefield, it changes how you see your environment. And as a consequence, that has changes on your actual behavior. The big lesson, though, that I try and draw big distinction here is between tactical and political decision making. That if we're asking what does this book really argue, it argues that drones affect political decision making less than they affect the behavior of the pilots. The pilots themselves are taking orders from someone up.

And I think the real secret here is to start to think a little bit about what does it mean when political leaders have access to the technology? How does their risk calculation change? How do governments and also nongovernmental actors think about this? So what I tried to do in the book was to push just beyond the military, although we'll talk a little bit about that today, and to look at a number of different actors—not just the United States—and a number of different domains. So within the book itself, I look at targeted killing and warfare, but also surveillance, peacekeeping, humanitarian NGOs, and other types of developmental actions. And to look at how everybody from the U.N. to NGOs to state actors begin to change a little bit about what they're doing over time as a consequence of having the technology. To really trace it out in multiple domains, to really make the argument that something, in fact, actually does change.

Now, I'm going to talk about two different examples from the book before I wrap up that really get to the core of the argument, and these are just two different pieces of the chapters. The first is I try to argue in the book that drones are actually important in a different way than we were used to or traditionally used to the technology being used in ungoverned spaces for counterterrorism missions. But in fact, they're now operating in crisis zones around the world. Those of you who've been following the recent military conflict between Armenia and Azerbaijan will no doubt be familiar. The Turkish drones are being used in the conflict and we're starting to see the introduction of drones into much more sort of hot conflict zones. And this is producing new dynamics around deterrence and around tacit bargains. The idea that we have a bargain about what I will do and what you will do and we don't break that.

The introduction of the technology we're starting to see is more and more countries beginning to kind of test the limits of traditional deterrence because it's an unmanned pilot. And that itself is reordering the strategic calculations of states. You see this in a couple different dynamics. The first is incursions, but no casualties. From looking at what's happening, for example, in the Armenia and Azerbaijan conflict, we're seeing real casualties. But if we're looking at other cases where drones are being flown across the line of control, for example, in India and Pakistan, to test the nerves of other states, we're starting to see more and more cases where incursions are used provided that they produce no casualties.

And I'll show you something interesting about U.S.-Iranian relations in a second. Secondly, that they can be used with plausible deniability. The Saudi Aramco attack and here you see some pictures of Ababil drones that were used in that attack in September 2019 were done and masked with missiles. And the essential attempt was to deflect blame to the rebels for doing so. And so the technology, because it is small, because it is mobile, because you can use it in ways that at least confuse radar, can be used with a degree of plausible deniability that other forms and platforms can't, in fact, actually work. And finally, we're seeing a dimension to which you're moving towards assassination or what I would call the individualization of violence. That violence is being conducted against more and more individuals, in part because we have technology that allows us to act more precisely and to be able to operate more effectively in conflict zones. And so all of these changes that I can do incursions without casualties, that I can do with plausible deniability. In some cases I may be able to make individuals rather than collectives pay a cost for the decisions are beginning to change the logics behind deterrents and are beginning to suggest that the interactions with human states are, in fact becoming very different as a result of having the technology. One example of this I like to throw up, just as an interesting kind of side note. Now we think about U.S.-Iranian relations. They traditionally had a quiet proxy war for a very long time. And the tacit agreement has always been not to knock down each other's aircraft. Well, starting in June of 2019, what we started to see is increasingly tit for tat between the U.S. and Iran surrounding aircraft. So we know that Iran took down a U.S. aircraft in 2019. In June 2019, a U.S. drone. The U.S. disabled an Iranian drone in July 2019. And in September 2019, the Iranian government authorized the attack on Saudi Aramco. While we take a look at these tweets, what we see is a really interesting dynamic. First, is President Trump acknowledging that in response to the first provocation, that he was unwilling to respond because it was going to cost the lives of 150 people and that wasn't worth it for a drone. Well, we start to see there is a crude moral calculus around drones where states are saying, well, I'll engage in this kind of incursion or engage in a tit for tat, but provided there are no casualties. Comes out later that the same thing happened with the Iranians, that the Iranian government authorized the attack on the condition that no casualties, civilians or Americans were killed. And the problem that the book tries to talk about here is that once we start to wear down our existing tacit bargains and once we start to see risk taking on the basis of no casualties, the question is where you get to that natural limit, reestablishing our kind of tacit bargains about what do we think about with drones and how actually can we interact with them becomes extremely important. And we have to reset what we mean by the natural limits of the technology in the national ones, the technology it produces this kind of risk taking.

When we come to goal displacement, this is something where an organization gets under enormous amounts of pressure as a result of having a new capability. And then one of the chapters of the book, I talk a little bit about how demands for intelligence, surveillance, and reconnaissance is really putting enormous pressure on the Pentagon to collect the whole haystack to learn more than it ever has and is having more operational consequences. The same dynamic is also true when we look at the United Nations. U.N. drones are now deployed as eyes in the sky for peacekeepers. You can see a U.N. fixed-wing drone here in Goma. And U.N. missions, when this was originally done, Ban Ki moon said know the development of a capacity for drones is largely about improving timely decision making. The more information we have, the better off that our decision will actually be. But if we look particularly about how drones were used in the Congo, they were gradually shifted from just producing information to decision makers to engaging what they call active deterrence against militants, that the drones themselves were used to fly over refugee camps to intimidate militants, to signal that they were collecting data on militants to protect the vulnerable. Now, the intentions here are very good, but what's started to happen was the technology once in the hands of U.N. peacekeepers began to push pressures to do something very different and to actually change the nature of the mission itself. And this changed expectations for civilians. Reports came out saying if you have a drone in the sky, why the U.N. can't respond more effectively to these incursions. And also, it has placed pressure on the U.N. itself to start to think about what it needs to do in terms of an intelligence capacity, that the technology itself is changing the U.N. to rethink what it means about peacekeeping and rethink what it has to do in order to be effective. And so in both of these dynamics, with changing risk calculations and also changing the kind of organizational goals and what the organizations seek to do, we can see that the technology is producing real effects.

So to kind of wrap up this brief overview of the book itself, what would I argue? The first is the book is a call to say we do need to rethink a lot of the strategic interactions around drones. That operating with the assumptions that these will operate like manned aircraft pr operating with the assumptions that the risk profiles and the risk nature among politicians, again, not pilots, will be the same is probably not true. If the technology itself is producing goal displacement, rethinking things from a more rational choice vantage point. In other words, the logic of conflict and logic, interaction between states and non-state actors is important.

But equally—I think there's already been some very good work done on this—thinking about the psychology of decision makers is extraordinarily important. If it isn't that decision makers consider the technology to be very different, then we can assume they will do different things with it. And I think that's an urgent need in the research itself is to start saying, well, how do we start to understand the psychology of decision making? How does this technology change what we're in fact willing to do? And this is becoming even more important because what we're seeing is that technology is changing in some really big ways that are going to affect 21st century warfare but also it can affect how states and non-state actors interact in other ways.

We're seeing the technology itself move both into land and the sea use, especially with increased investment in underwater submarines, both for interdiction, but also for for electronic warfare. With swarming with the ability to fly a large number of drones in tandem to conduct the operation, what impact does that have when we enter into a battlespace? How risk taking or risk averse are states? Once that technology is present with integration with artificial intelligence, which I know is something that Mike has written quite a lot about, once we have a case that drones themselves are being integrated in some way, artificial intelligence, and there is no realistic prospect that artificial intelligence will be pressing the button to engage in violence.

But what we will expect is that militaries like the United States will use artificial intelligence, provide a first cut of data and to accelerate the targeting practice, to make information intelligence collection more rapid and more efficient. When that starts to happen and we speed up the pace of activities, turning war into extremely fast and potentially extremely lethal, what kind of consequences does that have, especially when we don't have somebody in the cockpit? Finally blended unmanned systems. When we start flying something like Boeing's Loyal Wingman program, where you have drones and manned aircraft moving in tandem, how does that change the strategic interaction given that you have risk from the manned aircraft and less so? How do we change the kind of moral calculation in the strategic calculations associated with that? And finally, miniaturization of drones and ammunitions. What we're seeing with the technology itself is that the medium altitude drones like the Predator and the Reaper are becoming less and less important.

And drones are either moving into the kind of high military space or getting smaller and more lethal, as we see with things like the switchblade drone. And as drones get smaller, are we going to start to see more kind of assassination style missions? Are we going to start to see more, what I sometimes call the individuation of violence going as a result of the technology? And so the book, in short, says the technology is already beginning to change the patterns of war and peace as it begins to develop further, it becomes urgent that we start to work through new rational choice vantage points and also start to think about the psychology of decision makers once this technology is in play. And with that, I'm happy to answer any questions.

Michael Horowitz Well, thanks a lot, Mike, for that super interesting presentation, which I think was both excellent in the way that it laid out the history and provocative in the way it talked about some of the current challenges. And you and I could talk about like drones all day. We'll get to audience questions in a few minutes. And as a reminder, drop those in the Q&A. Let me start with the big picture question. I think it would be—and you can push back me if you want—I would characterize some of your views on this and a little bit, this book is skeptical in some ways about the impact of drones on international stability. So imagine that you got to run American defense strategy or military strategy when it came to the use of drones. What kind of changes would you think about from a policy perspective? What should the U.S. be doing differently?

Michael Boyle Well, it's I think it's fair to describe me as a sort of skeptic for this. And I think also no one would ever let me near running U.S. defense policy.

Michael Horowitz Me either, it's fine.

Michael Boyle But thinking about this, what would I recommend? Well, a couple of other things I would recommend. The first is, you know, when we think about drones for targeted killing, in a perfect world should you use that technology, you probably should, but you should use it rarely. So when people say you're opposed to all drone strikes. I'm opposed to drone strikes when they're used as a regular tool or kind of counterinsurgency, not as something that you would use, for example, to take on a very high level leader in a situation where there were no other alternatives.

Michael Boyle So my preference would be the drone strikes. It's not that you can never use them, the technology exists, but that you would make them rare. The second thing that I have real reservations about is the growth of this sort of institutionalization of the technology, especially in the U.S. context, entirely within the executive branch. Because what we see with targeted killing is that policies are being set by the president with relatively light scrutiny by the Congress, relatively light scrutiny by the courts and tactics of the courts to deny jurisdiction. And if you look at the end of the Obama administration, there was an attempt from around 2013, 2016 to do more transparency, to produce some data, to accept some more sort of scrutiny. And that was entirely ripped up in 2016. And most of the program was pushed back into the shadows in the Trump administration. And I think that problem is not just a kind of specific behavior of the Trump administration, but is actually about the way that the U.S. handles it. I would rather there be institutional checks on a technology that would allow this kind of activity. So, for example, a stronger congressional oversight, a stronger degree of even pressure from the courts in order to keep policymakers honest, in order to restrain it in a way that is not, in my view, currently restrained. So my concern here is what we've seen with that is stronger institutional checks.

Michael Horowitz  So do you think that we need those stronger institutional checks for all uses of force or just sort of strikes involving drones?

Michael Boyle I think I have particular concerns. In all uses of force it's worthwhile. All right. I think in all uses of force. But there are effective mechanisms for a lot of the regular combat activities that we have in Afghanistan and Iraq. I'm not suggesting that you would need, just to kind of recycle all of that. I think particularly with the drones program, the initial position in the Obama administration was to deny most of the details and deny certainly the courts access that was then brought out into the public in 2013 to 2016. And then that was simply wiped away. I would try to institutionalize that in order to make sure that there was a playbook, very high standards. The Obama administration actually got there at the very end in around 16 with much higher targeting standards, much more clear transparency, much more information being provided to other branches of government, and it simply got wiped away. There has to be a way to institutionalize that. So a future president can't do it. The other thing I think would be important to do in terms of beyond the targeted killing question is to start thinking about codes of practice or the use of drones and how you start to enforce, for example, not permitting the technology, particularly on sales and export license, to be used for human rights abuses. And there are already procedures in place like the State Department's certification process that are designed to stop that. The Obama administration actually moved this way in around 2015, 2016. They introduced a code of practice and got states to sign on. But the problem here is that they didn't get enough states to sign on. China, for example, didn't sign on. Other states that are major exploiters didn't sign on. And once you're there, you know, it essentially falls apart. So a renewed effort to build an international code of practice to governance use and more institutional checks. My short answer.

Michael Horowitz  So let me turn to the chat for the kind of interesting history question from from Frank Hoffman, who we both know, who asks about the—thanks, Frank. The origin of the term targeted killing. We don't really think about untargeted operations, right? Or at least maybe we haven't since, like area bombing and World War Two or so or something like that. Where does the phrase targeted killing come from? Like where did you see that sort of like pop up and kind of the historical research you were doing?

Michael Boyle Yes, it's really interesting question. So I actually talked about this in the third chapter of the book. I have a section called "A Short History of Targeted Killing." And the short answer to that question is the term itself was actually not used prior to Israeli practice in the second intifada. And so really, it was Israeli. Yeah, the term itself, if you go back and try and look for the phrase targeted killing in the 1970s and 1980s, you won't find it. And in fact, actually U.S. position was that that was equivalent to assassination. And what eventually happened was Israel, in the course of the second intifada, argued targeted killings are different than assassinations. Assassinations are conducted against leaders in peacetime. But once you're conducting a war, then you are permitted to take out people responsible for military actions and that you shouldn't, in fact, actually be limited to do so. And so the practice itself of targeted killing prior to September 2001 was largely pioneered by Israel. And in fact, actually Martin Indyk, who was one of our ambassadors to Israel at the time, made a statement saying essentially the U.S. does not support targeted killing, that we do not view this as different, and that that position that there is a kind of legal distinction between targeted killing and assassination was one that even the U.S. was uneasy about in 1999, 2000, 2001. The U.S. then adopted that practice in 2002, 2003, 2004. So as a term, it's one of those terms. It was relatively rarely used and really boomed after that point. And it's a good example when I talk about the language of drones. You know, I talk very briefly about technique and language. John argues that technology produces its own language. And this is what I mean. You can sometimes see that with phrases like kill box and kill list, but you can also with things like targeted killing, quite literally, the practice was pioneered by the technology.

Michael Horowitz And those that are that are watching, if you look in the chat, you'll see I dropped down a little fact check suggesting that Mike is correct about that. And if you do a Google books search of targeted killing. You see this huge spike right as the second intifada takes off. So it's a good point. So let me shift to something slightly different. We got a couple of questions come in about the use of drones by non-state actors. And I wonder if you could maybe expand on, you know, you called out at the beginning the way that many non-state actors now have drones and are using them for targeted attacks. What kind of trends are you seeing there and what countermeasures can governments take to stop those kinds of attacks?

Michael Boyle Yes. So I talk the whole fifth chapter of the book is on the way that drones are being used by terrorist organizations. And I think there's important to draw a distinction that I think gets lost in the literature between what I call terrorist drones and rebel drones. That's a slightly facetious way of putting it. But there is a difference between using a drone for a terrorist attack in, for example, Philadelphia and using it in a theater of war. It's a much more permissive environment. And so sometimes what we seem to see in the literature is, well, terrorist drones are going to come to New York or London because they can be used. You know, Islamic State is using them in Iraq and Syria. That's not quite true. In terms of the overall trends for it, there's been a relatively rapid diffusion of it. Most of the time, you know, you get sometimes in the literature, those say, well, it's going to lead to leveling the playing field with government actors. And that's not true. It doesn't lead to leveling the playing field with government actors. What it does, though, is distribute vulnerability in ways that it never had before.

So in Iraq and Syria, for example, American soldiers for the first time had to deal with threats from the sky by relatively crude drones connected with explosives that were conducting kamikaze attacks. The technology is spreading and was being adapted in different ways. Al-Qaida has tried to do it, but has a relatively weak capacity. The Islamic State, actually did industrial size production of kamikaze style drones or drones at least packed with explosives. And they managed to get a lot of fixed wing commercial drones that you can ordinarily buy on a normal market from Europe, get it down there and essentially use them for explosives. So we're seeing a lot more relatively increasingly the kamikaze attacks or attacks where you drop explosives and more often not using them with commercial technology. The kind of drone that you and I might find at Best Buy or slightly above that level, that's what we're really tending to see. And what makes that strategically important is not in any way that it levels the playing field. It doesn't. But what it does do is by distributing vulnerability, it gets the U.S., for example, to spend a lot of money defending it. And so counter drone technology is very expensive. And as an investment to protect people on the ground, you know, you're spending a lot of money to stop a relatively crude couple hundred dollar drone from dropping explosives on you. And we're also seeing a relatively good growing sophistication in that respect as well in terms of their ability to launch strikes and their ability to do things like, for example, to coordinate artillery strikes or even to film propaganda videos. Certainly in the Iraq-Syria example, we can see examples of that.

Michael Horowitz So let's take this domestic from an American perspective. It's a question that asks—it isn't necessarily domestic—but at what point might countries take measures to mitigate the risk through measures like a restricting frequency, bandwidth, or licensing the distribution of drones the same way we would license the destruction of small arms?

Given the way that that drones can be used for for terrorist activities against lots of different targets, I mean there were cases a while ago of people accused of trying to armed drones for to like attack FBI buildings. You know, those kinds of things. So, you know, is the U.S. thinking about these kinds of restrictions or at what point do countries, including the U.S., do things like that?

Michael Boyle Yeah. So I talk a little bit about this in the book. There are attempts to sort of deal with the question about how do we stop the spread in the domestic airspace in some sort of way. Part of the question with restricting radio frequencies is it's difficult to do. Difficult to do legally. And once you do it, it can be hacked. So part of the problem here is that that's a relatively difficult thing to do, terms of enforcement. I think one of the things that the U.S. has done right and interestingly enough, has not been done in the rest of the world, including in places in Europe, has been to insist on a degree of attribution from drones so that, you know, what they're doing with the registration system is to work very much like a car's bin number to attribute a drone to an individual.

And that's some way of at least trying to make sure that you can't buy the technology, pass it off to somebody, do something nefarious, and avoid responsibility for it. I think the interesting other problem that they're running into with counter-drone technology for a lot of these things is that, you know, blocking radio frequencies is one thing. But you also have to think a little bit about what you would do if you had a threat and how you would get it to the ground. And so, you know, detection is a problem. Interdiction is also a problem. Under current U.S. law, until fairly recently, you couldn't shoot down a drone. It was equivalent legally to shooting on an aircraft. And the FAA argued you didn't have the right to interfere with a flying drone, just like you did with an aircraft. And we've had law enforcement in the United States appealing for exemptions to say, no, we have to have an ability. And more recently, there's been exemptions granted to say you, the department of Homeland Security, you can't be legally stopped from bringing down a drone in your space. And so you have a problem of the legal barriers associate with what degree you have the right to interfere with aircraft. How do you bring it down safely so it doesn't hurt someone, restricting radiofrequency can do some of those things. But there hasn't been a consensus on counter-drone technology. Everybody's doing this. You're seeing a lot of different types of technology. You can try by a lot of different actors, varying different degrees of effectiveness.

Michael Horowitz So another question of the chat from friend Alan Luxenberg, thanks for joining today, Alan. For those of you that don't know, I until very recently ran the Foreign Policy Research Institute, a big foreign policy think tank, in the Philadelphia area where both Mike and I have been affiliated. And he asks and, you know, this is this is timely. Do you have thoughts on the use of drones to surveil domestic protests such as has happened with BLM protests in the U.S.? You made that point about local law enforcement earlier—you didn't say local. I asked you about law enforcement earlier. But as as drones—in part through A.I.—become easier and easier for non experts to operate, they become increasingly plausible for especially shorter range systems, for large varieties of local law enforcement to be able to use them to conduct surveillance. How concerned are you about this or is this this good because it gives them better situational awareness? How are you thinking about it?

Michael Boyle That's a really good question. This is something I consider a little bit in the chapter on surveillance, which is to say, we've seen this not just in drones. The first thing I would argue is that this isn't just about drones. Manned aircrafts can do this. If you look, for example, at the persistent surveillance program that was used in Baltimore that was able to record large swathes of the city of Baltimore for crime. That was largely done with manned aircrafts. So the general problem of aerial surveillance can't just be reduced to drones. But that drones could sort of superimpose power, it becomes a serious problem.

Michael Horowitz [00:49:03] If it changes the cost, essentially, like if you if you would. You know, an inhabited aircraft, maybe if you think about the fuel and, you know, like it might be a lot more expensive at the end of the day.

Michael Boyle [00:49:13] That's right. It can radically lower the cost. And what we're also starting to see, aside from the use of manned aircraft on this, we're starting to see federal agencies beginning to use this for monitoring of the border. Customs and Border Patrol, for example, is doing it. The FBI has also run its own drone surveillance programs. Now, officially, they're not allowed to be doing it, for example, to survey a protest. The private part of the problem here is disclosure. It's often very difficult to get data on what the overflights were and what they were, in fact, actually doing. And this is part of my argument for greater disclosure for it. You know, the ability to get into the sky certainly as U.S. drone technology move not just at the federal level, but to a lot of state agencies, to a lot of local agencies.

You have a lot more people collecting aerial information and not a lot of clear standards on how you might release that information or how that information could be used. So could it be misused to identify protesters? Could you use it in a way with drone surveillance that reinforces racial prejudices, for example? Absolutely. And one of the things I think about this technology that is important to understand, is that it's going to have real civil liberties consequences in a democratic state. But ultimately, you can resolve a lot of that through the courts. Domestic surveillance in authoritarian regimes is is even a more serious problem and is one that we haven't really thought much about. If we start to get surveillance drones being used in countries with very strong authoritarian governments, not only could it be a threat to the civil liberties of the population itself, but it could have much greater consequences in terms of, you know, using A.I. to sift through information and to triangulate who's on the ground and what they're doing. And are they going to get punished in a later way? So we have to be very careful about surveillance technology in the United States from a more basic civil liberties vantage point. But from authoritarian governments, it has a much more serious consequence. And we're starting to see early movements towards that.

Michael Horowitz So let's the shift back from the domestic to that to the sort of international security environment sorts of issues that you and I've spent a lot of time thinking about is a question asking about essentially, how our adversary is going to respond? And in that say the elusive terrorists who, you know, to quote a question tries to impose costs with attrition over time and the persistence of lethality, the persistence and lethality and low cost of uninhabited systems and strike operations could be could be efficient to sort of counter those kinds of terrorists. How do we essentially, how do you think about if war is a game of action and reaction? How did drones fit into that as adversaries react to the use of drones and developed systems to try to generate attrition over time?

Michael Boyle So a couple of things that I would argue on this. The biggest one I guess I would make is I think one of the dangers that the U.S. faces—and I think there's been some recent moves by the Pentagon to adapt to this—has been the danger to assume that the states will use drones much in the way that the U.S. has. But actually, what we're going to start to see is the technology will be adopted in different ways to maximize advantages in different sets of spaces. So China, for example, is going to use drones very radically different than the kind of traditional U.S. model flying over ungoverned spaces and taking your time to strike. And that's part of what I try to argue in the book, is that although that debate around targeted killing is very important, that's one set of practices unique to the United States and a small number of other states that are emulating the U.S. in some limited ways. That's very different than the battle front that the U.S. will have. Well, we think also about the way that adversaries will react to the U.S. The U.S. has also gotten used to extraordinarily information rich warfare. To being able to have as much information as they can in the battle space, harnessing technology like A.I. to be able to accelerate the processing of that information and to use it in a really efficient way. And what my concern here is that if you're thinking about the use of drones, what you would do is use drones, for example, to interrupt that. Whether that would be underwater drones, I would interrupt that, whether that would be other kinds of drones or in Europe in contested airspace. But we can't assume the adversaries can act in the same way that we could. We have to assume that their use of drones will maximize our vulnerabilities. And my concern with the U.S. is that, you know, the degree of information to which warfare depends or the degree to which we depend upon that, you need a very robust systems in order to survive what might happen if an adversary tries to capitalize on our use of that kind of technology in a very specific way.

Michael Horowitz So a lot of your book focuses, not all of it, but a lot of your book focuses on the essentially "current generation system." The discussion of surveillance, targeted killing, things like that, but you ended with some of those trends. And I want to pick up there with some questions from the Q and A, one of which asks, do you think we'll see armed air to air combat with drones? I mean, certainly we know we're probably unlikely to see that in a serious way with current generation systems. But as you know, you know, there's a lot coming down the pipeline. So where do you see that heading?

Michael Boyle So I don't think it's gonna happen in the short run. I don't see armed drone to drone combat in the short run. In the long run, I think that's certainly something we're going to start to see. And one of the things my concerns is that will probably be if it happens, it will be on among militaries in very deep pockets. Right. So that's a potential scenario with the U.S. and China rather than with a lot of other states. My concern here with that would be going back to my question about rethinking strategic interactions. A lot of our models are based on the assumption that we can calculate cost and credibility based on the loss of lives in a manned to manned aircraft. Once we move to unmanned to unmanned, what happened to the risk taking calculation and does that begin to change? So with combat unmanned aerial vehicles,  I expect that to happen five to ten to fifteen years off. My biggest concern, as you know, will states be more risk taking as a result of it, both incumbent themselves and also on the incursions that lead to the kind of salami tactics that get you to that point? So shorter answer is not in the short run, but in the long run. I would expect that to be something that would happen, but possibly less for direct combat and more to test the nerve of other states.

Michael Horowitz So here's a related question in a way, or at least a question that is related to the way where people think some of this is headed. As you recall, it is, you know, a couple of years ago, there was a big protest at Google about their participation in Project Maven, which was an A.I. program about automating the processing of drone surveillance footage. And that was emblematic in a way of how animated in some ways people get about about issues surrounding drones, in some ways more so than in other more dangerous military technologies and more dangerous in terms of, you know, ability to kill lots of people in some ways. What is it about drones that makes them seem so—why are we so animated about it? It makes them make them seem so sort of offensive or scary to sort of quote one one question.

Michael Boyle It's a really good question. You know, I think there is something about the technology that triggers a degree of fear that maybe is outsized to its actual impact. And you can make an argument that the Hollywood depiction of drones hasn't really helped things, that we tend to think of them as kind of unthinking, unfeeling eyes and this guy watching us. I think if I was going to trace it, it's partially the concern about being viewed from the air. I think that it actually makes people uncomfortable, but it's also a degree to which it is the unaccountability of it that when you see a drone, it's not quite clear who is recording, who's recording you and for what purpose. And that does seem to excite outsized amount of fears with it. I also think that there's a degree to which people are reluctant, particularly in the United States, around surveillance and they're reluctant the United States to trust the government. I think this is partially an artifact of a failure, of the trust of government. If we don't trust our government, then we're not going to trust law enforcement to use drones. We're not going to trust other actors to use drones. So I think there's something about the remoteness of the technology, the degree to which it sort of minimizes risk, the degree to which it can enable a very powerful state that triggers some very deep fears in the United States. And we've always generally in America, been very, very suspicious of having extraordinarily powerful federal state. And I think a lot of those fears are about, you know, what happens when the technology is watching us from the sky and absolutely unaccountable. So I also think, to be honest, using terms like "skyborg", which some of the programs have used that seem a little bit very similar to Terminator kind of language is probably not helping.

Michael Horowitz "Skyborg" just wants to give you a hug and make you feel good and make you feel comfortable. All right. Another question here. The asks about the language we use to talk about drones and you talk about sort of manned and unmanned systems. And, you know, do you think that how we use these gendered terms in some ways affects how people think about drones? As a side note for those academics out there, my PhD student is writing a dissertation on the intersection of gender and military innovation. Go check out her work. Anyway, back to you, Mike.

Michael Boyle That's fine. That's great work. That's fantastic. The short answer is yes. I mean, I think it's very hard to look at a lot of the security military affairs and not see a gender lens through it and not to see gendered language recurring all the way through debates. You know, if we were to go back in the nuclear era about the missile gap and so wanted to look at gender, gendered language there. And also with drones. And I think the unmanned and manned language that we use around drones, first off, I think it does have a clear gendered lens, as we see in a lot of the defense and security establishment. And secondly, I also think it's it's very inaccurate because, you know, when we think about unmanned aircraft, we generally assume that just because there's not a pilot in it, but in fact, actually the tail of people, not just men that are responsible for is is very large. There's a lot of intelligence analysts and lawyers and so on that are working whenever we deal with unmanned aircrafts. So in a sense, it's a misnomer. And this is part of the reason why the Air Force, for example, prefers the term remotely piloted aircraft.

I think there definitely is a gender lens to it. I also think that that language is going to get less and less helpful when you start to see the blending of manned and unmanned systems. When we start to see systems integrate in a way where the platforms are flying in the same space, the platforms are operating the collaborative way, starting to think in a distinct way about this isn't really gonna be very helpful. So I think that language can lead us astray. And I would argue the drones themselves have really produced an interesting set of evolutions with language. And this is back to the work I was talking about without changing your field of vision and changing your language. Because if we start to think about even terms like "kill box, kill list", none of these things existed before you started to see drones. And even among pilots, there are terms that have been used, derogatory terms about targets, for example, that emerge as a function of the way that they see the technology and the way the targets appear to them. So I think there's a lot of really good work about how the technology shapes the language. It shapes the way that you see problems coming from it. And I definitely think the language that has a feedback affecting how we, in fact, actually use it. So work done on a gender and gendered lenses, I think is extraordinarily important.

Michael Horowitz So we've spent a lot of time talking about aerial systems, but of course, there are what would be called robotic systems that are all over the potential battlespace, including sort of land, you know, under sea surface, et cetera. What are the kind of trends that you're seeing there? Because in theory, you know, if UAV's are a thing, then we should start to see these kinds of systems appearing in other domains of warfare as well.

Michael Boyle Yeah. At the end of the book, I talk a little bit about the move to land and the move to sea. The move to land seems at least to me to be going a little slower. We are starting to see unmanned systems move in the context on land, but it's moving in a much more support capacity. And I don't see it sort of transforming operations in the same way. The development of unmanned submarines, though, I actually do think there's a strategic consequence. And once you get to the point where you can do it and there's a lot of good reasons why you might do it for it, for example, for interrupting communications or detecting radiation, for example, but also clearing mines. And China has spent a lot of money investing in underwater subs. And I think once we start to see the United States and China both again investing in underwater unmanned technology, again, the call is to say we've removed humans from the equation, so what's the strategic calculation of both actors having that? And how does that change the risk calculations? And you might argue that with subs it doesn't or that it has a much more muted effect than it would have in the year for various sorts of reasons. But I think the strategic calculation, once we see the of growth underwater vehicles, underwater unmanned vehicles, especially among major players, the U.S., Russia and China, I think is extraordinarily important. And we'll go forward. I'm not as convinced that we're going to start to see the immediate kinetic effect underwater. I think it's more likely to be used for scientific research, radiation detection, electronic interdiction mines, rather than than shooting each other, hopefully. But I do think there is something to be said for how that changes the calculation underwater as well.

Michael Horowitz What do you mean to say a little bit about the about changing norms of warfare and, you know, almost like a logic of appropriateness in some ways about violence against civilians in this context. We have another question in the Q&A. It says, you know, civilized people, the 1930s thought gassing cities was going to be effective and humane. We've come a long way if we're looking at slaughter bots. For those who don't know that was a video produced by the campaign to Stop Killer Robots, which is which is not even worried about drugs. They're worried about like autonomous killing machines. How are these kind of norms changing and what does this mean?

Michael Boyle  Yeah, it's a really good question. I mean, I think I'm always suspect of any claim that technology is going to completely, radically eliminate warfare. I mean, we saw Tesla say that with drones. We've seen that in a lot of other places. And it just generally doesn't tend to be true. And humans are remarkably adaptable in finding ways to use technology to harm each other. I do think there's something going on that I think is really interesting when you think about the way the norms are changing—and I kind of hinted at this a little bit—and talk about what some does call the individualization of violence. Maybe it is not the most sort of pretty way of saying it, but it's about changing norms that we are now more able to attack individuals rather than groups. I mean, if you were to go back 20 to 30 years ago, we generally didn't attack individuals except as part of intelligence and tradecraft. So you might see assassinations as part if intelligence, for example, but you didn't have the military attacking individuals. And certainly from the practice of targeted killing, you see much more individualization of violence. We're gonna take up this specific terrorist and almost treating them as a criminal, for example. And the danger there is that this individualization of violence, I think, is actually changing our norms a little bit.

Now, you can argue it's a good thing rather than attacking large masses of people, we're talking just the people that are responsible for a specific crisis or specific points. The danger is if the individualization of violence becomes more frequent or it becomes one of these things that were done in a more capricious way and that has second order and third order effects. And that's something I've raised in previous work that I've written to raise the concerns that when we talk about certain that when we talk strategically about it, we have to think about that. You know, you can really look at something like the recent story that came about, the ninja Hellfire missiles, missiles that are flown directly into a car and kill just the target and no one else as an interesting example of what happens when we have the individualization of violence. That increasingly the U.S. is trying to seek to kill just that individual person. Again, there are humanitarian—you can make a humane argument that's more humane to kill one person than many. But I would worry more about what that does in terms of permissive behavior. And second and third order effects. So the establishment of that norm isn't necessarily good.

Michael Horowitz So we've had a couple of questions that ask about what these trends will mean for militaries and in some ways what what recruiting and retention will look like. That as people are increasingly fighting, and you get to this a little bit in your book—you talk about the psychology of this. But what if people, if pilots are being recruited, to sit behind a desk, to sit with a joystick rather than like do fun top gun stuff, how is that going to affect you the way that militaries function and who joins?

Michael Boyle It's a really good question. So this is something that the interaction, the technology is having organizational effects. I argue that it's having real pressures on the Pentagon to collect more information and to process more information than it's having consequences. But it's also having consequences for pilots. There are a lot of stories about the degree to which the desire for information is exhausting pilots that we're having situations where pilots are being able to work particular shifts and also doing in some cases of post-traumatic stress. But we're also seeing cases under which the military is having to adapt its recruitment networks in order to to fill the demand for pilots. For example, do you allow contractors to fly drones? Do you allow people who don't go through the same pilot training? And there have been cultural issues as well as this as well about, you know, how do we reward an appropriately incentivized drone activity.

Michael Horowitz Like the drone medal controversy.

Michael Boyle Drone medal. And it was dismissed by people from within the Air Force as the Nintendo Medal. And that's an interesting example about how the culture can kind of reject the drone technology itself and say, well, no, this isn't the same as flying a manned aircraft. So I think when we think about what this does in terms of recruitment, you're seeing a vast organizational change and I think an understudy organizational change inside the Pentagon and inside the military itself about what happens when this technology is introduced and what does it do. And then that's producing cultural stresses. It's also producing stresses in terms of recruitment and retention. Right. How do you keep people? How do you keep people into that job? And how do you kind of incentivize them to continue to do it, given the pressures that they are under.

Michael Horowitz All right, let me ask you one more question and then we will wrap up and that's and that's what's next for Mike Boyle? You've just written this awesome book on drones, I presume, like all academics, you have that list of projects that you hope to do someday. So where are you turning your attention next?

Michael Boyle That's a great question. So I haven't just finished this, I have started research on it but haven't really done enough yet. I'm interested in looking at variations in technology adoption by non-state actors. And that's an extraordinarily difficult thing to get to because you don't have data. Right? When we say at least twelve terrorist organizations have it, we know that more have it. And how do you get good data on it? But I think there's something to be seen about variations in repertoires of use. So even if we're not trying to quantify it, but who has many, what we see as different styles and patterns of use begin to emerge. And my question is, is that a function of the threat environment? Is that a function of the strategic orientation or is that a function of their internal organizational structure? So is it because the different terrorist organizations, for example, can learn better and adapt better? Maybe it is the organizational structure of non-state actors that in fact actually helps them use things like unmanned technology more efficiently. So that's a long term project. I'm hoping to turn that into another book at some point down the line. But I have promised my wife that there will be no more books for the next year or two at least until our new baby is born. I promise new books. But at a certain point, there will be another book a couple of years down the line.

Michael Horowitz Well, congratulations and thank you. Those that don't know, we tried to schedule this event around some important life events. And we appreciate you're making the time. And, you know, thank you for those insights. And I hope everybody will go buy and check out Michael Boyle's book, "The Drone Age." It's always great to have you back at Perry World House. And hopefully someday when we're in the after time, you can join us in person at Perry World House as well. We are out of time for today. I thank all of you for joining us. And we hope to see you next week when we have four exciting events as part of our upcoming colloquium on the "U.N. at 75: Coronavirus and Competition." More information and links to register are in the chat. As always, you can access a recording of this conversation on YouTube and find out about our upcoming events by joining our mailing lists or checking out our web page or following us @PerryWorldHouse on Instagram, Twitter or whatever social media platform the kids are using that I don't know about. Thanks again to Mike Boyle for a tremendous conversation and writing a really cool book. And I hope everybody has a great day.

Michael Boyle Thank you again, Mike. Thank you, everybody.