Anastasiia Voitova -- Encryption is easy, key management is hard

0:01Chris RomeoAnastasiia Voitova is the head of customer solutions and a security software engineer at Cossack Labs. She works on data security and encryption tools and their integration into real-world applications. Anastasiia joins us to discuss end-to-end encryption and how that fits within the modern web or mobile application. We talk about how user experience affects encryption. some of the threats that end-to-end encryption mitigates, and then we dive into some of the popular messaging apps and how they are impacted by end-to-end encryption. And spoiler alert, they're not as secure as you might think they are. So we hope you enjoy this conversation with Anastasiia Voitova. Are you trying to build a security champions program? Everyone is these days. One challenge of rolling out security champions is how do we educate all these new folks? Security Journey has your answer. We provide a Security Dojo environment with level-based security education that gives your newfound champions a path to follow. And the best part? It requires almost zero administration by you. Visit www.securityjourney.com to set up a demo and learn how you can use the Security Dojo to connect with your security champions. Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo, CEO and co-host of said podcast. I'm joined once again, as always, by my friend Robert Hurlbut. Hey Robert, how you doing? Hey Chris.

1:38Robert HurlbutYeah, Robert Hurlbut here, Threat Modeling Architect. Really excited about the topic today on encryption.

1:43Chris RomeoEncryption's one of those things where a lot of people are scared of this and Probably for good reason, but we're gonna unpack a little bit about why we need to be scared, maybe why we not need to be so scared. I don't know. But our guest today is Anastasiia Voitova. She has been on the podcast before. So, the previous episode and interview that she did with us was on the topic of Use Cryptography, Don't Learn It, which I originally found Anastasiia because I saw this talk that she had done. I was looking at the slides and that original title was like, kind of like caught my attention, like, wait a minute, use cryptography. Don't look like we don't want to learn. I'm kind of a teacher at heart. I want people to learn about this stuff. But she makes a very compelling argument for why people need to use cryptography but not necessarily understand all the depths that sit within it. So you can go back and listen to that episode. But today we're going to talk about something else. Robert, why don't you kick off our first question here to get us going?

2:44Robert HurlbutSure. So welcome, Anastasiia. First question is, what is end-to-end encryption? Now, we've heard about it, but tell us about it. What is it?

2:54Anastasiia VoitovaYeah. Hi. Hi again. So, our topic for today is cryptography again, right? And this previous time we discussed and learned it, but today it looks like we are going deeper. Okay. So, end-to-end encryption is one of the application layer encryption ways, right? Because encryption is just like an umbrella term and there are different kinds of encryption. We say TLS, which is data, um, like motion encryption, right? We say data at rest encryption, like database encryption, for example. Both of them are encryption, but, but both of them are different types of, uh, like protocols that use different types of ciphers to solve, uh, to protect from different threats. So application-level encryption means that application is responsible for encrypting and decrypting data, right? Application-level encryption works together, obviously, with TLS and with database encryption, with basically any other encryption. It's just another layer of abstraction. When we are talking about end-to-end encryption, in most cases, this is just application-level encryption But done in a way that applications are communicating with encrypted data, right? So data is encrypted on the application side before it gets from the application. Technically, no matter what kind of application, technically we can say mobile application, we can say web application, right? It's important that application encrypts the data before it sends through the network, through the servers, and the backend doesn't have decryption keys. So only another, only target application can decrypt the data, building this end-to-end channel, which means that data goes from the source to destination being encrypted all the time. If this data is being processed by backend, that's fine, it's encrypted. If this data is being stored in a database, that's fine, it's already encrypted. If this data is appeared in our logs, our backups, sorry, it's encrypted, right? Often we talk about end-to-end encryption when we are talking about mobile applications, and, and in this it's not only application itself but the user as well, right? So often we link application layer encryption with the user as like a person and their Trust anchor, password, biometrics, right? So the users are the ends and the data got decrypted for these particular users. But of course it's not, it's just one of the cases. We can talk about end-to-end encryption on the server level without any users involved, right? The main idea that only particular application can decrypt the data, which makes it really interesting from key management perspective, because obviously we need different kinds of keys for encrypting and decrypting data. And most likely, Many, many end-to-end encryption protocols, they don't rely on symmetric encryption itself, right? Because symmetric encryption means like one key, secret, secret key. And obviously when you want to build this tunnel, you don't want to have only one key. You want to have the key hierarchy. You wanna have all this, you know, nice and shiny key encryption key, data encryption key, customer, like customer master key that encrypts key encryption key, yada, yada, yada, yada. So this is where it gets interesting.

6:28Chris RomeoSo, I want to just summarize for a second and make sure that I understood. Here's how I'm thinking about this. Tell me if this is a correct kind of conclusion based on what you just described for end-to-end encryption. End-to-end encryption then provides data at rest and data in motion protection because the application— because the ability to encrypt and decrypt happens at the application layer, If that information gets passed through a backend, the backend can't decrypt it because it doesn't have the right keys. All the backend sees is a blob of ciphertext, cryptotext, whatever, that's in the backend. And so that's kind of a fascinating point for me that I never thought about the fact that you could have data at rest and data in motion in the same sentence.

7:20Robert HurlbutYeah.

7:21Chris RomeoSo are these 2 things really happening at the same time?

7:23Anastasiia VoitovaYes, absolutely. Imagine like a JSON model, right? Some JSON file, JSON-like object with some fields. And let's say we want to encrypt email, right? So I'm building mobile application to encrypt email field from my JSON model. I will encrypt only this particular field, right? So our JSON can be plaintext, email is encrypted plaintext. And then I will send it to you and you, for example, you have another mobile application. So this JSON, first of all, it will appear on our backend using TLS, right? So our email field is kind of twice encrypted because first it was encrypted on my mobile phone and then we use typical TLS, obviously, because everyone uses TLS, right? So technically we have 2 layers of encryption on the email field. Right? Then, for example, it's gonna— it's stored in the database. And let's say we use some database, I don't know, let's put it in MongoDB and let's use some data at rest encryption built into the database, right? So another, like, second layer of encryption. But even if we don't use TLS, even if we don't use data at rest encryption, we still have this email field encrypted until it comes to your mobile phone and you have keys To decrypt it. So yeah, application-level encryption allows us to have data encrypted as long as we want. If it goes through transit, data in motion, it goes— if it's stored, data at rest. At the same time, it means that we can still use all the things we use right now, right? So in some parts of our data flow, data are going to be twice, 3 times encrypted, and that's fine.

9:15Chris RomeoYeah, I can see how that, you know, for a modern web application, modern mobile application, how that really provides a lot of protection for our data and gives us a certain amount of assurance that our data hasn't been, you know, hasn't had the, you know, CIA kind of triad: confidentiality, integrity. I guess availability could still be, could still be messed with, but confidentiality and integrity are going to be intact because, you know, you're at least encrypting once. Sometimes maybe that data may be encrypted twice, maybe encrypted 3 different times as it's making its way through your architecture. So yeah, that makes sense to me as far as— and yeah, I'd never thought of it that way though. So you're expanding my mind here a little bit to think about there's a way to do more than just one of those properties.

10:08Anastasiia VoitovaI really like to think about it from threat modeling perspective, right? Because this application level encryption is like like ultimate security control, right? Because it protects from threats during transit. It protects from different threats of data leakage, data loss during storage, right? It also protects from the threats with I don't know accidental like. server misconfiguration, leaving S3 buckets in public, you know, from leaking backups, from leaking this data in logs. So like application-level encryption is like ultimate security controls that can like one shot to protect from different threats. But of course it comes with a cost, right? We cannot say that, oh yeah, this is shiny thing. Yeah, it's shiny thing, but it's a costly thing. Because if data is encrypted all the time, it cannot be processed. I mean, we have encrypted email, right? It means that database doesn't have any encryption keys. We cannot search through email field because it's encrypted. It's just a binary blob of something. Of course, if we use strong encryption, our binary blob is just gibberish, right? It's, uh, it's very complicated to brute force it, to try to understand, to do some frequency analysis. If you use strong encryption, our data is encrypted very well. Um, obviously in the modern world we have some techniques to solve search as well, because different searchable encryption approaches are possible from cipher's perspective. Like, we can use different ciphers to decrease security guarantees but increase ability to search. Through encrypted data, but at the same time we can use modern approaches like blind indexing, for example, to make this email encrypted but searchable at the same time. I really am a big fan of application-level encryption. I really like think that, uh, soon, especially when our infrastructures, you know, they have hundreds of microservices, so using nice encryption thing that gets data encrypted all the time, even when most of these microservices are vulnerable, it's a great idea. So, having all these searchable encryption techniques is really like our way to the future.

12:36Chris RomeoSo, the S3 example of the bucket, a wide-open bucket, is a great— it really resonates with me because that's one of those ones where it's, you know, even today, if you create a new bucket on AWS, it comes with public turned off now. Like, you specifically really have to jump through some hoops to to make a bucket be world accessible, but yet they're still out there. They're still, you know, people are still doing it, unfortunately, because it's a usability thing. They're like, ah, no one's going to ever know about this particular bucket. But the fact that end-to-end encryption can solve that because you don't, you don't, you're not worried about the privacy of it at that point. Like it could, that you could advertise that bucket as being available and tell people, come and get it, come and download whatever you want. As long as you've used a strong enough cipher, you know, enough long enough key and stuff, as long as you're not using DES encryption or something on it, you know, something crazy that people have been able to break. So that leads me to another question though, that's a little bit off the script, I guess, but all of that sounds great, right? Like you got— I'm with you so far. I'm tracking with you that, you know, we should do end-to-end encryption for everything. There's got to be some downsides, though. Like, it can't be— and you alluded to the fact that it can't be— if it was easy, everybody would do it.

13:59Anastasiia VoitovaYeah.

14:00Chris RomeoAnd so, what are the downsides, though? What are the challenges of end-to-end encryption? You mentioned searchability, but you kind of had some answers for that. But why would I not— like, what's the pain that I'm bringing into my life by bringing end-to-end encryption into my architecture?

14:15Anastasiia VoitovaYeah. So searchability is one of the most like important ones because again, this is a trade-off. Either we have nice security and resilience against frequency attacks, right? Or we have like searchability, operationability. I would put it this way.

14:32Chris RomeoYeah.

14:33Anastasiia VoitovaThere are some ways to work around it, but still we, right now, fully homomorphic encryption is not here yet. So we cannot have all the wide range of data operations on encrypted data yet. So large minus, another performance. Obviously when we do application-level encryption, even if we selected only part of our data to be encrypted, right? Not the whole JSON model, but one field in the JSON model, we still, we still add some, like we still have this performance penalties. And so the most important one probably is key management because encryption itself, it's, it's very fast. It's very easy, encrypt the data. Have plaintext, now we have binary data, easy as that. Key management is a problem. With application-level encryption, like with end-to-end encryption, I need, as one of the ends, I need keys to encrypt data, and you as another end need keys to decrypt data, right? So we need to get these keys from somewhere, we need to put it somewhere, we need to have a way— all, basically all these NIST guidelines on key management. Key generation, rotation, revocation, expiration, sharing, yada yada. We need to build this. And one of my talks, I really like this topic because encryption is easy, as they say, key management is hard because key management affects the UX of your system, right? It affects how you basically, how you build these built-in calls, API calls in your application to get those keys. Do you store them on the backend? In this case, it might mean that this is not an honest end-to-end encryption, that backend still has some access to the keys. So not, not so honest, you know what I mean? Right? So the trade-off is to build a system that gives good enough security guarantees, but it's not catastrophic in terms of all these additional layers we need to build to make key management also secure. And usable.

16:38Chris RomeoSo, one more question before we transition to kind of more the UX, because I want to go in that direction. And I know Robert's got a question on that. But one more thing on the performance, just from your perspective as someone who is thinking about cryptography and is a cryptographer that's doing a lot of work in encryption, like my modern— my phone, my iPhone 11, Is it reached a point where the encryption operations are all being done in hardware and it's really not— it's not expensive then as an operation to do end-to-end encryption from the endpoint? Is that a fair statement?

17:17Anastasiia VoitovaYeah, yeah, yeah. In most cases, like, let's put like small asterisks there, but yeah, in most cases, encryption is really fast and cheap in terms of resources. Because most modern ciphers, when we're talking about symmetric encryption, AES, right, AES-GCM, for example, it already has hardware acceleration for years for all the modern devices that we use, right? So when we are talking about mobile applications, easy. Of course, with some caveats. Of course, when we are talking about encrypting huge amount of data, yes, obviously it's gonna take some time. But when we are talking about chat applications, when we are talking about Like working with this selective encryption, encrypting 1 field, 2 fields, 10 fields out of huge model, right? These are just bytes, kilobytes of data, small, small pieces. It's really fast and effective. Another idea is IoT devices, right? Small devices for like small specialized devices. Fortunately, we have ciphers that work for them. Yeah, iOS might be a bad, uh, bad solution for small stupid GPU CPUs, right? But we have lightweight cryptography. We have special ciphers designed to be stupid and work on these small devices. So yeah, encryption is really fast right now, especially comparing with all the languages, Python, Ruby, that we have on our backends, right? Compared with all these huge monstrous infrastructures with microservices that make calls to another microservices that make calls to another microservices.

19:01Robert HurlbutYou mentioned about UX and some of the things that could affect it, like performance and so forth, and also mentioned honest end-to-end encryption. I want to know a little bit about both of those more, about the UX as well as this term you mentioned, honest end-to-end. What does that, what do those mean? And how does that impact, for example, UX?

19:26Anastasiia VoitovaYeah, 'cause let's put it this way, like end-to-end encryption is a cool marketing term, right? You can see many applications use, yeah, we use end-to-end encryption because it sounds cool. At the same time, like honest end-to-end encryption, honest, I mean, with all these honest ideas that only end applications or end users have access to encryption decryption keys. The all infrastructure in the middle, the all backend infrastructure should not have access to decryption keys, right? So this is like the honest approach. Decryption keys, encryption keys only on the ends, only on this services that are target to this data, right? So it's very complicated to build because of the key management, because of this, the keys should be generated, keys should be put to the other side somehow. Securely, right? So usually we are talking about wrapping techniques, different wrapping techniques when encryption key is being wrapped with another encryption key. You know what I mean? And often applications they combine typical client-server encryption and end-to-end encryption. One of them, like one of the great examples, is Telegram, which answers both questions about honest encryption and UX. You can see if you have used Telegram before, you see that they market a lot that they are super secure messenger, right? At the same time, they state in their docs that they use typical client-side encryption for chats and for channels. Client-side means that the data is encrypted on mobile side, yes, but gets decrypted on the backend. By design, right? And this typical client-side encryption will cover like 90% of all the features in Telegram. They use end-to-end encryption only in a very, very narrow use case, in special secret chats. Secret chat that you create as a user from your mobile phone, because only mobile applications support that. You create the secret chat to another user on their phone, and this secret chat is like, it's like peer-to-peer in a sense. It goes through this exact installation of this exact application of this exact user. Data gets encrypted there and data gets encrypted on this exact installation, exact device of the second user, right? And Telegram, they kind of combine this typical client-side encryption for most use cases, for most users, and this very special end-to-end encryption for paranoid users with better security guarantees. And these secret chats, if you use them before Signal, by the way, Signal does the same in terms of secret chats. They show you all these notifications, like be aware that this user has this public key. It's your responsibility to check their public key. You know, they increase the security like guarantees, but at the same time, they make UX more complicated for typical user, for not security-aware user, because of all this, check the keys, make sure that this chat is only synced between 2 devices and that's all. So you won't have a copy of this chat in your desktop, right? And many applications. So yeah, my point here is that end-to-end encryption is complicated in terms of key management. So, many applications use hybrid approach. Even when they market as end-to-end encryption, they can use typical client-server encryption for most cases and end-to-end encryption for narrow cases because it affects UX. It affects, like, user perception of what users need to do.

23:18Chris RomeoDoes that mean that those, from what you're kind of describing here, like, should we be less trusting or less positive on these secure— I'm going to put in air quotes— secure messaging apps? Like, should we be more concerned about them based on this? Like, because I didn't know this.

23:40Robert HurlbutI would've—

23:41Chris Romeoif you would've asked me before the interview started, hey, Telegram, Signal, all these types of things, like, are they doing end-to-end encryption so that people, you know, the backend is not decrypting and not knowing what's happening? I would've said yes, but I think I would've been naive in my answer by saying yes. So does that mean, do you trust these messaging apps less because of their architecture?

24:03Anastasiia VoitovaWell, it depends on how much they say us, right? Telegram has this in docs. So technically they can say we are using end-to-end encryption because they do, but you know, for narrow use case. And I did a comparison of Telegram, Signal, and WhatsApp. And from my perspective, Signal is the most kind of trusted one because they communicate in docs exactly what they do, right? They have this end-to-end encryption flows all the time. WhatsApp, again, from my very opinionated take, is the most questionable one, right? Because end-to-end encryption, even when application uses end-to-end encryption, it doesn't mean that all your data is like private because security, privacy, and encryption are 3 different things, right? So imagine that, yeah, for example, let's say that WhatsApp uses end-to-end encryption everywhere, but they can still grab your data, sell it elsewhere before encrypting, right? Using end-to-end encryption itself, having this like encrypt line in code doesn't mean that the data is being encrypted instantly. The data can be first analyzed, stored somewhere, put into some analytics before encryption, right? So privacy and encryption are different things. Again, application can have really good security in terms of what we understand from good security right now in terms of WASP guidelines. At the same time, application can have really good privacy but no encryption, like no end-to-end encryption at all. That's fine. Right? 3 different things, security, privacy, and encryption. Like, combine, select 2, you know?

25:47Robert HurlbutYeah.

25:49Chris RomeoI mean, Signal has been my go-to as a messaging app for the reasons that you described. Like, just by reading what it— what they advertise as far as what they do, it seems like they're the most transparent amongst all of the messaging apps. But now I'm realizing that Now that you've explained architecturally, that's also a better— it's a better choice because they're, once again, transparent, but also using the end-to-end encryption approach more clearly than either of the other messaging providers that are out there.

26:23Robert HurlbutYeah.

26:25Anastasiia VoitovaAnd as Signal user, you can feel the UX penalty, right? Because if you tried using Signal desktop application, for example, You know, like you can use mobile application and desktop application. Their desktop application, this, it's not a standalone application. It should be linked to your mobile app. So mobile app is a source of truth, right? And, and this is, this is like they have this linking process where you need to scan a QR code, yada yada. They have all this limitation how many desktop applications you can have simultaneously. So this is like a UX penalty for using end-to-end encryption because they don't— they cannot provide you a freedom of installing 1,000, like, okay, 5 applications, but still have end-to-end encryption. That's very complicated.

27:16Robert HurlbutBut those are some of the trade-offs, right? And that kind of goes back to, you know, application design and UX and so forth.

27:24Anastasiia VoitovaAbsolutely. Yeah.

27:24Robert HurlbutThe various trade-offs once you decide What's my privacy solution, my security solution, encryption solution, and so on, and still trying to build an app that's usable and secure and so forth. But, you know, those are all trade-offs. And it sounds like— I mean, it's good to know a little bit about some of the others. I don't know if we were planning to do that in this podcast, but it's pretty cool to hear about some of the differences. But again, trade-offs. We all are—

27:52Anastasiia VoitovaTrade-offs.

27:52Robert HurlbutThinking about trade-offs when we're building apps and how do we make them security versus usability, privacy, and so forth that we have to think about.

28:00Anastasiia VoitovaAbsolutely. Another trade-off into this connection from an end perspective is engineering amounts, like engineering efforts and UX, right? So we can build, we can make user life easier, improve UX, but make our life as software engineers hard because we need to build this really multilayered system, you know, to keep in mind all these use cases and corner cases. And this is what I, I often deal with when designing encryption layers, because the more, the easier you want to do system for user, usually the more complicated it's going to be for developers, for engineers to make, to build and then maintain, support, and other things. So, yeah, trade-offs, trade-offs everywhere.

28:53Chris RomeoYeah. The maintenance and support are some of the most challenging pieces of that because when you're thinking and designing and building something the first time, you're understanding how all the pieces come together. The challenge is like 12 months from now when someone gets a bug and they're going like, why are there 7 layers of encryption things happening here? And I don't know what's happening in any of them. And then they have to relearn it, you know, to be able to support it. So, you know, keep it simple is always a good design principle in— I don't care what you're designing, if it's a swimming pool or an end-to-end encryption, keep it simple. Because if you make it complicated, no one's going to be able to figure out what it actually does. And that, I guess, in some regards could be more secure if nobody can figure it out. But somebody will figure it out. somewhere on earth.

29:43Anastasiia VoitovaThat's why, that's why typical end-to-end encryption protocols are so complicated. If you read, if you ever read the description of Double Ratchet end-to-end encryption protocol behind Signal, right? Signal protocol, it's very complicated. The key, okay, not like very, but quite complicated. The key management, they have different steps of key management itself, right? Before even the encryption, And they need to build in all this protection against replay attacks, for example. So yeah, a lot of trade-offs. Security engineers often try to make security systems as easy as possible, but real life is still here with all the threats we have. And every time we need to make this decision if we wanna protect our system against these threats and how much it'll cost for, for us as engineers and how much pain we will add into people's life. You know?

30:38Chris RomeoYeah. So, I guess from a key takeaway or call to action perspective, what would you like our audience to do as a result of this? From your perspective, is there something they can go study? Is there something they can go watch? Is there something, some action, something they can go do to make the world a more secure place with end-to-end encryption?

31:05Anastasiia VoitovaAbsolutely. So if like, if you ever see, I don't know, Jira ticket that says let's add encryption here and it's estimated like 3 hours, it will never be 3 hours. Okay. So the most important key takeaway is that encryption, it's not just an encryption. It's the whole system of deciding what data to encrypt, which cipher to use, what library to use. How key management should look like and what the trade-offs that we will make in our system security, usability, development efforts, yada yada. So encryption is never encryption. So if you want to build anything with encryption related, beware. Probably most likely there are already ready solutions or building blocks or open source libraries that can make your life easier. or go talk with people who write, like, who build encryption-based systems, just because probably you don't right now understand how deep this, like, hole, this abyss is.

32:10Chris RomeoWell, thank you so much, Anastasiia, for joining us again on the podcast. And, you know, I know you expanded my mind. I'm sure Robert's gonna agree as well. Like, it's, you expanded my mind in a lot of different ways about encryption that, I just hadn't thought about before. But I love this idea of end-to-end encryption. I love the idea of it being a user layer thing where it doesn't matter who can see it on the backend, doesn't matter what nation state gets access to it, it's not available for them. Even if it traverses their country and sits on a server in their country, they can't unlock it. And that's a very powerful thing that I think as we move towards a more secure and more private world and a more secure and private internet, these are going to be— this is going to be one of the building blocks that a lot of things have to be built upon, this whole idea of end-to-end. So, once again, thank you so much for being here with us, and we look forward to another conversation in the future where you teach me a whole lot more about cryptography and encryption. So, thank you very much.

33:12Anastasiia VoitovaMy pleasure. Thank you for having me.

33:14Chris RomeoThanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast and on the web at www.securityjourney.com/resources/podcast. You can also find Chris on Twitter @EdgeRoute and Robert @RobertHurlbut. Remember, with application security, there are many paths, but only one destination.