Dan Lorenc is a Software Engineer at Google and lead for Project Sigstore a Linux Foundation project. Dan talks about his history at google with projects such as skaffold and minikube which inspired his work currently with Project Sigstore. Dan currently works on a cloud native project called Sigstore which is a non-profit, public good software signing & transparency service. Software supply chains are exposed to multiple risks. Users are susceptible to various targeted attacks, along with account and cryptographic key compromise. Keys in particular are a challenge for software maintainers to manage. We talk Software Supply Chain Security and Software Supply Chain Devops along with our mutual love of our home of Upstate NY. Dan provides some valuable advice on how to protect your software supply chain and this is a very fun episode! On June 18th 2021 - we will be holding our first Root Key ceremony on June 18th at 2pm Eastern on June 18th at 2pm Eastern on CloudNative.tv (CNCF twitch). Please join us more details at this link: https://blog.sigstore.dev/a-new-kind-of-trust-root-f11eeeed92ef
On June 18th 2021 - we will be holding our first Root Key ceremony on June 18th at 2pm Eastern on June 18th at 2pm Eastern on CloudNative.tv (CNCF twitch). Please join us more details at this link: https://blog.sigstore.dev/a-new-kind-of-trust-root-f11eeeed92ef
Timeline/Topic
00:00 -- Sigstore Key Ceremony June 16th 2021 - EXCLUSIVELY on CloudNative.tv
00:15 -- POPCAST Opener (Like and Subscribe and leave a comment!)
00:23 -- Introduction to Dan Lorenc Google Software Engineer and lead for Sigstore
01:00 -- "Lorenc" pronounced like "LAWRENCE"
01:46 -- the Dan's talk Upstate NY / Freihofer's Bakery but specifically Cookies
04:33 -- Dan's Journey to Google
08:36 -- Dan talks Skaffold
11:09 -- Dan talks Minikube
13:08 -- Secure Software Supply Chain... whats the problem we need to solve?
15:43 -- Dan provides some advice on how to Secure Software Supply Chain
21:22 -- How a company's culture can help shape better security.
23:43 -- Sigstore / Cosign - what is it and why you need it.
27:44 -- What a Sigstore Key Signing Cermony is (a full explanation)
34:20 -- what work are you most proud of?
Please leave a comment if you enjoyed the episode! it helps the show!
Brought to you by:
***Teleport***
Teleport allows engineers and security professionals to unify access for SSH servers, Kubernetes clusters, web applications, and databases across all environments. You can download Teleport at https://goteleport.com
***Sysdig***
Run Confidently with Secure DevOps Security for containers, Kubernetes, and cloud
***COCKROACH LABS***
Discover @CockroachDB the most highly evolved distributed SQL database on the planet.
Kubernetes-native and built from the ground up to help companies of all sizes including Bose,
Comcast, and Equifax scale fast, survive anything, and thrive everywhere.
Sign up for a free 30-day trial and get a free t-shirt at https://cockroachlabs.com/popcast
***Styra***
Learn how to operationalize Open Policy Agent at scale with Styra: https://hubs.ly/H0Pnkm20
***CIVO***
Civo is an alternative to the big hyperscale cloud providers.
They've launched world's first managed Kubernetes service powered by K3s.
With sub 90 second cluster launch times, a simplified Kubernetes experience,
and predictable billing, Civo is on a mission to create a better developer experience.
Get $250 free credit to get started. Sign up today at https://civo.com/popcast
Episode Links
Sigstore - https://sigstore.dev/
Sigstore Root Key Ceremony Blog Post - https://blog.sigstore.dev/a-new-kind-of-trust-root-f11eeeed92ef
Ken Thompson Paper - https://www.cs.cmu.edu/~rdriley/487/papers/Thompson\_1984\_ReflectionsonTrustingTrust.pdf
Dan's Malware with Falco blog - https://dlorenc.medium.com/hunting-for-malware-with-falco-834b19b398c9
POPCAST SHOW DETAILS
YouTube: https://bit.ly/3xgmmCj
Audio Podcast (Apple, Spotify, and others): http://bit.ly/35MXfte
Follow us on (Twitter): https://twitter.com/PopcastPop
Follow us on (Linkedin): https://www.linkedin.com/company/the-popcast-with-danpop
- [Announcer] Join us for the 6th Door Key Ceremony. Only on Cloud Native TV this Friday, the 18th.
- Hello everyone. And welcome to the POPCAST. Look, this is my namesake here. This is my guy, like from day one. From day one, this guy, like this. Not only 'cause we're the name, but we're gonna get into it. We're from the same parts of upstate New York. Okay, we're gonna talk.
- I think the name helps though.
- It helps. So all right, let me run my show here, Dan, okay? Google people. All right, so this is Dan Lorenc, he's a software engineer at Google. Welcome to the POPCAST.
- I'm so happy to be here.
- I'm so excited to have you, buddy. I didn't really rehearse that. All right, so listen, let's talk about little Dan Lorenc. Normally, you'd see somebody like named Lawrence, The last name is L-A-W-R-E-N-C-E. Did you ever get that question? Like, hey, Lorenc, why is it spelled like that? Let's talk about that before we get into the journey.
- This is the first time anybody's ever brought that up actually. I never realized.
- Really?
- Of course not, no.
- Try going through life with a name POP Andrea, pal, All right? Come on.
- Yeah. No, it's a L-O-R-E-N-C. I've heard pretty much every pronunciation. I don't bother to correct people, it doesn't bother me at this point. You said it right. It was a pleasant surprise. I don't remember ever correcting you. So where did you hear it from?
- I don't know, I stalked you. So that's probably the reason.
- I see.
- All right, so listen, my man. Let's go back. Let's go back to little Dan, all right. Little Dan's growing up. Where'd you grow up, bud? Tell the world.
- I grew up and upstate New York. If you're not from Upstate New York, or you're not from New York in general, then that term's a little confusing. I define Upstate New York is wherever you live in New York state, and then above that. Right? So we get the people like Dan POP here, who are from 20 minutes outside of New York City that consider themselves Upstate.
- Whoa, whoa, whoa, time. Timeout. I'm gonna stop the interview right there. I am from a little... I was born in Brooklyn, everyone, but when I was five, I raised in upstate New York, a little town called The City of the Hills, Oneonta, New York. Known as the place where Brooks Chicken comes from, everyone. A little history lesson for you. And where are you from in relation, Mr? You're actually big city man at the capital district, buddy. So tell me a little bit about where you're from, pal.
- That's right, that's right. So it's a relative term, like I'm saying. It's wherever you are and farther north. So I'm from Albany, New York, which is the capital. It's actually not that far north, right? So I'm giving POP some trouble here, but I used to get it too. There are these people that grew up on the Canadian border right by Montreal that consider themselves to be upstate, right? And they'd laugh at both of us. You know how it works. There's no line here, right?
- If you were like a four... Somebody who had a four wheeler or something like that, you're from upstate New York. So that's why, me and this guy bonded over that stuff. But the other thing we bonded over, okay, is normal in this Cloud Native space, is food, right? So we started to talk about... So we're gonna go in through to journey everyone. We'll talk about the Google and this Key Sign, we're gonna talk about all that stuff, but we're gonna talk about this first. So me and this guy started talking about this company. It's an upstate New York thing. It was this place called Freihofer's. Freihofer's had the delicious cookies, and delicious donuts. This just reminds you of upstate New York right now, huh?
- I can taste them, just thinking about it. This is like a factory bakery. This isn't a small family shop. Let's get that clear. But it's still, it's an amazing factory bakery, if you can put it that way. Every grocery store, you can get this stuff, but only in Upstate New York, I think. I've never seen it anywhere else. I remember the Fruit and Nut Cookies. Do you remember those?
- I remember the Fruit and Nut Cookies, yeah. They were Delicious. Delicious. And those chocolate covered donuts, I remember. This is me, okay? I've lost-
- The little ones, right?
- The little ones. I watched that movie, The Lost Boys the first time. And that was my thing. I had my Hershey's Chocolate Milk, and my Freihofer's. And that's why I was morbidly obese, by the time I was like 18, I don't know, anyway. So moving on, let's talk about this. So you grew up in the Albany area. Tell me about the path, the first time you picked up a computer, bud.
- Oh man, I don't remember. I did not like programming. I did not like computers though, that much growing up. My undergrad was actually mechanical engineering. And so you mentioned the ATVs, that's what I grew up doing. Dirt bikes, ATVs, used to race stock cars in Upstate New York on a mud track, that kinda thing.
- Yeah, smaller than that. Even more out there, it was in Albany Saratoga Speedway, I think it was called. If you look that up. So even farther upstate than I am, I was. Right up there. I was 16. You'd get these cars out of a junkyard. There's a limit. You couldn't spend more than 300 bucks on your car, total, I think it was. And that was it. You'd go and race them every Friday night.
- Jesus, you're the a man, I missed you. I'm here like, I should have asked you more questions, pre-questions.
- And then you went to Freihofer's Cookies on the way home.
- Oh, boy. So, okay. So we did the mechanical engineering. You're like, "I like to take things apart, fix them or whatever." And then what happened next?
- Yeah, I think the first real program and I did, we had to learn MATLAB for mechanical engineering to do all these simulations and stuff. And then from there I was like, "All right, fine, this is a little bit more fun." It's faster, you know? You don't have to wait and order parts from a shop, and have them shipped, reserve time in the machine shop, that kinda thing. So kinda got that fast reiteration cycle, and I got hooked after that.
- Where did we go to school?
- I went to school at a MIT. So still Northeast.
- Jesus, you're such... My God, you're a humble MOFO. Jesus Christ. So you go to MIT.
- I thought you said you stalked me?
- I didn't stalk you.
- You had the Yankees hat on, So you're Red Sox.
- And here's the other thing. Let's talk about Upstate New York from that. There's a line that basically people are Red Sox fans, because the proximity to Massachusetts, then there's the New York thing.
- It's about the same distance, yeah.
- Right? Like it's the middle. Like it's right in the middle. So you were more a Boston fan just because you went to school there and stuff like that, right?
- Yeah, I think I was. I don't remember the years now, yeah. I think I was there when they broke the curse and won that first world series. It was a special time to be in Boston.
- God, I hate you. All right, so moving on. So we went to MIT, how do we get to Google? Tell me the steps we got to Google.
- Sure. It wasn't too long. Yeah, I stayed right on the I-90 there in Albany, Boston. And then back out the other way, I was in Seattle for a couple of years, working for Microsoft, then moved down to the Bay area. I was at a startup for a little bit and then got a job at Google.
- So let's talk about Microsoft. You're glancing over a couple of jobs just as you do, 'cause you're a humble dude. But let's talk about the Microsoft piece. Like what did you do at Microsoft at the time?
- It was about as boring as you can get, honestly. I didn't have much of a computer science background or anything like that, so I was lucky they took me then. Because I was doing mechanical engineering, like I said. I was working on installers for a box software. It was like this CRM thing' that people bought on DVDs still. That's how it worked at the time. And we shipped once every three years, and there was an installer that would fire up when you put that DVD into your computer. And that's what I worked on for a couple of years.
- And then you were like, okay, there's better stuff out there. Like, let me go take a look around. So you ended up in the Bay-
- Oh, I learned a ton. Yeah, it was a good experience. But yeah, I moved down to the Bay Area. I worked for a startup. The startup was called Disqus. It was like commenting. You probably see it all over without knowing about it. Yeah. So I worked on like some ad engine stuff there. I wasn't there for too long, maybe it was just under a year, I think. And then I hopped across the street over to Google in downtown San Francisco.
- All right. So let's talk about Google. All right. Dude, that's how I first knew of you through, obviously, minikube and Skaffold and stuff. And you know I've had And once that integration happened with Skaffold, I'm such a fan of the hipster demo, right? And that's the one I kinda was, okay, every one of the SEs at sysdig, I was like, "This is how you can demo a microservice." It's the most incredible thing. It just deploys. And then the way that Skaffold works, I learned that and I'm like, "Wow, this thing just deploys into a registry, of the GCRs just beautifully, I can build locally." And that just takes it such incredible stuff. So let's talk about, first Skaffold, and then we'll talk about minikube. So talk about beginnings of Skaffold. What's the story there?
- Oh man. Have you had Vick on the POPCAST? You know, Vick?
- I do. He'll be on. I wanna have him live when we go to LA.
- Okay. Good, good, good. Yeah. I think it started with him actually. I'm trying to remember which KubeCon it was. It was like a random hallway conversation with Vick. And he was like, "There's three or four steps you gotta do to deploy Kubernetes. These are so hard. Why don't we just make a tool to do it?" But it's only three or four steps. So do we really need to make a whole tool just to do those? I was like, "Well..." We were sitting in a hallway at KubeCon. I'm really trying to remember which one it was. Oh man, we could check the years and figure it out that way. I can't remember which city we were in. We were like, "Well, there's thousands of people here at KubeCon that are all doing those same three or four steps. Why don't we just go make that tool?" And so that's how Skaffold started. It's the same little loop of like, you save your file, rebuild the Docker image, figure out the tag, change the tags, stick that into your Kubernetes YAML, and redeploy that thing. So four things that you gotta do in a row. Everybody had their own little bash script to do it. And that was the first version of Skaffold. We did each of those as fast as we could, and it's kinda grown from there.
- Dude, this is your MO. And I'm trying to tie this all together. There's the mechanical engineering piece of you that was just... Anything that I've seen you work on, just the simplicity of it, it just does a thing and it does it right. And we'll get to other stuff, right? But that to me is your MO. Anybody who works with you on your team, you're gonna be this humble guy that you are and I love you for it, but that is the thing that I think that draws everybody to what you do: is you take a ridiculously hard concept like Skaffold, right? Making those four different things that everybody does and makes it like this. You do this same with cosign: like this Let's talk a little bit about minikube though. So with minikube, it's the problem... And I had Thomas on, we talked about it at length. We're gonna go deep into it, but I wanna talk about that day you're like, "I'm so sick of this, this and this deploy. I wanna do this locally. Here's how you do it." Tell me about that day.
- Oh, that one was completely different actually. I had never written any Go code before. I hadn't really done much with Kubernetes at all. I'm trying to remember that. That was my first couple of lines of minikube were my first step in this whole Cloud Native journey. I had used the Vagrant a bunch. It's an awesome tool from HashiCorp at the startup I was at, and it was awesome. You could run one command, you got a whole VM running locally. It didn't matter which type of work station everybody at the company was on. You could write these little scripts and get that all set up. And Kubernetes didn't have anything like that at the time. There was no real way to spin up a Kubernetes cluster on your laptop, which was a big problem 'cause GK is great, network at Google. All the cloud providers are great now but back then they weren't as great. And they were pretty expensive and they were hard to use. And so getting your own Kubernetes set up... I think this is even before Kubernetes the hard way. Maybe it was right around the same time, but there wasn't even a hard way written down. So the stuff was not easy to get started with. This was right around the time of 1.0, maybe even pre 1.0. It's hard to remember back then. And I knew about VMs a little bit. I had done a bunch of the Vagrant work before. I was like, "Why don't we just throw this into something people can download? One command, just like we talked about before, has everything packaged inside of it and get some Kubernetes?"
- Yet again, think about the pedigree. Does one thing, does it really well. Okay.
- [Woman] Run confidently with Secure DevOps, security for containers, Kubernetes and cloud. Learn more and sign up for an upcoming webinar at sysdig.com That's S-Y-S-D-I-G, .com. I had Maya talk about Secure Software Supply Chain, but what's the problem today in security?
- It's tied in actually, 'cause the story about minikube is kinda where I first started to realize there was a problem here. So I'm at , I made this whole minikube thing, it packaged it up, put in on GitHub. This is a thing that creates virtual machines on your computer, right? This means it's gotta run as root. You're downloading this thing, running in as root on your computer. And it frankly terrified me that so many people installed this on their computers that I had made, stuck on GitHub. Nobody had an idea of where it came from and they were just running this as root. And that was terrifying to me. I went through, I read all the literature I could find, figured out the best way to do this, so I wasn't gonna go get tons of people owned in the process of installing my tool, set up on my CI server the best I could. I think I just saw that Kubernetes was seven years old yesterday. Is that right? I think I saw that on Joe Beda.
- Joe Beda just put that out yesterday. By the way, he's airing on the 16th, everyone.
- Sorry. Yeah, yeah, yeah. A couple of weeks. Not super far off. Yes. Well, Kubernetes just turned seven. And this is probably five years ago now, and that terrified me. So I did the best thing I could, and setup my CI system the best I could. And this is complicated too 'cause we were shipping bills on windows, on Mac. I had a Mac mini set up under my desk at work, that kinda thing. I kinda just got nerd sniped, I think that's the right term. I'm just trying to set that whole thing up as securely as I could. And I spent the next five years just confused and nobody else cared about this. Everybody's shipping all the software in the cloud that's running as root in people's data centers, and nobody had any idea where it came from. So that's how I got to this point with the crazy hair and the beard. And I've been working on this problem for a long time now.
- If you solved the problem, will you cut the hair? It's not me asking. That's a Twitter question.
- Sure, sure, sure. Yeah.
- So again, it is a large problem. And again, I know of it 'cause I'm in the security space. and then what we do at sysdig. But in terms of when Maya was on, and also just knowing you over the last year or so, it's been like, "Wow, this is a problem without a doubt." And last year, we saw it hit hard with what happened at that place, at that name of that company, right? That something happened, right? And so, riddle me this? How would you say somebody can solve for this? What are some basic ideas that can assist somebody?
- Yeah. So much of just the advice that everybody hears and nobody wants to do 'cause it's hard. There's no groundbreaking, magical, no silver bullet. But it's not even that hard, you've just kinda gotta do your homework. The biggest thing, I think, if everybody could do one thing to improve the state of their supply chain, I think it would be to treat your CI system like a real production system. I keep saying that. I was just guilty of it, right? We were running Mac minis under my desk at work to do cross-platform bills on Mac 'cause that was the only option for minikube, like I said. But that's not how you would run a production system. Nobody would stick a server under everybody's desk in the office and then treat that as prod. But we do that with our TI systems. We do that with the systems that build the things that make it into our production environments. And that's been okay up until now, but it's no longer okay. We can talk about why I think the supply chain attacks are a thing, and becoming a bigger thing, if you want to. But I think that's the one piece of advice I would give everybody, is to treat your build system, treat your CI system like a real production system, because it is one.
- [POP] Civo was an alternative to the big hyper-scale cloud providers. They've launched the world's first managed Kubernetes service, powered exclusively by K3s. With sub 90-second cluster launch times, a simplified Kubernetes experience, and predictable billing, Civo's on a mission to create a better developer experience. Get $250 free credit to get started. Sign up today at civocloud.com/popcast. That's C-I-V-O.C-O-M/P-O-P-C-A-S-T. Go check them out. Let's go by the question you just basically posed, right? You set me up here. I'm not that smart, man. I need any help here.
- So why is this a problem now? And yeah, it's complicated. I don't know the right answer, right? I have a theory. We've known about this since... We'll go all the way back now to before I was born. Ken Thompson wrote this paper in the 80s, called Reflections on Trusting Trust. If you've never read this or looked at it, you should. It was a series of kinda exercises and almost pranks he did on his colleagues in the 80s at Bell Labs, to show that it's turtles all the way down. And unless you built all the software that has ever built any software you use today, you have no idea what you're running. He did crazy pranks, like inserting a backdoor into a compiler that would insert a backdoor into everything it compiled. And then this thing... He worked with some really smart colleagues, right? Everybody's heard of Bell Labs. He made this so if it was compiling a compiler, it would throw that same backdoor into the next compiler it compiled. These people knew how to disassemble binaries and look for back doors like this. So he made it so if this thing compiled the disassembler, it would put in special loops to hide the backdoor. And he did all of this way back then and it puzzled everybody. And then he went and gave an award speech where he explained all of this, and it kinda terrified everybody back in the 80s. And then for some reason, people just forgot about it until 2020. So how many years is that now?
- Yeah, it's like almost 40, right?
- Right. Yeah, we've known about it for a while. I think the reason it's become such a problem is because back then, Ken Thompson was the only one that could do that. It was really hard, and there were so many easier ways. Up until only the last couple of years that people have been doing things like two factor auth. I've seen stats for how many people are doing that on GitHub, on pypy; it's terrifying. So even though the stuff is still out there, developers aren't taking it super seriously. But they're starting to, right? We're starting to do HTTPS and SSL everywhere, thanks to stuff like Let's Encrypt. We're finally not sending passwords over plain text on the internet at conferences. But I think it's a relative thing, right? Attackers are always gonna take the easiest way in. We've done such a good job as an industry, particularly in security and hardening all the other ways, that the supply chain attacks are becoming the easiest way in, relatively. They're not really becoming easier on their own, they're just becoming easier than everything else 'cause we've done such a good job blocking everything else down.
- And again, we'll have link to the the paper that you just mentioned here, just so people can go in and take a look. But to me, and I've been on countless security things recently. And this year, just because of my job, with Vulkan runtime security. And I just feel like, again, it's like, you have to take a cultural thing and also a process thing, meaning you have to adapt your security team and developers to understand that is you have to embed security at every layer of it. Like you said, from the software supply chain perspective. Even at a runtime perspective, and that's what we do at Falco. So it's not just a single posture. And then that's the thing, like you said, and especially when you bring up Kubernetes, right? You all have had the secure Kubernetes at the board level for years. So you all are the experts from that perspective. It's not just an application that runs in a model with all of this. It's literally like the control plane, the backend, the application that's running on it, and all of those things. I mean, it's a daunting task. What would you say? And this is leading to a question. What would you say, again, somebody just off the shelf right now from a security perspective, and maybe even just in general; how do you shape culture to understand security and adhere to it as an organization? What are things you can do?
- This is a tough one. I think the simplest one is to reward it. That's an easy answer though. I think security is a tough one because it takes one bad day, it takes one slip up. And if you're protecting something valuable enough and you're in a huge company, and you've got enough data, you've got enough money that people will wanna attack you, it takes one slip up. And especially this last year across tech, there's hundreds of thousands of people, millions of people all over the world, different cities, different setups at home. One slip up is enough for somebody to get in and ruin your company's day or ruin your week, your month, depending. Maybe you didn't get executive orders which we'll get into later. One slip up is all that really takes. And I think especially now with security, it's a big buzzword. The easy thing to do is pay attention to these big flashy attacks. And that's what our organizations do. They respond to stuff like that, they reward responses to stuff like that. But really, it's the mundane that it gets you. It's the forgetting to patch your build server. It's the forgetting to actually use two factor auth on every single website, forgetting to not reuse passwords. And so it's figuring out ways to reward the mundane, and not figuring out ways to get people to care about security, 'cause everybody cares about security. How many people have heard of the fancy ones with the logos and the websites like Shellshock and Heartbleed, and that kinda thing. But how many people are guilty of not updating every single dependency and every single service they've got running? 'Cause that's what happens with the Apache Struts and some of these other big attacks where credit records-
- Any crypto level thing is without a doubt, because something's being opened. But when you think about it, again, it's a stat. This is a real stat. You can look this up. $5 trillion dollars it cost the industry. 5 trillion in misconfiguration security. It's from techerati. So to me, it's like you said, it's the mundane things that the people don't care about. That people, they're like, "Ah, it's all right. We just need to ship this." That mentality to has got to stop.
- Yeah.
- Without a doubt.
- Automate the mundane stuff, and reward people for doing it in the meantime until you've done that, I guess.
- No doubt. So let's talk about something. I think one of the tools that I, and again, when I've seen and I used it, I was floored by it. Again, it goes by that same thing that I think is your pedigree. Does awesome things and does it in a simple way, And does it really well and people can collaborate on it, and make it even better. Let's talk about cosign/sigstore.
- Sure. Yeah. So this is a fun one. Maybe it ties into what you're talking about before with simplifying tools and stuff like that. Cosign is a tool for signing containers. That's really all it does. It can sign containers all the different ways you can imagine, and you can verify those signatures. We started it back in late... This is June now. Maybe February or March. There have been a couple of attempts at signing containers in the past. You can get into the whole history of it if you want to. But it's not widespread yet. It's still not widespread even though we're working on stuff now in cosign and sigstore. But we're really trying to lower the barrier to entry, explain signatures in a way that makes sense to people, and let them do it in ways that will help them, and actually improves their security.
- [POP] What you build and where it takes you shouldn't be limited by your database. Cockroach DB helps developers build and scale apps with fewer obstacles, more freedom and greater efficiency. So you can forget about the database and trust that it just works. Kubernetes friendly, open source and indestructible. Cockroach DB makes it easier to build and scale apps. It gives companies the freedom to serve customers anywhere, and it's backed by world-class documentation, and excellent dedicated support. Discover Cockroach DB. The most highly-evolved and distributed SQL database on the planet. Kubernetes is native and built from the ground up to help companies of all sizes, including Bows, Comcast and Equifax scale fast, survive anything and thrive anywhere. Sign up for a free 30-day trial and get a free t-shirt at cockroachlabs.com/popcast. That's C-O-C-K-R-O-A-C-H-L-A-B-S.C-O-M/P-O-P-C-A-S-T. Got it. And so I need to understand. So cosign is the tool. What's sigstore? Explain that to everybody.
- Yeah, sure. Yeah. So cosign is a tool. Github.com/sigstore/cosign. It is a tool just for signing containers. It can do a couple of other, you know, it's got some fancy bells and whistles bolted on. But if you wanna sign containers and verify those signatures, cosign is where you go. sigstore is kinda an umbrella project that we've been working on. The community has been amazing, and it's been incredibly responsive to this whole effort, and helping out, sigstore is an umbrella-
- I'm a contributor, everyone. I'm a contributor.
- You are. I think I merged your PR. Yeah, it was awesome. You get a badge.
- I got a badge for that.
- Yeah, it's an umbrella effort. We were really inspired by kinda what Let's Encrypt did for web security. But we wanna make signing and verification of supply chain metadata containers, how the containers are built, all that stuff ubiquitous. And so to do that, you've gotta run on, it's more than just tools. It's tools, it's services, it's new technology. We've gotta do all of this somewhere in the open as a public benefit for everyone. Even the tools and systems, they need to sign stuff and get value out of the metadata they're generating and signing. So if you're just looking to sign containers then come hop on board, we're happy to help you out in cosign. If you're looking for other things for your supply chain, how to get keys to sign things, where to publish these signatures, the rest of it, and how to check stuff on artifacts that are coming in your organizations, then the rest of sigstore's for you.
- There we go. All right. So everyone, this is airing on the 16th. But the 18th, we're doing something extremely special. Let's about that. So that's gonna be on, everyone. Plug. Shameless plug. It's gonna be on Spotlight Live which is my live show on Cloud Native TV. Dan, let's explain what a key signing ceremony is.
- Sure. Yeah, so this is fun. A key signing ceremony. It's like a key signing party, if you've heard of that before. It's different from a key party, if you've heard of that.
- Yeah, this is definitely not a key party, everyone. Disclaimer. All right, moving on.
- Definitely not a key signing party either, if you've been to one of those before and are bored. So this is a little bit different. You're dead.
- Just a little bit different.
- A little bit different, a little bit different. Yeah. So to get into it. I'm going to explain what it's for and why we need to do on, and that kinda thing. When you're doing signatures, there are a lot of misconceptions about how they work, especially digital cryptographic signatures, right? They're way harder to use and harder to get value out of than people tend to think. They're only useful in limited circumstances. And to be honest, for containers, even though people wanna sign them today, it's hard to actually get value out of those signatures in a way that you can't without signatures right now. And this all kinda ties back to this other concept. I'm gonna do another shameless, little shoutout for a blog post from a friend of mine. Signatures without PKI are kinda hard to use. PKI stands for Public Key Infrastructure. A lot of people will kinda have heard of this and not really paid much attention to it. And I was guilty of that until just a few months ago. That's all the stuff that's going on behind the scenes in order to make it so you can verify signatures and stuff like this. So when you load a website in your browser, PKI is how you first figure out what's to trust and how to know who you're talking to. It's not magic. You just kinda have to have some root. Let's say, you and I just randomly walked up to each other on the street and started talking to each other. We know we're both from upstate New York. You start talking about all these things that we know in common to kinda prove that I know I'm from upstate New York. You can remember this conversation. You asked me about a cookie company, right? So we tie our common understanding and knowledge of each other back to Freihofer's, right? That is our root of trust in that situation. Once we get to there, we know that we're both from upstate New York, right? And that's what PKI does for the web. Two people just passing by in a desert or on the ocean somehow, they can't figure out like who they are, or if they're who they say they are. They need some common root of trust. And there's no magic there. You've gotta set up some kinda root of trust, get it all over the world, get people to kinda take that in one thing as blind faith, and then from then on, you can work out the rest of these details. So that's what we're doing on the 18th in two days from when this airs, I think. Correct me if I'm wrong.
- Right. That's right.
- So two days from when this airs, we were doing the sigstore route of trust, or key ceremony. We're establishing the first keys that we're gonna publish as widely as we can. That's why we came to you. You've got such a huge audience. We're gonna publish these keys. We're gonna publish the public keys. We're gonna do the whole process live. You're gonna watch us take these YubiKeys out of their bags, hold up the tamper proof seals to the camera, get this recorded and have the audience watching and participating, get all this stuff signed and published. So everything we do from then on can be tied back to that moment on your show, on Cloud Native TV, going forward.
- [Man] Learn how to operationalize open policy agent at scale with Styra. To get started, go to the link at H-T-T-P-S : / / H-U-B-S-.L-Y /-H-0-P-n-k-m-2-0.
- Again, it's truly an honor. But it's also... I love the story of it, right? I've never even heard of this concept. And I was talking to you about this. And he's like, "Hey, I'm thinking about doing this." And I'm like, "Dude, let's do it here." And so we kind of moved in to be able to do that. I'm really excited about it. But also, dude, again, like again, I've gotten to be like too tight with you over the last couple of months, or whatever. And so I'm like, two upstate people, man, signing keys and doing fun some fun stuff, man. So I appreciate it. It's really good stuff. And again, I'm excited for the project because I absolutely saw value from day one from what you all are doing. And I want everybody to do that, to see that. So without a doubt, I'm gonna ask you-
- It's gonna be fun. I said, it's gonna be fun. These things happen a lot. They're normally done at companies, if your company is using HSMs or fancy equipment where the keys are baked in. Probably did one of these. It's just a little different when it happens at a company. It's a bunch of people going into a room, looking at each other, and making sure the boxes get opened up correctly, nothing got leaked in the process. It's pretty rare for these to happen in an open source community where nobody works at the same company, we wanna distribute these geographically, we wanna build up trust that can last beyond companies and people and teams and that kinda thing. So we've designed it to be fun, to keep people engaged over the years as we go. And we wanna make this available for any other open source product who wants to use it. This is hard. We're trying to make it fun. We're gonna have fun with it. But a lot of times, they're boring, and people can gloss over it because of that. So we wanna make this available to any other open source product that wants to use this route of trust going forward as well. So we're trying to do the hard parts, the hard boring parts, so everybody else in open source can take advantage of that.
- That's what I think, again, what's great about this community is this kinda stuff. It's basically like doing things like this so people can understand the value of obviously the project; that's number one. But also you can see that this is something that it's for the benefit of the community. It's gonna make everybody have a more secure, like in a posture from that perspective. So really I'm looking forward to it. I think you all should definitely check this out. again, in two days, cloudnative.tv. It's gonna be at 2:00PM on the 18th of June. All right, and then there'll be replay as well. We'll have it on YouTube later for replay value and all that fun stuff.
- [POP] Teleport allows engineers and security professionals to unify access for SSH servers, Kubernetes clusters, web applications, and databases across all environments. You can download teleport right now at goteleport.com. That's G-O-T-E-L-E-P-O-R-T.com. Dan, namesake, my friend, what work are you most proud of?
- Oh man, it's gotta be this last sigstore work. I love it so much just because it really is a public benefit. And I feel like everyone's incentives are actually aligned. There's gotta be probably some recency bias in there. And I love to see all the laptops of people running code that I wrote and stuff like this, but I love the pure kinda... I don't wanna say pure goodness and stuff out of it. But I like that with sigstore, we're all coming together as an industry to do this for everyone. And even competitors, right? We all compete, but we're all consuming the same open source code. Nobody can make money off of this. We've gotta do it all together. We've gotta do this for the benefit of everybody. I'm sure there's a little bit of recency bias in here, but that's what I'm most proud of right now anyway. And not just this, kinda the overall open source security effort in general. This is a software that we all use, we all rely on and companies have come together to start trying to fix it. It's hard enough to make money on open source in general, we all know that. You've had a ton of great podcasts about that, but it's awesome to see so many companies willing to give back to open source. And we've done some cool stuff in the open source security foundation. I think that's actually where I met you, right? I was doing some work there to try to look for malware and package managers.
- Yeah. Yeah, and you wrote an awesome blog about Falco. And again, it was all based on, we were talking on Twitter and you're like, "Hey, I'm trying to solve this problem." And I was like, "Hey, why don't you take a look Falco and stuff?"
- Yeah, exactly. 10 minutes later, I had it working. Now I remember the history there. A common problem in... There's so many problems in open source security we've gotta solve. But a common problem is that developers try to take advantage of people, or sorry. Attackers try to take advantage of people including developers, and malware gets inserted into the packages that people install. There's a couple of different ways this happens. Some people try to slip the malware in, so it hides until it's in a production environment. You see some crazy attacks like that lately. Another one though that people don't think of as much is attacking developers personally, right? So if you install something on your laptop to do some local development, there are a lot of packages. These keep happening. They keep getting found and taken down months later, but when you install something that gets to run arbitrary code on your computer, which is terrifying. Outside of a sandbox, this thing is running with whatever credentials. If you run PIP install or npm install, this thing is running some setup scripts, and it'll go through your home directory, look for any credentials they can find, those. Look for your cryptocurrency, whatever you've got stored in your own directory, that kinda thing. And yeah, so this was part of an effort where we were all setting up a honeypot or something to install every single package on pypy and look for what it was doing. We're looking for ways to monitor behavior there, and got Falco set up and running. We didn't find any malware of that sort yet. We did find some spamware though. Already, there are some attackers trying to trick developers into clicking on websites, and free discord credits, and that kinda thing. I can't even quite figure out what a lot of these people are doing. It's like those spam phone calls you get when they're leaving your messages that are so nonsensical you don't even understand what value there is to them, if you didn't listen to them and follow them through these few steps. Yeah. We found some stuff like that and got it taken down from package managers.
- Nice. Well, Dan, we did it. Like I said, I appreciate you. It's been awesome working with you on different projects and stuff like that. And so, dude, definitely appreciate you being on the POPCAST. Thank you so much.
- Keep going. Hopefully the next one will be in person.
- We'll do it. All right.