Ville Aikas
Ville Aikas is one of the original "godfather's" of Kubernetes. He is the "George Harrison" of Google Cloud/Kubernetes, built much of the underlying tech used in cloud native technology being largely adopted globally and has stayed quiet about all his amazing accomplishments which are many:
* Founder and later on tech lead and manager of Google Cloud Storage from 2008-2014.
* One of the earliest members of Kubernetes project within Google.
* Tech lead / manager of Google Cloud Deployment Manager.
* Kubernetes Service Catalog / Open Service Broker / Helm OSS contributions
* Founder / Tech Lead on Knative Eventing / Member of Technical Oversight Committee
Ville finally speaks in depth about his cloud native journey direct from the Isle of Ville.
POPCAST SHOW DETAILS
YouTube: https://bit.ly/3xgmmCj
Audio Podcast (Apple, Spotify, and others): http://bit.ly/35MXfte
Follow us on (Twitter): https://twitter.com/PopcastPop
Follow us on (Linkedin): https://www.linkedin.com/company/the-popcast-with-danpop
Timeline/Topic
00:00 - Opener/Sponsors
00:11 - Introduction to Episode Links
00:44 - Ville's Journey
17:44 - Early Google and Seattle
24:40 - Yadda Yadda Yadda... then we started google cloud (Plan B)
29:46 - Scale is just an illusion.
35:32 - Kubernetes in its infancy
40:00 - /hack directory
42:35 - Godfather Joe Beda
44:00 - Did Ville envision Kubernetes would get this big?
49:22 - What does the Kubernetes community mean to Ville?
52:07 - Contributor of the Month / Community Chop Wood Carry Water
54:10 - the Challenges and Advantages of Open Source
57:14 - Ville's Island aka the Isle of Vile.
59:31 - What work is Ville most proud of.
Please like and leave a comment if you enjoyed the episode! it helps the show!
Episode Links
Ville's Twitter - https://twitter.com/AikasVille
Knative - https://softwareengineeringdaily.com/2019/02/08/knative-serverless-workloads-with-ville-aikas/
Duck Typing in Kubernetes: https://github.com/vmware-tanzu/tgik/blob/master/episodes/145/README.md
Brought to you by:
***Cisco***
Check out Cisco's super informative blog https://ciscotechblog.com/
***CIVO***
Civo is an alternative to the big hyperscale cloud providers.
They've launched world's first managed Kubernetes service powered by K3s.
With sub 90 second cluster launch times, a simplified Kubernetes experience,
and predictable billing, Civo is on a mission to create a better developer experience.
Get $250 free credit to get started. Sign up today at https://civo.com/popcast
***Teleport***
Teleport allows engineers and security professionals to unify access for SSH servers, Kubernetes clusters, web applications, and databases across all environments. You can download Teleport at https://goteleport.com/popcast
***COCKROACH LABS***
Discover @CockroachDB the most highly evolved distributed SQL database on the planet.
Kubernetes-native and built from the ground up to help companies of all sizes including Bose,
Comcast, and Equifax scale fast, survive anything, and thrive everywhere.
Sign up for a free 30-day trial and get a free t-shirt at https://cockroachlabs.com/popcast
***Styra***
Learn how to operationalize Open Policy Agent at scale with Styra: https://hubs.ly/H0Pnkm20
- [Announcer] This episode of the POPCAST is brought to you by these sponsors.
- Hello, everyone. And welcome to the POPCAST. I got this guy right here. He's fifth, fourth, fifth Beatle. He's a silent, but he's the member. He's the George Harrison of Kubernetes, the silent Beatle. He's Ville Aikas. He's the conciliary of Kubernetes, Senior Staff Engineer at VMware. Welcome to the POPCAST, Ville.
- Thank you so much. Super pumped to be here. And I can't wait to go ahead and chitty chat with you for the next, however long we go.
- Let's go all night, Ville. You ready?
- All right, let's do it.
- So you're a young Finish boy, and you're like, I have compute, I want to start computing. Or, just talk to me about your beginnings.
- Sure. So, 12 years old Ville is seeing his friends got Vic-20's, playing computer games. I was able to go ahead and sucker my parents into getting me a computer. And they surprised me with the C-64.
- Wait a second. Commodore 64, Ville?
- You know it.
- Everybody take a shot. Anytime somebody comes on the show, and they have a Commodore 64 story, which we all, it all seems to emanate from the Commodore 64. We had Joe Beda on. We had Brendan Burns. Everybody seems to have a, I was gaming, and I had a Commodore 64. Okay, great. Let's go.
- Absolutely. So meanwhile, the only thing I'm doing is basically playing computer games, and whatnot. Meanwhile I had a whole bunch of friends who were actually in the demo scene, doing a lot of great stuff. And it was just mind blowing. And one of my friends tried to go and teach me, and I just have no patience for it. I was too deep into playing soccer, hanging out, and doing all kinds of things that we can't talk about, but. For the next 10 years, just playing computer games. And that all changed when I came to the States as an exchange student. It's kind of a funny story, because all my friends who were going to the exchange to Australia, or LA, or cool places that. And I was like, I'm still waiting to know where I'm going. And they were basically like, okay cool. Get in to Seattle, and we'll sort it out there. So there I am, 16 year old Ville boarding a plane, and like, well we'll see what comes out of this one. So anyway, so I basically board the plane, come to Seattle.
- Wait wait. Wait wait, wait wait. So I need to understand. This is the exchange program. All your friends are scattering all over the world. Seattle, did you spin the globe and just? Was it chance? Or it was. Tell me about that.
- It was just basically going to the States. So USC basically got a chance to go and choose where you might want to go. One of my friends went to Australia, but a few of us came to the States. As far as choosing the location, no I had nothing. And that was the tricky bit. That basically the day I was boarding, I didn't know where I was going. I didn't even know if I was going to Seattle. I just knew that okay, you're gonna take a plane to Seattle, and then it'll get sorted. All my other friends knew their families where they were going, and yada, yada, yada, and had been exchanging letters. This is before the internet, and so forth. So I just hop on a plane and I'm like, well, we'll see what happens.
- Wait wait, I gotta gotta. Dude. I just, I want, I gotta get the double tap on that. That takes such courage, man. Look, like Ville. My wife is from Poland, and she came to America, whatever. But man, you just leaving at 16. And you're like okay, I'm gonna go to Seattle. You're crazy, man.
- Yeah, and I was as bit. I was also a little bit kind of bummed about that initially, because I looked at the map. And again, this is before Google Maps. I'm basically go going to the, whatever, it's not a thesaurus, it's basically. It's not a dictionary, but you basically had those. Encyclopedia, that's what it is, encyclopedia. And I'm like, Seattle. Where is it? And I look at it, it's right on Canadian border. I was like, damn it. I wanted to go somewhere nice and warm. And here I am in the cold. Because I didn't understand that at the time Canada, to me, was pretty much okay, it's just Finland. So it's gonna be cold and dark, and everything else. But I was like, all right, whatever. As far as sorta kind of doing. You said courageous. Another way of putting it is reckless. And turns out that if you talk to my friends, they might still consider me as such, or doing things in a whim. And that's fine. But yeah, so I get there, get off the airplane, and go where everybody else is going, and there's a sign there. And then again, they're like okay, you're gonna stay here for a bit, until we figure out what we're gonna do with you. And I'm like, okay fine. And so I stayed with them for a few weeks. And then one of their friends basically took me on, and said okay, you can stay with us for a year. You don't seem too crazy. So I stayed there for a year.
- Wait, wait wait. This is a random family? This is just a friend of, or friends of them? How did you know them? And I'm sorry, I don't want to pry.
- No, no no. Totally, yeah. No, no no, totally. It was actually the, the family that I went to, they had had exchange students before. So it was their daughter who had never had any, or the daughter's family, so it was basically three boys. They had three younger boys, so they foolishly thought that I would be a good role model for them. They took on me as the big brother, still suffering.
- Incredible. And so, okay so how did? So we got the Commodore 64 early, and then you're in the States. Put the two things together for me.
- Okay, so. At that point, basically I'm like okay, awesome sauce. Now we are back to the, now I find Nintendo. And I'm like, this is so much easier. So I basically switched to Nintendo, however, at the end of that year, because as I go high school, the things I really cared about was math and physics, hard sciences. And I don't know anything about computers at this point. But then I found this in September about robotics. And I was like, that looks really cool. I want to do that. And so turns out that the South Seattle Community College here had a robotics degree. So I was like, that looks fun. I want to do that. So I decided to apply for the program, again not knowing anything about it. But I thought, that looks fun. I should learn something about that. So I applied for that and, and I get into it with some conditions. But I have to go back to Finland for a bit until I turned 18, to come back with a different Visa. So I come back, jump through the hoops, and take all the tests. I should have had to go one more year of high school, but I just took a test, I forget what it is. Something, SA, no. Something. GED. So I took a GED. So, I never even graduated.
- Timeout again. People look, there's this concept of gate keeping that we hear. I have my GED. So does he. It's like, look sometimes traditional school. Look, I'm not advocating, and I'm sure Ville isn't. Its just, it is what it is. But sometimes there's a non-traditional path to get into this. And that's what I think is pretty evident with your story. Sorry to cut you off, my man.
- No, no no, totally. I totally agree. There's many many, many many, many paths. There's a lot of turns and twists, and don't be. Everybody's different, so do your own thing. Be you. So anyway, so I get into that, and we start doing things. And we're doing basic electronics, and everything else. And it's super awesome. I'm all into analog electronics. And I'm really finding an amazing, and so forth. Then we have this class. This is where we kind of come back into the C-64 days, the first course that we are doing on the microprocessor. So microcontrollers course 6502. So now I'm like, okay. Now I'm back into this. I remember somebody had told me about accumulators, and registers, and things like that. So that kind of completes the loop back to the C-64, where I had seen some of that code. But none of it made sense then, and now I was kind of like, okay cool. So we started doing that thing. And there's a couple of amazeballs moments in my professional life. I'm not gonna call it a career, but my professional life. And one of them is when I was able to go ahead and take an input into microprocessor or microcontroller, and then be able to go ahead and blink the light in response to that. You could do that in analog. But as far as be able to program that, at that point it was like a light bulb went on. And I was like, oh my God. Now you can go and take physical world, take some inputs, program it, and produce a different output. And that was like, now you can do anything. It's just like, okay. That completes a loop for me.
- That was like alchemy to you, pretty much.
- Yes, absolutely. At that point I was like, now you can't stop me.
- Incredible.
- Anyway, so then I started kind of getting into that. And I was really excited, read a whole bunch of stuff about that code, as in wanting to learn more and more and more. And then I got into this. One of my physics professors hooked me up with this mechanical hacker. He was an OG hacker. He would build all kinds of crazy stuff, but it was all mechanical. So he didn't really understand things about electronics and things like that. So he put two of us together, because he was basically building this ozone machine, so air purifier. And he was kind of like, hey you should talk to this dude. We got together and we basically spent about six, nine months putting together this contraption, if you will. But it was pretty cool. It basically had all the ingredients for fun-ness. It basically had a ozone generator, which basically means we have to go and use these traffic light generators, or transformers, to go and get the high voltage. And then, of course, we needed to go and have propane to go ahead and chemically go and dispose of the ozone once it's done. And it's all enclosed in this metal cage, and it's all relying on us. So one could say that it's pretty lucky that we are still alive, but here we are. So that was sorta kind of fun. And I learned so much about that, just all kinds of crazy stuff. Anyways, so. So then I. I was really heavily into radar kind of stuff. Again, just kind of physical world and everything else. But I really enjoyed dabbling in the software stuff. But applied for the UDaB, got in, and I was gonna go into Double E Department. And then, this is again where one of those random chances kind of entered into my life in the sense that I used to carpool with a buddy of mine, Louise, a Peruvian friend of mine. And we both applied for the Double E, because that's what we we're doing. And then he, one of the car rides in the morning, he basically was kinda talking about, oh, it's so good. We have to get into the Double E, because it's so, it's impossible to get in this computer engineering program. Because nobody gets into that program, and it's super hard. So I was like, oh yeah? It is that a white glove you just threw down? I can't get into that? So I'm like, hold my beer. So I applied for the department, and I get in. And I get this package. Again, this was back in the day. So I basically get this package and I'm like, well, there's all these courses. And I'm like, compilers? I'm like, I don't know. They don't sound interesting, But then there was one class that was operating systems. And by this point I had actually gotten myself a 386, with the turbo of course. So anyway, so there I am, hanging out. I get applied into it. I see one class that I'm interested in. It's just, I wanna understand how operating systems work. So I decided to switch my major from Double E to Computer Engineering, based on that one freaking class. And it's a 450 some class, so I have a lot of classes to go through just to get that point. But luckily because of Comp E, I also got a chance to go and take electronics, and signal processing, all kinds of neat stuff that, I think, has helped me later on, as far as being able to wire things together in my head, on sort of kind of what they look like. And yeah, so there's all that. Oh, so when I got the Comp E, this is, I got a student job at the UDaB, working for the Network Operations Center, so NOC, basically writing tools. And that's sorta kinda is where I've gotten stuck ever since, in the sense that that's where we start writing tools. And I kind of feel that's what I still do in my life, which is basically write tools, whether they are tools, or things that make people's lives better, is what I lie to myself, and every now and then I even believe in. But I totally just kind of want to build tools. I can't remember the last time I've actually worked on a product, if you will. Except the one shareware program I wrote, but geez. But anyways, so that, and again-
- What operating system was it? Was it Solaris back then, or AIX? Or what was it?
- The AIX.
- Okay.
- Yeah. So, yeah. And then of course, once I got in the department, there's the whole Linux stuff. We were building Linux a bit, and so forth. And then once I finally. So anyways, I was super early in the internet stuff, before it was even popular, when I was like, oh my God. Telling all my friends this, foam at the mouth. This thing's amazing. It's gonna change your life. Oh my goodness, and they're like uh-huh, whatever. You're gonna get pizza from that? I remember somebody asking me this. And I was like, yes. One day you will get a pizza over this. And they're like, uh-huh, whatever. You're drunk. But anyway, so I learned a lot of stuff there about the networking and so forth. And then, okay, so then the second point that is sorta kind of interesting is, in the OS class, the, again, the only class that I wanted to really take. So in the Operating Systems class, me and the professor, Brian Bouchard, had a lot of conversations on the sides and everything else. And he didn't think I was totally clueless. So he offered me to do some research with some of the grad students. So I was an undergrad, obviously. I still am. I never got anything more than a Bachelor. So he wanted me to go and come help with some of the research stuff, which was super awesome pants. Because we were basically doing low-level binary rewriting stuff, and this modular kurnels kind of stuff. So there was a lot of really neat stuff there. We spent about a year on that. And then out of that comes a, we spun off a company. So few of us basically spun off a company out of that. And that lasted 10 years, and by the end of that, I was just completely and utterly burned out. And I was kind of like, okay, now I need to go back into the helms of UDaB. So I got a full-time job at the UDaB, for two reasons. One, UDaB is great about, if you want to do outside work. If you want to go and put a startup together, nobody's gonna care. Whereas you go to any company today, and they ask you to sign, you can't do anything outside of work. Well, yada yada. It's extremely difficult. Whereas at UDaB it wasn't. And I was like, well, UDaB is fairly straightforward. So I go in there, work there. Do good stuff, I think. And then one of the people from the previous startup, we had an idea for location-based services. We've been talking about that. That's the next thing. So we put the startup together, working on that. And then a couple of years into it. Oh, and then there was another startup. So, a full-time job, two startups on the side, and a family. So it's fun times for everybody, but anyway. A few years after that, then we sold the company. At this point I was like, I really want to just one job. I have three jobs now, whatever. And I was like, I just want one job. And this time I had a bunch of my friends, who had gone over to Google and they have been pinging me every now and then. You should come here. You would really it. And I was like, I can't right now, because I'm still, I'm not done with this.
- Can we talk about early Google? I'm gonna talk about that a little before we get into it. So it just seems. I've had some folks from Seattle. I've had some folks from UDaB, that were at UDaB as well, and talking about you can start. I had Mitchell, and Armon from Hashi. They were UDaB folks as well. Had Brendan Burns on. You're all just lock, stock. And Joe, right. So it's lock, stock, and barrel all Seattleites, in terms of that. And it seems folks would go to Amazon, Microsoft, or Google. And so it was like that's the kind of natural progression. So talk to me about, 'cause it was first, wow. It's a search engine company. I mean, this is back in. I remember it, I'm sure, it was just, this is a search engine company. How am I gonna apply the ozone machine stuff that I'm working on, all this other stuff. Were you saying to yourself, this may not work? Were you saying to yourself, that? I don't know. Tell me about that.
- Sure. So again, Google was a little bit of a fluke, in the sense that, there was a lot of very interesting and exciting things going on at the time there, from what I had heard. But it stopped there. It's like, oh, exciting and interesting stuff. And nobody could ever say anything about it, because everything was super secret back then. And I guess it's still is to the most extent. But they were no, no no, trust me, trust me. And these are some people I do trust, and so forth. Now this was sorta kind of, I have no idea on sort of kind of what market value is, or what do people pay. I had been at the government, UDaB, government job. I think it is government, whatever. So I did, I went to the Amazon, and I actually interviewed there. I did the loop there. It was funny, I think it was for an S3 at the time. Or I don't know if it was. Yeah, it doesn't DWS side of the house. And it was for the, it was for the S3 side of the-
- 'Cause early on you felt your specialty was storage early on, or like?
- Oh God no. No no, no no no. At this point I have none of this. It's just basic standard distributed stuff, and yada yada yada, fairly applicable, and so forth. So UDaB produces, they have a high reputation for producing clue full people out of the U, so I'm sure that helped. Anyway, so I got an offer from them. And then at the time I was actually thinking about moving to Europe, going back closer to home. And Google had an opening in Zurich. So I actually applied for the Zurich office, but the interviews were all down in Mountain View. Well they wanted me to fly to Zurich. And I said, well I don't want to just fly there for a day. I'll go there for a week. And they're like well, let's do the Mountain View. So anyway I go there, I get an offer from them as well. I was kind of pondering these two options in my head. And I kind of pinged my friend, Brian, Brian Bouchard, my old CS professor from OS days. I was kind of like hey, can I chitty chat with you a little bit about, get some of your thoughts? Because he's very plugged in. He knew many of the folks, in common, from the Google as well as from AWS. So anyway, so I got them together, or I pinged him. I was like, let's go get a beer, chatty chat. And I was like, look. I got these two offers. And he was also, he was kind of like, oh I am actually looking for a job as well. And he was, and one of the options for him was Google as well, but I don't remember what the other ones were for him as well. So we kind of chatted for a while about some of the pros and cons, and what was. And then it was like, okay well, when one of us decides let the other person know, and so forth. I just thought about it for a while. And I was like, look. I'm just gonna pick Google because I think the potential there was really, I thought it was, it was more. Because Seattle Google at that point was just starting up. Actually it didn't even exist then. So the only option would have been on the East Side. So when they called me up, when I decided not to move to Zurich, and just to stay in Seattle. I actually called Google up and I said, look. I don't think it's gonna work, because I don't want to go to Zurich. I'm willing to move to Zurich, but I'm not willing to move to East Side of Seattle. So I guess, I'll just stick with the Amazon because they are on the good side of the lake. But anyway, so then they said you know what? We are actually opening a Seattle office. So it was like, the timing wise, it was pretty good. So they were opening up in a month or so. So I was like okay, well cool. Let's do it. Plus they send a nice gift basket.
- So everyone, everyone, the key to Ville's heart is gift basket.
- Absolutely, or a case of Reiner.
- Writing this down, okay.
- Anyway, so yeah. So chose Google. At the time I think the Seattle office, geez, maybe 20, 30 people tops. And those were the folks that have been working probably on the East Side, and now they were closer to Seattle side. So it was tiny, tiny, tiny back then. So my starter project was Google Voice. So again, something I had no idea about. I hadn't been doing SIP, or any of those VoIP stuff. I had been doing a lot of HDTV, or things like that at the time on the UDaB networking side of the house. But voice? No, not so much. So I though, cool. Well again, if you're seeing a trend here. It's like, cool, I don't know anything about that. So let me figure it out. So off I go, and basically go to Google Voice, spent about a year on that, and then, due to reasons it was basically moved down to Mountain View. So our project that we've been working on for a year was moved down to Mountain View. I went down there for, I don't know, a week or so, and basically trained the folks, like a handoff. All right, here it is. Yeah, good luck. And I get back to Seattle. I'm like, okay, well now what? This is when I really, sorta kind of first time really remember meeting Joe. So Joe sort of kinda had just shifted off of something else as well. So we kind of started chattering, the few us, sort of kind of, well what the hell are we gonna do? Our projects have moved away from Seattle. What are we gonna do?
- This is godfather, Godfather Joe Beda, everyone. Godfather Joe Beda.
- Joe Beda, yeah. So anyway, so we're kinda chatting, and yada, yada, yada. We are passing around ideas. And we start talking about this whole cloud thing. And we created this mailing list called plan B. Because none of us had plan A, so the original cloud mailing list in Seattle was plan B. Because bunch of us had no jobs. So we're like okay, well let's go figure something out. And this is. Early on you said like oh, you were doing storage. So, no. But we'd be basically kind of divvied up. Here's the certain kinds of things we want to at least start tackling. And one of them was obviously Compute, storage, and-
- Who was in that original team? Because here's the thing. We all know, and again, Brendan, Beda, McLuckie, Tim Hawkin, Wes Hutchins, our friend Wes, was like you gotta talk to Ville. He's like you gotta talk to Ville. Brian Grant, whatever. You gotta talk to that him as well. So tell me about who that crew was.
- So the crew at the time was, basically, it was Dave Hansen. This is in no particular order. I'm just pulling things out of a hat. Dave Hanson was, he had been at Google for a while. Super awesome. Everybody's awesome. I'm not gonna have to say that for everybody. So Dave Hansen, there was Michael Sheldon. There was Joe. I don't remember if Gus was there from the beginning. And David Erb. Did I say David Erb probably? David Erb was my boss in Google Voice. So him, Joe, Michael Sheldon, and myself. If I think of somebody else I will put in. And if you're listening to this, and I forget your name, I suck, but you know that already. So anyway, so we kind of sliced and diced, and sort of kinda like who wants to do, and what? I mean fairly informally. And Joe basically was super pumped about Compute. Personally, I kind of fiddled with, I had been twittling that with the low-level stuff for so long that I was like, I don't want to do any more of that for now. Kind of like I don't want to do networking. I've been there, done that. And not in a way that I understand all the everything's. But I had spent enough time there I was like, I want to do something else. So that's when I decided that I wanted to work on storage. So to your earlier question, ah, so you know storage? I was like no, I don't know anything about storage. So let me go figure it out. So we basically started building what we called Big Store at the time. So everything at the time, for the Google GCP was big something. 'Cause the idea was that, the whole point about the cloud is that it's big. So everything was big. So we had the Big Star, Big Store, which became GCS after a couple of renames. We had Compute, which was Big Cluster. And then we have BigQuery. That's the only thing that still retains the name. And I'm bitter about that, because I wanted Big Store. So, anyways. So Sheldon and those folks started working on, Mike Sheldon started working on BigQuery. Joe, and a few other folks there, again, that I'm sort of kind of forgetting right now, started to work on Compute. And I and David started working on storage. How deep do you want go here?
- It's up to you. This is your show.
- All right. Anyway, so yeah. At this point also there was App Engine. So App Engine was kind of around as well. So we went down to San Francisco, and we were hoping to go ahead. Because they also were working on something like blob store kind of thing. So we were gonna try and unify forces. So it's like, well, let's not do it twice. Let's do it once.
- Was that the early pit bindings of Colossus?
- At this point it was still CFS. Then it became CNS, but yeah, on the road to it. I'm sure at the time it was like, oh, it's just right around the corner. Just everything at Google. You've got the deprecated, and not ready yet.
- But talk to me. Is this early? This is Borg that we're working on, or this as Kubernetes, eventual Kubernetes? I'm trying to figure out.
- Yeah, no no. This is all Borg stuff. So this is circa, I don't know, 2000 and whatever, five, seven, maybe. Let's see, 2008. Yeah. So no, all of is Borg. And, oh okay. So back to the second light bulb moment. So earlier on, remember, take an input from the world, blink of light. That was the first one, oh my goodness. The second one was the first week, first or second week at Google, when you basically launch 10,000 Borg tasks, and you're like, boom done. And that was the second point, which is, scale is just an illusion. So anyway, no all of this is in Borg. But however, we wanted to fist try to see if we could build this on top of App Engine. So we wanted to go and eat our own dog food. So can we build storage on top of App Engine? We basically wanted to go and validate things, like APIs and everything else like them. So we basically put together a App Engine app that acted us basically S3 compatible, blob storage for Google. It was all right. If had 10 meg limits. We had few internal customers that needed something like that, whether they were at positions, and they were still storing stuff on S3. So they were like, well, we should have at least something that they can migrate to. So we used that. That worked fine. And then we went to work in earnest, which is like this is not gonna scale. And it had all kinds of issues at the time, that I'm sure we could have worked through. But we decided that we had just kind of build it into its own product, it can stand on its own. So off to work we go. And so we write the first version in C++. Because that's what we have been working on at that point. Get into that a little bit, but one of the things that always bothered me was, at the time at least, S3 had this eventual consistency. Which basically I found goofy to reason about, which is like, I was like I think, from a developer's point of view, it would make sense that we'd be all right. So if you basically go ahead, and write something, you should be able to read it back. So we started looking at things. And at the time we had some technology at Google that allowed us to do basically the pack, so we could basically give the strong consistency guarantees. So I'd been on this mad coding binge, and I basically ported it from C++ to Java, to go and demonstrate that we could do that. And this is all just kind of the metadata stuff, no byte pushing or anything else. That doesn't really matter. So we converted that. And I think that was really the interesting. That was a great choice. I mean, I'm happy we made that choice for various reasons, because it's been kind of a gift that keeps on giving. Because then later on, we were able to do things like object notifications. This is something that we'd have customer advisory boards. Folks would ask well, give us a listing that is faster. And I would ask them, why do you need a faster listing? And they were like, so I know if something comes up, new stuff comes up. And I was like, look. I'll give you, how about I tell you. I'll tell you if you get new object. How about that? So we added that capability, and it was really. It was really cool. And if you look at it, you can sort of kind of see in that, even proven computing, sorta kind of, later on with Lambda and so have taken off big time.
- [Dan] Civo was an alternative to the big hyperscale cloud providers. They've launched the world's first managed Kubernetes service powered exclusively by K3S With sub 90 second cluster launch times, a simplified Kubernetes experience, and predictable billing, Civo's on a mission to create a better developer experience. Get $250 free credit to get started. Sign up today at CivoCloud.com/POPCAST. That's C-I-V-O dot C-O-M slash P-O-P-C-A-S-T. Go check them out.
- Anyway, so that's Big Store. That was fun. I did that for about maybe four or five years, or something like that. And by this point it had gotten pretty big. The kinds of things that I was doing was not something that I was really enjoying. Okay, now we jump back again to the early days of GCE. So when we were talking about the Compute, and everything else. So Joe and I, and a few other folks would always talk about. I was not convinced that a virtual machine is the right thing to be shipping. And I was biased by this point, because I had come to Google, and they were using containers everywhere. I never saw a machine, unless I broke something very badly. I would have the SSH in the box. But you only deal with containers and everything else. So I was kind of trying to float around the idea that we should be-
- Can we talk about that? So again, for everybody who knows this, I'm sure, C-groups were invented at Google, or that concept. Based on Solaris zones, or whatever. I mean, that concept. And so it was let me container that for you, or whatever. And so you're coming into this being okay. Like you said, you couldn't SSH into a box, 'cause everything was packaged, you were good to go, and all of that fun stuff. So again, that's a key thing to work into Google. You're so far ahead of the curve, in terms of that stuff. So it's like that. So now you're all in the GCE team, or GCP team. And it's like, okay. Joe's there, you, and you're like. How does this? I'm sorry. I'm using the DeLorean, and we're going forward now. How does this translate into, shit, let's Kubernetes? Let's give this to the masses. Let's Kubernetes this. How does that happen?
- Great question. So at that point, basically the response was that the market is not ready. Unless you're at Google, maybe this is not how you think about things. And I was like, okay fine. That's cool. I'll buy that. And then, so then we fast forward for about four years. Now, in that four years. So Brendan came into the mix. I don't remember exactly how, in what context I ran into Brendan, but somewhere in the cloud business. And Joe and I had been kind of talking every now and then, hallway chats, and Brendan and everything else, where I was like, I really want to go ahead, personally. Think about the service model. They're just like, you're exposing all these microservices, and make that super easy, and trivial, and so forth. And again, that's just like a Borg service, or stubby service, or GRPC, if you're not familiar. And that's how everything worked there. You're basically wiring protobufs together, and so forth. So I would every now and then kind of revisit this, because that's what I wanted. And that's what I wanted folks outside to be able to go ahead and do. So then about, again, about four or so years ago, kind of things aligned where I'm doing the Big Store stuff still, but not super excited about it anymore because different things, not as exciting to me. And then Brendan had put together this prototype that he was showing around. And this was, it was basically the very early versions of Kubernetes ideas, in the sense that you basically had a signal master. And then you had the minions that you would have, and so forth. And we would kind of play around with that, and then. But, yeah. That's kinda what I want. Is just, give me a container and make it into a service. So I was like, I want to come play with y'all. And they're like, yeah please. I left the team, and went to work with Joe and Brendan. And again, this was all kind of cobbled together, which is fantastic because that's all you need to do, is show the idea, and so forth. So a big plus one for them. Then we started. And again, this is the. It was all Java, and yada, yada, yada. But we all kind of wanted to learn Go. So Go was kind of the new hotness at the time. And you're not gonna learn anything unless you build something pretty reasonable size with the language, so you can understand the quirks, and how do you hold it, and so forth. So we decided that we were gonna write it in Go, so that we actually learned Go. The hillary that we had in the beginning, about putting the packages. Coming from Java land, it was just super goofy. And we did so many things, I think, poorly, or made some choices that I still kind of chuckle when people point to, well, Kubernetes does it like that. I'm like yeah, but we didn't know what we were doing. So don't follow that model. And, yeah. So you know. Joe was. We did some helper stuff. So we did some helper scripts, and everything else like that. And Joe would, he would add a little bit of Bash here and there. And I was like, Joe, let's go and write that in Python, bud. Let's go and do it in Python. He's like, it's just a little bit of Bash. And I was like, Joe, you and I both know what's gonna happen. That's gonna turn into a lot of Bash. So we seemed to kind of make this compromise where we put all those things in the slash hack, because I felt that they were hacks. And I still kind of think they're hacks.
- The infamous slash hack directory. So a couple of people asked me to ask you about that. Tell me about the origins of the infamous slash hack directory.
- Yeah, so again, this goes back into the, we didn't really know how to structure projects. And by no means were we expecting it, or I was not, maybe Joe and Brendan were, it to get this big. So when we were putting things together, we were, again, just kind of learning the Go together. And at the time, at least, there weren't very many projects that were, I guess multi-language, with things like some helper scripts and things like that. There was some Bash. And it didn't really belong in the scripts directory, or it didn't belong in the root, I think, they started. So I really kind of wanted it to be Python, because Bash can be a bit unwieldy. And it's great to go and do one-liners and little things. But once it starts growing, you can't test it, and yada, yada, yada. And Joe kind of talked me into, like, okay it's fine. But it's not. We put those in the hack. That was kind of the compromise, in my mind, was like okay, well let's put them into hacks. So it's clear that's not where you should go look for good ideas.
- What you build and where it takes you shouldn't be limited by your database. Cockroach DB helps developers build and scale apps with fewer obstacles, more freedom, and greater efficiency. So you can forget about the database, and trust that it just works. Kubernetes friendly, open source, and indestructible, Cockroach DB makes it easier to build and scale apps. It gives companies the freedom to serve customers anywhere, and it's backed by world-class documentation, and excellent dedicated support. Discover Cockroach DB, the most highly evolved and distributed SQL database on the planet. Kubernetes native, and built from the ground up to help companies of all sizes, including Bose, Comcast, and Equifax scale fast, survive anything and thrive anywhere. Sign up for a free 30 day trial, and get a free t-shirt at CockroachLabs.com/POPCAST. That's C-O-C-H-R-O-C-H-L-A-B-S dot com, slash P-O-P-C-A-S-T. Can I ask you this? It seems like, again, you've been tied at the hip, to a certain degree, to Joe Beda for a long time. What would you say? How are you so simpatico with each other, in terms of how you work together? Talk to me about that work, just the relationship, the personalities together. 'Cause I love Joe Beda. Joe, Joe, Godfather Joe, you know I love you. He's a wonderful person, just a wonderful person. But I've never worked with him. So I don't know that relationship.
- Had a great time working, smart guy. We got along super well. I have nothing but good things to say about him, or anybody else in that matter. So at this point, it's just me and Brendan, and Joe. And Greg, of course, is doing great things. So the three of us are coding, trying to learn Go. Okay, what is this channel thing? The first version of the cube proxy, well seven proxy at the time. Because everything was seven at the time. That was my way of learning how channels work. So it was like the first couple of ones had deadlocks, and all kinds of things, and yada, yada, yada. But that's how you learn Cut their finger off, and-
- So you all grew together. Just throw these things together, and it just worked. Let me ask you this question. We're gonna go. Take the hat off, come to now. Take the DeLorean, we're present day. Do you look back and say to yourself, Jesus Christ, I cannot believe this cobbled together thing hit like this hit?
- Absolutely not.
- You had no preconceived notion of this? You were like, no.
- No, I did. No, not. No, absolutely not. And I think a lot of those things. Look, there's a lot of great things about Kubernetes. But it's still too damn hard. I think it was necessary, but not sufficient. And I always, I've used the term. It's the cloud assembly language. It has a lot of very low level constructs. And if you get them all right, then good things happen. But it was still hard, like the conventions. You have to know what you're doing, and that's cool. It was still enough to go ahead and get people excited. You might not be at the North Star, but you can sort of kind of put enough points in there where you can kind of go like, I can see where it's going and I kinda like it. And I think, it became the community aspect of it, early on, that got people really excited. And I think that it was such a welcoming, and like hey yeah, let's do it. It was something even. I don't know. I feel it was kind of a magical bit, as far as the early coming from few, just three of us. We kind of started showing that at. I think the first time we, I actually, I think even, even met Tim and Brian was, so GCP, Google Cloud, or TI, I forget what it is. But then there was basically these summits where we would go together, and get some people from cloud and TI, Technical Infrastructure, and just kind of chitty chat about stuff, share some stuff, like here's some new stuff that's coming and yada, yada, yada. And I remember that's where we gave, I think it was Brendan, Brendan or, I can't remember. It wasn't me, certainly. I can never be trusted with that. But anyway, so they'd basically give the presentation. And it created a lot of buzz for it because, especially with folks Tim and Brian, who had been involved in Borg. I'm sure they had also been thinking about how do we go ahead and make this available to folks? That's where we kind of started getting more internal exposure into, and excitement, as far as, and also talent. So if you think about the kind of talent, because they have been working, and living, and breathing that stuff for awhile. They could come in and go, yeah. If you do it like that, you're gonna be screwed. Also, I think, one of the, I think one of the great things about Kubernetes that necessarily was not apparent early on was the importance of the API and this. I remember Brian Grant did a lot of work on this one, on the API surfaces, and making sure that each one of those, the fields that we were doing, matched up. And if you squint your eyes and you look back five years, and you're like now you have CRDs. And because of that common definition, you can basically go ahead and treat everything always as the same. Maybe he knew that that was gonna happen, maybe not. It was not apparent to me at the time. But that's where a lot of the value, I think, they're starting to now come in, where it's not just the Kubernetes, the baked in things. But it's really the platform, tool building tool, if you will.
- [Announcer] Learn how to operationalize open policy agent at scale with Styra. To get started, go to the link at H, double T, P-S colon, slash slash H-U-B-S, dot L-Y slash, H zero P-N-K-M, two zero.
- I believe that Kubernetes is the greatest open source community, community-based open source tool that ever was. And here's what I mean by it. You have Google, and you all did what you did. You created this thing. Y'all had Hockin, and you had Brian Grant. You had all of you guys. Put it together, gave it to the masses. Seeing the open source people and community adopt this, and grow it, and stuff like that. To me, this is the poster child for how communities should be built, should be run, should be done. That takes, you should be proud of that every single day. 'Cause this is the gold standard from any project going forward. Do you ever say, again, now you're like, oh we didn't think it was gonna hit. But do you ever say, yeah, I'm really proud of this? And let me ask you this. How do you feel? The Kubernetes community. What does that mean to you? Give me some of the vibe about the Kubernetes community, just in general.
- Vibe wise, it is just amazing how welcoming, and how helpful it is. So I think, I mean I'm still involved, very rarely, and every now and then hop into that as well. But even when I go back into Cube Con, or go ahead and look at some of the issues, like some of the duct taping stuff that we've been doing in K Native, and try and go upstream that. It's just so awesome to go back there, and have those kinds of conversations, where it's just like, how can we do these things, and yada, yada, yada. And everybody's kind of, it's not like. How would you put this? There's people who are sort of kind of like. There's no titles. Meaning, it's just conversation, the kind of conversation where you're like, hey would it be bad if? And then you're like, yeah that's a terrible idea. Or wait a minute. No, I don't know. Let's talk about that. So in a sense, it's very, very opening. And it's all about sort of kind of like yeah. I want to go. How do we make this happen? And how do I help you? I don't know. It just blows my mind. So you asked about the community, what does it mean? Every time I've got the Cube Con, and I'm like oh my goodness. All these people are basically, their lives are easier, their lives are better. They are basically being able to do things they couldn't do before. And there's companies built. It's just mind boggling to see what all it has enabled. And yeah, I certainly never. I just want it to go and help my fellow dot com person, expose some web services, and be able to go ahead and automate all that, and bring the goodness to that we had seen at Google, and really simplify the model. Where you don't have to go and worry about, what's in my, more hosts? How do I set up the DNS? How do I do this, that, and everything else? Get away from things that don't really matter.
- Teleport allows engineers and security professionals to unify access for SSH servers, Kubernetes clusters, web applications, and databases across all environments. You can download teleport right now at GoTeleport.com. That's G-O-T-E-L-E-P-O-R-T, dot com. It's interesting, and again, we talk about community. And one of the things that I feel like we came full circle. I was so influenced by you all. And again, being welcome in the community, I talked, when Joe Beda was on, I talked about how man, the guy just was like. Anytime I asked a question, he was responding to me on Twitter. Props to Joe. But now I feel like, again. We had something in Falco. We had the contributor of the month. And I'm like that was my idea. And I was like, now you all in K Native are using it. And I'm like, you know what?
- We stole that.
- No, you don't because I think. No, it's not stealing, as we're a community. I'm so happy and proud that you all did it. Because it's like now I want to give it back. I want everybody to think of this, is you want to give back to the people that you learn from. So that's just, that's a beautiful thing that's from the community. And I was touched by that. But I'm also happy that you all found a use for that. So that was cool.
- Absolutely. And it just, just to kind of riff on the contributor of the month. And I think also one of the great things that Kubernetes did was the chop water, carry wood thing. So basically acknowledging those kinds of things. Because far too often, that doesn't happen. So that was an amazing idea, whoever did that. I can't, I don't recall. But I think that being able to go and recognize those kinds of contributions, especially in the open source projects. There's all the kind of features, and everybody's doing the new shiny, and everything else. And then there's just the whole bunch of stuff that is just kind of keeping the lights on, keeping things running, and it doesn't show. When nothing breaks, nobody comes, yeah thanks for not breaking things, but.
- And that's the hard part. And we were talking about this earlier. It was in terms of hard part of open source. I mean, open source is getting involved in this, PR. It's hard to write code. It's hard to do code review. And these types of things, and that's why you need the power of a community. And that's again where, I think, you bring people into the project, start on documentation, learn what's going on, do code review and all those things. Do you think that's an advantage of open source, versus closed source like commercial stuff? Is that piece of it, is to being able to get people in quickly, that type of thing.
- I definitely think so. And I think, because so many of the tools that are being used in open source, in many other places, they are very familiar. It helps when you move in between different projects. Whereas when you work for a company, they have their own tools, and they have their own processes, and yada, yada, yada. And I think the open source, at least they tried to go and take the best practices from what I've seen. Especially in Kubernetes, everything is standardized there. So that's a lot of mileage for people. So from API machinery to something else. You're like, okay. I know what's up. Whereas if you're in a company, and you work on a different project or product, it could be a totally different code base. Who knows what they're doing there? And Google of course is the mono repo, so everything was the same, which was great. But yeah, I think just the welcoming aspect. And everybody understands that if I go in, it's an investment. Just like, if you think about it. We get a new contributor that is filing an issue, and/or asking for help. For them helping them, first of all, oh my goodness, you're using our product? Thank you. Let me help you. And I'm so sorry that we didn't document it properly, or we, I, nobody else writes bugs. But I wrote a bug in there that you have to go, and go through, and yada. Let's go fix that up. And then if people have a positive experience, then they're more likely to at least come participate. And kind of to your point about writing code, hard. It's one of the hardest. Or another trickier bit is to go ahead and, and not enable that communication early on from. If I want to contribute to a project, coming in and locking myself in the basement, writing a huge BR, and slapping it in, and going well, they didn't take it. If go and start it out by kind of like, hey by the way, here I am. I'm thinking about doing this, kind of a two-way street, then it just gonna be easier all the way around.
- When you think about it, the majority of the people that aren't the maintainers are usually, that's not their day job. Their day job is not this. And so that's why, again, we talked about this. The chop wood, carry water, and the contributor of the month ideas. And this is something also, I talked to somebody that was in the Docs team. And shout to you Zach, Zach Orlsian, and Courtney on that side. And that's where I got this idea. It's, you gotta give back to those people. They're taking their time, and being able to do this. And it makes, and enriches the project so much, so much. I want us to switch gears, man. We've been technical for awhile. You ready for this? All right.
- Yeah yeah, yeah yeah.
- So, look. Tell me about Ville's island. Can you tell us about Ville's island? You see how Matt Moore, and Scott got me in trouble here? Jesus, you fed me a lot of stuff. So it's all good. Tell me about Ville island.
- Yeah, well as Matt calls it, it's the Isle of Vile. So for the longest time, Matt has been basically changing my name from Ville to Vile. I'm not gonna read into it. I assume that's just good, clean fun. But so he renamed it. It's not Ville's island. It's Isle of Vile. Because apparently it rhymes. But thanks man. So being Finish, there's a couple of things that I really would like to have. One of them is sauna, which I have here. And the other one is a cabin. I've been wanting to get a cabin for a long, long, long time. So a few years ago, my wife and I started looking for cabins in Finland, in earnest. And we asked her parents to go take a look, 'cause we were actually here at the time. So they go in, and they go look. And they're like, nah this place is not all that great. It's pretty close, and there's a lot of traffic there, and yada, yada, yada. But how about this? And they basically, they basically have gone and checked out this little island. And I was like, oh my God, an island? An island, island? Like there's only, like it's our island? And they're like, yeah. And I was like, oh my goodness. Of course. Why wouldn't I? That sounds amazing. Yeah, so we got the. Yeah, so we got the cabin. And it just turns out that it comes with the island of its own, which is great, because we have dogs. We have our own privacy, and so forth. So yeah, we ended up getting a little island.
- That's incredible. Maybe someday we can have a POPCAST from the Isle of Vile. What do you think?
- I would be very disappointed if that never happened.
- Ville, I got one last question for you, my man. What work are you most proud of?
- What work am I most proud of? I think GCS. Yeah. And the only reason it's because it actually, there's a lot of interesting technical stuff. But there was also a lot of team building, growing folks, and so forth. I'm really proud of that, given where it came from, and all the things that it had going for that. I think that that's gonna have to do it. There's a lot of good ideas there too.
- Well, Ville. You're silent no more. We've had your story. George Harrison of Kubernetes. Finally got your story out there. Stop taking all the glory, Tim Hawkin. stop taking all the glory, okay? All jokes aside.
- He deserves it.
- He does.
- He deserves it.
- You all do. You all do. You're fantastic people. I appreciate you being on the show. Thank you so much.
- Well, thank you so much. I appreciate it. That was awesome sauce.