Thomas Strömberg has 25 years of experience in systems automation, open source, and building user-focused engineering teams. He was a vital leader for the minikube, skaffold and many other developer tools at Google In this episode we go through Thomas' journey from hacking his brothers saved games all the way to Google and Open Source. Thomas also gives a world premiere REVEAL only on THE POPCAST... of the next company he is moving to after 15 years at Google. We hid the reveal in the episode so you will have to watch the whole thing to find out :) An absolute pleasure speaking to Thomas and i think you will thoroughly enjoy this episode with one of the best people in our community
Timeline/Topic
00:00 - Sponsor - Stormforge - stormforge.io/popcast
00:58 - Opener
01:06 - Introduction to Thomas Strömberg
01:39 - Thomas Journey
12:43 - How did Thomas get to Google? the Google Years.
23:44 - Skaffold
28:58 - Thomas talks about Kubecon and the Kubernetes Community.
30:01 - The Reveal ??? or a free t-shirt from https://cockroachlabs.com/popcast.
32:47 - the End User loop / Hallway Track
34:17 - Minikube
53:24 - Kind vs Minikube - Why not combine them? Differences?
58:18 - Amateur Bike Builder
01:02 - what work is Tom most proud of?
01:05:13 - The Reveal ??? or a free t-shirt from https://cockroachlabs.com/popcast.
Episode Links
http://stromberg.org/t/
https://github.com/kernelcafe/welcome— Public infrastructure for open source developers (author)
https://github.com/google/triage-party
https://github.com/kubernetes/minikube — Local Kubernetes for everyone!
https://github.com/GoogleContainerTools/skaffold
Episode Sponsors
COCKROACH LABS
Discover CockroachDB, the most highly evolved distributed SQL database on the planet. Kubernetes-native and built from the ground up to help companies of all sizes including Bose, Comcast, and Equifax scale fast, survive anything, and thrive everywhere.
Sign up for a free 30-day trial and get a free t-shirt at https://cockroachlabs.com/popcast.
STORMFORGE
StormForge’s Kubernetes Performance Testing and Optimization platform is the easiest way to ensure your applications behave the way you want them to, while cutting out unnecessary resources and time spent manually tuning.
In alignment with this, StormForge is asking for your help reducing the amount of unused Cloud resources making both a financial and environmental impact in the world today. We want you to help make an impact with us, so visit https://stormforge.io/popcast to learn more about how you can help ERASE Cloud Waste, and take the Cloud Waste Pledge.
While you’re there, why not try out the FREE TIER of the Machine-Learning-backed service to start saving resources and getting better performance today.
POPCAST SHOW DETAILS
Watch (YouTube): https://www.youtube.com/c/thepopcastpop
Listen (Apple PODCAST and others): https://popcast-d9f7b6dc.simplecast.com
Follow us on (Twitter): https://twitter.com/PopcastPop
Follow us on (Linkedin): https://www.linkedin.com/company/the-popcast-with-danpop
Stormforge Voiceover:
This month's popcast sponsor is StormForge. StormForge's Kubernetes performance testing and optimization platform is the easiest way to ensure your applications behave the way you want them to, while cutting out unnecessary resources and time spent manually tuning.
Stormforge Voiceover:
In alignment with this, StormForge is asking for your help reducing the amount of unused cloud resources, making both a financial and environmental impact in the world today. We want you to help make that impact with us. So visit https://stormforge.io/podpast, that's S-T-O-R-M-F-O-R-G-E.io slash P-O-P-C-A-S-T. To learn more about how you can help erase cloud waste and take the cloud waste pledge.
Stormforge Voiceover:
While you're there, try out the free tier, the machine learning-backed service, to start saving resources and getting better performance today.
Dan:
Hello everyone, and welcome to the popcast today. Look, you probably wouldn't be doing half the shit you do on your home computers, trying to do things, interacting with Kubernetes, understanding, learning Kubernetes, if you didn't have this guy do what he does. This is Thomas Stromberg. He's a Kubernetes open source contributor himself. He's a carefree father of two. Team lead and manager of Google's container dev X team. Again, this is Thomas Stromberg.
Thomas:
Well, thanks for having me on, Dan.
Dan:
Appreciate it. So let's talk about, hey, the brass tacks, all the way to where you are now, your journey. You started from ... You're coding, and you're like, "I love this. I want to do this." Talk to me about that.
Thomas:
I think, for a lot of people in my generation, computers really stemmed from gaming. And my first computer, which was a brilliant machine, the Timex Sinclair ZX81. We're talking the chiclet keyboard, two kilobytes of ram, with a 16K add-on pack at the back. And this was a wonderful machine. This is the machine that you would see in the catalogs in the '80s of buy your computer for less than $100. I think $99 was their shtick.
Thomas:
And so I had one of those. But I didn't have any storage. I had no tape drive or anything to hook it up to. Some games, for me, came from magazines and books of here's the source code to the game. And so, basically, me at nine or ten years old, I'm tapping away at all the source code to play a stupid game.
Thomas:
And, at the time, I was living in south Florida, which is famous during the summers, when I actually got time to use the computer, famous for thunderstorms, like 2:00 PM every day. And the thunderstorms would knock the power out, and I'd lose my game. And it would piss me off so much. But I got really good at typing and debugging. Because you spend like a day typing the code in, maybe an afternoon typing it in. And the next two days debugging all the typos that you introduced from typing this program. Get a couple of days of gameplay. And then, boom, power goes out. You do it all again the next week.
Thomas:
And that kind of got me into ... I guess I consider my superpower to be debugging, at this point. And I think it all stems back from having that stupid machine.
Thomas:
So that got me into understanding basic programming. I wouldn't say I was a great basic programmer, but I understood how to do it. And I got into operating system internals in the same way. All right, early '90s, there used to be something called Novell DOS. And I don't remember why we have this [crosstalk 00:04:07].
Dan:
Novell with two L's, right?
Thomas:
Yes, exactly.
Dan:
Novell with two L's.
Thomas:
One of the awesome things ... So, my brother and I, we used to cooperate on this one video game. And he didn't want me to go ahead in any levels without him. So he password protected the directory. That was something you could do in DOS back then. And so I got so pissed, because I knew I wasn't going to see him for weeks, over a summer. So I was like, "I can hack this. Let me look at the file system layout with a hex editor, and see if I can go ahead and figure out how do I turn off the password using a hex editor?"
Thomas:
And, a couple of days in, I figured it out. And I was like, "Okay, I understand file system layouts now. And so it went from there, as basically just this whole journey of being given this problem, find a way to work around it, without asking anybody for permission, and keep on moving along. Sometimes not asking for permission gets me in trouble. But I think it's actually, as a whole, been pretty successful in my life.
Dan:
So, I mean, again, this is the arc. And I've had a bunch of people on the popcast that start from the magazines, the Commodore 64 is hacking away, debugging. Everybody seems to have cut their teeth on it.
Dan:
I talked about this as well. My father had this computer for accounting. We owned pizzerias. And he was like, "I had to hack into it, install some MicroProse games that came on 5/2 floppies, to be able to play the Punisher, right?" It was like, "Wow."
Dan:
But that's incredible, just that whole trajectory, and then getting into this. So, again, you're debugging. And this is like, "Okay. Now I'm debugging, basically hacking into my brother's directory." But what happens next? How do we get into where we're at now with Google?
Thomas:
So I blame BBS's. Fully blame BBS's. So, for people who aren't familiar, listening to this, BBS is where you basically ... People have these modems hooked up to phone lines. You can dial your friends. You can dial a central place. Some people used it to share games. Some people used it to share messages. I did a little of both a teenager.
Thomas:
But, nonetheless, I realized, here was something for us geeky kids, a way for us to connect. Because we couldn't really see each other in person. Just getting permission to go and like visit someone's house that's across the city, or in the next city over, it wasn't happening. And, quite honestly, us really hardcore computer geeks, first of all, we were too scared to ask our parents anyways. But even if we did, we didn't really have the means to get there.
Thomas:
So BBS's really drove this human connection of knowledge sharing, and just camaraderie, and a sense of community between us geeks. And I fell for it hook line sinker. And I said to myself, "I want a job that uses computers to connect people. Someday I want to do that." And the opportunity came up, fell into my lap a little bit. After graduating high school, I moved to Sweden with my family. And I didn't really have anything going on. And my dad was sick of seeing me mope around the house, basically just playing way too much GTA.
Thomas:
And so he set up a interview for me at a local school. It happened to be the school my sister was going to. And he basically just dropped it on the breakfast table in the morning, said, "By the way, you've got an interview at 10 o'clock in the morning." "In doing what?" "System administration for a school." I had never had a real job. I didn't really know what a system administrator was. At that point, I just knew that I can automate stuff. I knew how to get by with whatever challenges were faced.
Thomas:
And, like many schools, even now, but especially in the '90s, there was these curriculums that we wanted to get everybody to know computers, but we didn't want to spend money on computers. And so ridiculously underpowered machines. This is '96, '97. And everything was starting to migrate to Windows 95. And these machines only had 20 meg hard disks. I'm not saying 20 gig hard disks, I'm saying 20 meg hard disks. And you actually can't have a full install on Windows on 20 megs. But we were expected to make this work anyways.
Thomas:
So we figured out that we could actually move the system [inaudible 00:08:56] directory to network file share. And we could do this really hacked up installation. I learned a lot about using cURL to automate windows. And then, okay, we need authentication. And, oh my God, the Microsoft licenses are too much. Let's figure this Linux thing out. And let's figure Samba out, and let's do all this. And then that drove us a little further. And then Linux, at the time, was notoriously unreliable in certain releases. And that's when I learned about FreeBSD.
Thomas:
And I think learning about FreeBSD is when I became a bonafide operating systems nut. All of a sudden, I could get usernames longer than eight characters .I could get by without having to reboot my machine every couple of days. FreeBSD, back then, was the solid choice for everything. So I learned quite a bit from that experience of just becoming ...
Thomas:
I mean, just using FreeBSD, or any of the BSD's, gives you a little bit of extra insight and knowledge into what Unix is about, that I feel is missing in the Linux monoculture that's been built. Although I highly appreciate the fact that that monoculture exists, because it's definitely better than the ... Here are the 20 flavors of Unix you have to make your application work, on that we suffered through the '90s.
Dan:
Oh yeah. And also just the ease of ... It's for the hardcore folks like us, that have cut our teeth at the early ... It was a Linux. And then it's like, "Folks, I just installed Ubuntu 20 for my kids on their laptop." And it took like a millisecond. And installed all the hardware drives. I'm like, "You don't know the pain of having to compile an audio driver or video drive." I mean, it was just nightmares for us.
Thomas:
Oh yeah, it absolutely was. So it's really hard for me to say, because I don't know what it's like to grow up in today's generation of technology. But I have to wonder if ... I feel like the arc to get to being a successful software engineer or systems engineer is different now, because there are so many more layers of abstraction that you have to learn. And it's not in your face immediately, that you have the power to control everything.
Thomas:
For somebody growing up on, say, an iPad, if you don't like what the iPad gives you, tough shit, you got to use something else. And you don't really have that opportunity of, okay, let's let a hex editor edit the file system, to get by this protection. No, you've got DRM everywhere. And so there's definitely pluses and minuses for everything. But it's a very different upbringing.
Thomas:
And it didn't really strike me until someone on our team ... So we have somebody on our team who's getting involved with minikube. And someone reported a Windows issue. And we were like, "We actually don't have time to look at this. Do you mind looking at this?" And the person told me, "I've actually never used Windows before." Never in that person's life have they ever been exposed to a Windows machine, and didn't know DOS. They only knew Unix based operating systems. They got into computing when macOS X was starting to be a thing. And they learned Linux after that. So that just blew my mind.
Dan:
Crazy.
Thomas:
A different world.
Dan:
Different world, definitely.
Thomas:
But that person is also so much more productive than I ever will be. So, just because we have experience doesn't mean we're better, it just means we're different.
Dan:
It's just different ways of attacking the problem, right?
Thomas:
Exactly. Yeah.
Dan:
Again, I want to talk about you get to Google. What brought you to Google, from being this system administrator in Sweden? I want to know that story. I want to get from there to-
Thomas:
All right. Okay. So I decided to move back to the United States. And, thanks to IRC, I got a job in the US through IRC. BBS's had basically ... They were gone. IRC had taken over everything. And somebody was looking for a specialist in FreeBSD and Solaris. And I had just happened to make a name for myself, primarily in FreeBSD.
Thomas:
And so I went to work for a startup in North Carolina. And it was a very scrappy little startup, but we did well. We did a lot of ... I'm probably one of the laziest system administrators you'll ever encounter, in that I will always script everything. It really just got-
Dan:
Lazy? Or just trying to automate yourself out of a job? What do you-
Thomas:
Well, it's a constant effort of trying to automate myself out of the job, so I could goof off the rest of the time. So even if it's something that I think I will just do once, if there's any possibility I ever have to do it twice, I will write a script. And, back then, it was a lot of cURL. Eventually, it became a lot of Ruby.
Thomas:
The thing that I loved about the scripts, that they were always self-documenting in nature. If somebody asked me, "How did you do this?" I could point them, "Here is the URL, here's the file. Run this." No more documentation. It's always self-described. And I'm pretty meticulous about documentation, because I'm also extremely forgetful. And so I found that just to be super useful for myself. [crosstalk 00:14:39].
Dan:
Can we sidebar on that real quick though? And, again, I don't know what it ... Your documentation for minikube is fantastic. It's like the original steps of this ... And we're going to get into minikube, kids. We'll get into it. But I'm sitting there ... And this is, again, I needed something on my local box, just to be able to test a couple of deployment scripts that a customer gave me.
Dan:
And I'm sitting there and I'm banging my head. I'm like, "Okay, so I'll just run this in a container. And I'll just pretend like it." And then I saw your thing. I'm like, "This is minikube." And this is, what, three years ago, we're talking really early iterations of it. Blown away by it. I'm like, "Look, I can, I can use the local hypervisor. I can use Mac. Or I can even take this same thing, move it on my Windows servers, or a couple of my nucks back here, and it just works." Right?
Dan:
And I look at the documentation, and it's flawless, man. It is goddamn flawless. It's like, "Do this, cURL this, grab this, move it to here, minikube start." Cool.
Thomas:
I wish I could say all of our documentation was that good. Some of it is. I am a strong believer that documentation drives adoption. And it's not just an open source thing. It's also even for internal software. For me, the biggest drivers of adoption for any technology is documentation. And it goes hand-in-hand with inclusiveness. If you can't structure your project in such a way that people feel welcome to participate in it in a longterm, your project will eventually be replaced by what is inclusive.
Thomas:
And this is really the success of Kubernetes and Linux, in a nutshell. We're not all running AT&T Unix at this point, thank goodness. And, in large part, because these software vendors were very ... They were willing to take changes, even if they didn't necessarily find them useful personally. But they were willing to take contributions from the community, have a well-structured process, have great documentation, and really investing in a constant onboarding process.
Thomas:
Because, quite honestly, with open source, a lot of contributors are drive-by contributors. And you're always going to have to optimize for that path, where somebody is a contributor for maybe two or three months, and they move on to the next thing. Like so many of us geeks, a lot of us get a hobby for two months, and then we'll move on to the next hobby. And it's a path we have to optimize for.
Thomas:
So, anyways, getting to Google, I got really good at automation. And it turned out that it became useful. I met the love of my life in North Carolina. I followed her to Indiana. And I didn't know who to work for in Indiana. So I ended up working for the university that she went to, because they were looking for people to build out super computing capabilities. And automation is everywhere when you're talking at scale.
Thomas:
And it turns out I made a name for myself, and Google actually called me, like, "Hey, you want a job? Or at least interview for a job." And I said, "Well, where are you hiring? Because I'm going to be moving to Atlanta in a couple of weeks with my girlfriend." And they're like, "Well, sorry, it's Bay Area." So I said, "Call me back if you ever open an office in Atlanta."
Thomas:
So I worked in telecom in Atlanta for a couple of months doing automation stuff, most miserable job I've ever held in my life. I do not recommend telecom to anybody, anywhere. I don't care what the name of your company is, it's all miserable. I won't name and shame here. So when my six months were up, and I could get somebody ... And this time this job was through LiveJournal, by the way. So going back and forward in time a little bit.
Thomas:
So I got the guy to LiveJournal his little hiring bonus. And then my wife actually started looking for a new job for me before I did. She saw how miserable I was. And she happened to notice that, "Hey, Google did open an office in Atlanta." Happened to be a data center. But-
Thomas:
Yeah, happened to be a data center, but there's job openings. I applied there and I got to be part of their hardware systems team, which is basically deploying hardware at scale across the globe. And I got very involved, started with data analysis work, trying to determine like, "Hey, this combination of this hard drive model and this machine fails this much more, or it has this much more problem in pairs." But eventually I got to, how does Google actually build out new clusters from a software stack? And it turned out to be a miserable... well, I shouldn't say miserable, it was genius at the time, 15 years ago, of basically trying to apply Python unit testing to unit testing a cluster and saying like, "If a test fails, oh, I have work to do. Go ahead and run this work stack, re-run the test afterwards."
Thomas:
The problem is deploying a new cluster like the software stack, Google software stack is incredible. We've got our own cluster file systems. We've got our own mock surfaces and it's a lot of the stuff you see in Kubernetes nowadays, a lot of the types of, I guess, abstraction layers. But we were basically trying to deploy from bare metal to, "Hey, I'm going to run Gmail." And it would take months to get that stack up and running on the machine successfully.
Thomas:
And part of the problem was that we were only deploying new clusters say, once a quarter. And software moves quick once a quarter. And inevitably we get her to play software again and things would break. And our entire team owned the automationing stack. We actually owned the automation stack for how do you deploy a cluster file system, rather than the teams that own the cluster file system. And they felt this made sense because, "Hey, we're a bunch of system administrators, [inaudible 00:20:57], but we can script our way out of any problem." It turns out that you can't script your way out of organizational problems. That there is no way you will ever keep up with the rate of change in automating the software stack that you are not involved in. And so inevitably there was always a change that we didn't know about. Like teams wouldn't tell us that, "Oh, there's a new flag you need." And we'd run the automation and then we'd have to spend days trying to figure out here's the missing flag.
Thomas:
So we changed to this distributed system, all giant pile of YAML that would basically talk to each distributed API server from each team and say, "Okay, are you good in this cluster? If not, do your thing." And then check later to make sure you're still good. And we ended up getting this process that was months down to days, which not only saved Google a pile of cash of cooling and power, but quite frankly, opportunity cost wise. If we can't deliver a cluster's new locations quickly, other people'll beat us to the punch. Early on, Google didn't have a lot of competition in scale, but now they do. So we did that and that worked out great. But then we ran into a problem of how do you test this? And how do you test changes before they hit production? And it got me involved in an effort called Virtual Cluster at Google, which is basically kind of similar to minikube, honestly, in that, how do you emulate production locally, and how you do it at scale in a virtual environment? Because you can only run so many nodes locally.
Thomas:
And so I got very familiar with virtualization and automation of deploying nodes, it was fun for a while and necessary, but I got tired of working on projects that I couldn't talk about. And it eventually led me back to the open source world where there was that community and inclusiveness that I was looking for. And Dan Lawrence, who is the original author of minikube, and also Kubernetes contributor, basically posted a job opening internally and said like, "We'd love help, somebody. Please help us with minikube or Skaffold or something. Join our team and we'll figure it out afterwards." It was exactly the invitation I was looking for. And so I joined it. Didn't know a lot about Kubernetes at the time, but I knew Google's equivalence internally. And it was clear that it needed a lot of help. User-friendliness, wasn't quite there yet. So yeah.
Dan:
That's why we're here. So it says like, again, in terms of, I want to talk about minikube last because I want to talk about others, I want to talk about Skaffold. So I had a met on Al Balkin.
Thomas:
Yeah, I love Al, man. He's awesome.
Dan:
We all do it. He's great. He's got a restraining order against me because I'm a stalker. So I talked to him about the microservices demo, the hipster demo and I told him how it is the most complete and perfect demonstration of the capabilities of deploying. I mean, you can deploy it on mini, you can deploy it anywhere. It gives you STO and all this stuff, and Skaffold, that's where I got first introduced to it. I'm like, "This is amazing. I can click, it does all of my applies or, puts my daemons in the places I need, treats it as code, checks it into GCR if I need to. It's incredible." So talk to me about Skaffold a little bit.
Thomas:
I think you nailed it as far as the description. I would say my vision for Skaffold is that you can take any arbitrary, open source or closed source project. Like you go to a GitHub page and it doesn't matter how complicated, how many are microservices are actually deployed. But the idea is that you check out something like the hipster-shop, you check out the GitHub repo and you apply it to your local development environment and you say, "Hey, looks great. I'm going to deploy it to test. Okay, looks good, I'm going to deploy it to prod." And you didn't have to write any configuration at all. The configuration was already baked in. All you have to do is select a different destination. And that is for me, what I see is as the vision of Skaffold, basically a descriptive language of how is this set of microservices architected together. But also have a wicked fast local dev loop, so that you can make a change and iterate on it.
Thomas:
We aim for zero delay. We're not quite there, but yeah. So you make your changes locally or to the Kubernetes cluster, you promote to production. So this tool had been laying around my team for a couple of years and we did do the 1.0 launch right around Thanksgiving of last year. And one of the cool things that we ended up doing is we did it in cooperation with the Cloud Code team at Google. So Cloud Code is a set of IDE extensions for VS code or IntelliJ. And it's basically just, a lot of it is a UI over Skaffold. So they really pushed us to make the friction-free experience. Because one thing I've learned about trying to collaborate with user interfaces as, I'm a command line person, but being part of a graphical user interface really pushes you to make sure that everything works without additional user input.
Thomas:
Because we can't just say an error message in Skaffold and say, "Oh, by the way you need to do this." Because the person who's using it through a UI may never see that message. Like it may never get thumbed through on a prompt. Those on minikube and Skaffold's sake, we really have to be very cognizant of, never interrupt the user, just do what they wanted. Try anyways, try to do what they wanted. And a lot of the changes for our minikube and Skaffold in the last two quarters have been driven by the fact that a client could use those tools or not.
Dan:
It's fantastic. And you know, one of the things I really like about Skaffold, as well, I love, this is genius, where you included the ability to do the Docker builds in Cloud run as well. That is such cool things. Think about it, let's just say you have an underpowered machine, if we talk about this whole folks' adoption and folks utilizing the [inaudible 00:27:47]. Think about somebody who's running this on a very low powered Chromebook, how they have the power of X amount of cores to be able to do their deployments and all of that. That makes it so accessible for folks who don't have the resources they need to make some amazing code. That's brilliant.
Thomas:
Right. And I think that's where the world is moving to, especially in large businesses. This is something that I feel like every decade, it waivers back and forth of like, "Is everything going to be remote? Or is it going to be local?" I am kind of a local person myself, but I do see that the direction that we're moving to, of like IDEs for instance. I think when you start talking about large businesses and I'm talking about your Verizons and JP Morgan's out there, that they want to be able to put a new person at a desk and have them productive in their IDE and deploying to production ASAP. And that means a fully configured IDE in the cloud. And it means the ability to build images, even if you don't have local access. One of the interesting things that I didn't realize until going to KubeCon, and by the way, KubeCon is just such an amazing thing. I wanted to call it out. I would not be a participant in Kubernetes, such a hardcore fan. If I had never gone to KubeCon. Like, it just changed my life.
Dan:
An amazing community.
Thomas:
What?
Dan:
It's just such an amazing-
Thomas:
Yeah.
Dan:
Again, here's the way I see it. It's like, big fan of yours, I've used your tool for X amount of years, right?
Thomas:
Uh-huh (affirmative).
Dan:
I've been to a bunch of KubeCons. We've probably grabbed beers together, unknowingly maybe, at some point, but at the end of the day, it's this community. Everyone's so accepting of each other. It's like, and I've said this in a bunch of popcasts, there's no other community like this in the world. I don't think so.
Thomas:
I haven't seen one, no. So one of the wild things that I learned out of KubeCon and particularly at KubeCon, San Diego, was this idea of a lot of these larger companies, they don't allow virtualization on their personal machines, and that includes Docker-
Dan:
What you build and where it takes you. Shouldn't be limited by your database. CockroachDB helps developers build and scale apps with fewer obstacles, more freedom and greater efficiency. So you can forget about the database and trust that it just works. Kubernetes friendly, open source, and indestructible, CockroachDB makes it easier to build and scale apps. It gives companies the freedom to serve customers anywhere and it's backed by world-class documentation and excellent dedicated support. Discover CockroachDB, the most highly evolved and distributed SQL database on the planet. Kubernetes native and built from the ground up to help companies of all sizes, including Bose, Comcast, and Equifax. Scale fast, survive anything, and thrive anywhere. Sign up for a free 30 day trial and get a free t-shirt at cockroachlabs.com/popcast. That's C-O-C-H-R-O-A-C-H-L-A-B-S.com/P-O-P-C-A-S-T.
Thomas:
... and so, how do you give these people a great environment to iterate on their tooling, on their application, if they can't build a Docker image? So one of our other projects is Kaniko, which is basically, allows you to do Docker builds trivially in any Kubernetes cluster. So it's one of the targets that you can do for Skaffold. It's very similar to how you do with say Cloud Build Google's hosted service, right? But you can do this on any on-prem cluster. And that's one of our more fun projects that I really want.
Dan:
That's very cool.
Thomas:
But yeah, a lot of these larger companies, like Verizon actually have a presentation about this, like, "How do I work around the fact that I can't do anything on my local workstation?" And I found that really interesting, especially in the hallway track afterwards, hearing a bunch of other people, primarily financial institutions, who were like, "Yeah, we're totally there. How do you do it?" And people were just sharing ideas. And it really gave me a lot of food for thought about how important this idea of completely remote development, remote managed development environments, is I think really going to take over in the next two to three years. I personally will still probably develop locally, but I am also the one with an unreliable internet connection, so.
Dan:
Let's hope it's reliable for the interview here, Thomas.
Thomas:
Yeah, yeah! So far so good.
Dan:
And one of the things too, you talked about the hallway track, developers typically, and operators and those things, never get the opportunity to see the end users. That's what's the beauty of KubeCon, like you said, being able to do that, being able to talk to, "Hey, we're doing it this way, or, Hey, we're using [inaudible 00:33:06]." You know what I'm saying? You see customers that are using it, but being able to see someone using our open source projects and using it in their projects and how they're using it, to me is so invigorating.
Thomas:
It is. It is absolutely invigorating, and it's one of those things that I felt like I lost out on when I joined Google, because there's so many levels of indirection, it's such a big company that the people who use your software are not necessarily going to be in the same time zone as you, and you may never meet them. But to have this idea, and Google has actually done similar things of having like tech conventions before, but to have a convention full of people who are so hyper-focused on a particular technology, but are also super inclusive and just willing to welcome anybody, that combination I have not seen in the industry. And I think it's a huge reason why [inaudible 00:34:06] is as successful. They invested so much in their community and continue to do so and that's why I love to be part of it.
Dan:
That's fantastic. So let's talk about minikube. So again, you mentioned, was it Dan, the person that was the original person who did it?
Thomas:
Yeah, Dan Lawrence, yeah.
Dan:
Dan Lawrence, okay. Shout out Dan Lawrence you'll have your picture right here. But again, talk to me about, again, the beginnings of that, and when you took it over in the development of it, because again, it's a tool everybody out there uses.
Thomas:
Yeah. So this was a tool that, it came up as a proposal, fairly early on in Kubernetes' life, probably around 1.0, 1.1 somewhere around there, like fairly early, like, "Oh my God, we have to solve local development." And Dan took this on, and Dan is a bonafide genius. So many great ideas have come out of him, half the tools that our team works on are actually in a large part inspired by him. So there was that inspiration, but as I said, like with many things, geeks love to work on a project hardcore for a couple months and then move on to something else. And I think that's what happened to minikube before I joined it, was that minikube had been, I wouldn't say stale, but it hadn't been very actively developed for nine months. Which in Kubernete's universe is an eternity. It was getting releases every once in a while, but not really a really active power behind it.
Thomas:
So I joined the team and when I looked around at like, "Here are the projects I could work on," I saw minikube, and I was like, "This is my jam." Operating systems, internals, lots of debugging. I think of Kubernetes as crash heap driven development, the way that a cluster starts out is, okay, it's less crazy now than it used to be at like say, v1.10 but you bring up a cluster and then there's this eventual consistency, everybody comes into harmony eventually. Like yes, etcd and API server, they're talking with one other somewhere, and it drives people crazy when they try to debunk problems with start up of a cluster. And I love that kind of thing. I think the thing that I really noticed wasn't there, was at the time, when you started minikube, you had to have a lot of insider knowledge still. At the time, when I first tried it, I was trying it on a Linux box and it was like, "On Linux your options are virtual box or KVM." And I was like, "Okay, I know KVM is native to Linux, let me use that." And what I didn't realize is that I was going to have to install this whole other pile of things. So the documentation didn't capture that at the time.
Thomas:
And as I said, I'm super forgetful. So the first thing I did was I ran through the steps and like, "Okay, that doesn't work, let me edit it. Edit the dox and run through it again later. And eventually like the dox did get to where you can basically cut and paste. That's my ideal, if you can cut and paste an entire doc and never have to understand the words in it, that is perfection for me. So we eventually got there with minikube, there's this huge focus on, make it easy for people who this is their first exposure to Kubernetes. I don't want people to have to think about what Docker is, honestly, or what KVM is, or virtual box. It's hard enough to learn all the new concepts-
Thomas:
That's like, I want to have... It's hard enough to learn all the new concepts of Kubernetes. Why give them this extra contextual tasks? Let's just do it for them. And that's been really the [inaudible 00:38:14] ethos for minikube [inaudible 00:38:16]. Just do it for them. Figure out what we can do to just make it work, do the thing and just let them move on to learning Kubernetes. Because really, I think, when we do our survey data, easily 30% to 40% of people who are running minikube, are doing it because they want to learn Kubernetes.
Thomas:
Because it's integrated and we talk about the inclusiveness of the community. If you look at the documentation and it says, learn Kubernetes, what's the first thing you see, you see minikube, you see deploy minikube and go and then the folks, honestly, that end up using it the first [inaudible 00:38:55] away. And then they go to [Kelsey's 00:38:57] the hard way, they're like do the really hard steps, right? And then I'm sure somebody... There's a Tom in the backward that's just crying. He's like, "Oh, it's so much easier with this," and, and [inaudible 00:39:08] ABM people. But I look at it again because it's exactly that, if I just want to just deploy that engine X pod and do all the tasks that are in the Kubernetes documentation to understand how to do it, it's like this. It's amazing.
Thomas:
I [inaudible 00:39:23]. You mentioned [inaudible 00:39:24], and I want to give a huge shout out to that effort. The development [inaudible 00:39:32] simplified minkube you so much.
Thomas:
You can see the pedigree in there, like the way that you all do the manifests and [inaudible 00:39:40] to be able to do [inaudible 00:39:42] and all the Kube API, it's incredible. And it's also like if somebody wants to learn how all those components works, they can just go into those Docker containers and see that it writes to this director. It's very cool.
Thomas:
Yeah, it is so much nicer than it was. And I think really, that was part of like when I joined the team, part of the technical debt that we had just resolved is we finally committed to Kubeadm is the only way to go. Like when we first started, there was no Kubeadm. So we were basically hammering and nailing this thing by hand. And it was great until it wasn't, like when a new release came out and they would change like, here's the [inaudible 00:40:26] list of work, here's their order. Oh, I've got new flags. You would be sitting there with your hammer and nails again like, "Okay, I'm going to patch this thing." Kubeadm just simplified it. We have a common API to talk to any version of Kubernetes. That's huge. I think I, I just, I think it's one of the best efforts that we've done as a community coming together and identifying that as a huge shortcoming is how do you automate deployments?
Thomas:
Yeah. And again, it's like you can make your master, you can join Kubeadm. And I want to ask you this, and this is a very super geeky question, but you know there's the feature flags enabling, [inaudible 00:41:06] the Kubeadm side influence the minikube side where you can... I'll give you an example. If you want it to enable audit logs in minikube, you'd have to have... There was a flag for that. Did [inaudible 00:41:18] or did ADM do it? Who influenced the other? Or was it a mutual thing?
Thomas:
Honestly, I don't know. I do know... The name is just [inaudible 00:41:28]. So some of the early Kubeadm contributors were also minikube contributors.
Thomas:
Got it.
Thomas:
So it probably went both directions. I feel like [inaudible 00:41:46] might've started with Kubeadm, because the dark period of minikube and I [inaudible 00:41:53], I call it the dark period. There was something called local Kube that we use to make use of as a good strapper.
Thomas:
I remember.
Thomas:
And it was... You can think of it actually sort of spiritually similar to what you see with K3s, like let's try to optimize everything by [inaudible 00:42:16] everything's the same binaries, but it wasn't a very maintainable stack. And it got us to wherever we needed to be eventually, but Kubeadm just overtook it, because Kubeadm was more maintainable. With local Kube, you'd have to wait for them to update to the latest version and fix all the things. I will admit, when I saw K3s, I was super skeptical. I was like, "Oh, here's the next local Kube. This is going to be fun to maintain." Who does [inaudible 00:42:42] for actually maintaining this thing and seeing it through in the long term? I've been blown away by that effort. I think in many ways, they are showing the rest of us the way to make Kubernetes work on low resource machines.
Thomas:
Fantastic. I had Darren Shepherd on and we talked about-
Thomas:
Oh, yeah.
Thomas:
It's phenomenal. But again, it's still the same concept as a minikube or Kubeadm, it's like make this thing accessible for somebody to, like you said, learn Kubernetes or deploy this on an edge device, or I need a simple cluster. I want to test some things and use Kubeadm to do it. But the one thing I think, and going back to the feature gates thing, and I think it's such a great feature with minikube. I'll give an example, I work for Sysdig. We have Falco and Falco is this behavior rules engine, and we can actually interpret Kube audit logs. So we can take minikube, right? And basically have those feature gates and be able to show people like if somebody wants to kick the tires and what Falco does, it's a fairly immediate to do that. If we didn't have that feature gates thing, we wouldn't be able to show people the power of all of these different projects that are out there.
Thomas:
Yeah.
Thomas:
Think about that. See how much you're enabling everybody with this?
Thomas:
Yeah.
Thomas:
Props, dude.
Thomas:
Thanks. So this comes... One of the large efforts for minikube, so when I say 40% of our users are just learning Kubernetes, the other 60% are really just trying to emulate their production environment and do local development. And the emulating the production environment part is really... Well first, really difficult, but it is like our raise on [Detra 00:44:29]. This is why we exist and why things are really complicated. If you look at minikube's support chart of here are the different drivers and different runtimes and everything, we have that incredible amount of options. We have too many quite honestly, but it's because we're trying to emulate every possible production configuration locally. There's not really a line that we'll draw and say, "No. Of course, we won't do that."
Thomas:
I don't think we'd set that yet. Clearly, we do have this idea of do the features that fit, but we've found over time that most Kubernetes features just [inaudible 00:45:12] anyways. You might suffer a little bit of start-up latency when you add certain features, but it's really important that we want to give a high fidelity simulation of your production environment. If your production environment requires a custom Linux kernel like [DACA 00:45:32], for instance, we want to support that. You have a custome CNI, we want to support that too. There should be no Kubernetes feature that is inaccessible [inaudible 00:45:44], because if we have done so, we have failed it in production.
Thomas:
The [inaudible 00:45:50] psyllium is a great example, right? It's an example where if I saw the announcement, I could see I can run it in GKE cluster. Awesome.
Thomas:
Right. Mm-hmm (affirmative).
Thomas:
I just want to kick the tires on it at home to understand what psyllium does, EDPF functionality, right? With Falco or whatever, I can literally just go and again, it's this idea. It's like that accessibility, somebody just needs to just understand just enough for them to deploy the applications and you all have succeeded in that without a doubt.
Thomas:
Thank you. So we've gone to great lengths, but it does mean that minikube does have a very high support for them. [inaudible 00:46:33] a look at our open issues over time, we exceed that of many [inaudible 00:46:39] mainstream Kubernetes efforts. And I think this become this is for two reasons. First, there are so many options. And a lot of the people who are trying minikube, this is their first exposure to Kubernetes anyways. And so they don't yet quite understand why something happening. They're not the professionals who are going to be opening issues on [inaudible 00:46:59] because you can see that [inaudible 00:47:02], and a lot of the people who are turned out to professionals are going to be open just like [inaudible 00:47:07], "Oh my God, this is my first time using any virtualization, they told me to learn Kubernetes, and I don't understand why I get this [inaudible 00:47:13]."
Thomas:
So we have a lot of things like that, but we also have a lot of things like, "I enabled this feature and this particular thing didn't work the way I expected." And we actually for most of the [inaudible 00:47:25], we just pipe it straight to Kubeadm. So we ended up actually supporting... Oftentimes, I'll tell people we actually don't do anything special, talk to Kubeadm, but we try to do it respectfully because we are [inaudible 00:47:37] inclusive community. And same with all the new people, it's driven us like the number, the two things that such a high number of issues open every day against our repository has driven us to do is first, give really clear, easier, actionable error messages. You may notice if you run into an error in a modern version of [inaudible 00:47:58], we will often point you directly to, here's a get out issue, it can be about where other people solve this issue.
Thomas:
We have this whole map of like, here's all layer of possibilities, but we have also done... We basically have written our own triaging tools that have kind of like permute [inaudible 00:48:17] have proliferated around Kubernetes now, something called triage party. Basically, we got tired. So we have this biweekly triage meeting as a project where we invite people from the community to come help us triage incoming issues. And one of the things we noticed is we used to basically walk through them in [inaudible 00:48:34] and we would walk through a hundred issues I guess, every week, we'd walk through a hundred issues and that's really not any fun to do in [inaudible 00:48:44]. No one wants to sit there and listen to a hundred issues. So what we did was we devised this dashboard where we could split the work and where every person can work on an independent thread and triage their own issues.
Thomas:
So if we had six people in the room, we got out of that meeting six times faster. When a question was raised, we'd talk about it as a team. And it was really good because people had a chance to drive the discussions that weren't driving the discussions before. And so I think that has really given us a great deal of better responsiveness to all these user issues. So I think now, our average is more three days instead of 40 days. So-
Thomas:
That's very cool.
Thomas:
Yeah. And so now it's also being used by other parts of Kubernetes, and I'm really glad to see that effort going forward.
Thomas:
That's very cool. And one of the things, again, I think in the latest couple of releases, the speed of deployment, I mean, I did a tweet about it. It's fantastic. I did it compared to an older version and it's amazing. I mean, you shaved off a ton of time. Talk to me about that effort.
Thomas:
So this is definitely a team effort. I definitely want to give a shout out to [inaudible 00:49:54] who contributed a lot of the performance improvements. We basically got together in right after QCon San Diego last year and said, okay, this is great. User friendliness is there, but oh my God, this is [inaudible 00:50:07]. Who is actually waiting at a command prompt for a minute? That's not fun. So, we got it down to... On my machine, it's 24 seconds to get the command run. But actually, that's not the time I optimize for.
Thomas:
The time that we're really optimizing for is zero to first deployment. Because that's what developers care about. They don't care about when minikube [inaudible 00:50:30], they care about when when their app is up. When can they talk to their app? So that's really the thing that we're focused on iterating on and now, we're faster than any of the other local Kubernetes for that benchmark, roughly like 50 seconds, but that's still too slow. I want it to be 10 seconds and getting there is going to be really tough, but I think that's really going to be one of the big things for next year.
Thomas:
Let me ask you this, I'm going to ask this, you want to get onto the site because look, I don't ask you to have the answer now, but what do you look at for those things? I mean, is it the speed of deploying the components? Is it the hardware components? I mean, I want to get deep on that because I think it's like I want to understand that train of thought.
Thomas:
So my train of thought is if you look at... We have for all your ADHD listeners out there, you have at most 10 seconds that somebody is going to be staring at a command prompt before they move on to an edit. And I want to make sure our tool gets their app there in that timeframe that their attentiveness. How do we get there? This is a great question. I have no idea. Well, actually I say I have no idea, but I have an idea and I want to sell it to everybody out there. Like all the other efforts, that I think like ultimately were how you get to this level optimization is by doing less. And my idea for doing less is use a basically hibernate the memory state of the Kubernetes cluster to disk, put that in a Docker container or an ISO.
Thomas:
And basically, it's like freeze, dried food, like bam, you've got a Kubernetes cluster. That's what I want. And so there are some really cool technologies out there like [inaudible 00:52:29], for instance, that let you... This is something Google uses quite a bit actually where we will freeze the process and move it to another node entirely. And that's how we deploy a lot of these cloud services in a way that we have zero downtime because we can move it between hardware.
Thomas:
I want the same thing for Kubernetes and for local Kubernetes. That's what I want to see by... That's my vision for the end of 2021 is where this is a normal accepted thing. Clearly, if you're defining your own flags, like I have feature gates I want to enable, you may not fit in that super fast flow because we might have to restart containers. But for the... I have no stats, but I would wager that 90% of minikube users are just using minikube start and I want to optimize for that case of just instant Kubernetes. Here's your app. That's where I'm going.
Thomas:
That's awesome. I'm going to ask you something and I'm sure you've been asked this before because there's folks that use [inaudible 00:53:33], there's people use micro [inaudible 00:53:34], and there's people that use minikube. [inaudible 00:53:39] a project within Google as well. Is it a similar idea? I want understand where you see either the difference or whatever, why couldn't those efforts be united? I don't know. Or maybe they're just two different tangental things.
Thomas:
Yeah, that's a great question. And certainly something that we've had discussions with [inaudible 00:53:59] team before on this, and I think ultimately it came down to the reason why we are not the same project is that our focus is different. [inaudible 00:54:09] is focused on integration testing Kubernetes components at here's a known good configuration. I want to keep testing [inaudible 00:54:19] basically. I want to use the latest container D and many key has been focused on like, I want to emulate production. So I want to Kubernetes release at six years old. I want to use this like I want to use the Docker run time. It's really optimized. Like mini cube is optimized for application developers and kind is optimized for Kubernetes developers that said, we definitely share a lot of common ideas and architecture.
Thomas:
So the mini cube Docker driver for instance, was heavily inspired by the kind effort. And we definitely had some discussions with them on how does it work? How can we make it better? So we actually do use their base image right now, but we slap her on Kubernetes on there because our configurations are substantially different as is, are like support for different run times. So like one of the things that we many to exposes because it's focused at application developers is we expose like a Docker D endpoint. So you can build your containers directly against many things and just have that like very quick iterative loop, because that's what we care about is like very fast iteration times both across scaffold and, and kind of doesn't do that. They, they do have some peripheral registry that you can push through.
Thomas:
It's a little slower, but it's actually more proper in many ways because your production cluster isn't necessarily going to have like Docker points accessible to users. So there is some different focuses here. Ultimately, I think what I really love is that I'm so glad that there is this competition here because every effort has done things that has inspired the other efforts like micro Kate's. Their documentation is first-class and has inspired us in many key, for instance, to like up our game on the getting started documentation. Like if you look at getting started micro cakes and getting started in NICU, you'll see that we took a lot of inspiration from it mainly wholesale
Thomas:
Let's let's think the analogy, if you think about like metal or you think about like the Beatles and the beach boys, let's go and talk metal, napalm death, they kind of inspired like all like different like iced or inspired, like it made things better. And then you crash mentally, everything kind of iterates. And so it made everybody better. It makes all of our products better. Right.
Thomas:
And this is totally their thing. Like local Kubernetes, like even like Docker, desktop, like the fact that they have the Kubernetes Damon sitting right there. Right. Super. User-friendly not very flexible. It's super user-friendly. And I would love that idea of like, here's a simple UI, but as, as an open source tool, right.
Thomas:
... simple UI, but as an open source tool. This is something we've talked about before. If anybody out there wants to work on UI stuff or Kubernetes, holler at me.
Dan:
The link in description. With the link in description right here.
Thomas:
Yeah. But yeah. Likewise, we have all rubbed off on one another, and we're all driving forward to the same goals. And it's really like we've got independent evolutionary paths to get there, but we learned from one another. And I love the fact that there is this competition out there, because it's making us all better, quite frankly. Like K through D, I love their startup times. And I think a lot of the inspiration for us, realizing that, "Hey, we can make Minikube much, much faster, "I think came from them, honestly. We saw how they did it, and we're like, "How can we do that, but turn it to 11?" So I love all those guys. I think we're all making a beautiful community together. Sometimes strife can happen, I can imagine. But yeah, we're all aiming for the same goal.
Dan:
All for common good.
Thomas:
Yeah.
Dan:
And with the common good, let's move on to something that's more near and dear to your heart. So you are an ... I want to understand this, the amateur bike builder thing. And I know you built one out of bamboo, and we'll talk about this, but where did this start? Is this from Sweden? Tell me where this came from. Like, "Hey, I'm at home. I'm bored. I don't want to code anymore." Talk to me about that.
Thomas:
Yeah. I think everybody should get some time away from the keyboard every once in a while. And so my time away from the keyboard has usually been cycling. And I love cycling around like the hills in [inaudible 00:58:56]. I'm definitely like a cycle to the next bakery kind of person, like a bakery coffee. Like let's do this. I'm not like the competitive cyclist at all. But I joined this effort, there's this big bike ride in California, AIDS/LifeCycle, where people, to raise money for HIV/AIDS awareness and treatments, 2000 people will ride from San Francisco to Los Angeles, which is 575 miles give or take. And so we'd all do this. It's basically like a rolling party for a week. And I got involved in that, met many of my best friends there.
Thomas:
But then one day I was cycling to work and I broke my foot. I somehow managed to run over myself. Still a little confused on how it happened. But I was having one of those great mornings where I'm just like pushing the pedals, having so much fun. And then I ran over myself. And I break my foot and I couldn't cycle anymore for like two months. And so what am I going to do as my off time, away computers? And I remember seeing this article, I don't remember where from, but somebody had posted about like, "Build a bike out of bamboo." I'm like, "Yeah, that sounds kind of crazy. It sounds like it would take a lot of time." And then I broke my foot, and I'm like, "Yeah, time is what I have now. So I'm not doing anything better." So I tried doing it. I bought a kit from Calfee, and I sat in the garage.
Thomas:
And I learned very quickly it's actually much harder to build stuff with a broken foot than I anticipated, because I was trying to use an office chair and rolling around and getting things. And that was a disaster. So I built the bike, and it lasted about six months before it fell apart. I learned a lot about like joinery, and how do you join two pieces of wood in a really stable fashion? Ultimately, bamboo, if you ever get to Asia you'll see massive like 10 story scaffoldings build out of bamboo. It's a really strong material. But it's really the quality of the joins that dictates whether or not your bike will fall apart. And as a complete [inaudible 00:01:01:06] when it came to anything carpentry oriented, or like how to epoxy, like I did all the worst things. I mean, I actually ended up building it out of medical bandaging tape to join the pieces of wood, which works if you do it right.
Thomas:
I've since moved on to other things. My favorite way to join now is ... So I've just built like three. My favorite joining technique now is a mix of hemp and flax, and a ton of epoxy. And it gives you a really great compliant feel, because bamboo has like a natural bit of shock absorption to it. So it feels very different than, quite honestly, any other element. It has some similarities. I think of it as like carbon and steel mixed into a frame, is kind of how it feels. But it still, when you hit a big bump, it still feels a little bit more relaxed than either. So I love it as a material. It's completely unpredictable, because there is no two pieces that are ever the same diameter, or even the same curvature. So I think of it as distributed systems, where you're given basically like, "Here's a pile of components that don't fit together, make it work anyways."
Thomas:
And instead of [inaudible 01:02:27] I've got just a hell of a lot of epoxy. And it all works until, if you ask my wife about the time that I stuck two gallons of epoxy in the oven, it all works until then. And then it's a big mess.
Dan:
So we just put together a bike building all the way to distributed systems. Try doing that, The Cube. Just kidding. Anyway, shout out to The Cube people. Alrighty. So let me ask you the last question. What work, Tom, are you most proud of?
Thomas:
What work am I most proud of? I think I am, in many ways, I would say it's probably a lot of the automation work that I did at Google for like nailing down like, "Here's how you deploy a cluster in a maintainable fashion," and trying to distribute out the responsibility across multiple teams. That was a huge game changer for me mentality wise, in that it's this idea that you cannot do it all yourself, that you have to delegate to people you have never talked to before. You have to trust them to do the right thing. And kind of building that mental model of shared responsibility was just a huge game changer for me. And something that I get to do in an open source world now in Minikube. Minikube is so complicated, there's not a single person that knows the whole code base. And so I do have to rely a lot on people.
Thomas:
I have no idea how [inaudible 01:04:08] works, to be honest. But I know there are people that I can delegate that to, write great integration tests, and they got this. So yeah, I'd say that. But I'm also very proud of user friendliness of Minikube at this point. I think it really does a great job in attracting people who don't yet know why Kubernetes is a great thing. Just giving them a taste of that. Like, "Here's a cluster. We helped you out. We gave you the helping hand to get this far, now do ... [inaudible 01:04:41]." I love that.
Dan:
And you've done such an amazing job, you and the team. And again, thank you so much for being on the pod. It means so much to me, because you're one of the people that like ... I picked this Kubernetes world up, and I use Minikube. It's probably like the first couple of weeks I was at the job. And I appreciate you, man. I want you to know that.
Thomas:
Thank you.
Thomas:
(music).
Dan:
Thomas, a lot has happened to you since we talked last. First off, some big things with Minikube. Talking some big things with you. So bring me up to speed [crosstalk 01:05:37] Minikube.
Thomas:
There's-
Dan:
There's so much.
Thomas:
I think honestly, it's less Minikube and more what's gone on in the industry since we've last talked. So we talked in September, and there's been some big, big movements in the cloud native space, particularly around ARM64. And this is something that a lot of us saw coming, but really it really started picking up in November in a hardcore way. And I blame Apple for this. Like Apple made some announcements, "Yeah, we're going to ship ARM64 hardware." And we all said, "Yeah, we'll believe it when we see it," and then they did it. And the transition, I don't know what kind of vocabulary I'm allowed to use on the air, but let's just say the transition has been difficult, I think. Apple-
Dan:
You can use shit. That's all right. [crosstalk 01:06:34]. Okay.
Thomas:
Transition's been a clusterfuck. That's what it's been. And Apple went up on stage, and they showed off like, "Docker's going to run on the ARM64 hardware." And then, when the machines got released in November, Docker had not actually had a chance to test on them. And so Docker only this last week, five months after the machines have gone for sale, Docker finally has like a stable release of Docker for ARM64 on Mac.
Dan:
Today was the number one, somebody posted, the number one article on hacker news, was basically porting just trying to figure out getting the development environment set up on it. Yeah. Crazy.
Thomas:
Yeah. It was a wreck. We tried to get developer hardware from Apple. Knowing that this was coming, we reached out to Apple, and we're like, "Hey, we want to make sure that things like Minikube," and I'm sure Docker did this on their end, "that they work on day one." And they did give us hardware, but the hardware had virtualization disabled. So all of us were like, "What are we supposed to do with this? You can't run Docker, and you can't run a VM. We can't do anything." So our machines basically sat idle for a long time. And when they finally went on sale, we moved heaven and Earth to make Minikube ready for the early adopters who got the Mac hardware in December and January. We didn't get it up and running until January. And similar projects that we work on on the container tools team at Google, we really pushed. Even before the go compiler had stable support for ARM64, we were pushing these releases out, trying to get everybody up and running and unblock everyone.
Thomas:
So Skaffold's got it. Minikube's got it. Handful of our other tools have it. But it's been a real ugly transition. And it's going to continue to be a transition this next year, because one of the things that hasn't been solved is if you're doing local development on one platform, and you're iterating, you're using a tool like Skaffold or Tilt to do iterative development of your Kubernetes app, and you happened to be on an ARM64 Mac. And then you're like, "Okay, click the button to deploy to prod, in your prod environment. You don't actually know what architecture it is. As an application developer, you don't really know this. And you shouldn't have to know, you should just click the button promote to production, and it'll do the right thing. It'll deploy to the right architectures. But right now, very few of the great automation tools understand architectures. Everybody assumes that you deploy to the same architecture that you develop to locally.
Thomas:
And that's going to be a place for all of the developer tools to grow in 2021. I think it's going to be a huge transition year for ARM64 on the server side. Some cloud providers are faster than others in getting ARM64 up and running. So kudos to AWS and Equinix for having that hardware available today for use. But I think a lot of the industry has to catch up, and not just in the cloud offerings, but really just the software support to make ARM64 awesome. I mean, from a price performance perspective, it's an amazing value proposition for everybody. So I think by the end of 2022 it's going to be a default thing that we deploy to, that everybody deploys to. I'm really looking forward to that. And then I'm looking forward to the whole RISC-V transition after that, but that can be a different popcast.
Dan:
Without a doubt. So listen. Everybody tuning in, you waited a couple of days. We had this announcement on Monday. And you ready to have the big reveal, the world premier reveal of where you're going, Thomas?
Thomas:
Yeah. So I made an announcement this last week that I left Google. And I've been there for just over 15 years. Amazing experience. The fact that they paid me to go travel to 10 different countries, some of which paying me to travel by motorcycle to go between a lot of these different places, was kind of wild. But 15 years, it's enough to get to the point where it winnows away at your ability to have perspective. So I got out, which it's sad. I'm going to miss my team. But yeah, I'm heading to Equinix to work on the team that ... I guess to lead the team that works on the Tinkerbell project there, which is a CNCF project, open source provisioning engine, highly recommend you all go over to a tinkerbell.org to check it out, if installing operating systems at scale is your kind of thing. Absolutely check it out and join in.
Dan:
Awesome.
Thomas:
Yeah.
Dan:
And again, there's a lot of great people that we obviously both know, is Jason [inaudible 01:12:01] is over there. [inaudible 01:12:02] over there.
Thomas:
Yeah.
Dan:
The crew Equinix is great. So again, congratulations to you from the [POP cast 01:12:09]. Thanks for coming back and doing this reveal. And dude, you know I'll be following you as I always do.
Thomas:
Yeah, absolutely. Yeah. I'm super, super pumped about this. And the fact that there is so much overlap from the Kubernetes community, not just in some of the personalities, but really that welcoming vibe that was going on. Like I came to Tinkerbell to solve my own problems of like, "Hey, I want to run this open source lab at home, a lab for open source developers to basically tinker around on these random architectures like ARM64 and RISC-V." And looking around for solutions and I saw Tinkerbell. I'm like, "Okay, this sounds interesting. It's CNCF. Let me check out the community office hours meeting," because I use that as like a judge of like how inclusive is this crowd? And how fun is this crowd? And I was blown away. It gave me the warm, fuzzy feels that I got for many of the Kubernetes meetings that I went to early on. So I'm super excited about this opportunity. I think we're [crosstalk 01:13:17].
Dan:
Super excited for you, man. An Equinix [crosstalk 01:13:19].
Thomas:
Thank you. Yeah.
Dan:
[inaudible 01:13:20] sponsor the POP cast at some point, all right? All right. Thanks man. Thanks for coming on.
Thomas:
Yeah, no problem.