In this episode of the POPCAST, we talk to Jimmy Zelinske about his past from Bittorrent to CoreOS to Operators and the Operator Framework. Jimmy gives great insight on Centralized Authentication with Authzed... and why you need it!
Jimmy's BIO:
Jimmy has been both a product manager and critical engineer on various container and orchestration projects.
His passion for building products sees light even in his spare time as he contributes to open source projects and standards.
He is a co-founder of Authzed, Authzed is an API service for developers to build permissions into their applications without the usual pitfalls and distractions.
Timeline/Topic
00:00 - Sponsor - Stormforge - stormforge.io/popcast
01:00 - Opener
01:08 - Intro to Jimmy Zelinske
01:24 - Jimmy's Journey
04:10 - Bittorrent in his beginnings.
11:36 - The CoreOS Story.
15:10 - Quay - What is it and its origins?
16:28 - Joey and Jake and how Jimmy is the "tie breaker"
21:08 - CoreOS and Product Management
23:30 - Docker and Kubernetes as the early choices for Kubernetes/CoreOS
29:56 - Operators and the Operator Framework
37:00 - Authzed. Centralized Authentication, Jimmy explains why you need it.
49:46 - What work is Jimmy most proud of?
Episode Links
https://jzelinskie.com/posts/youre-not-running-vanilla-kubernetes/
https://authzed.com - Find out more about Authzed
https://play.authzed.com/ns/example/document (Play with Authzed)
Today’s episode sponsor is Stormforge:
StormForge’s Kubernetes Performance Testing and Optimization platform is the easiest way to ensure your applications behave the way you want them to, while cutting out unnecessary resources and time spent manually tuning.
In alignment with this, StormForge is asking for your help reducing the amount of unused Cloud resources making both a financial and environmental impact in the world today. We want you to help make an impact with us, so visit https://stormforge.io/popcast to learn more about how you can help ERASE Cloud Waste, and take the Cloud Waste Pledge.
While you’re there, why not try out the FREE TIER of the Machine-Learning-backed service to start saving resources and getting better performance today.
POPCAST SHOW DETAILS
Watch (YouTube): https://www.youtube.com/c/thepopcastpop
Listen (Apple PODCAST and others): https://popcast-d9f7b6dc.simplecast.com
Follow us on (Twitter): https://twitter.com/PopcastPop
Follow us on (Linkedin): https://www.linkedin.com/company/the-popcast-with-danpop
POP (00:00):
This month's podcast sponsor is StormForge. StormForge's Kubernetes performance testing and optimization platform is the easiest way to ensure your applications behave in the way you want them to while cutting out unnecessary resources, time spent in retuning. In alignment with this, StormForge is asking for your help reducing the amount of unused cloud resources making both a financial and environmental impact in the world today.
POP (00:27):
We want you to help make that impact with us. So visit https://stormforge.io/popcast, to learn more about how you can help erase cloud waste and take the cloud waste pledge. While you're there, try out the free tier of the machine learning backed service to start saving resources and getting better performance today.
POP (01:08):
Hello, everyone, and welcome to the popcast. Today, I got Jimmy Zelinskie. He is one of the founders of Petricorp.io. Fantastic dude, I mean, everybody I talked to from the CoreOS side is always like, "Jimmy. You got to talk to Jimmy." So, Jimmy, welcome to the popcast.
Jimmy Zelinskie (01:23):
Hey, thanks for having me.
POP (01:25):
So, you grew up in Philadelphia, the suburbs of Philadelphia. Talk to me about your humble beginnings, talk about first how you started computing, all that fun stuff.
Jimmy Zelinskie (01:36):
Yeah, sure. I guess it's not super humble. Basically, I'm just a complete computer nerd. I grew up just always spending time in the computer. I played lots of video games in the computer. Eventually, started hosting-
POP (01:51):
What games did you play in the computer? Let's talk about that.
Jimmy Zelinskie (01:53):
Like MMOs, literally going back to the earliest days that starts with Habbo Hotel and shit like that. Really old, precursors to MMOs and I think console games mostly. But what actually happened to me was I started hosting Counterstrike servers and things like that in game forums. That kind of showed me the world of IRC and Linux.
Jimmy Zelinskie (02:18):
Basically, from 13 through maybe 19, I guess it's my first job technically, hosted services just like as sys admin. It's actually kind of crazy because people playing games would be vicious. They try to hack each other's forums. They would like distributed denial-of-service each other. Literally, at the same time, there's news going on about Anonymous and all these people like DDoSing literally in the Middle East as vigilante justice.
Jimmy Zelinskie (02:51):
Crazy shit was happening where these 14-year-old kids are DDoSing with way more bandwidth on my tiny little Linux servers on Linode. I'm like, what the actual hell is happening in the world? There's giant press on these crazy huge attacks. Meanwhile, my tiny little internet forums are getting completely destroyed by 10 times as much.
Jimmy Zelinskie (03:16):
Honestly, that experience drove me to be super clever. You learn a lot of internal details like crazy lower level details of networking and Linux and really forced me to kind of get out of my comfort zone pretty early. I basically used a Pentium 4 until 2012. So I had a super fucking slow computer. I couldn't even run the latest Windows. So then I run like Arch Linux, and now, I use a Mac.
Jimmy Zelinskie (03:45):
But yeah, really weird beginnings just like forcing myself to always be clever. It wasn't until college when I started programming though. So it's purely sys admin work.
POP (03:55):
But again, I think being that DIY kind of gaming, I have a similar kind of with what I do. I started doing a lot of Usenet stuff and all that fun stuff, and BitTorrent and stuff. So you actually wrote a standards doc for BitTorrent. That's in a tracker that's used for orchestration. So talk to me about that.
Jimmy Zelinskie (04:17):
Sure. Basically what happened was, like I said, I never started programming until college. There's this kind of meme that goes around when you're younger that if you learned programming on your own, you'll pick up bad habits and then you'll be forced to unlearn them. That is entirely false, yet that like mind virus basically stopped me from learning how to program almost entirely up until I got to college and I started taking courses.
Jimmy Zelinskie (04:44):
I took courses that were C++ and then Java eventually. I remember just getting to the course where we learned about threading. And I was like, there's no fucking way that this is how people actually program like parallelism. The future is going to be tons of CorseOS on computers. That's the only way they're going to get faster. How the hell is the shitty thread pool API like the thing that people are actually going to use?
Jimmy Zelinskie (05:10):
I was super upset with parallelism and Java and that was at the point I started looking at other languages. I looked at Erlang and then eventually Go. And so, this is kind of where it all started in the Go ecosystem for me. It's around 2010. Around 2010, I started playing with Go. One of the projects I went to work on was BitTorrent tracker, which is the thing that lets you find other peers in BitTorrent. And then once you connect to them directly, that's when you actually share the content.
Jimmy Zelinskie (05:42):
So it's kind of like a router for BitTorrent. And so, I just decided BitTorrent was cool. I was a kid in college, so I was trying to get everything for free. So you're kind of scrappier then. BitTorrent software seemed like really cool, and I just started working on that. Basically, what happened was I noticed some parts of the protocol that were widely adopted were not actually written anywhere. So it's kind of one of those things where somebody wrote the original spec, and then a couple of reference implementations kind of try to make everybody compatible with each other but then nobody ever came full circle and wrote that back into the spec.
Jimmy Zelinskie (06:19):
I started standards draft for a thing, part of the UDP protocol just to get that actually written and codified even though literally everyone already implemented it. So that's really cool to be in the credits for something so popular. But yeah, peer-to-peer and distributed systems were super interesting thing for me at that time.
Jimmy Zelinskie (06:42):
What eventually happened was one day maybe it's like junior year, I ran an IRC channel for a project. A bunch of other people contributed to it. Basically, someone, not necessarily from Facebook hits me up and is like, "Hey, we want to use this." Or actually, I'm pretty sure they told me, "Hey, we're already using this."
Jimmy Zelinskie (07:04):
Then I had an interesting discussion. Basically, there's an orchestration system at Facebook called Tupperware, and Tupperware is a thing that actually schedules software. It's basically their Kubernetes, predates Kubernetes. It's actually pretty janky. It's effectively a weird mixture between container orchestration and configuration management because it's actually just a bunch of scripts that move RPMs under different machines.
Jimmy Zelinskie (07:33):
It's kind of a weird cross between Chef and Kubernetes. But one of the interesting things is that it distributes those RPMs via BitTorrent. It was super cool.
POP (07:46):
So let's talk about that. That's so much efficiency to that. That's pretty ingenious. Taking again that scrappy ... The thing about you as an admin, it's like what can I do to keep my channel up and you get DDoSed. I mean, I did something simple like the IP sets stuff, you're able to block. I was getting hacked constantly from all types of Russian and Chinese IP. So I just block the subnets.
POP (08:06):
You got to get scrappy if you're doing this because you're not going to go buy an IDS/IPS system. And then that's funny that Facebook was just like, "Hey, I'm seeing this guy doing a ton of stuff writing a standards doc for open source," and then there's a tracker called Chihaya. And so, being able to use that, that's again being able to contribute to these enterprises of companies with this open source like scrappy roots. That seems like your story, Jimmy. You know what I mean? They're having that piece there.
Jimmy Zelinskie (08:35):
I really think it's a scenario of like if you build it, it will come. If there's a reason for you to build it, you look around and no one is doing something and you need to solve the problem, solve it. People will also need it as well. If you see the problem, you solve it as well.
Jimmy Zelinskie (08:53):
The interesting thing with Chihaya actually is that you get these crazy efficiencies out of using BitTorrent and you see Alibaba actually doing quite similar like PouchContainers nowadays like many years later, or I think it's Dragonfly. Yeah, Dragonfly is the actual framework but PouchContainers are the thing they distribute.
Jimmy Zelinskie (09:14):
But actually, the thing I'm super impressed with, with Facebook is generally their network. They could do lots of optimizations so that they could assume basically ... They can look at an IP address of an individual machine and because they've actually planned out their whole network, they can actually look at the IP address, look at what subnet it's in and say, "This isn't the same rack with me." And then they don't send any data over the backbone across data centers they can totally optimize.
Jimmy Zelinskie (09:40):
So a lot of the stuff you could do in Chihaya that you couldn't do with any other BitTorrent software was actually to encapsulate that knowledge. Chihaya would actually prefer traffic that was local to them so they could keep most of their traffic in rack. And honestly, we still are nowhere near that level. It was kind of jarring for me when I started working at CoreOS because you start working with enterprise customers and you realize their networks are cluster fuck of just random things glued together.
Jimmy Zelinskie (10:06):
You acquire somebody. Now, you have overlapping subnets. No one has a cohesive network, and yet Facebook has a completely planned cohesive network. Everything is IPv6, and there's only a couple of things that do IPv4 and they're dual stacks. So they have both stacks. It's just the hyperscaler is working on another completely new level. That's not your average mom-and-pop show or even like-
POP (10:32):
Because initially from that start, they architect them in such a way that they were built in that way. And so, yeah, it's [inaudible 00:10:41]. And then when you hear, "Oh, yeah, we're just going to dump a Service Match in here and all of my networks are going to be all right. Well, it's not going to work. And no offense to you Service Match people. I love you ... It's just the reality is that underlying kind of foundation, if that's somewhat flawed no matter what happens, you're not going to have the Google level scale. You're not going to have the Facebook level scale, whatever it might be.
POP (11:04):
So let's shift gears a little bit, before we get into CoreOS, I want to talk because I think this is a natural segue. I mean, you also worked on OCI. I think was that in pre or kind of in the later stages of CoreOS? I'm just curious about that.
Jimmy Zelinskie (11:19):
It started in the later stages of CoreOS. It wasn't necessarily at the beginning. But yeah, standards body is kind of like a thing for me, I guess at this point having done BitTorrent and now OCI.
POP (11:33):
Got you. So let's talk again. CoreOS, it's very near and dear to my heart. I have a couple of friends, friends that are from there and I've met you through a couple of my friends that work there and stuff like that, Jimmy. Talk to me about ... It's a special place. Everybody I talked to and I want you to probably relay me a redbeard story, but let's go ahead and tell me like your, "Hey, I just interviewed with Brandon and Alex," and talk to me about that.
Jimmy Zelinskie (12:01):
Yeah. So it's actually like a pretty interesting story because I'm in New York. I've always worked in New York. And CoreOS has been in San Francisco so how the hell did I get a job? That's the initial question. Basically, what happened was I graduated school and moved to New York. I know I wanted to be in New York. I have a family here. My sister moved here after college.
Jimmy Zelinskie (12:25):
I basically lived in Brooklyn the whole time. And I was more I didn't want to move in the West Coast because I wanted to be close with my family. They're in Philly. I can always hop on the train and go home. And so, kind of West Coast was off limits in my head. I'm not going to San Francisco. I really liked New York and I was convinced I could find a tech company out here.
Jimmy Zelinskie (12:46):
What happened was CoreOS basically acquired Quay, which Jake and Joey, my co-founders from my new company because I've been working with them this whole time. And that kind of opened up a New York office and I just saw a Hacker News post that was like, "Hey, we're hiring in New York." And I knew CoreOS because I had worked on Go really early and Doozer was this project that was the testament to Go at that time.
Jimmy Zelinskie (13:13):
It was the first Paxos implementation in Go, which is basically it's the spiritual predecessor to Etcd. Basically, what happened was people at Heroku did this and everyone kind of rang the bell to be like, "Go is a real program language. You can implement Paxos." And then Blake Mizerany actually started working on Etcd at CoreOS with Xiang. And basically, Etcd became the successor and used Draft instead of Paxos.
Jimmy Zelinskie (13:41):
So, I knew Etcd and I knew CoreOS through that, and was like, "Holy shit, I worked on peer-to-peer systems, distribute systems. It's all in Go. CoreOS is the fit for me." I was actually interviewing at Google at that time, and just bailed halfway through the Google interview. I also interviewed at Datadog at the same time, and it was kind of this choice between them and I ended up bailing and then going to CoreOS because I thought that was more fun.
Jimmy Zelinskie (14:09):
But yeah, my interviews were in person with Polvi. So, Polvi had just happened to be in New York at that time. And honestly, the way I got the job was just a cold email to Polvi. I saw the post on Hacker News, shot Polvi an email and was like, "Hey, I worked on peer-to-peer like BitTorrent stuff in Go. It'd be really cool like Facebook uses BitTorrent to move around their containers. Why don't you?"
Jimmy Zelinskie (14:32):
And that was kind of the pitch, and that kind of sold them. The interview was really interesting. They made me describe a DHT like how DHTs work, distributed hash tables. And now, after the fact, I thought Polvi was really grilling me. But then after the fact that I talked to him later and he's like, "Yeah, I have no idea how they worked genuinely. I just wanted to learn from you how they worked."
Jimmy Zelinskie (14:56):
So, a super cool experience. And I immediately came out at the gates working on Quay. And at that time, Quay, I don't even think Docker had their private registry out yet at that point.
POP (15:08):
So will you take a step back? So let's explain to folks, again, there's a lot of folks who know what Quay is but for uninitiated, what is Quay?
Jimmy Zelinskie (15:16):
So, Quay is a Docker registry. And what a Docker registry does is story or images in a centralized place. Think of it like GitHub but for Docker images, container images. So at that time, Docker had their own, but it was basically just public images that were there. It didn't really have anything private. So the thing that Quay did initially was actually, Quay private images, private repositories and Jake and Joey actually had initially built it as a periphery service.
Jimmy Zelinskie (15:52):
They were building a cloud IDE and they just needed to have private containers. They were using containers to actually secure and let people execute code on their servers as part of their IDE. And they needed something private. So they assumed that other people would pay for it, and they did. Basically, they ended up dropping their IDE product just to go run in with Quay because so many more people cared about Quay.
Jimmy Zelinskie (16:17):
But yeah, I started working on that with them. I'm trying to remember my train of thought. Quay was the major project and eventually-
POP (16:28):
Can we talk a little bit about just Joey and-
Jimmy Zelinskie (16:30):
And Jake?
POP (16:31):
Yeah, and Jake. I want to talk about that, because I've met them and they're just a joy I've spent some time with like in San Diego a little bit. Man, that guy, all of you are just really dudes. You think of things so out of the box and it just seems like you naturally gravitated towards each other just in terms of like do you enjoy ... What's that cohesion you have where one ends and the other one picks up? Because that's, I think, the best part of partnership is like naturally fitting their talents together. What would you say like how do you work well with them?
Jimmy Zelinskie (17:10):
I think it's actually a really great experience having the three of us. Basically, what happened was Jake and Joey met at Google working on the APIs team there. So Joey is actually the creator of source maps, the thing that maps like JavaScript. Basically, they compiled JavaScripts to the language you're compiling to. Say if you're using like TypeScript or CoffeeScript the way your browser can actually tell you where you messed up in your code in the source language and not in the compiled JavaScripts. Joey invented that while he was there. He worked on protobuf while he was there as well with Quentin.
POP (17:42):
Man, I owe my career to him. Sysdig uses protobufs, right? And so, Joey is ... Like I said, I talked to him for a good hour in San Diego, and that dude is just a fucking brilliant dude. And I'm sure Jake as well, but I'm sorry to cut you off, but just Joey, man. That guy. Joey, props to you, brother.
Jimmy Zelinskie (18:04):
Yeah, I find that Jake and Joey kind of ... They're like opposites in certain ways and they kind of butt heads in a really healthy way. And then I kind of fit in as a weird tiebreaker. So sometimes I leaned in towards the Joey arguments, sometimes I leaned in towards the Jake argument. And that was kind of how we worked for years.
Jimmy Zelinskie (18:26):
When I joined Quay, ran on a single EC2 machine. And as that scaled up and up and up, it was just a three of us building out all the products, building out the service entirely. And whenever there is technical decision, you listen to both sides and then you kind of wave the flag as a tiebreaker and be like, "All right, this side wins."
Jimmy Zelinskie (18:47):
And so, honestly, I think that is such an effective way to work. People talked about, what was it? Like the pizza team or something like that at Amazon where it's like if you can't feed ... A team should be no larger than the amount of people that you can feed with one pizza. I think really small teams like that can actually get a lot of cohesion. And if they jive together socially, it can be extremely, extremely healthy.
Jimmy Zelinskie (19:16):
Basically, the way that we've worked together as a team is just like proven so well like I've seen a bunch of teams now. It's been a couple of years. And honestly, that's my first real major team out of school that I've spent a long time with. And now, I've looked around like, "Oh, damn. I got so lucky," so extremely lucky getting put in that position because there's tons of unhealthy teams. There's giant teams where nobody gets any personal attention or growth.
Jimmy Zelinskie (19:46):
There are things that I was able to teach Jake and Joey even though they had been in the industry for a decade before me. There's tons of things that they taught me. I kind of just sponged it all up.
POP (19:57):
It's almost like you're the quorum. And I'm not to use that [crosstalk 00:20:00]. It's like, "Look, this is the decision. This is the decision." You're the quorum. It's like you have this great decision three of three. If you have two of those guys and then have you, it's like where's the tiebreaker, right?
Jimmy Zelinskie (20:13):
Yeah.
POP (20:14):
Wow. It's such a great thing. You all have done so much stuff. Again, Quay, Clair, apps registry. Operator Framework, I want to talk about the Operator Framework a little bit at CoreOS. I am such an unabashed CoreOS fan. And I told like I had Asesh on, it was one of the [crosstalk 00:20:33]. And I was like, "Man, you guys tapped in shit when you bought CoreOS. This is fantastic." All the things that OpenShift doesn't have, you're able to fill. And I had Szumski's on too.
POP (20:48):
And when I Szumski met, god, it's just like that fucking raw talent, all of those people. And again, just like redbeard said. Redbeard said all of these folks at CoreOS have had to extend their careers at different places because you all set this foundation. You guys are the like the fucking Rolling Stones or The Beatles. You set this foundation for Kubernetes.
POP (21:08):
Google gets all ... Respect to Google, respect to Red Hat, but CoreOS, man. You guys were the ones that are writing things that everybody else emulated and everybody else picked up. So I want to ... Props to CoreOS. There's a question here.
Jimmy Zelinskie (21:21):
I totally agree with all of that. I worked with Rob. Between Rob, myself, Christian Vogel and then Reza Shafii, we were the only product people at CoreOS. So eventually, I transitioned to be a product manager. And then when we got acquired at Red Hat, basically, Ashesh was my boss. So I've worked directly with the VP you're talking about.
Jimmy Zelinskie (21:45):
I think the really interesting thing about CoreOS was that we all had a vision. We knew what the future was. Kubernetes and everything we built with and Container Linux, for example, like everything, it was a means to an end. Everyone was subscribed to the vision that CoreOS had, and we just knew we had to get there.
Jimmy Zelinskie (22:04):
So it's just a matter of building whatever had to be built to get to that point. It could have been a completely different software stack. It wouldn't have mattered. We knew what the vision for the future was supposed to be. We just had to get there. And I think that's a really compelling thing and it helped kind of guide everyone at the company working towards that shared vision.
Jimmy Zelinskie (22:26):
So, prior to Kubernetes existing, we worked on a thing called Fleet and Fleet basically was distributed Systemd. So people hate Systemd right now, but what if you could just write a Systemd unit file and it ran a group of machines, rather than just on a single machine? That's what we had initially tried, because we knew what the future was. We were trying to make the gap bridged. But the second basically we heard about Kubernetes, because Google actually reached out to a bunch of companies before they made it public and said like, "Will you back this thing if we make it public?" And you kind of had to commit.
Jimmy Zelinskie (23:01):
The second, basically, we were asked about that, deprecated Fleet on the spot and started investing in Kubernetes because we knew Borg was basically the thing you needed. You needed a Borg or a Tupperware. You just had to have the thing for the future for containers.
POP (23:17):
I had Paul Bird on, again with CoreOS and he was talking about Brandon being Babe Ruth, being able to call the shot, being to say like, "This is the technology we should use," and stuff like that. And one of the things I will also say, do you think also Docker in the desktop version ... Let me container that for you, was just like, "Hey, it was just a speck. It didn't seem like somebody could just pick it up and go."
POP (23:44):
But do you think Docker popularity also helped the CoreOS and the Kubernetes thing?
Jimmy Zelinskie (23:48):
Absolutely. [inaudible 00:23:52] if you ever heard that pronounced, just let me containerize that for you. That had no adoption. LXC really had no adoption or we're targeting a different use case. LXC seemed to target machine-level images kind of like you would target a VM versus the application focus. I think honestly, without Docker, we wouldn't have gotten that point.
Jimmy Zelinskie (24:16):
But packaging can look completely different. And you kind of see it now with Go from Google and a couple of other projects that are specialized. This is how you should package Java JARs because they need to be packaged differently in containers. It's not just like start with a Linux-based image and add shit.
Jimmy Zelinskie (24:33):
And I think as we go to the future, you'll see more people doing things like that. But that's actually how, for example, internally at Google, Rapid is the system that they used and that's completely packaging differently. They don't necessarily start with a base like Docker style, Linux file system and packaged up their software. They normally should have just make it a binary, but they packaged all their dependencies together in that system.
Jimmy Zelinskie (24:57):
The thing that Docker had was mindshare. They got people in a place where they could quickly create a container and start seeing the benefits of containers. But there could have been various other ways we could have gotten there. Like I said, CoreOS had the vision. We're going to adopt whoever was there, and at the time, the same thing with Kubernetes.
Jimmy Zelinskie (25:19):
Kubernetes wasn't initially all-in on Docker. It was originally written in Java and they kind of had that pivoted towards the Docker, Inc. ecosystem because that's where the traction was. And so, if you're trying to get from point A to point B and there's a bus going there, you get on the fucking bus. So we definitely invested in Docker. Docker is super critical to it. The place where everybody butt heads with Docker was honestly just because they wanted to own the whole market. They wanted to own everything end-to-end.
Jimmy Zelinskie (25:49):
And CoreOS already had this long-term vision of like, "Hey, we're going to own a platform all the way out here." And Docker was kind of holding everyone back to being like, "We've captured the market here. How can we actually make everybody pay?" And so there's this butting of heads where we're trying to move everyone forward and commoditize things as fast as possible to get to the end goal. And they were trying to kind of slow everybody down to capture the market so that they control everything.
Jimmy Zelinskie (26:17):
But honestly, this is where Brandon stepped in. Brandon is actually the most incredible person at being able to kind of communicate and work with the community. If I was going to have a technical board of oversight like the bootstrap committee, the initial bootstrap committee for Kubernetes is the most impressive collection of people, I think, like on the planet. All these people are incredible at working-
POP (26:41):
You know what's so funny? The only person that I've not had on our show has been Brandon, and Brandon keep on avoiding me. I will have you on my show.
Jimmy Zelinskie (26:51):
Brandon is a busy guy. He's a busy guy. He's got family-
POP (26:55):
Not too busy for the podcast, you know what I mean? You know how much love I have for CoreOS for God's sakes? Come on, Jimmy.
Jimmy Zelinskie (27:01):
He literally just had another child, so I'm pretty sure that's why.
POP (27:04):
I know, I know, I know, I know. You know I love him and Polvi. I know it's all good. By the way, what are we drinking, Jimmy? What are we drinking over there, buddy?
Jimmy Zelinskie (27:11):
This is Hakushu. It's a Japanese single malt.
POP (27:15):
Awesome. Well, gesundheit. So again, we changed the industry. We did what we did with CoreOS, right? We got acquired by Red Hat. You lived out of a hotel for the first few months of 2018 in San Francisco. Talk to me about that. Which hotel? Was it at least a nice hotel-
Jimmy Zelinskie (27:34):
Oh, no. It was an awful hotel. Basically, all these things happened at CoreOS. I go out to San Francisco to do a quarterly product planning. So, I'm not there ... It's like a couple of weeks for that. I think it's two or three weeks for that. So, I'm there for that, staying in a hotel in Union Square in San Francisco where the shittiest ones. Not going to name which but it's not a great one. It's near a pretty good ramen spot though.
POP (28:02):
Holiday Inn Express? Let's just call-
Jimmy Zelinskie (28:04):
No. It's not a chain. But I get back from basically three weeks of that being there. I fly back. I'm in New York for less than a week and then we announced that we're being acquired and then it's like you have to come back to do basically planning integration planning. And then it was another bunch of weeks after that. I basically went back straight to the same hotel.
Jimmy Zelinskie (28:31):
I remember after coming home post-acquisition meetings, just being like, holy shit. It was January. It started in January because we got acquired at the end of January. I think I spent more time in San Francisco so far this year than in my own house, than in New York. This is where I live, what the hell?
Jimmy Zelinskie (28:51):
And then later back in May, I was back there for a Red Hat Summit which is where I eventually had met you. It's pretty crazy going from KubeCon EU directly. So I was in Boston for a little bit for a Red Hat thing and then back to New York, then went to Copenhagen for KubeCon EU, then went from Copenhagen to San Francisco, and then finally back to New York. My body is just completely destroyed, jetlagged. I had no idea where I was.
Jimmy Zelinskie (29:22):
I flew to Copenhagen in WOW Air which was Spirit Airlines, reseller of Spirit Airlines. It was literally a hundred dollars to get from Newark to Copenhagen. It's like the shittiest flight possible.
POP (29:36):
Was there like chickens on the thing? Do you have [inaudible 00:29:40] the whole time?
Jimmy Zelinskie (29:40):
There's not. But the seats are like folding chairs. They're basically like you get on the plane and it's just like rows of folding chairs. You can move them physically. If there was something, you just go flying everywhere. It's absolutely crazy.
POP (29:54):
That's insane, man. You were pretty instrumental in terms of the Operator Framework. Again, I had Szumski on as well and talked a little bit about this. But I'm going to talk to you about just the Operator Framework and your work on it.
Jimmy Zelinskie (30:08):
Sure. Rob and I were like this when we're working on Operator Framework. What basically happened was the origins of it are ... Tectonic is the CoreOS Kubernetes platform. Underneath it, the whole strategy is auto-update. We had the auto-updating Container Linux. Our whole mindset and value prop to Tectonic was that it's been like Kubernetes, which I've also read a blog post on why calling a product Kubernetes is bullshit, but it's been like Kubernetes plus auto-update.
POP (30:45):
We'll have a link in the description, everyone. We'll have a link.
Jimmy Zelinskie (30:51):
Our whole value props are on automatically updating this stuff. So, how do you automatically update Kubernetes? Well, you take down the control plane and replace it with new instances of the same software. What does that and how do you do that? Well, the answer is, well, another layer of Kubernetes basically. You have a controller. The controller manager is doing all of your software and you create another controller that knows how to manage the software itself.
Jimmy Zelinskie (31:18):
So, we bootstrapped Kubernetes. That's the first thing. That's the first goal. The first goal is have Kubernetes running under Kubernetes. So, you can look inside the Tectonic system namespace or the kube system namespace and see the controller manager. You can actually see Kubernetes is running on itself. And once you have that, you can do rolling updates and now, you can actually orchestrate updating Kubernetes the same way you would orchestrate updates literally with any other software.
Jimmy Zelinskie (31:46):
Once you have that step, then you're a basically custom controller which is what everyone is doing now. They're like, "Whoa, operators. What if I take my app and I write a controller for it? Wouldn't that be cool?" Well, we had that idea super long time ago and it was so that we could update Kubernetes itself. We saw it's the only way you could actually update Kubernetes itself because it's so complicated.
Jimmy Zelinskie (32:05):
That was kind of our strategy. And as we kind of like controlerized so many things, we realized like, "Hey, other people are probably going to want to do this. And more of our stuff needs to be delivered in this way." So, a lot of our strategy later in CoreOS was about how would you get Amazon style services on your Kubernetes clusters? How do I give you an RDS? How do I give you something that knows how to run itself?
Jimmy Zelinskie (32:35):
The only way you can do that is with the Operators. So, we kind of started with two different like a two-pronged approach. Rob took the angle of helping developers build Operators and making it easy as possible. So that's the Operator SDK. And he largely focused on that. And then I took the route of actually like updating in a lifecycle aspects of Operators.
Jimmy Zelinskie (32:56):
Now, you have got an Operator in a cluster. Who updates the Operators? Now, we have yet another meta level of, we were updating the clusters. Now, you have Operators in the clusters who updates the Operators and how do they know if the Operators are healthy or not. So now, you have an Operator-operator. Basically, it's an operator that manages all the other operators which is super confusing, but that's what the Operator lifecycle manager is.
Jimmy Zelinskie (33:20):
And so this was our way that we could have streams of updates you could subscribe to like, "Hey, give me the stable version of like the Prometheus Operator." And so that's what the end result of that would look like on your cluster. It'd be saying like, "I'm subscribing to the Prometheus Operator beta or alpha or stable."
POP (33:37):
If you think about the beginnings of this discussion that we had, remember we're talking about the whole Facebook thing. And then we talked about this is what you wanted really is what Facebook has this whole ability to basically add this ... I don't want to use the term fleet but I'm going to use this fleet ... of machines that are in a cluster that could be on this cloud, could be on a private cloud, could be on a public cloud. And I'm going to click a button and if I want Prometheus everywhere in the same configuration, to me, the best thing out is to have an Operator Framework do those things.
POP (34:09):
It's phenomenal, like you said, to have that same thing. If I'm using like ... Again, Amazon, you take for granted. I could just click a button and have MySQL or Percona database up and I'm ready to go, right, or whatever it might be. But in the Kubernetes' world, I have to have my manifest files and I have the blah-blah-blah-blah. What if it's running? If have to upgrade it and then I have to patch it to where the data source is and stuff like that.
POP (34:34):
Whereas, man, it's like this framework, it's open. I can do all these things. So, again, was this something like you've all went to go ... Again, I was a product manager. I know how it is. You go to a couple of customers. I'm suffering from this. Did you have any anecdotal aspects of that that we can talk about that there was like, "Shit, I knew we nailed it"?
Jimmy Zelinskie (34:55):
Yeah. It's actually interesting because it was kind of, there are multiple aspects where there's like the Henry Ford thing where if you ask somebody what they want, they'll tell you faster horse and not actual car. There was that and then, we had a lot of customers that were in that space. We had a lot of customers that were just already like in the future. They're like, "Holy shit. They built the thing that we have on our road map for like next year." They just solved the problem and already have it.
Jimmy Zelinskie (35:23):
Then there are people like all in between. Our whole point was we want to work with the customers that had built custom solutions that were kind of in outer space and make sure that we could make a generic one that would actually solve that for everybody else. So, there were definitely customers that just like you would walk in the room and you start talking to them. And they're like, immediately, they got it. They got it and they had already built a whole bunch of stuff in line with the same philosophy as you.
Jimmy Zelinskie (35:52):
And it was really a lot of just talking about how to make that real for everybody. And when you get in the room and you talk about that, they're like, "Hell, yeah. I don't want to have to maintain this shit. I want something that someone else is going to do for me, because I made this thing as bespoke. It only works in a small number of scenarios for us specifically. And if you want to take the burden on owning that and making sure it works for everybody and it's going to work for us in the future, absolutely, we'll pay you for that."
Jimmy Zelinskie (36:21):
Sometimes, it's not like if you build it they will come, but instead you look at what other people are building internally and like, "Hey, I bet a bunch of people could use that." And that's kind of the story of Kubernetes and a whole bunch of things. You look at Borg or Tupperware. And like, "Hey, the whole world could actually use that," which is also the same challenge we have for our new company but we can get into that later. It definitely is like that's the right way to tackle things. I don't even know if that answered the question.
POP (36:53):
No, it did, and it was good. I liked hearing that. And then like I said, you did my segue for me there. That was good. Jimmy, again, you're still working somewhat with your two partners there in Petricorp now. And so when did you say, "Hey, you know what? We need to do this. This is something we need to do," and talk to me about Petricorp.
Jimmy Zelinskie (37:21):
Sure. CoreOS gets acquired. Immediately, we're in a big company. Little did we know, it would be way bigger when Red Hat gets acquired by IBM. It's actually crazy because Jake and Joey were like DevTable which was their IDE company. Then got acquired by CoreOS, then got acquired by Red Hat, then got acquired by IBM. It's just like so many acquisitions deep.
Jimmy Zelinskie (37:49):
But effectively what happened was we got acquired from the beginning kind of every ... At least across the three of us, we're like, "So how long until we can leave and start a new company?" It really was a thing where we find freedom in operating as a small team and working in that startup environment and a scrappiness to it.
Jimmy Zelinskie (38:12):
And I had honestly never worked in a big company before, so I knew nothing else. Just like I came out of college, it was like, "Hey, this is how it's supposed to be because this is how Tupperware worked at Facebook and this is how Borg works at Google." I never knew the world of Chef or config management. I just went in being like, "This is what the world should be." My experience is I have no idea what a big company is actually like.
Jimmy Zelinskie (38:37):
I am actually really glad for the experience of working at Red Hat. It's a great place. I'm really glad for the experience of actually having hands on in the transition from CoreOS to Red Hat. That's in a once of a lifetime experience. I learned a ton, but the environment we actually really want to work on is a startup.
Jimmy Zelinskie (38:56):
The whole time we're there, we're kind of like spinning, working on our stuff. We really about the ecosystem, maintaining what we've done and then once we had felt like we had gotten to a place where we could ethically hand the things off, that was kind of how we transitioned and then decided to leave and start our own company. Now, we actually really literally didn't have an idea.
Jimmy Zelinskie (39:19):
This is another crazy thing that is really good advice. If we had spent that couple of years while we're at Red Hat, scheming on one particular idea, when we left and actually went to go to execute that idea, if it failed or our expectations or assumptions didn't meet reality, founders can get into a weird position where they force it to try to work instead of abandoning the idea, seeing it for what it was and saying this actually doesn't have traction and abandoning the idea for something else. So, you kind of don't want to be super biased and tunnel vision on one particular idea. You actually have to step back and take feedback from the world.
Jimmy Zelinskie (40:00):
And so following the philosophy, we're like, "Why bother having an idea? We'll just leave figure out our idea after we leave. We're just super ballsy. So, I spent the first month just me and Jake basically sitting in video calls, talking to each other being like, "Hey, what should we do? Here's like a couple of ideas. Let's talk them through and see where they fall apart, see where they have aspects that are really good, see where they have aspects that are really bad."
Jimmy Zelinskie (40:28):
And we kind of knocked it down to two ideas and kind of pitched two different ideas to close friends and similar companies. And eventually, it became clear that one of them was better than the other and so we kind of like shelved the other one. I just started working on this. And eventually, we pitched that idea to Y Combinator and did Y Combinator interview and got into that, which is pretty crazy.
Jimmy Zelinskie (40:53):
Basically, what the idea is authorization as a service. So, the whole pitch is very similar to what I was saying earlier. At Google or at Facebook, you have a team that's dedicated working on just authorization. Authorization is different from authentication, which is a thing that you can actually get as a service right now. Right now, if you went out and you look at like Auth0 or PingFederate or Okta, these are all services that basically aggregate OAuth providers.
Jimmy Zelinskie (41:22):
If you want people to be able to log into your application, you slap one of these things on it and then they can log in with Google or Facebook or GitHub or whatever. But that's where it ends. It says like, "This is who this person is versus authorization." Authorization basically is who has access to what. So, it's a permission system inside your application.
Jimmy Zelinskie (41:47):
What happened at Google was they started building Google+. And Google+ is about social network. I shared a post. They had circles if you can remember where you would share with different groups of people, and you could overlap them and share between these two or only the intersection of these two or remove this group from the other one. And so to do that, you need a really sophisticated ways of saying who can access the thing and who can't. So, they built a dedicated service towards actually tracking who has access to what.
Jimmy Zelinskie (42:20):
And so literally, we take an inspiration from that and said, "You know what? No one outside of Google or Facebook has systems like this." It's kind of like the next step. So, the first thing in the cloud is like, "Hey, let's get our databases. Let's get all these services up." And now it's kind of like the next step up the stack which is a service that's actually ... It's kind of like bookwork. It's not necessarily the core part of the domain logic in your application but it's something everyone has to do.
Jimmy Zelinskie (42:45):
Everyone that's writing their app, they basically have to model the same thing as if it's like an enterprise-style app. So, if it looks like GitHub where they have organizations and teams and users, you could build these systems. And then that's a whole bunch of work. None of the open source libraries that you find in Rails or Flask or Django, really actually even scale to that point. They make it really hard to even do that model.
Jimmy Zelinskie (43:12):
But then the crazy stuff that the teams at Google will have is they'll have replicas of that literally everyone in the planet. So it will be fast no matter where you are. But then on top of that, you can actually share once you have your authorization rules in a centralized thing.
Jimmy Zelinskie (43:31):
So, the classic example is I embed a YouTube video in a Google doc. By doing that, if you have access to the Google doc, you should transitively have access to the YouTube video that's embedded in it. But these are completely separate services. So that means the permission system somehow have to know about each other's nouns. They have to know that this is a video and this is a Google doc, and you have to be able to make that relationship modelable.
Jimmy Zelinskie (44:01):
And this is a problem that basically everyone has that introduces a second service. So at Quay, we got into all kinds of shit basically by having a super-mature authorization system. We basically just used off-the-shelf libraries in a Flask ecosystem just what pretty much anybody who would do right now. You picked that up and what happened was Docker changed a thing and we wanted to make that feature-enabled but it would require us completely refactoring or rewriting all those authz code.
Jimmy Zelinskie (44:32):
At this time, we had petabytes of data, tons of users at this point. And literally, we just couldn't add the feature because we would have to spend so long changing all that stuff. It's all security critical. If you make a mistake in it, that means people can access things they shouldn't be able to access. You get security audited if you're going to actually to play it to certain customers. It just totally breaks down.
Jimmy Zelinskie (44:57):
And then when you have Clair, for example, and going back to Service Mesh, Service Mesh just makes it so ... You can identify that Quay and Clair can talk to each other securely but how does Clair know that it's allowed to access particular images? If someone was to hack Clair, they could access all the data in Quay because Clair doesn't act on behalf of the user, it just has free access.
Jimmy Zelinskie (45:23):
And so, it's kind of like ... And this is basically how everybody builds their services. Once you add a second service, basically your problem is everyone does Mutual TLS but they don't have any permissions that are shared across them. So, it's basically like having two websites and just because you connected to the website basically with TLS, it means you can ignore the login box. When I go to Google, I still log in and [inaudible 00:45:50] and then Google can actually control what I'm accessing. It's not just TLS that's securing everything.
POP (45:57):
Because the backend, they have all those components that's essential. Like you said, this is a centralized thing, so you're trying to bring that centralized piece to the populous pretty much such that if they have all of these individual ... I'm sorry, I want to boil it down because at the end of the day, somebody is listening to this and is trying to build this from scratch, good fucking luck because it's real tough.
Jimmy Zelinskie (46:19):
It's a really hard problem, right? The more services you have, the harder it is. It's really the people that have multiple services that realized. And what happens is companies look around one day and they hit a make or break scenario, Gusto had this somewhat recently, the payments processor. They basically looked around and went, "Oh, shit. We need this system because we have to share things across multiple services," and they basically had to stop everyone.
Jimmy Zelinskie (46:46):
You can't do any product work anymore. You have to build this thing. We have to integrate them all together and then product feature work can resume. And a bunch of people are going to hit this problem. A bunch of people are not going to be able to be confident in changes in their authz code. And it's just one of those things that, how can you compete with a hyperscalers? This is the whole problem.
Jimmy Zelinskie (47:06):
The cloud is supposed to give you the services that the hyperscalers are using as a competitive advantage. That's why it works. And these next generation services are the ones that don't exist yet. There's IAM, for example, from Amazon but you can't add your own things in IAM. You can't teach IAM about things inside of your application. And that's really what we're trying to do, is say like, "Hey, what if you could extend that model? What if you could teach the centralized permission systems about the things in your applications and then share those?" That's basically the idea.
POP (47:43):
And there's also an app that you guys has been putting out. So, Sharewith.io is built on this platform. So you all can see this and we'll put a link in the description as well of this in action. And the other thing is definitely, you all are looking for folks to kind of kick the tires and open up this platform. Again in the liner notes, they have an email you can send to be part of this. And if anything these three touch turns to gold, I wish you the most success with this. This is phenomenal stuff.
POP (48:14):
Is there an open source component or anything so people can contribute? Or this is mostly closed doors at this point?
Jimmy Zelinskie (48:19):
Right now, everything is closed source. ShareWith is the first app that's built on top of the platform. So, we're going to eventually have an open platform where people can use our SDKs and integrate their applications into that platform. And probably simple cases where you want to do local testing, things like that, will be open source.
Jimmy Zelinskie (48:41):
The idea is to keep basically everything that we have, like everything that we would need to control proprietary just because we're going to be moving fast and changing a bunch of stuff. But really the advantage you get out of using an authorization system like this at a global scale is that we're going to run it at a global scale for you, which is really hard. That's why there's dedicated team at Google or Facebook doing this work.
Jimmy Zelinskie (49:07):
So, you actually want to pay for the SaaS and you don't actually care about that side. But everything that you'll possibly care about will be open source and maybe eventually ShareWith will actually be open sourced as an example application so that people can actually see highlight, take this SDK and build a real app with it. So that's kind of the long-term goal.
POP (49:28):
Jesus Christ, Jimmy, man. I'm like floored that you all are going to do really well with that because I hear that in a lot of discussions I have with customers and then users. I mean, it's just so awesome stuff. Again, good luck with this and all the best but I got my last question for you and that's what work are you most proud of in your career?
Jimmy Zelinskie (49:52):
This one is a really good question for me. I really enjoyed this question because there's kind of two aspects to it. I work half the time as an engineer, half the time as a product manager. As an engineer, kind of the growth and seeing Quay, for example, come from running on like ... At that time, we would call it like a cellphone tier like EC2 instance. But cellphones, they are now so much more powerful, I can't even say that anymore.
Jimmy Zelinskie (50:16):
But seeing Quay basically grow from nothing to something that had millions in revenue, petabytes of data, tons of traffic suggesting that from end to end is such a crazy personal accomplishment. And [inaudible 00:50:30] to people that I have worked with on that. But then also as a product manager, because you're actually orchestrating and helping people meet the means to an end and working at CoreOS, it's actually just ... It's almost weirdly like accomplishment of everybody. The cloud native ecosystem and Kubernetes, I feel like I've had an impact on that and helped guide people on that way.
Jimmy Zelinskie (50:52):
And nowadays, I feel like I'm telling mostly people about the past, being like, "Oh, no. Hey, we thought of that in 2015 and this was the direction we were taking," and just providing that context with people and shepherding the direction of the community. I feel like I get credit basically for everything that they've done. And so for me, the biggest accomplishment is kind of like the impact on the industry. Everybody has kind of unanimously adapt the cloud in this way. We've kind of turned the adaption of cloud more away from proprietary services towards more open and sharing ecosystem and one that is actually more of the future what we want to see.
Jimmy Zelinskie (51:31):
So, I'm kind of stealing. It's kind of a cop out. Everybody's work is my work is kind of my point. But yeah, honestly, the impact at CoreOS is that just that legacy of seeing the future and kind of forming and making sure that we were in a world where everybody is using Docker's form right now.
POP (51:51):
Well, Jimmy, thanks so much for all your work and thank you so much for being on the podcast.
Jimmy Zelinskie (51:56):
Yeah, thanks for having me.