POP (00:11):
Hello, everyone, and welcome to the Popcast. Today, we're going to talk observability and high performance software engineering team practices and MMOs with principal developer, our advocate at Honeycomb, Liz Fong Jones, aka Liz the Grey. Welcome to the Popcast, Liz.
Liz Fong Jones (00:27):
Thank you for having me.
POP (00:29):
So I love this quote and, uh, you know I'm a fan of yours, and if anybody wants to see how big a fan I- I am of her, if you look at the Escape Show last year, I literally like got all like tongue twisted. I was like, had to talk... You know, how can you talk like golden signals with like somebody who, like literally kind of started that whole kind of, you know, mindset? So I was like, yeah, uh, flabbergasted.
POP (00:50):
Uh, so I, I love this quote you made. It was, "I made developers on, on your list, uh, your dur- your", um, your website, and I'll put this as a link in the liner notes, but, "I make developers operators and workers as whole more productive empowered." Talk to me about that quote.
Liz Fong Jones (01:05):
Yeah. I think it really encapsulates why I do what I do. Um, so I work both as a labor advocate, and also as a developer advocate. Um, so my goal is to help people have a stake in what they're doing, and be better at what they're doing. Um, so whether that's developing better developer tools to help people, uh, create, uh, create their applications and have them run successfully in production, or whether it be helping people champion the rights of users and courtesy rights within their workplaces. I, I think that it's really important for people to be productive and feel happy about what they're doing.
POP (01:42):
And where did you get your start in terms of just doing, you know, uh, uh, being an SRE and, and understanding, like you know, what, what- the progression to where you are as a developer advocate at Honeycomb right now?
Liz Fong Jones (01:53):
So one of my very first jobs was actually at a massively multi-player online game studio. Um, I started doing technical support there, and I eventually got taken on as an intern, and they helped me really level up my systems administrations skills and my coding skills. And that's really how I got my start in software engineering. And then I went to college briefly, dropped out, went to work for Google as a site reliability engineer to kind of really grow that SRE expertise. And then over time, I found myself more drawn to, instead of being an SRE for one system, instead helping teach and popularize what I had learned at Google to many other companies outside of Google, so that everyone can be as productive and as reliable as Google's site reliability engineers.
POP (02:40):
Got it. And that's wh... And then, you know, you, ye- ye... What makes teams high performance in your eyes? Like what are the things that make those teams high performance?
Liz Fong Jones (02:48):
I think that a team is high performing, not just if the individuals on the team are high performing, right? But instead that the team is really [inaudible 00:02:57]. So when I think about a high performing team, i think about, is the team able to shift code to production? Are they able to understand the code that's running in production? Are they able to validate that what they're building is actually being received positively by users? Right, like, I think that that virtuous feedback cycle is what makes a team high performing. And there's a large number of kind of specific technologies that are involved with that, right? Like, and also socio-technical practices, right?
Liz Fong Jones (03:22):
Are you practicing some form of agile delivery? Are you practicing continuous integration and delivery? Are you practicing observability so that you can understand the impact of [inaudible 00:03:32] in production, right? Like, I think all of those things are really, really important.
POP (03:36):
Got it. And then basically, um, so now you're at Honeycomb, uh you're working with Charity Majors?
Liz Fong Jones (03:42):
Yes, that's correct, uh Charity is my boss.
POP (03:45):
Awesome. I mean sh-very well regarded in past, uh, I hope to have her on the show, to be honest with you. Uh, phenomenal person. So what's it like being literally, like in that environment, like literally just be the, the bet of kind of a-all day working observability? That's, you know, how, what is, what's it like?
Liz Fong Jones (04:04):
Yeah I think that observability is a means towards and end, right? Like the end is, as I said, you know, how do we help teams become more high performing? And observability is one axis, right? So I help people with understanding their software sprawl. I help people understand, like, you know, "Why is my system breaking down?" "How do I improve my system?" "How do I architect it better?"Tight? So I think that that is where I shifted my focus, rather than kind of, you know, I still continue being a general accessory; I am on call for Honeycomb's production systems. But at the same time, when I talk publicly, I'm talking a lot of the time about kind of those specific practices in the domain of observability because that's where I've chosen to specialize for now.
POP (04:43):
Got it, and, and you know, one thing, I'm gonna sidebar on a t-uh, that talk we talked about earlier in terms of like, you know, uh, the Escape Conference. What I love that you did for that was, you did, you had accessibility, you had the, you know, c-the closed captioning as part of your talk.
Liz Fong Jones (04:58):
Yes.
POP (04:58):
And I ju-I just love that, because it's such a, it's such a touch, touching thing, it was like, you know you wanted to make sure that your talk was accessible to anybody, and that, and that's just, I think that was a very good, you know, part of your character. I just wanted to highlight that. So.
Liz Fong Jones (05:12):
Yeah definitely, I feel that we have to be maximally inclusive, and that means making sure that, you know, number one, we're, we're not, you know, overtly discriminating against people on the basis of their race, gender, gender identity, et cetera. But also that we are inclusive of people who have disabilities, right? And that means that we need to take care of closed captioning so people who are deaf can understand what we're saying.
POP (05:36):
A-and just as an FYI, I'm going to closed caption this episode, it'll be the first episode that is closed captioned, and that's because, because of your example.
Liz Fong Jones (05:45):
Excellent, yes, transcripts.
POP (05:47):
Transcripts, right, yeah. Um, the other, the other thing is um, so back on course, I'm sorry, I kind of veered a little, but you know, I just, uh riffing there. So in terms of uh, how does OpenTelemetry play in, in observability and, and [inaudible 00:06:01] functions?
Liz Fong Jones (06:04):
Yeah, so OpenTelemetry is a vendor neutral standard for people to add instrumentation to their applications, so that they can send it to any number of different providers for, for understanding what's happening inside of their applications. So it's very much a mechanism for people to be able to try out various providers and see which ones, or combination of, of which ones helps them achieve observability. So this is a, uh, change that we, that we as a community of observability practitioners really, really pushed for, because people were afraid to adopt observability in a vendor specific fashion, right? People were afraid of lock-in, people were afraid of their tools not intraoperating, and OpenTelemetry really empowers people to have high quality telemetry that is as ergonomic as possible, and then to ingest that into the right tools that make sense for them.
POP (06:55):
Got it. And then in terms of like, do you feel like, you know, tracing functions and, uh, you know, i-is tracing functions are different than metrics that are different than events, and all those things, how does that all play in terms of, what d-what does observability, is it a combination of all of these things, or it's basically kinda one size does not fit all?
Liz Fong Jones (07:14):
Observability is not the data itself, right? Observability is a socio-technical capability. Do I as an operator and developer have the capability to observe and understand what my systems are doing without having to push new code, right? Can I ask the system new questions and get the answers that I need to my questions? And definitely the telemetry plays a part in that, right? Whether it's logs or metrics, or events, um, and honestly our viewpoint at Honeycomb is that those are actually all the same data type; they're just different manifestations of that, of that data. Right? Like, a metric is just a summarization of a number of events that are flowing through your system. [inaudible 00:07:53] a bunch of interconnected events that have a causation to them, right? Like, so that's how we look at it, is, you have your data, right, but your data is, is useless unless you can actually analyze it at scale, and in realtime, without having to push new code, and without having to preaggregate it.
POP (08:10):
Got it. In terms of, um, the, the you know, and again reflecting back to your book that you, that you've written, and I know there's a subsequent one you'll hopefully talk a little bit about that today. Um, so the book it h-you know, how SRE relates to DevOps, and I, you know, so the question there is, a lot of people, like, thinks, "Oh I'm just gonna pick up the, you know, Liz the Grey book, or Google's SRE book," or what, what have you, and they're gonna understand what DevOps and SRE... So to explain to me what the differences are versus, and then an SRE, and, you know, give me more meat on that?
Liz Fong Jones (08:40):
Yeah, I think that, so this is referencing the first chapter of the Site Reliability Workbook, um, so it's a compendium volume; I published only one chapter in it. Um, but the, how SRE relates to DevOps chapter argues that contrary to peoples impressions that DevOps and SRE are competing with each other, or are in conflict of each other, they're actually compatible with each other. That SRE is a very specific opinionated way to approach DevOps. That it satisfies the key areas that we expect out of a DevOps practice; mainly around trying to make changes safer, or trying to iterate faster.
Liz Fong Jones (09:15):
Um, however I think that the popular conceptions are definitely a situation where you see companies spinning up a separate DevOps and SRE team, and I-I look at that a little bit strangely, right? Like, I think that definitely people have this stereotype that DevOps people are only interested in continuous integration, continuous delivery, right? That you would put a DevOps person on your release engineering processes, right? Like, it's like, no, that's a entire discipline, right, they, that's the art of release engineering, right? And instead we as DevOps and SRE practitioners are the people who think about, how do we make the systems more reliable and resilient? How do we make our-ourselves able to move faster, right? As a holistic practice.
Liz Fong Jones (09:55):
So I think that's kind of where a lot of this confusion is, because of the fact that DevOps hasn't really had a rigorous definition because it's kind of a set of cultural practices, and I think that, you know, when SRE came onto the scene, it was definitely this highly opinionated thing that people initially didn't know what to make of it, but now we can kind of see that they're satisfying the same goals; they're trying to achieve the same thing.
POP (10:18):
Uh, you know what I thought was kind of interesting, the kind of sidebar on this one two; the fact that like, uh, um, I saw some tweets that you had done that was, literally it was an outage that had happened somewhere, and you were literally live tweeting, you know, the [inaudible 00:10:27], it was so, so revealing about like, just the overall process. And I'm sor-I gotta go back to those tweets, and I'll have the tweets in the, in the linear notes of the episode because I thought that was really cool. It was literally like, "Yes this is this. We think it's this," and then it ended up being something different and then you were like, "Okay this is what happened, "I just, just love that post mortem-there was literally a whole stream of tweets on that, I-I just, I thought that was really cool that you did that.
POP (10:51):
So just a little props.
Liz Fong Jones (10:53):
Yeah definitely one of my favorite things is live tweeting events, although I get to do much less of it now that we're all in quarantine.
POP (11:00):
(laughs)
Liz Fong Jones (11:00):
Sorry there was a bug. Um, I-I much less-
POP (11:03):
All good. It's all good.
Liz Fong Jones (11:03):
Oh my goodness, there we go. Got it. So I definitely do much less live tweeting of events now that, um, now that they're all online, they're al recorded and people can watch them in realtime. But it's definitely been one of my favorite ways to share what's happening in a conference that may not be recorded, that's only in a specific physical location, to share, uh, what [inaudible 00:11:22] are talking about, and also my impressions and commentary on it with the wider global community.
POP (11:27):
That's definitely great. Um, one quote you have is, "Grumpy humans are really bad at running systems," can you kind of elaborate on that one?
Liz Fong Jones (11:37):
Yeah, I think that, too often people take an approach to SRE and DevOps that's focused on tools, and we really should be thinking about, how are we making the lives of individual operators better, right? How are we making the lives of the software developer that's on call, or the SRE team that's, that's helping with on call? How do we make them better? Because, at the end of the day, if you're being repeatedly woken up at 3:00 AM, you're not going to be productive, you're not going to be, uh, creative, you're not going to be able to come up with engineering solutions, right? You're just going to be putting bandaids on things night after night, after night. So I think that's where it comes from when I say that grumpy humans are bad at operating software, right? Like, you have to make sure that people are getting adequate rest in order to really tackle engineering as opposed to, uh, as opposed to just firefighting.
POP (12:22):
Do you see like mul-multiple on call sta-I mean obviously the smaller companies it's harder for them to-to-to have like those on call resources, so what would you prescribe for that?
Liz Fong Jones (12:32):
I think that the number one thing that's important to do is to reduce your paging volume, right? And the best way of doing that is switching to something that we call uh, potential cau-oh sorry, it's switching to something that we call, uh, symptoms of user pain. Right? Instead of, instead of paging on every single potential cause of an outage, you want to look for, "Is my service actually impacted?" If my service is not impacted, why does this, you know, why does this random alarm matter? Um, so turn off all your noisy alerts and really adopt this service level objective [inaudible 00:13:03] based alerting approach. And that'll make your on call life a little, a lot quieter. And that enables you to wake up less often.
Liz Fong Jones (13:11):
Um, the other thing that, if you kind of can't bring it under control that way, another way is to really divide up your on call schedule, right? Like, have people on call maybe, you know, twice as often, but for half the time for on call shift. And that really means that people burnout less, right? Like, I-I argue that if you're in a noisy environment, you just can't do, uh, uh 168 hour shift. It works out really, really poorly. So if you can say, you know, one, one person takes the nights, the other person takes the days, right? Or actually we've been encountering this recently at Honeycomb with lockdown; people with parenting duties, right? People cannot do on call during specific hours, so we're trading around days of, day of the, uh week and hours in the day in order to make sure that people can do on call sustainably.
POP (13:51):
Definitely agree, I think there's the, you know, there's a rash of just high level of alerts and, and not being able to like, provide that kind of, you know, like you said, time off to people to address those, those things, that's also, you know, some things we do at [inaudible 00:14:05] in terms of just like, providing more of the context in terms of the alerts that are being pr-provided, so there isn't that rash of, you know, alerts that are 1000 alerts, it's literally just set to the things that you want to monitor. Or protect, when it comes to our security product.
POP (14:20):
So one of the, the last kind of, uh, tech question, you know, in the SRE vein before we get into the, the fun we're gonna talk about MMOS, uh, can there be too many dashboards?
Liz Fong Jones (14:33):
Yes, anything more than two dashboards is too many. Um, you kind of want one set of dashboards that's focused a lot, along, along the lines of USE RED Metrics, um, so things, uh, Utilization Saturation Errors, um, RED stands for Rate Error Duration, right? So those kind of basic APM metrics are handy to have because they tell you at a glance, you know, how my traffic is my service doing, have there been any spikes in amount of traffic? Have there been any spikes in errors? Am I saturating my [inaudible 00:15:01], am I in danger of having requests take too long and pile up on each other, right?
Liz Fong Jones (15:05):
But I think beyond that, you also need a set of dashboards around service [inaudible 00:15:09], right? You want to know, what are my SLOs, right? Rather than what, what's my [inaudible 00:15:13] duration rate? And SLOs are really tied to business outcomes, right? What are the business outcomes my customers are intending to achieve? And are they [inaudible 00:15:20] succeeding?
Liz Fong Jones (15:21):
And I think belong that, right, like anything else that you do is going to be unique to the situation; it's going to require investigation and drilling down and digging in. And no [inaudible 00:15:30] dashboard can do that for you, so in the past I've encountered situations where teams are tempted to create, you know, 20 dashboards, each with 20 different graphs on them; that's 400 graphs you have to stare at to figure out what line wiggled at the same time as that other line. It doesn't make sense, like, it doesn't work as far as having a prompt resolution of issues, or understanding of what's going on. And more importantly, it encourages you to adopt this kind of fixed mentality, right? Like, of, "If it's not an existing dashboard, then I don't see it," right? Rather than treating your data as something that you can visualize any way that you need in order to understand and debug an incident, right? You shouldn't have to create new dashboards for everything, and ideally, you know, if you do create a new dashboard, prune or trim away another dashboard, right? You know, you should really only focus on those kinda of two, maybe three dashboards that are most important.
Liz Fong Jones (16:18):
So use RED, SLOs, and maybe you know, some-something else around your most common sources of initial points of, starting points for debugging. But your dashboards are not the end of your debugging; they're the start of your debugging.
POP (16:31):
Y'all see why I'm such a fan of her? You see that? It's, that, you just [inaudible 00:16:36], I mean it's all just practical, amazing advice, and, and, and again, this is from years, years, like she talked about, of, of seeing this and-
Liz Fong Jones (16:44):
Yep, fif-16 years now. 16 years I've been working in this industry.
POP (16:48):
Well w-w-w-we're glad to have you. And I'm glad to have you on the Popcast.
POP (16:53):
So let's, let's move on. You got your s-you know, let's bring her out. So you got your start, you know, doing some of the, you know, I saw you work in, in, at, doing an M-MMO, so, um, talk to me about your first MMO, or R-RPGS, or whatever you were doing, and on what system?
Liz Fong Jones (17:11):
Yeah my first MMO was a MMO called Puzzle Pirates, um, which interestingly enough, as a result of the COVID lockdown, is suddenly doing as many concurrent users as it was during it's peak in 2005 and 2006. Um, so Puzzle Pirates is back in style. Um, it's a java based, uh, MMO, that is nautically themed, and involves, instead of kind of a combat mechanic where you know, you're clicking on, you know, you're clicking on buttons and smashing buttons, or like shooting people, right? Instead it's focused on puzzle fighter gameplay, right? So the extent to which you do well at a give task depends up how well you execute a-a-a puzzle game, right? And I think that's that's a super fun mechanic. Another interesting thing about it was that it's player base was I think more than half women, which was unusual for the t-for MMOs at the time.
Liz Fong Jones (18:01):
It also was one of the first MMOs to introduce micro transactions in the Western World. So it kind of has a large number of claims to fame, um, you know, all of these things around you know, player created experiences, uh, like, you know, uh, player politics and fighting over control of who has control over the islands and the ocean, right? Like, and I think that that's what made it a lot of fun, and you know, I started playing it in 2004, I'll say, 2003, 2004, and eventually I started doing things like, you know, helping out with the tech support forum, right? I was one of the earlier kind of Linux users, uh, of, of Puzzle Pirates out in the wild. So I was helping [inaudible 00:18:42] with that, and eventually the company offered me an, an internship to kind of join-
POP (18:46):
[crosstalk 00:18:46] so let's double click on that. So your kind of love for MMOs, and you just, you know... And this is a common theme of folks that talked about in the Popcast, like, you started, and this is, I think I was Commodore 64, and they were like, "My love of gaming got me into DevOps and, you know", and, and, and what have you. So like, so you're basically like, "I love this game so much that like, I'm, I'm on the forums, I'm getting involved", which to me is early streams of OpenSource like, you're getting involved because you love this tech so much. I'm sorry I had to kind of double click on that one.
Liz Fong Jones (19:19):
Yeah, yeah, I think that it's super, super cool that there were entryways into tech from whatever someone is, is most passionate about, right? Like, as long as there's a spirit of people can volunteer to help out, right? And that when you help out, to a sufficient extent, like, people entrust you with more responsibility, and kind of give you a say in the project.
POP (19:40):
Got it. And was, um, is there any other MMOs you, you like to play? I know we're gonna talk EVE in a little bit, but is there any other ones during your progression, uh, that you've played?
Liz Fong Jones (19:49):
I've definitely picked up a number of MMOs along the way, right, like you know? I definitely stopped playing Puzzle Pirates as much when I became a developer for the game, right? It's inevitable that you kind of can't participate in politics if you're a developer. And the politics were kind of the more interesting part of that game for me. Um, I also have at varying points, played World of Warcraft, played EVE Online. Um, this has kind of been my go-to three MMOs: Puzzle Pirates, EVE Online and World of Warcraft.
POP (20:16):
Did you ever play City of Heroes and City of Villains?
Liz Fong Jones (20:18):
No I didn't.
POP (20:20):
Great game. So, again a kind of sidebar, I-
Liz Fong Jones (20:23):
I kind of have these interesting constraints, right? Like, I have been a user of nothing but Linux for 16 years, so-
POP (20:28):
Got it.
Liz Fong Jones (20:28):
It really constrains which MMOs you can play, like you can only play ones that either natively work with Linux, or work with [inaudible 00:20:34]. Um, it's a, it's a pretty narrow set.
POP (20:36):
Got it. And so, and so you play Worl-and so I played, like, I said, World of Warcraft, and I played City of Heroes and City of Villains, I met, I actually met in real life a lot of the folks that I played with. And it was like-
Liz Fong Jones (20:45):
Oh yeah, I actually, um, funnily enough, I had the opposite sit-situation with EVE Online. Um, with EVE Online, I started playing EVE Online with a bunch of people that I knew in real life from c-from working with them. And it actually was a huge, huge, huge advantage. Uh, EVE Online is a game where you can, where you can betray, uh, anyone in your, in your organization if you rise to a position of power within that organization. And therefore people are very, very restrictive about, you know, how they promote people, who they hand out roles to. But because we were a group of friends and coworkers in real life, like there were ramifications; if you, if you stole from the group, like, you know, there, people would not, wouldn't want to, uh get lunch with you anymore, right? So it's kind of this environment where we could trust each other to a much higher degree and that was a strategic advantage because we could, you know, be a lot more nimble. We could give people the accesses they needed to do things, and not trust, uh, and not have to worry about them backstabbing us.
POP (21:39):
Our guild was always the same way, it always, we would have like, and, and same thing with World of Warcraft, its like, okay, somebody knows how to craft these things, it's like you have at least the start off with, uh, a good set of armor for you to get from 10 to 20, to do all the dungeons... And you had a group to do dungeons and raids with, it was like, okay great, you know? Um, for the-
Liz Fong Jones (21:55):
Yeah it's kind of interesting, right? Like, you know some games take the approach of you know, if, if you're, if you're building gets stolen from, then, then they'll ban the, the thef-the thief. In other games it's like, you know what, like, if you get stolen from, it's your own fault for trusting that person, right? And those are two different views of the MMO sandbox.
POP (22:13):
No doubt. So for those who, who are uninitiated to EVE, let's talk about EVE a little bit; talk about the game, and what do you love about it?
Liz Fong Jones (22:20):
Yeah, EVE Online, so it is a game that is, what I would describe as libertarianism simulator.
POP (22:27):
(laughs)
Liz Fong Jones (22:27):
Uh, set in space. Uh, the theme of the game is you ha-you are a capsuleer, right? Like, this person who is immortal in the EVE universe, because you're conscious can get downloaded in, into another body if you get killed. Um, and all of the government is, is player run. Right? Like all of the control over individual start systems is who has the most power in that system at the moment? Um, all all of those organization are, are run by individual people. Right? Like, and therefore you can have these large organizations that have tens of thousands of people in them who are all contributing towards the, a common goal, or at least pretending to before they backstab someone else.
Liz Fong Jones (23:04):
Um, so it's definitely a fascinating place to learn how to kind of manage people, to corral people, to get them to all work together. Um, so you know, it definitely took some of the mechanics that I loved about the diplomacy and, and politics in Puzzle Pirates, and it definitely turned it up to 11. Uh, definitely my favorite part of Puzzle Pirates was the uh, political puzzle, um and now, you know, now I do a lot of stuff in EVE Online involving again the political puzzle; helping you know, establish relations with humanities that don't trust each other, helping kind of figure out what are the right government strategies to keep the organization healthy?
Liz Fong Jones (23:43):
And turns out a lot of those things apply to management too. Um, one of the stories I like to tell is, I had to fire my previous boss in EVE Online, and I would never you know, in, in probably even 20 years of, of work experience in a tech company, have to deal with that situation, but I'm, I, uh, much, much more well developed manager as a result, uh, as a result of having that experience with EVE Online.
POP (24:09):
Fantastic. Can we talk about of s-Of Sound Mind? Can you kind of, uh-
Liz Fong Jones (24:13):
Yeah.
POP (24:13):
Illuminate for the audience?
Liz Fong Jones (24:15):
[crosstalk 00:24:15] So, Of Sound Mind is the EVE alliance that I, uh, played a role in over the past seven years or so. Um, it is a organization that... And like many other EVE Online organization, has a code of conduct, right? Like in the tech world, we're familiar with [inaudible 00:24:31] having codes of conduct. That's not true of a lot of EVE Online [inaudible 00:24:35]. Uh, a lot of EVE Online alliances are kind of anything goes, or you know, they're very tolerant of specific kinds of behavior that you wouldn't want in a professional setting. But in our EVE Online group, we're a, we're a professional community, right? Like, we're, we started off as a kind of alliance of a group of people who are very very focused on new player experience, and a group [inaudible 00:24:55] who are very, very focused on kind of this, we're a bunch of coworkers who like to play it, play online together. So that kind of combination has lead us to this kind of very professional culture which I really, really like.
Liz Fong Jones (25:07):
Um, so we've played in basically all of the various uh situations of all the various, uh, uh, gameplay styles that EVE has. Uh, you know, ranging from uh high security space to the wild, wild west [inaudible 00:25:22] space politics, um, and now we're currently in Faction Warfare, which is kind of a intermediate ground between those two things. Um, but kind of, the specific kind of play doesn't actually matter, right? Like, what matters is building that community and kind of having that sense of cohesiveness to us.
POP (25:39):
Awesome. And again, I'll put links to it, 'cause I, you know, I read uh, you know, your kind of mission and, and I thought it was very cool. So I'll put a link to that.
Liz Fong Jones (25:45):
Yeah. Yeah. Goodness, we're going to have to watch out for spies applying [inaudible 00:25:50]. (laughs)
POP (25:51):
So, the last question I ask everyone in the Popcast is uh, work, what work are you most proud of?
Liz Fong Jones (25:57):
I think the work that I'm most proud of is my work with organizing the tech industry. Um, you know, I can't say that [inaudible 00:26:04], that unilaterally, um, or single handedly, but I definitely was involved in a lot of the ethical discussions around, uh, around product decisions Google was making, labor conditions that Google was imposing upon it's employees and subcontractors. And that work had really, really blown up over the past couple of years. Um, I've been at it since 2010, uh in terms of trying to organization the labor industry, and it's been amazing seeing the wave of worker protest against unethical government contracts, against sexual harassment of employees, like, and against mistreatment of contractors. Like, I-I'm so glad to see that finally, you know, people waking up and paying attention to it.
POP (26:45):
Thank you so much for being on the Popcast, you know, again, you are a hero of mine, you know that. And um, I really appreciate you taking the time out of your schedule to uh, talk about MMOs and all the fun stuff that we talked about, about SRE and DevOps, so I appreciate it, thank you so much.
Liz Fong Jones (27:02):
You're very welcome, take care. Stay safe.