Born To Run: All Things Kubernetes Adoption

In this episode, co-host Joel talks with Randy Abernethy,  Managing Partner at cloud-native consulting company RX-M,  about cloud adoption and, more specifically, the value (and struggles) of Kubernetes.

To keep the conversation going, join our Slack community and follow along on Twitter

Connect with Randy on LinkedIn or pop him a message at


Joel Parks:


Hello and welcome to Cloud Unplugged the show where we take a lighthearted but honest look at the who, what, when, where, why, how and WTF of cloud adoption. In today’s episode, we have industry veteran and all around good guy Randy Abernethy, to talk about cloud, Kubernetes adoption, application architecture and all sorts of other goodness, this interview covers a lot of territory and I think you’re gonna like it. So with that, I will get out of the way we can get straight to the interview. So I am very privileged to be joined on the podcast today by Randy Abernethy. Randy, would you like to say hello?Randy Abernethy:


Hey, how’s it going?Joel Parks:


So Randy, why don’t you give the listeners a little bit of an intro or bio on yourself, you know, let them know who they’re talking to.Randy Abernethy:


All right, happily. So I’m a managing partner at RX-M. And we are a cloud native training and consulting shop, we spend pretty much all of our, our time knee deep in Kubernetes, and various other related projects and because we are super geeks, we don’t satisfy ourselves with just building things, we love to just tinker and explore and probe the depths and then kind of develop training and curriculum around all of the nuanced things that we discover. We were also involved in all of the certification developments. So the Certified Kubernetes Administrator, application developer, security specialist, and now this new one, the, the Kubernetes, and cloud native associate, we’ve kind of been helping out the community build all those and so we offer, you know, courses that prepare people for those and also a lot of like, kind of custom solutions for companies that are making the transition to cloud native. So you know, we can kind of help them with the consulting side. And and also, you know, build a training regime that’s going to work perfectly for the tools they’re using.Joel Parks:


Fantastic. So yeah, all right. So you’re, you’re hitting on the Kubernetes thing, which is probably my favourite slash most frustrating thing to talk about. Because it’s, it’s such a, it’s such a beast of a platform and it can do amazing things. But everyone seems to get hung with Kubernetes in one way or another. It’s either at the beginning with training, which I would I want to touch on, on a deeper level here in a second, or it’s, it’s down the road with scaling their their operational muscles around Kubernetes. From a training perspective, I’m fascinated to know when when you are getting students coming in to learn and to upskill on Kubernetes, do you do you find that they come from consistent backgrounds? Are they kind of all over the map? I guess, you know, where’s the starting point? For most of them?Randy Abernethy:


That’s a great question. And I think there’s two different vectors. There are people who, as individuals want to know what’s up with the cutting edge stuff, and they want to beef up their resume and things like that. And so those those folks are just in it for the, you know, for the the knowledge and the cachet of having a certification or what have you. But then there are also companies and people who have, you know, sort of vested interest in the success of the business or, or the, you know, some business leadership saying we want to, we want to help turn the boat. And I think, some, in my opinion, very smart CTOs or CIOs will lead a significant transformation with training. Because if you ask an engineer to solve a problem, to do something, they’re going to use tools they know that they trust that they’ve worked with before. Yeah, absolutely. So yeah, if you’re if so if you’re in a leadership position, you want to get your people to use something different. Yet, you need to make sure that they know and trust and understand it, if you want them to actually use it, otherwise, they’re just gonna keep falling back into their old practices.Joel Parks:


100%.Randy Abernethy:


So if, if you can kind of get people comfortable with that tool and, and familiar with it, and get them to the point where they trust it, and they see the value, right? Show them the value proposition, then they’ll just go there naturally. And it you know, you got to realise that your engineers are making hundreds of decisions developers, you know, operations people, hundreds of decisions every day. And they’re making those decisions based on the best information that they’ve got and if you train them around the tools that you want to use and adopt and go forward with, not only are they going to to more likely use those, but they’re going to use them effectively. And you won’t find yourself having to back out of a blind alley that you got down because people didn’t really understand what they were doing. And I think it’s, you know, it seems to me kind of a shame. how often that happens, how often people charge into something. Without enough, yeah, preparation, you know.Joel Parks:


Oh, Kubernetes, especially because if you rewind a few years, it was the new hotness, it was the shiny object that everyone was sort of gravitating towards, without, I think, in a lot of cases, a clear sense of the huge paradigm shift that it required, both in the way that they operate, and also the way that they’re developing and shaping their applications. I don’t, I don’t think most companies truly appreciated how big of a change it was going to be versus what they had been doing before, which is primarily just like, sort of VM management.Randy Abernethy:


You said that Brother, it is a completely alien landscape, to a lot of organisations, you know, and I think, you know, I look back at Kubernetes 1.0. 1.0 Should be, right, that’s, that’s, yeah, we can build things on. No, no, Kubernetes probably 1.7 was where we started, you know, is suggesting to some customers that it might be good to use for certain types of things. You know, unless you were a martyr, or, you know, really had already that kind of stuff happening in your world.Joel Parks:


Or crazy, like me and John, were we jumped in hardcore at 1.2.Randy Abernethy:


There you go. And, you know, in some companies are, you know, that’s their job, right, is to get there before everybody else, so they can help the rest of the world and, you know, so that makes sense. But for your, for your typical enterprise, it’s just that this, none of this stuff matters to them. They’re just trying to build software, you know, you’re looking at where we’re on version 1.21 and so a third of the history of Kubernetes was probably not really even, you know, of a calibre or maturity that you would be wanting to useJoel Parks:


It was an interesting science project for it. a long time.Randy Abernethy:


Yeah. Things like, you know, know, our back. So, whatever, whatever, 1.5 or something. So, yeah, so it’s been, it’s, it’s been, it’s amazing how fast it’s gone from nothing to this gigantic monster that is so sophisticated, so capable, so able to operate anything in the cloud native sphere, you know, is pretty impressive as a platform.Joel Parks:


Yeah, completely agree. So from a, from a student perspective, when, you know, they’re they’re entering, entering into this world of, uh, what are what are, if you had to rank it, you know, what is maybe like the top two or three challenges that almost everybody doesn’t quite grasp at the beginning?Randy Abernethy:


Well, you know, from an enterprise standpoint, the two things we see that are attributable to failure, most often. Are number one, people don’t understand how to do state, in this cloud native world. Yeah. Because, you know, they’re the old model of just put everything in a relational database and, you know, let Larry Ellison worry about it. That model doesn’t work, right? It doesn’t scale, it doesn’t, you know, it’s, it’s it, you have to completely change the way you do things. And you have to adopt a scalable, cluster oriented kind of view of storage and of state. And then you have to understand how the micro service model works. Because a stateful service is not a micro service, micro services are stateless, by definition. And that’s sort of a mind bender for a lot of people. And so you end up with these, a lot of times weird, funky, hybrid services that have some state and because they do that it doesn’t scale like it’s supposed to. And Kubernetes thinks it’s stateless. And so it just tries to treat it that way. And things blow up. Yeah, that’s a bad one. The other one is culture. And I think that you have some businesses where you’ve got, you know, all these cascading management structures built around the old monolithic world where, you know, this is development, this is QA, this is, you know, the people who own the data, and it’s all these five things in vertical slices of the puzzle. And when you try to get these cross functional, you know, teams so that they can iterate really fast and own their own services and do all of that, that changing the organisation and its behaviour, that sometimes there’s no political will to do it. I kind of relate it to this whole you know, climate change scenario, everybody agrees that it’s a good idea. But then as soon as it comes out of my pocket, forget it, you know? And nothing happens.Joel Parks:


You mean, I have to change? Oh, yeah, that’s a different story.Randy Abernethy:


I have to suffer during this. Wait a minute, you know? Yeah.Joel Parks:


Yeah, I’ve told this story on the podcast before, but you hit the nail on the head with culture, because I’ve had so many bizarre conversations with with customers that are sort of somewhere along the adoption process for Kubernetes, where they just aren’t catching that, actually, their process and the way that they’re approaching it is actually what’s wrong. Right, the technology does exactly Kubernetes does exactly what it was designed to do. What how you’re trying to use it is wrong. Right. Case in point is I had a conversation with an IT service management director, who ran like the ITSM. Team. And he was like, alright, so show me how we can get the container IDs when they’re scheduled into our CMDB. And I was like, whoa, back up a second. Why? Like, what what, what are you trying to do? It’s like, well, we have a process, we have to track everything. You know, in the CMDB, I’m like, you realise that those are going to get created and destroyed at lightning speed? Like, there’s like, there’s no way that that’s going to be a viable path forward. And he just fought me tooth and nail for a while. over that. But yeah, I think you’re you’re exactly right. Like the the culture the the process has to has to adapt at the rate that the technology is otherwise it becomes unusable.Randy Abernethy:


Yeah, you have to, you have to retool, you know, at a pretty deep level into the organisation to, to really get the advantages and you can leg into it, you know, you can do things incrementally, and there’s nothing wrong with that. But you need to see the end of the road and keep on trucking to get there if you really want the benefits, if you really want to get the advantages.Joel Parks:


Yeah, absolutely agreed. So, I mean, those are sort of two very leading or strong indicators of failure, I think you you touched on one strong indicator of success before which is having, you know, very strong positive leadership from, you know, the CTO down into the organisation to guide and lead resources towards, you know, the the healthy ways of working, you know, the the transformational aspects, are there other leading indicators of success that you’ve seen?Randy Abernethy:


Hmm, that’s a good question. I think when when the when the leadership that that you’re interacting with, sort of show up as mentally flexible, you know, that, that’s helpful, because I believe that every organisation is a little bit different and because the, the way that you do things, you know, that that’s the interesting thing about Kubernetes, is it wants you to, you know, behave a certain way and organise your applications a certain way. But there’s a lot of sort of freedom and latitude around that. And I think that, you know, being able to, you know, to adopt that, that sort of flexibility and find the patterns that work for your organisation, I think, is really important. For example, like, you know, you, I think it’s important, if you’re using the cloud at all, right? You you want to create tags that you can use to track things and identify, you know, how critical is this particular thing? What team owns this particular thing? How do I charge you know, you know, different, you know, cross accounting types of functions for resource use, and that sort of stuff. And in a dynamic environment, you know, that the cloud introduced a lot of people to that, and when you go to Kubernetes, it just jumps up another level, you know, the, the dynamic nature of things. And, you know, that, that, that, that offers you a lot of opportunities to do some pretty exciting things and from a cost benefit standpoint, you know, your server density could really, really skyrocket it, especially, you know, for folks who have a lot of existing infrastructure, you can do a lot more with what you’ve got with these kinds of tools, but you just have to be careful about all of the, you know, slippery slopes where things, you know, Hey, why are these 17 things, you know, up and running here for the past, you know, nine months that nobody owns or knows what they’re doing? All that sort of stuff? Right?Joel Parks:


So, I guess the question that I’ve got is, so you’ve got people coming in, they’re getting educated, they’re, you know, making sure that they’re able to to absorb these new ways of working, really understand what’s going to be required of them to to operate Kubernetes at scale, how many of your customers or your trainees would you say, are still looking at Kubernetes as an on prem technology versus, you know, utilising a cloud managed service of some kind?Randy Abernethy:


That’s a great question. I think that because we’ve got such, you know, I think, good maturity, with a lot of the managed services, most enterprises are going to go that route. I do think there’s a lot of on prem Kubernetes clusters, cropping up all over the place, because people, you know, maybe the first movers would use their own thing that they would stand up in bare metal, or in, you know, a cloud. But then second movers would maybe use, you know, your EKS, your AKS, your GKE, and things like that. But now we’re getting to the point where the the managed service providers are strong enough, that you can, you can just, you should, you can just go that route, even with your own dedicated infrastructure. I mean, there’s, there’s a lot of organisations that are, you know, that have huge bare metal environments, and they’ve been managing them with, you know, tool XYZ, whether it’s VMware, OpenStack, or some other homegrown thing. Kubernetes is a is a nice option in those scenarios, and you can get it, you know, at a very, very low ask, because of the the managed service providers. But I think the mistake is to assume that therefore, you don’t need to know Kubernetes, then you’re going to fall down and hurt yourself.Joel Parks:


Yeah, absolutely. Yeah. It’s, John and I have talked about this before that those managed services are great, because they abstract away some of the problem. But it’s, I think, I think there are a subset of customers that look at them and say, oh, cool, they made kid proofed Kubernetes and that is flatly not the case. It’s it still does require you to understand, you know, the basic operational constructs of Kubernetes. And how to diagnose problems when they go when they present themselves, whether it within your app or within Kubernetes proper. The, I guess, where so if most, if most people are kind of, especially those second movers, the ones that didn’t have an investment in their own tooling, or their own on prem? Implementation, if majority of those are gravitating towards the the managed services? Are there, does it change from a from a training point of view? How you approach educating someone about Kubernetes? You know, whether they’re going to be doing it on prem or using a managed services, does that present a fork in the training paths?Randy Abernethy:


I think there’s a natural fork, you know, kind of anyway, in that, if you’re an operator, if you’re running Kubernetes, then of course, you know, you need to know a lot of the CRI. And, you know, CNI and CSI and all that kind of wiring. And then, you know, if you’re using if you’re like, the, the lowest ask is I’m using, you know, I’m using GKE, I go into a dashboard, set some things that I need to understand, yes, but then I click the button, and Google does all of it. And, you know, storage just works. And my, you know, secrets are encrypted at rest, and not, you know, ingresses, just working and all these other things, and I didn’t even I don’t even own anything, and it’s just all there. Then when you go to on prem, and you have a managed provider, now, all of a sudden, it’s like, well, these are our machines and they have a physical position in the world and they’re connected by equipment that we own. And, you know, and you start getting into, well, how are we going to integrate, you know, authentication, and how are we going to, you know, do the networking stuff and the ingress and, you know, in, oh, well, we have these requirements inside our company, and it’s inside our walls, and it has to fit with all this stuff. And so I think that there’s still a lot of people who need to understand Kubernetes like an operator, even if they’re not operating it the network guys, the the people who are responsible for the machines, you know, that sort of stuff. And then then the the developers no matter where it goes, they need to understand Kubernetes too, because, you know, first of all, they have to understand cloud native so that they get the, you know, stateless service and storage and solutions like that. Right. Then they also have to understand Oh, you know, well be Things need access from the outside, they have to understand what ingress is, and whether they should use that or a load balancer service. And they need to know, you know, not to use daemon sets and you know when to use a stateful set and when to use a deployment and whether maybe they don’t want to use deployments at all, and they want to use a CD tool that does the deployments for them. And, you know, there’s just a whole lot of, you know, questions that a managed platform doesn’t answer, you know, the icing still needs to be put on the cake. And a lot of the people specifying those requirements are the developers.Joel Parks:


Yeah, so let’s let’s pick on the app devs, a little bit for a minute. So in your experience, obviously, they do have, if they’ve got legacy applications, in all likelihood, those legacy applications are going to need refactoring. In some shape. Yeah, what I guess, how do you find approaching? Or how do you approach those discussions with with App devs, in order to get them from a legacy application architecture way of thinking to more of a cloud native 12 factory way of thinking?Randy Abernethy:


So I think, you know, I’m a big fan of take the vehicle that you just built from scratch out onto a road and go really slow with it and see what happens. Look at the street is there oil, and you know, things like that. And then if everything’s working well step on the gas and go faster and faster and faster, progressively. Right, but that initial, get the first thing kind of dialled in, right, I think is a great learning experience. And you want the smallest possible, you know, ramifications from the iterations of getting that wrong until you actually dial it in. So I think, you know, starting with a small piece of that monolith saying, what is what is the thing that because there’s two things that you would maybe care about, you know, peeling out of that architecture first number one is, what’s the thing that customers care about the most, that we’re that we need to iterate on, and that we need to improve? Right, so something that you want to have fast cycle times with so that you can, you know, unlock, you know, innovation, that would be the first thing I would look for. And then you know, sometimes if there’s a if there’s a scale issue, I would say, then look for the thing that needs to scale, where you want elastic scalability, and, you know, cloud scale type of stuff. So those are the two kinds of services that I I would say, let’s, let’s use a brownfield migration approach where we leave the monolith in place, we peel out one service, and then another, and then another, then we just put stubs in, in place of them. And this works really nice with GRPC. Or with Thrift, if you want to have like an RPC kind of a, you know, model for breaking those services out. Or you could do, you know, a RESTful API, but you know, the monolith isn’t using get post and put verbs internally on resources. It’s, it’s making function calls, so so so RPC works pretty nice for that. And so once you peel out those first, that first service, and you get the hang of it, the learning there, there’s so much learning there, that then you can make a lot better and more informed decisions about the next step and the next step. And so I really, like, you know, that sort of approach. And what I find personally, I’m a big fan of, if it ain’t broke, don’t fix it. And so if you’re brownfield monolith, you know, after you pulled out all the important parts that you really want to focus on and develop, if there’s a big chunk of that thing still laying around, that does whatever it does, and everybody’s happy with it, I wouldn’t convert it, you know, I would just leave it. You know, unless there’s some reason to do something with it, you know, just leave that legacy piece doing its legacy thing and focus on the new things that are going to evolve and grow.Joel Parks:


Yeah, I would absolutely concur with that. I think that’s a that’s a really good approach. I mean, either looking for, you know, the place where you need velocity, or the place where you need scale. And sometimes those are related, right. You know, it’s sometimes sometimes they have more than more than one consideration, do you find that, you know, the commonly modernised components of those, those legacy stacks tend to fall into one architectural layer or another? Meaning like, does it kind of tend towards UI or front end stuff? Or does it typically exist in like, middle tier sort of business logic? Where whereas it just all over the place,Randy Abernethy:


I think, yeah, sadly, it’s usually a slice of that, you know, hamburger from from, you know, the top all the way down because you, you, when you start going to the microservices, you start focusing on business functions. And now the sudden you get a little piece of presentation, a little bit of business logic and some of the data, you know, the high velocity data, and so you end up turning that into a service that owns some state moving that out and, yeah, it’s, it’s not always as simple, you know, as it as it seems, when you start, which is a good reason to start small, you know?Joel Parks:


Yeah, so you touched on each, we’ve touched on state a few times now, you know, when you when you start to slice through that, that hamburger, and you get to data, my experience, and maybe maybe that my experience is just sort of unique here. But what I’ve seen in the past is, a lot of times, when you start trying to pry the data structures apart, it becomes a real nightmare. Because a lot of times they’ve been commingled, they’re not well understood that, you know, there, they could be sitting behind stored procedures that, you know, nobody’s quite exactly sure what’s going on. Is that a pattern that you’ve seen? Or am I just working with incredibly backwards customers?Randy Abernethy:


No, no, that’s a, it’s a very common, and it is a difficult thing to manage and I think, you know, in a perfect world, you want, you want all your state to be immutable. And, you know, you want to append only log type of activity, because that’s the only thing that you can really, you know, ultimately rationalise it at scale, if you want to scale out rights. And, you know, you need to be able to think about, you know, partitioning, you know, from a right scale out standpoint, and the, the idea of replicating data, especially when it’s immutable, that’s, you know, that’s, that’s a cheap and easy way to get scale. But the, it’s, everything I just said is completely opposite. From the way that you know, things are, are done in a, you know, kind of a traditional environment with a relational database where, you know, you’ve got all these, you’ve normalised the data to the point where it satisfies every possible application. And, you know, that’s, that’s an anti pattern in cloud native, and, you know, you you, yeah, you want to save the data, that way, you use it in a cloud native environment, so that you get speed and, you know, simplicity versus that that whole, you know, other thing. So, it can be tough, and what a lot of times, I think is a good way to handle things, is to leave the old thing in place. So that you have the surety of the data model in that old world, and you create a new thing that has, you know, sort of the go forward solution and it statewise and the new thing, you know, using some mechanism like a Kafka or something like that is going to produce state into the old back end, so that it can keep functioning the way it was. And as the as you lean less on the back end, the old the old back end, more of the activity moves to the live set of data that’s being produced by the more microservices side of the world. And you get to a point where you can start making transitions and retire things. I think trying to retire. Long standing solutions too soon, can be disastrous. You want to create a parallel universe as much as possible for the microservices. That you know.Joel Parks:


Yeah. Otherwise, you’re just asking for there to be a pain, heartache, explosions, lost weekend’s right. It’s, yeah, that’s that’s a bad yeah, that’s a bad way to fly.Randy Abernethy:


You step on one of those landmines. And everybody. Yeah, everybody doesn’t want to do this anymore.Joel Parks:


Yeah, yeah, exactly. Yeah. I mean, it That’s exactly right. I mean, you talk about really sort of rewinding the clock on any sort of cultural transformation, you’ve been able to affect, you know, if you have a series of those, those painful events in a row, you know, you can really demoralise the entire team or cause them to, you know, participate in the great resignation, as they’re, as they’re calling it.Randy Abernethy:


Right, right.Joel Parks:


I want to go back to something you said a minute ago about the the the set of tools that surrounds Kubernetes, you know, sort of the ICA this set the ecosystem of tools that you really need in order for it to be a viable platform that can be, you know, operated by the operators, but also consumed by the developers. We have sort of this internal discussion at work about the the CNCF landscape, which is an attempt to, you know, sort of talk about the different tool classes, and you know, the different things that are that, you know, are represented around, you know, this this ecosystem. And I’m curious to see where your head’s at, because there is one camp that that thinks that it’s, it’s become unreadable to a degree, and it has kind of lost the intent of what it was intended to convey. Another group is like, No, this is really good at conveying the the expansive nature of the the ecosystem. I’m curious if you have thoughts on this.Randy Abernethy:


You know, I think a lot of people have a problem with it, because they grew up with it. They saw it, when in the beginning, you could understand the whole thing, and it all made sense, you know, but the way I see it is it’s more like a dictionary, right? Yeah, read a dictionary, you look stuff up in the dictionary. And the the landscapes really useful from that standpoint, because it does capture a lot of information. And it lets you, if somebody you know, if you’re on a call, and somebody mentioned some project, you’re like, Well, let me go look that up in the landscape real quick, you can see, you know, what kind of category it fits into. And obviously, some things, you know, are hard to categorise, but at least it makes an attempt, and some things are easy to categorise, so at least you can get the category of thing that it is. And maybe that’s all you need. To just know, what is this thing? Is it a storage thingy? Or is it a database thingy? Or is it a container runtime? What is it? And then if you need to know more, you know, you can find the URL and all sorts of good stuff in there to, you know, point you at more information. So I like it. I understand why, you know, it’s definitely an eye chart. But if you’re gonna look stuff up, it’s not a bad way to go.Joel Parks:


Yeah, I like the description of it being a dictionary, we, you know, we have some some salespeople internally that get hives every time they look at it. Because I think from their, perspective on it is, you know, oh my gosh, this, you know, this, is our competition. I’ve tried to, you know, maybe reset the expectation here of oh, not really, you know, there’s a lot of stuff on that map that is unrelated to what we specifically do. But I mean, it’s, it’s, it’s good to know, do you think that that most people coming to the subject of either Kubernetes or cloud native for the first time understand the context of the different classes in the, in the landscape? Or does that come with more exposure to the platforms in the tools?Randy Abernethy:


Maybe not. Yeah, I think that, that, that comes after time, you know, the Kubernetes is like a crate motor. You know, at a minimum, you need, you know, a transmission and drive train some axles, tires, and, uh, you know, some some other stuff before you can actually drive the car. And, you know, and then if you get want to get fancy, you can, you know, start doing all sorts of other things and really take it down the road. And as you as you learn, you know, what’s necessary to actually have a usable platform, you start understanding the different pieces of the puzzle. Yeah, that makes guidance there to I guess, to write these different categories of things that are that exist? Well, yeah,Joel Parks:


I think it’s probably a situation of you don’t know what you don’t know. So, you know, you may see a category or a class on the the landscape and you’re like, that’s interesting. I’m not exactly sure what that is. And then with some experience, and some time, it’s like, oh, that’s the thing that I need. You know, you didn’t necessarily know it going in. But as you get more experienced with it, you you start bumping your head on the problems that have already been solved, to a degree, or at least have there’s there’s attempts at solving it. Well, Randy, it has been fantastic talking with you. How can people get in touch with you?Randy Abernethy:


A bunch of different ways @RandyAbernethy on Twitter, and anyone is welcome to reach out through RX-M and my you could you know, always I mean, I’m happy enough to give out my email address. It’s Parks:


Well, fantastic. Randy, thank you so much for joining us. It’s been a pleasure speaking with you.Randy Abernethy:


Super awesome.Joel Parks:


Well, that was a fantastic interview. Thanks again to Randy for participating. That was a lot of fun. And I hope that you enjoyed it. As always, please rate and review us on your favourite podcast app, or you can tweet us @cloud_unplugged. Emailing us also works and you can also find us on youtube under Cloud Unplugged for episodes, transcripts and bonus content. As always, thanks for listening, and we’ll see you next time.