[{"content":"This story serves as an example of how leadership can come from unexpected places, and often we get better results by leading from the front instead of pushing from behind.\nMoving the Cow In the spring, we had an older cow that was set to return to my dad\u0026rsquo;s farm. She had been living at my house for about a year, hoping she was with calf. Unfortunately, she was not, and it was time for her to return to her home.\nThe situation was, I had to get the cow from inside my barn, through a fairly narrow door, and into my stock trailer. There are a few truths about cows. First off, there is not any point in my life where, if a cow didn\u0026rsquo;t want to move, I use brute force to move it. The other truth is that cows are not overly inclined to go through tight spaces and into enclosed trailers.\nThe Stubborn Cow\nTypically, the plan is to get them moving through the narrow passages and into the trailer before they realize where they\u0026rsquo;ve gone. If you can get them into the trailer before they realize they\u0026rsquo;ve moved, and then close the door, you\u0026rsquo;re set. That was my plan, I\u0026rsquo;ve done it before, and I\u0026rsquo;m ready to do it again.\nThe big day arrives, and I had my trailer set up, my pathway laid out, and the cow is in the barn. I follow my original plan, open the door, and get behind the cow to shoo it down the pathway I had laid out.\nAs planned, the cow moves through the door and starts into the trailer. I am feeling good, I laid out the pathway and the cow is following it just as I planned. However, not planned, as the cow gets closet to my planned destination, she trips walking up the ramp. This immediately change the calculus, and the abrupt introduction of a challenging situation and the discomfort with the new environment caused the cow to now resist the plan. She immediately backs up, straight back through the door and back into the barn when we had begin. Now, the element of surprise is gone, and it\u0026rsquo;s just me, a 1,400lb cow, and a narrow door that the cow has no interest in trying to go through again. The pathway had been built, obstacles had been cleared, but unexpected challenges led to opposition.\nThe Destination\nYou Can\u0026rsquo;t Push A Cow At this point, I am trying every trick in the book to get movement from the cow. I\u0026rsquo;ve tried to entice the destination by placing feed in the trailer, showing a reward for the cow going through with my plan. I\u0026rsquo;ve gotten behind the cow, applying force trying to push it towards that destination, and received nothing but an odd glance back and a snort. I\u0026rsquo;ve taken a gate, closed off all other avenues, hoping that showing this as the only possible destination will entice the cow, and that doesn\u0026rsquo;t help to provide any encouragement.\nAt this point, I\u0026rsquo;m exhausted, the cow has become stressed and more dug in, and we\u0026rsquo;re both perplexed. I\u0026rsquo;ve tried everything I know of to force this cow to move forward, and no matter how much I push, no matter how much I apply pressure, and how much I put treats into the end point destination, there is no movement.\nThe Hero\nI stood there, arms on the fence, sighing and wrapping my head around the situation. I am looking at the cow, staring at the door, and an interesting thing happens. Out the door, I see Mee, one of our two barn cats, walk down the gate, hop down into the pathway I had built, and walk straight up and into the stock trailer. My exhaustion and frustration grow, immediately I think \u0026ldquo;Now I have two problems\u0026rdquo;. The cat will likely scare the cow more, or if I get the cow up there, the chances are the cow gets stepped on. At best, I have a cow and a cat locked in my trailer. This is not what I needed, and now I have two problems.\nThis was not my plan. I had planned to have the cow move in that direction, and now this cat has walked down the path I laid out, and is where I think he shouldn\u0026rsquo;t be.\nLeading Through Curiosity Mee, brother to Ow (Mee and Ow) starts sniffing around the cow feed that I had dumped in there previously. At this time, the darndest thing happens. The cow doesn\u0026rsquo;t get more stressed, but instead gets curious. Interested in having seen the cat walk down the pathway I had built. See that the journey was safe, and the reward at the end was achievable. The cow starts sniffing toward Mee, the cat, and takes a forward. Slowly, ever so slowly, she takes another step, and then another.\nBefore I knew it, the cow has walked out the door. Still seeing Mee in the trailer, she continues to let curious lead her, and following the cat into the trailer.\nI stood there dumbstruck. The pathway to the destination didn\u0026rsquo;t change, the rewards didn\u0026rsquo;t change. However, by taking those first steps himself, by showing the way instead of just pointing it out, Mee had created an environment where teh cow willingly went to the exact location I wanted it to.\nAfter what felt like a minute, but was really a few seconds, I snapped out of it and snuck over to shut the door on the trailer. Mee, being the absolute greatest barn cat imaginable, immediately jumps out the window of the trailer and back to the cat life of naps and mouse hunting. Just like that, once the cow was placed where we needed, Mee moved onto the other things on his schedule, and the farm had solved a problem that I couldn\u0026rsquo;t force through from the back.\nShowing The Way\nYou Can\u0026rsquo;t Push Engineers, You Must Lead Them The takeaway here is that your engineers are much more likely to follow if you lead from the front. How often have we seen team leads take the approach of trying to force change? Demanding new adoption, dictating changes to workflows through the power of discomfort and threats. This regularly leads to unhappy team members, folks digging in their heels, and a general lack of enthusiasm about adoption.\nI tried as hard as I could to force the change by applying force and discomfort to move that cow, and it just led her to dig in her hooves. However, Mee took a different approach. One that led from the front, showed the way and presented it as a safe passage. He walked the pathway first, he led the way to the destination. Instead of force, he led with curiosity, and this spread and ultimately enabled the cow to buy-in and move to the preferred destination.\nYou are much more effective as a Platform Lead, a Manager, a Leader in general, when walk the path first yourself. You learn the tool, you develop the skills, you do the hard work of building the workflow ahead of your teams. You bring back your experiences and expertise, and build the understanding and curiosity with your engineers about where the team needs to go. You show the way and clear the path for your team to follow. When leading your team, be the cat, not the farmer.\n","permalink":"https://pigs-pods.com/posts/leading-from-front/","summary":"\u003cp\u003eThis story serves as an example of how leadership can come from unexpected places, and often we get better results by leading from the front instead of pushing from behind.\u003c/p\u003e\n\u003ch2 id=\"moving-the-cow\"\u003eMoving the Cow\u003c/h2\u003e\n\u003cp\u003eIn the spring, we had an older cow that was set to return to my dad\u0026rsquo;s farm. She had been living at my house for about a year, hoping she was with calf. Unfortunately, she was not, and it was time for her to return to her home.\u003c/p\u003e","title":"Leading With Curiosity"},{"content":"This story dives into the hidden cost and lessons learned in rushing the move to production, and a failure to invest in the underlying architecture that protects your critical workloads. Through an assumption that the new environment is as good as the old one, I spent an entire summer patching issues, chasing runaway workloads, and making little progress on anything else.\nThe Initial Deployment The thing about goats is, they are the ultimate example of a poorly written application. Left to their own devices, they will go all the places you don\u0026rsquo;t want them, consume all the plants you want protected, and once they find a weakness in your infrastructure, they will continuously exploit that until it is fixed. They are also the ultimate example of Garbage Cleanup. They will eat almost anything, and especially seek out the foliage that other animals won\u0026rsquo;t touch, like multiflora rose, poison ivy, and briar bushes. You can place them into your environment, and they will clear out all the unwanted junk that chokes out the useful plants for your other \u0026ldquo;workloads\u0026rdquo;.\nWe had this exact problem, and needed to clear pasture for our eventual cattle. Perfect use case for goats. On top of that, we had existing infrastructure that we could put them in. A runtime environment that was proven solid, and we knew would be up to the task.\nThe workloads\nIn the exploration of deploying our weed-eating workload, my dad told me about an adage with goats, \u0026ldquo;If It Can Hold Water, It Can Hold A Goat\u0026rdquo;. The meaning of this was, your plan for infrastructure needs to be rock solid. For certain types of animals, you can skimp on the boundaries, because they are easy going. A small pony, you could put up a piece of string and they\u0026rsquo;re likely to stay. We felt confident though, and moved forward with purchasing the goats and placing them in the pen. They were deployed to our preconfigured, robust environment with little concern.\nStarting out, we had a nice little area for our goats, strong fence, and lots for them to eat. The goats did their job, the small field we had for them was cleared, and everyone was happy. However, as time went on, we needed to expand their pen as they were eating their way through the available resources. Much like expanding the resource limits for a containerized application, we had to provide more room for the goats to continue to eat, and we needed to deploy our newest workloads, cattle, into our freshly cleared field. I had successfully deployed my goats to my pre-existing environment, and was planning to move them to the new environment. I started to think \u0026ldquo;I know more about keeping goats in than my dad!\u0026rdquo;. He may have 40 years of farming experience, but I kept my goats in easy\u0026hellip;\u0026hellip; that was a mistake.\nInvestment, or lack of, in the New Environment As I mentioned, the failure was in not understanding the workload. The goats would not so easily give up their old pen, where they slept every night and where they had eaten for the past year. When evaluating building infrastructure to run our applications, it\u0026rsquo;s not enough to understand the process of building, or the material that is needed. You must also understand the underlying applications, the application needs, schedules, and tendencies.\nToo often, we see Platform Engineers concentrate on the infrastructure, and ignore the application. It is important to remember that the Platform serves the Application, and success of the Platform is directly tied to a successful deployment and execution of the Applications that exist on top of it. I would learn the hard lesson that rushing the infrastructure, and failing to spend the time to plan and build robust guardrails would cost me in both time and money.\nPoor Infrastructure\nSo, in my rush to get the goats into their new environment, I bought the cheap fence, cheap posts, and strung up a fence on the hillside. It took me a few days to get what I thought would be a sufficient solution in place. Unlike their previous environment, this one was built from the ground up, with no pre-existing solutions in place. I believed that because they ran in the old environment, surely this one would work too. I moved the goats from the pre-existing environment to the new one. Two strings of electric fence, some water, surely I\u0026rsquo;ll be good, right?\nIt took about 2 whole hours for my newly deployed workload to escape the boundaries that I had set for it. They were out in my yard before I could go back in for dinner. And once they found that they could exploit the weakness in my infrastructure, it would keep happening\u0026hellip;. over and over and over and over.\nAs the summer went on, they killed my wife\u0026rsquo;s flowers, they pooped on our porch, they slept in our driveway. I woke up in the morning one time to a goat looking through my window at me. The ultimate noisy neighbor, killing the plants, dirtying up the environment, and crashing when they\u0026rsquo;re not supposed to. I had a runaway workload, and no possible way to easily retrofit the environment to counteract the holes they were exploiting. Once an Application is in production, it is much more difficult to make sweeping changes in how those Applications are managed. Deploying sweeping changes to Production comes with friction, the need for change control, the risk of impacting other things in the environment, the cost of thoroughly testing and validating those changes.\nIt became personal. I spent that entire summer trying to patch fence, add electric strands to decrease the exposed fence line, spending hundreds on electric chargers. Adding more fence posts, more strands, more anything just to get this workload under control. Eventually, you couldn\u0026rsquo;t tell if I was trying to stop goats from escaping, or the T-Rex from Jurassic Park. I had completely over-engineered the environment, but to little success.\nThe reason is, I was trying to retrofit a now pre-existing environment for a workload that it was never properly designed to handle. Once you\u0026rsquo;ve opened the environment up, and then try to bolt on additional guardrails because you\u0026rsquo;ve under engineered it, you simply expose the ugly under belly of the solution. My solution was never designed to withstand the requirements that this workload came with. I never understood how stubborn, how creative, and how persistent they would be at testing the edges of my solution. I knew how to build A fence, but I didn\u0026rsquo;t think about the proper fence for what I was going to be placing into this environment. Then, trying to turn that solution into the proper solution without ripping it out just led to disjointed and ill-fitted add-ons that cost money but did nothing for efficiency.\nBad Neighbor\nNow picture this, you\u0026rsquo;re the wife of a farmer, drinking your afternoon coffee. It\u0026rsquo;s 93 degrees outside, and you look in the yard. You see a goat sprinting by, and then your farmer husband sprinting with a net above his head. He catches one, puts it back into the pen, and another escapes behind him.\nEventually, I just gave up, they did what they wanted, and I tried to mitigate the damage. I moved them back to the old environment, we ripped out the entire solution, and I built it properly from scratch.\nNo Kidding, the Bahhhhd Truth The old adage of, you can do it fast, cheap, or good\u0026hellip;. pick 2. I chose fast and cheap, and it ended up being an absolute poor fit that ended up costing me more in the long run. There are many lessons we can take away from this story. Platform Engineering isn\u0026rsquo;t just about building infrastructure. It\u0026rsquo;s about understanding the differences in what you need to provide from environment to environment. What Applications will be running in your Platform, what capabilities do they need, and how much room do they need to expand. My new pen looked great, until I placed a workload in it. Had I been putting a horse in there, it probably would have been sufficient. However, I wasn\u0026rsquo;t, and my failure to take time to understand the workload requirements meant I couldn\u0026rsquo;t account for their tendencies in my initial design.\nThere is a time investment that must occur up front, before any workloads ever get scheduled. Do you have the proper controls in place to enforce standardization, such as Policy Enforcement around workload declaration to enforce resource limits? Do you have proper observability, so that you don\u0026rsquo;t find out your workload is running amok due to your wife finding her tulips as just stumps on the ground? Is the automation in place to seamlessly move that workload from the old environment to the new, to add new workloads in a controlled and standardized way, and to simplify the processes that manage and maintain that environment? You can build a pretty looking environment without much substance, but you will truly test those boundaries when you start putting workloads into it, and any weakness in the architecture is exposed. You\u0026rsquo;ll never know where those workloads may test your infrastructure if you don\u0026rsquo;t take the time to understand their tendencies.\nToo often, leads look to implement the new thing before taking time to prepare the infrastructure and teams building that infrastructure. Whether that is failing to invest in the proper hardware, or failing to provide adequate time to train the engineers who will support the new environment, or even allowing for the process to build the guardrails to provide a stable and consistent environment with which to run. When that happens, the engineers spend all their time chasing errors, runaway processes, and issues in the environment. Add to that the additional money spent on trying to patch a losing solution, all that time wasted trying to find quick fixes to a situation that inevitably would never work at scale. As a leader and decision maker, we must make sure to present the downfall of a lack of investment and preparation for a big move, while making sure to evangelize the long-term benefit in proper investing in planning and execution. We must fight for that investment in time, effort, and hardware. On top of what we\u0026rsquo;re already dealing with, AI workload implementation will challenge our understanding of what is necessary in the infrastructure, and test new boundaries and configuration to expose those weaknesses. Whether that is new scheduling requirements, changes in how we observe and react, or execution concurrency on hardware.\nThe Hidden Cost There are hidden opportunity costs to this lack of investment as well. There is A LOT I can accomplish on this farm. Improving the pig load out area, increasing the number of cattle we have, improving my water distribution system in the barn, or pasture management to improve grass quality. Instead of doing any of those, which would either increase automation to free up time, or improve quality and output from a financial perspective, I was chasing fixes for a solution that was not generating any net benefit at the time. This is the true cost, taking development and engineering time away from new features, and having them directed towards just trying to keep the existing runtimes operational. Lowering the overall speed to market due to rushing a solution to production that wasn\u0026rsquo;t ready.\nHow many of your engineers are sprinting across the yard with a net trying to catch that runaway workload simply because we avoided investing in the time and cost of a properly configured environment? How many developers have given up on solving the problem and are simply just trying to limit the damage? Your infrastructure isn\u0026rsquo;t just an environment, but it\u0026rsquo;s the boundary that keeps the workloads running where they should, when they should, and how they should. The lesson here is, buy the right fence early, build it strong, save yourself the headache, and allow your teams to move on to the next big thing.\nResults of Proper Planning\n","permalink":"https://pigs-pods.com/posts/holding-goats/","summary":"\u003cp\u003eThis story dives into the hidden cost and lessons learned in rushing the move to production, and a failure to invest in the underlying architecture that protects your critical workloads. Through an assumption that the new environment is as good as the old one, I spent an entire summer patching issues, chasing runaway workloads, and making little progress on anything else.\u003c/p\u003e\n\u003ch2 id=\"the-initial-deployment\"\u003eThe Initial Deployment\u003c/h2\u003e\n\u003cp\u003eThe thing about goats is, they are the ultimate example of a poorly written application. Left to their own devices, they will go all the places you don\u0026rsquo;t want them, consume all the plants you want protected, and once they find a weakness in your infrastructure, they will continuously exploit that until it is fixed. They are also the ultimate example of Garbage Cleanup. They will eat almost anything, and especially seek out the foliage that other animals won\u0026rsquo;t touch, like multiflora rose, poison ivy, and briar bushes. You can place them into your environment, and they will clear out all the unwanted junk that chokes out the useful plants for your other \u0026ldquo;workloads\u0026rdquo;.\u003c/p\u003e","title":"The Great Goat Breakout"},{"content":"Pigs and Pods What is this? I\u0026rsquo;ve always wanted to be one of those people that wrote interesting and impactful tech articles. The thought of having something to contribute to the Cloud-Native and Platform Engineering community was always an enticing goal.\nHowever, like many of you, I\u0026rsquo;ve always struggled with believing that I could contribute something novel to the conversation. There are so many talented and insightful viewpoints out there, how can I fit an original thought with all those experts.\nThe thought hit me one day as I cut my 4 acre yard. I have a farm, and I\u0026rsquo;ve often thought about the comparison between what I do on the farm, and what I do professionally at work. And although I don\u0026rsquo;t have the official count, the number of individuals creating OPA Gateway Policies and L7 Ingress Routes in the early afternoon, and shoveling pig poop in the evening is likely very low. And much like I have to clean up after my animals, professionally Platform Engineers are oftentimes cleaning up after bad implementations, poorly written applications, and impacts of limited resources.\nThe farm and Platform Engineering comparison goes very deeply once you take the time to understand both.\nWorkloads\nThe Workload The application is the whole reason to build a Platform. A space to run the application, boundaries to protect the application, and day to day management to keep the application running at optimal performance for optimal output.\nOn the farm, this is the animals, the livestock. We build fence to keep them within the area we\u0026rsquo;ve designated for them. We feed and provide them water, the resources they need to run. We clean up after them, removing the aftereffects of them \u0026ldquo;running\u0026rdquo; on the farm.\nThe Infrastructure\nThe Infrastructure The underlying infrastructure provides the runtime for the workload. The Memory, the CPU, Disk IO and throughput, Networking, and even the software infrastructure like Kubernetes. We build the infrastructure in a way that is optimal for the workload, not too large that we waste resources, but not too small that we can\u0026rsquo;t handle the workloads that we need to run. The infrastructure is an investment, that you get back by running your workloads. The farm itself is all infrastructure. The fence that we build to keep the livestock in the area we designated for them. The feed and water that we provide them so they can continue to grow and operate, much like memory and cpu to an application. The barn to provide security to the animal from the weather, or the electric that keeps your water from freezing at night. These are all costs, that enable you, as a farmer, to maintain your livestock. The Automation\nThe Automation Without Automation, we would be buried in toil. Whether it\u0026rsquo;s Ansible, Terraform, Github Actions, Jenkins, ArgoCD and GitOps, all these tools and methodologies allow us to take manual tasks and complete them in a uniform and efficient pattern. You couldn\u0026rsquo;t scale your platform, you couldn\u0026rsquo;t guarantee consistency, and you couldn\u0026rsquo;t maintain the environment without the usage of these tools. On the farm, this is the equipment. Without a tractor, there would be no way for me to move the payload that I need to move. Cutting hay and feeding it by hand would never let me get beyond a small number of chickens, purchasing the hay baler and hay mower allows me to provide for a much larger number of livestock. Though, I only have one piece of equipment that was manufactured before I was born, it serves the same purpose\u0026hellip;. just requires a whole lot more maintenance, no opensource contributors are remounting the wobble box on my New Holland 477 Haybine. The Engineer\nThe Engineer The engineer is the most critical part. They must be experts in security, networking, automation, policy, and cloud-native technologies. There is always something new to learn, always a new technique or technology. That is what a lot of Platform Engineers love, including myself. The daily challenge, the need to learn and grow, and the feeling of never being bored. The engineer is the farmer. I\u0026rsquo;ve had to learn to read the hay to know when it\u0026rsquo;s ready to bale, to understand the feed makeup for finishing market hogs, learning to weld to fix broken equipment, or understanding how to manage selling product to family, friends, and community members. That same desire to learn and be challenged exists in the farmer, and it\u0026rsquo;s what makes this such a great comparison. For all of those that dream of giving up the IT life to become a goat farmer, this is your chance to see what that is truly like, from one of your own. Read about the challenges that I\u0026rsquo;ve faced on the farm, the lessons learned, and how I connect them to the daily struggle we see in the life of a Platform Engineer. Along with two rambunctious kids, and one wife who, for some reason, tries to adopt anything with wings, this is Pigs and Pods.\nCredit This idea came about one day, riding my tractor after finishing the grass. More than anything, the credit should go to all those that have supported me in my life and career. From my parents, who own a much more impressive farm and have helped me start my own. My co-workers who have mentored me, supported me, and helped me grow as a professional. My friends who laugh when I share my experiences. Most of all though, my loving wife who has supported me all along the way, and I would never be able to do this without her.\nTags List of Tags\n","permalink":"https://pigs-pods.com/posts/pigs-and-pods/","summary":"\u003ch1 id=\"pigs-and-pods\"\u003ePigs and Pods\u003c/h1\u003e\n\u003ch2 id=\"what-is-this\"\u003eWhat is this?\u003c/h2\u003e\n\u003cp\u003eI\u0026rsquo;ve always wanted to be one of those people that wrote interesting and impactful tech articles. The thought of having something to contribute to the Cloud-Native and Platform Engineering community was always an enticing goal.\u003c/p\u003e\n\u003cp\u003eHowever, like many of you, I\u0026rsquo;ve always struggled with believing that I could contribute something novel to the conversation. There are so many talented and insightful viewpoints out there, how can I fit an original thought with all those experts.\u003c/p\u003e","title":"Pigs and Pods"},{"content":"My name is Isaac Omaits, and I am Platform Architect for FedEx in the Edge Compute space. Professionally, I lead through engineering and architecting cloud-native compute systems with the many constraints that exist on the Edge. A deep knowledge in Kubernetes, Cloud-Native technology, Platform and Application Engineering practices allows me to build efficient and effective systems for a life. I have been extremely blessed professionally with opportunities to grow and learn, and believe that we owe it those that taught us to pay it forward.\nIsaac Omaits\nIn my personal life, I grew up on a 70 acre farm, raising cattle, pigs, and chickens. As I became an adult, I looked back on my childhood, and realized that the success I have had professionally came from a lot of the lessons I learned growing up on the farm. I am the father of 2, and I wanted to provide that same opportunity to my children. Plus, I didn\u0026rsquo;t like having all my neighbors right outside my door. My wife and I made the decision to buy a small old farm, and build us a little slice of what I had growing up. Like all hobbies, it grew until I was running a full fledged farm, and now I spend my time bringing together the two things I love.\nI hope you enjoy my attempt to provide a new perspective into lessons that help us succeed professional, and find humor in the situations, challenges, and memories that we are building on our small farm.\n","permalink":"https://pigs-pods.com/about/","summary":"Information about the author","title":"About Me"},{"content":"The groundwork for this site was laid by the team behind FedEx Cookbook, so thank you to that team for both requesting the usage of HUGO and providing access to their code base so we could build a comparable site for the edge.\n","permalink":"https://pigs-pods.com/credits/","summary":"Credits","title":"Credits"},{"content":"© Isaac Omaits\n","permalink":"https://pigs-pods.com/_footer/","summary":"\u003cp\u003e© Isaac Omaits\u003c/p\u003e","title":""}]