An Emerging Job: Kubernetes Engineer

With each major shift in technology, we see new jobs with new titles emerge. Today, the biggest shift in infrastructure and application deployment is the widespread adoption of Kubernetes, and the latest job title we’re seeing is the “Kubernetes Operator.” This isn’t to be confused with the Kubernetes programmatic operator construct, which is a software extension used in Kubernetes to manage resources). Since the word “operator” has multiple meanings in Kubernetes, for the sake of simplicity we’ll use “Kubernetes Engineer” for the rest of this article.

This following graphic from the Kubernetes Gateway API site depicts the Kubernetes Engineer as a “cluster operator” who is responsible for managing Gateway functionality. The Gateway API is still in alpha, but this model applies equally to Kubernetes implementations using other ingress-egress options (such as the Ingress API).

The Kubernetes Engineer role is often covered by engineers with other titles, such as Site Reliability Engineer for web apps or System Administrator for more old-school IT departments exploring this brave new world. Cloud-native orgs and teams might make the role part of Platform Ops or Cloud Architect positions. But more and more, companies running modern apps or going down the cloud-native path need a dedicated Kubernetes expert.

This development harks back to the last big infrastructure shift, when virtualization replaced bare metal. Enterprises rushed to hire virtualization engineers to operate VMware and Citrix virtualization platforms. As in that transformation, there is a significant learning curve for Kubernetes Engineers. The vocabulary of Kubernetes is often confusing; a term also used in networking or apps may have a different meaning in the world of Kubernetes. At the same time, many newer terms — such as microservices — are closely tied to Kubernetes as the dominant deployment pattern for microservices. The service mesh, an entirely new piece of infrastructure designed to help teams manage and deliver microservices securely and reliably, is almost always associated with Kubernetes.

Going forward, almost every organization pursuing Kubernetes will need someone that fits the description of a Kubernetes Engineer. If they don’t have one already, they’ll almost certainly need one soon. This article will look at the role, what it means for your organization, and how you might shape the role to fit your needs.

Brian Ehlert and I were recently on the Day Two Cloud podcast where we talked about this new skillset. Give it a listen and check out the Q&A below for more on the in-and-outs of this emerging role.

What Are the Basic Responsibilities of Kubernetes Engineer?

Kubernetes Engineers are tasked first and foremost with making sure that all things Kubernetes run well for their organization. Responsibilities usually include:

• Security: Kubernetes does not come secure out of the box. It’s the job of Kubernetes Engineer to lock down Kubernetes and configure it so that developers deploying their apps on a cluster are not needlessly exposing APIs, allowing unauthorized traffic, and more.
• Performance and observability: While Kubernetes is well-known for its many resilience features, performance-tuning it requires extensive knowledge. A pod may appear to be running fine even when it’s under-resourced at the CPU or memory layer, which leads to latency, dropped packets, or repeated restarts. It is the job of Kubernetes Engineer to tune performance and to identify problems by looking at service and traffic metrics for indicators of problems that are more nuanced than “is the pod up and passing traffic?”.
• Networking: Kubernetes networking differs from traditional networking. It multiplexes Layer 4 and Layer 7 and runs everything through APIs. Kubernetes networking involves managing north-south and east-west traffic and tuning for the internal networking requirements needed to maintain critical services. Many traffic management tools are  unique to Kubernetes – for example, an Ingress controller is a Kubernetes-specific component that is required for advanced Ingress conventions such as header rewrites and traffic shaping. A Kubernetes Engineer should be a master of this novel and differentiated networking environment, and ready to set up and manage the networking plumbing for Kubernetes.
• Infrastructure: An organization can either opt to run Kubernetes itself or use a managed service. In either case, the Kubernetes Engineer is tasked with making sure everything is running the right way, is properly patched, and has sufficient resources to run apps. This may bleed over into IT territory. Rarely does the Kubernetes Engineer have the keys to the server closet, so to speak, but they’re often the ones who can tell that CPUs, memory, and other physical elements are failing or insufficient. With managed Kubernetes, the Kubernetes Engineer ensures the service is configured and can scale as needed without being overprovisioned.

How Much Does the Size and Infrastructure of a Deployment Contribute to the Need for Kubernetes Engineer?

The extent of a Kubernetes Engineer role(s) depends on three factors:

• The size of your infrastructure
• Your position on the Kubernetes adoption curve
• The complexity of your Kubernetes deployments

If you’re a startup with a limited number of apps running on a small infrastructure footprint, “Kubernetes Engineer” is likely to be just one facet of someone’s job, possibly limited to deploying containers and apps on a managed Kubernetes service such as Google Kubernetes Engine. Such services abstract away concerns about the physical hardware and networking layer and do a pretty good job of handling scaling, basic security, and basic network management with minimal manual work.

But think about this for a bit before you decide that a part-time Kubernetes Engineer will suffice. The larger the infrastructure and the greater the number of moving parts, the more complex and time-consuming the job becomes. The diagram below depicts a very simple view. Broadly speaking, managing and operating Kubernetes on bare metal is far more complex than with a managed service. But using a managed service means zero visibility into the physical layer and it can add some latency. You might be surprised how often problems, conflicts, and performance issues can be traced to multi-tenant infrastructure. If you think noisy neighbors are bad in the cloud, imagine what they’re like in a managed Kubernetes service! Ditto for restarts of larger apps residing on big containers and higher capacity Kubernetes clusters.

In addition to the infrastructure question, you need to consider how many clusters you’ll be running and how many pods and nodes are in each cluster. Multi-cluster Kubernetes is considerably more complicated because it requires additional customization of security, networking, and API and traffic management. It also requires managing and linking multiple Ingress controllers (more about that in a bit) and determining whether you need multiple clusters running for hot failover (expensive) or you can tolerate a bit of downtime while a backup cluster spins up from scratch. Observability and troubleshooting are also far less complicated with single clusters and a limited number of pods and services. As soon as you have to troubleshoot and observe performance of more than a handful of services running on more than a dozen pods, keeping an eye on everything becomes fairly taxing.

Do Kubernetes Engineers Deploy Apps or Only Manage Infrastructure?

This varies from place to place. In larger organizations, a Kubernetes Engineer typically focuses mostly on keeping Kubernetes environments safe and running well, rather than writing YAML manifests or overseeing application deployment. Typically working closely with the Platform Ops team (or as a member of that team), Kubernetes Engineers at larger orgs are more interested in setting up trusted catalogs of services and APIs or registries of containers vetted to work on their Kubernetes setups. In smaller organizations or where a team is more invested in running the guts of Kubernetes, Kubernetes Engineers may overlap with DevOps teams and be responsible for working with app developers to oversee deployments.

How Do You Know When You Need a Kubernetes Engineer?

As soon as you’re sure that Kubernetes will play a big part of your app infrastructure, you’ll  want to start thinking about bringing on a Kubernetes Engineer, or at minimum an app developer with some Kubernetes expertise and experience operating it in production. If you’re following the modern application design pattern of building a rock-solid Kubernetes cluster foundation before you start deploying apps – at F5 NGINX we call this pattern “cluster out” – then you’ll want a Kubernetes Engineer as soon as possible. There are many ways things can go wrong in Kubernetes, and problems with architecture, service design, and other flaws may not reveal themselves after you’ve “gone live” and the stakes are high. For this reason, developing your Kubernetes Engineering chops early in the adoption process is a wise move.

Is Kubernetes Certification Necessary?

While being a Certified Kubernetes Administrator (CKA) is definitely a plus, there are plenty of people who learn how to run a Kubernetes cluster the old-fashioned way with hard-earned experience and the scars to prove it. In fact, we believe that experience executing specific tasks is more useful than a CKA credential. The reason for this is two-fold:

• Kubernetes is far more complex than any certification course can capture.
• Invariably, any Kubernetes deployment is a bit of a snowflake once all the services are built, networking is configured, and security is set up.

The best of both worlds, of course, is experience and a certification. Pursuing a CKA after having spent time hands-on with Kubernetes is a great strategy.

Do Kubernetes Engineers Need Advanced Kubernetes Networking Expertise?

Not necessarily. Kubernetes Engineers need a strong grasp of Kubernetes networking concepts and experience configuring Ingress controllers, multi-cluster networking, and managing east-west and north-south traffic. But they don’t necessarily need experience with more advanced Container Network Interface (CNI) apps designed to provide something more like a network management and network security layer in Kubernetes. For the majority of Kubernetes use cases we see today, a CNI is overkill, making CNI skills a nice-to-have but not a need-to-have for Kubernetes Engineers.

Where Can I Learn More?

A solid foundation in Kubernetes networking concepts and tools is essential to success. That’s why we’re offering Microservices March 2022: Kubernetes Networking. It’s a free education program taking place March 7-31 that includes a series of events and self‑paced activities designed to take you from zero to hero. You can choose to complete the entire program or just the bits and pieces you need to fill in your knowledge gaps. The total time commitment is about 16 hours, spread across 4 weeks.

Four units progressively guide you through the essentials of Kubernetes networking:

• Unit 1 (March 7–11): Architecting Kubernetes Clusters for High‑Traffic Websites
• Unit 2 (March 14–18): Exposing APIs in Kubernetes
• Unit 3 (March 21–25): Microservices Security Pattern
• Unit 4 (March 28–31): Advanced Kubernetes Deployment Strategies

Each unit includes:

• A YouTube livestream featuring experts from NGINX and learnk8s
• A collection of blogs, videos, and ebooks to deepen your knowledge
• A hands‑on, self‑paced lab for experimenting with Kubernetes technologies
• Access to the NGINX experts via our Slack community

Sign up today to get free access to the program – and don’t worry! Even if March 7^th has already passed by the time you read this article, you can catch what you’ve missed on-demand.

Guest post by Jenn Gile, Sr PMM, F5 NGINX
Source CNCF