DevOps and Docker Talk: Cloud Native Interviews and Tooling

Kubernetes Pod Live Migration.

That's what Cast AI calls it when they
dynamically move your pod to a different

node without downtime or data loss.

They've built a Kubernetes controller
that works with CSI storage and CNI

network plugins to copy your running
pod data, the memory, the IP address,

and the TCP connections from one
node to another in real time Welcome

to DevOps and Docker talk, and I'm
your solo host today, Bret Fisher.

This YouTube live stream was a fun one
because I got to nerd out with engineers

Philip Andrews and Dan Muret from Cast
AI on Kubernetes pod live migrations

and how it works under the hood.

We talk about use cases for
this feature, including hardware

or OS maintenance on a node.

Maybe for right sizing or bin packing your
pods for cost savings or moving off of a

spot instance that's about to shut down.

Or really anytime you need to move a
daemon set that would cause an outage if

the pod had to restart or be redeployed.

I don't know of any other turnkey way
to do this on Kubernetes today, but

I've got a feeling that Cast AI has got
a winning feature on their hands, and

I'm glad that we got to dig into it.

Over 20 years ago, the virtual machine
vendors built this live migration feature

into their products, and finally, in 2025,
we're now able to do that in Kubernetes.

Let's get into the episode.

Welcome to the show.

All right, Both of these
gentlemen are from Cast AI.

Philip is the global
field, CTO at Cast AI.

What exactly does Global CTO do?

handling, a lot of our large customers.

a lot of the strategic partnerships,
usually new technologies, so working

with a lot of our, you know, customers on
showing them new technologies, helping in,

proof of concepts with new technologies.

it's been kind of a cool role.

basically on the technical
customer facing side.

I get to do a lot with our
largest customers in solving

some of the the hardest problems.

Nice.

when you don't know what the
title is, it makes it sound like

you just live in an airplane.

it sounds impressive.

and we've got Dan Muret or Muret.

I really don't know my French, so that's
probably, a horrible pronunciation.

Dan is here, he's the senior sales
engineer with CAST AI or one of them.

I'm gonna make you the sales senior sales
engineer so that you sound very elite.

Uh, welcome Dan.

who could tell me about what's the
elevator pitch for Cast AI because

I've known about you all for years.

I've visited your booth at KubeCon at
least a half a dozen times over the years.

what's the main benefit of Cast AI?

Sure I can grab that one.

when Cast AI was founded, it was to
solve a problem that our founders

had in their last startup years ago,
which was every month their AWS bill

was going up 10% regardless of what
they did the month before to try to.

Mitigate and manage costs
across the in infrastructure.

they sold that startup to Oracle, when
they finished that Oracle time, they went

and figured out how to solve this problem.

They said, well, Kubernetes is
gonna be the future platform.

We're gonna make our bet on Kubernetes.

And the only way to solve this
problem is through automation.

Because doing things manually
every month, one, it's tedious

and takes up a lot of time.

And two, it's just, not helpful, right?

You save a little bit every,
you know, it's two steps back

for every step forward you make.

With Cast AI, it's fully
automation first, right?

We made the effort that everything was
going to be automated from the start.

when it comes to node auto scaling,
node selection, node, right sizing,

workload, right sizing, everything
to do with Kubernetes, everything

we implement is automated.

And that's where the live migration
piece came in being able to automatically

move applications around within the
cluster without having downtime.

and that's where application
performance automation comes

in, Moving from this application
performance monitoring mindset.

Datadog has made a lot of money on that.

I dunno if I'm allowed to say that.

but they've done very well
and have a fantastic platform.

we love Datadog, but
you get data overload.

You end up with metric overload and
actioning on those is very hard.

Where we need to go from here,
especially with the AI mindset that

we're moving into, is automation
of that application performance.

And that's what Cast AI
is leading the way in.

Nice, when you all reached out and I
learned about the fact that you now have

live migration, it took me back to almost
25 years ago when that was first invented

for VMs at the time it felt like magic.

It did not seem real.

We all had to try it to believe
it because it seemed impossible.

To, move from one host to
another, maintain the ip,

maintain the TCP connections.

surely I'm gonna freeze up and
it's gonna be like a frozen screen.

we all just assumed that.

eventually, and maybe at first it
was a little hiccupy, I think, if I

remember correctly, like 2003, 2005.

It was one of those where it wasn't
quite live and they didn't, it was

like very short amounts of gaps.

then eventually it got good
enough that it was live.

I was running data centers, for
local governments at the time.

So I was very interested in this.

'cause we were running
both ESX and HyperV.

So I was heavily invested in
that feature and functionality.

So when I saw that you were doing
it to Kubernetes, my first thought

was, why did this take so long?

why don't we have this yet on everything?

Because it's clearly possible,
it's technically possible.

Obviously, it's not super easy and
requires a lot of low level tooling

that has to understand networking
and memory and you know, dis disc

rights and all that kind of stuff.

So I'm excited for us to get
into exactly how this sort of

operates for a Kubernetes admin.

And I really feel like this show's gonna
be great for anyone learning Kubernetes or

Kubernetes admins, because to talk about
the stateful set, the daemon set problem

of, we've got stateful work, everyone's
got, I mean, everyone I know almost

has stateful workflows in Kubernetes.

I would say, I don't know about you
all's experience, but to me it's

an exception when everything is
stateless in Kubernetes nowadays.

Do you find that to be the case

we talk to customers all the time, right.

And yeah.

Used to, you know, a few, just
a couple years ago, right.

It was a lot more stateless.

Web servers, whatever.

now we're definitely seeing a shift
to, more stateful workloads whether

it's, legacy applications being
forced into Kubernetes as part of a

modernization project or whatever.

We're seeing a lot more stateful
workloads in Kubernetes for sure.

Particularly amongst the Fortune one
hundreds, fortune five hundreds, right?

'cause you've got this modernization,
and I put it in quotes, where

modernization means taking some
crusty old 15-year-old application,

containerizing it and shoving it in
Kubernetes and calling it cloud native,

the flawed approach of lift and shift.

and you end up with a lot of
applications that are in Kubernetes.

They're listed as a deployment, but
you can't restart them without your

customer having a significant outage.

it goes against everything
Kubernetes was built on.

But that's the world we live in today.

when we first launched Live migration,
and I posted about it on LinkedIn,

some of the first questions I
got was, why is this even needed?

If you're doing Kubernetes correctly, live
migration shouldn't even be a real thing.

yes, but 95% of the customers I deal with.

Don't do Kubernetes, the right way.

Well, yeah, I honestly think we
could argue that's a great point.

when Docker and Kubernetes were
both created, it was like stateless.

'cause it's easy, you know,
move everything around.

It's wonderful.

But I think that to, I mean, this channel,
if there's anything consistent about this

channel over the almost decade that has
existed, it's that I, it's all containers.

Like I don't care what the tool
is, we're doing it in containers.

the large success of containers is
because we could put everything in it.

so many.

Evolutions or attempted evolutions
in tech have been, well, you're

gonna have to rewrite to, you know,
functionless, you know, or functions.

you're serverless, you're gonna have to
write functions now or you're gonna have

to rewrite this language or whatever.

and that's, I think that's like the
secret sauce of containers was that we

could literally shove everything in it.

It's also the negative.

And so there's, there's a thing that,
I don't know if I learned it from a

therapist or whatever, but often our
weaknesses are just overdone strengths.

And I feel like the strength of containers
is that you can do everything with it.

you can put every known
app on the planet in it.

They will eventually work
if you figure it out.

the overdone weakness is that we're
putting everything in there which makes

managing these infrastructures very hard.

you have to assume everything's
fragile Until you are sure

that it's truly stateless.

even stateless, people say stateless
and what they really mean is it's, it

doesn't care about disc, but it definitely
cares about connections, which is when

we're trying to talk about stateless,
that's not technically accurate.

Like when we say stateless,
we should probably mean also

doesn't care about connections.

at least once the connections are drained.

it's an interesting dilemma we all have
in the infrastructure of we have the

power to be able to move everything and
do everything, but also everything's

super fragile that we're running at the
same time, so how do we even manage that?

We've encountered a lot of teams
that, swore up and down they were

stateless, right up until you
started bin packing their cluster.

They said, wait, wait, wait.

why are we having all
these restarted pods?

We're like, because we're bin
packing and we're moving things, and

we're getting better, optimization.

They're like, but my container restarted.

Well, yes, that's what
containers do in Kubernetes.

Right.

And for those that are maybe just getting
into Kubernetes or haven't dealt with

large enterprise forever workloads
where they just can't be touched.

I've had 30 years in tech
of don't touch that server.

don't touch that workload.

It's fragile, it's precious, but
it's also probably on the oldest

hardware in the least maintained.

so the idea of, one of the performance
measures that any significant size

Kubernetes team is dealing with
is the cost of infrastructure.

And then we keep getting told, I think
this was just in the last year at

KubeCon, that even on top of Kubernetes,
we're still only averaging like 10% CPU

on average utilization across nodes.

Like we, we still are struggling with
the same infrastructure problems that

we were dealing with the last 30 years,
even before VMs, before virtualization.

That was the same problem we had then
because everybody would want their

own server and they always had to plan
for the worst busiest day of the year.

So they would buy huge servers,
put 'em in, and they'd sit idle

almost all the time because
they barely got 5%, utilization.

So I can see where like one of the core
premises of something like a application

performance tool is that we're gonna
save tons of money by bin packing.

can you explain the bin packing process?

what does that look like?

so one of the big things with Kubernetes
is the scheduler will typically

round robin, assign pods to nodes.

if you have 10 nodes in a cluster,
your pods will more or less get evenly

distributed to those nodes in the cluster.

You can manage that with certain
different scheduler hints and

certain scheduler, suggestions to
steer that towards, you know, most

utilized, least utilized, et cetera.

But at the end of the day,
you're gonna have spread out

workloads across your nodes.

Bin packing is basically the
defragmentation of Kubernetes right back.

You know, Bret, when you were first
starting out, when I was first starting

out, and you could actually defragment
a hard drive and you get to move the

little Tetris blocks around the screen,

days.

being able to do that in a Kubernetes
cluster can mean massive, massive

savings on the actual utilization of that
cluster because now you free up a bunch

of workloads, a bunch of nodes in the
cluster that are no longer necessary,

you can delete those off and when you
need them, you just add them back.

that's the joy of being in a cloud
environment, you can use the least amount

of resources when you don't need 'em.

So in, for instance, at, you know,
your off busy hours, your nighttime

hours, and then when you start needing
'em again, you spin 'em up, you add

more to it, you scale up during the
day, and being able to do that process

over and over again every day is how
you can optimize your cloud resources.

What we see is that.

People have so many stateful workloads
whether it's stateful in real state or

stateful in, this is a really poorly
architected application, or stateful in

this application takes 15 minutes to start
up and I, it's a monolithic and I can only

run one copy of it so I can't move it.

All of those things causes, so you
can't bin pack a cluster, right?

You can't move those things around.

So what ends up happening is people just
end up with these stateful workloads

scattered throughout all 10 nodes.

And even if the 10 nodes are only
60% utilized, you can't get rid

of any of them because it'll cause
some kind of service interruption.

And that's where live migration
allows you to move those

stateful sensitive workloads.

So now those 10 nodes can go down
to six or seven nodes without having

a service interruption, even if
there's less than ideal workloads

scattered throughout the cluster.

stateful versus stateless versus like,
where's the scenario for where we need

a live pod migration and like to those
that are perfect in all their software

and they run and they control all
the software that runs on Kubernetes.

I don't know who those people are,
but let's just say they exist.

Then this isn't needed.

Every database has a replica
or database mirror, so you

can always take a node down.

Every pod properly has proper
shutdown for ensuring that connections

are properly moved to a new pod.

By the way, I used to do a whole
conference talk on TCP packets

and, resetting the connection to
make sure it moves properly through

the load balancer to the next one,
and having a long shutdown time

so that you can drain connections.

that world of shutting down a pod is so
more complicated than anyone gives it.

everyone treats it like it's casual
and easy, and it's just not if

you're dealing with, hundreds of
thousands or millions of connections.

there is a lot of nuance
and detail to this.

and I often end up with teams
where they implement Kubernetes.

It's a sort of predictable pattern, right?

They implement Kubernetes,
move workloads to it.

They think Kubernetes gives their
workloads magic and then they just

start trying moving things around
and they realize when their customers

complain that, The rules of TCP IPS
load balancers and connection state,

like all these rules still apply.

you have to understand those lower levels
and obviously disc and writing to disc

and logs for databases and all that stuff.

that's still there too, I think.

I think the networking is where
I see a lot of junior engineers

hand waving over it because quite
honestly, the cloud has made a lot

of the networking problems go away.

So we don't have to have
Cisco certifications just

to run servers anymore.

We used to, but now we can get away
with it till a certain point, in

the career or complexity level.

And then suddenly you're having to really
understand the difference between TCP and

UDP and how session state long polling or
web sockets, how all these things affect.

Whether you're going to break
customers when you decide to restart

that pod or redeploy a new pod.

I love that stuff because it's super
technical and you can get really into

the weeds of it, and it's not, I wouldn't
call a solved problem for everyone.

my understanding of something
like a live migration is,

takes most of those concerns.

it doesn't make them irrelevant, but
it does deal with those concerns.

am I right?

in terms of networking we're talking
about live migrations, having

to be concerned with ips and,
connection state, stuff like that.

yeah, so with being able to do the
networking move of things, to your

point, reestablishing those sessions.

one of the big things we
see is long running jobs.

if you've got a job that's running for
eight hours and it gets interrupted

at six, you've lost that job.

Even if you try to, move it from one to,
there's some checkpointing involved, a

lot of times, like on a spark workload,
the driver will just kill the pod

and restart it if it senses any kind
of interruption in the networking.

So the networking's super important there.

Being able to maintain that.

Long running sessions, web sockets.

To your point, we've actually tested
this extensively with web sockets.

Web sockets, stay connected.

And we're still in those
early vMotion days.

there is a slight pause
when we move things.

it took vMotion multiple years before
they got it kind of really ironed out.

We're moving probably faster
to them 'cause we have a lot

of experience, you know, of the
experiences they went through.

and the research that's
happened since then.

So I think we're moving pretty fast
on shortening that time window.

But what we found is you queue up all
the traffic and once the pod is live on

the new node that traffic is replayed
and all the messages come through.

So even on something like a web socket,
you don't actually lose messages.

They're just held up for a few seconds.

And that's extremely important
from maintaining that connection

state, like you were mentioning.

one of our customers that we're
working with this heavily on,

they run spark streaming jobs.

So they're 24 7, 365, pulling off a
queue, running data transformations

and detections, and then pushing
somewhere else for alerting mechanisms.

If they have a pod go down, it takes about
two minutes to get that process restarted,

pull in all the data that they need.

Again, that's two minutes of backlog.

They have super tight SLAs.

They have a five minute SLA from
message creation to end run through

the entire detection pipeline.

So if you've got a two minute
delay on that shard in your Kafka

topic, that's a huge chunk of that
five minutes that you just ate up.

You're talking all the other pipeline.

It's very easy to start
missing SLAs there.

It's, you can't take maintenance
windows if you're 24 7, 365 and

you're doing security processing.

you don't.

You can't be like, well, security's
gonna be offline for 10 minutes

while we move our pods around.

Like, that's just, that's not
acceptable in that world mindset.

so keeping that connectivity, keeping
the connection state, being able to

keep everything intact, keeping the
Kafka connection, keeping the, spark

driver connection is all super important
with being able to move that entire

TCP/IP stack over from one node to
another, during that migration process.

Yeah, and I mean, we're really talking
about a lot of the different kinds of

problems that come with shifting workloads
like being able to say, you know, walking

into an environment and sort of being
your own wrecking ball and, your own chaos

monkey and saying, I'm gonna go over here
and push the power button on this node,

or I'm gonna properly shut down this node,
do you have everything set up correctly so

that connections are properly drained that
is a, such a. I would say a moving target,

especially because every time we've had
these processes where I've had clients

where we go through this like exercise of
we're going to do maintenance on a node

and we're even gonna plan for it, and
then we do it, and then we fix all the

issues of the pods and the shutdown timing
and, the Argo CD deployment settings

that we need to massage and perfect.

And then, you know, six months later, if
we do it again, the same thing happens

because now there's new workloads that
weren't perfected and weren't well tested.

if I can make a career out
of actually being like.

a pod migration guru, like that sounds
like my kind of dream job where we crash

and break everything and then we track
all of the potential issues of that and

we are like a tiger team that goes pod
by pod and certifies this is like, yep,

this pod can now move safely without risk.

because we've got everything dialed in.

We've got all the right settings.

I feel like that's a workshop opportunity.

maybe sell something on that because
there are so many levels of complexity

we haven't even talked about, like
database logs and database mirroring

you can't really spin up a new node
of a database and let it sit there.

Idle is a pod while you're waiting for
the old one to shut down, they can't

access the same files, blah, blah, blah.

it just depends on the workload,
on how complex this all gets.

But I'm assuming also that when we talk
about something like live migrations,

we're not just concerned with networking.

We're also somehow shifting to storage.

I'm guessing there's certain
limitations to that where you're not

replicating on the backend volumes.

You're, I guess you're just
using like ice zy reconnects or

how's that, how does that work?

we haven't really gotten into the
solution, but I know you're only on

certain clouds right now, and I'm assuming
that's partly due to the technicals of

the limitations of their infrastructure.

Right, exactly.

each cloud has different kind
of quirks, around how they

function, what the different,
technologies look like around them.

somebody had asked about being able
to move, you know, larger systems

are the limits around it and their,
it depends on the use case, right?

If you're talking spot instances,
being able to move from one spot

instance to another spot instance
in a two minute interruption window

on AWS, depends on how much data.

If you're trying to move 120 gigs of
data physics is working against you.

you don't have enough time in that two
minute window to get enough through

the pipe over to the new, system.

Now if you're talking small pods,
if you're talking less than 32 gig

nodes, you can move that fast enough.

64 gigs, maybe you're on the edge.

Depends on how much other network
traffic is tying up the bandwidth,

64 gigs is getting on the edge that
you can move in a two minute window.

that other example I was talking about,
those long running spark streaming

jobs, if they're running on demand,
live migration, is still a massive

benefit because now you can do server
upgrades without taking an outage.

You can create a new node running
your new patch version of, Kubernetes,

running your new Patched Os, and
migrate the pod from one to the other.

Your time to replicate is less important.

Even if it takes you three minutes
four minutes, or five minutes

to replicate the memory from
one box to the other, who cares?

It's not gonna be paused for that
long, because what we're doing

is we're doing delta replication.

So you replicate a big chunk, and then
a smaller chunk, and then a smaller

chunk until the chunk is small enough
where you can do it in a pause window.

And so now when you're moving a
huge service from one to the other.

Same thing.

If you're talking like NVME local storage,
we've got another customer we're working

with and it's a different set of problems.

They have a terabyte of NVME that they use
on local ephemeral disc on every node that

needs to be replicated from node to node.

When they do node upgrades, that
takes about 20 minutes, to replicate

all of that from one node to
another, even on high throughput

discs, on high throughput nodes.

But if it's happening in the
background, while everything else

is humming along nicely, who cares?

Keep replicating it over.

You keep going down to Deltas, and
then once your deltas get small enough,

you pause for six to 10 seconds,
depending on how big the service is.

And then you slide it over.

a lot of these things are being solved.

We're actively reducing these pause
times by being able to do more prep

behind the scenes, being able to do
more processing behind the scenes.

everything is, operating
as a containerd plugin.

I saw somebody asked, about on-prem
we will be supporting on-prem, we

will be supporting other solutions.

The big catch there is everybody
has some different flavor of

networking and different flavors
of things behind the scenes.

The actual live migration piece right now
could apply to any Kubernetes anywhere.

It's the IP stack that gets a little
trickier because you've got cilium

running places, you've got Calico running
places, you've got VPN, V-P-C-C-N-I

running places, you know, everybody
has different networking flavors.

So being able to maintain network
connections when you do the

migration is largely the more
difficult part of the whole process.

Hmm.

Being able to move the pod isn't that
bad, so if you've got workloads that

you can reestablish connections and the
connection resetting is not a big deal,

but you don't wanna have to restart
the pod, that's fairly straightforward.

We could pretty much do that today across
any containerd compatible Kubernetes.

It's specifically the networking
that causes a lot more hardships,

because everybody has a
different flavor of networking.

for AWS, we were able to fork
the open source AWS node CNI, and

create our own flavor of it that
now handles the networking piece.

So we're using the open source
AWS CI code, and we've modified

it, and now it works just fine.

for our purposes, we're doing something
similar on GCP, working with the gc.

GPFI recall is using cilium under
the hood for their networking.

So we're gonna be, building a
similar plugin for their cilium side.

Yeah, and the nice thing is, I
guess if you build it for cilium.

would it work universally
across any cilium deployment?

In theory, I mean, I'm just thinking of
the most popular CNIs and if you check

those off the list, it suddenly gives you,
you know, a lot more reach than having to

go plowed by cloud or os by os, you know?

and,

Exactly.

our first iteration of this back in
January, February, the first version

that we demoed was actually Calico.

A lot of people were like, I don't
wanna have to rip out my cluster and

rebuild it with Calico as the CNI.

we were able to figure out
a way to work with the, VPC

CNI, as a backing basis there.

So Calico's pretty much already built.

we've got AWS CNI now built,
cilium our next target, that saw

somebody asked about Azure, Azure
is probably gonna be early 2026.

we'll be E-K-S-G-K-E and then
we'll work on a KS, and then we'll

work on-prem solutions after that.

So on-prem will probably
be sometime in 2026.

Yeah, I can remember, going
back to the two thousands.

I can remember when we went from
delayed migrations or paused

migrations to live migrations.

I can remember reading the technical
papers coming out of, VMware and

Microsoft and they were talking about
the idea of, these deltas continually

repeating the Delta process until
you get down to zero or like you can

fit it in a packet and then that's
the final packet kind of thing.

I don't know why I remember that all
these years later, but I do remember

that I thought that was some pretty cool
science, like some pretty cool, physics

across the wire because back then we were
lucky if our servers had one gigabit,

200 gig workloads or anything like that.

this actually led me during my
research and, we could talk about

the idea that there are, there are
attempts in Linux over the years

to try to solve this universally.

I did some research before the show and
saw some projects around ML workloads in

particular a lot of, engineers, whether
it's platform engineering or just the ML

engineers themselves interested in this
because of the, sort of the problems of

large ML or AI workloads today where you
can't interrupt them if you interrupt

them, you have to basically start over.

it's sort of a precious workload
while it's running and it

might be running long time.

do you have, AI and ML workload
customers where they're.

Are they maybe part of the first movers
to move onto something like this?

I'm basing it on the KubeCon talks
and things that I've seen out there,

Large scale data analytics is
definitely, one of the big players here.

A lot of it's spark driven data
analytics that we're seeing,

because of exactly that problem.

A lot of these jobs will be running
for 8, 10, 12, 14, 16 hours and.

Running those on demand at the
scale that they're running them

at is extraordinarily expensive.

So the big ask is how do we get those
workloads onto spot instances where

when we get the interruption notice we
can fall back to some type of reserved

capacity and then fail back to spot.

So basically the goal is to move
to this new concept where in your

Kubernetes cluster you have some swap
space, whether that's, excess, spot

capacity, two or three extra nodes of
spot capacity or a couple of nodes of

on-demand capacity where if you get a
node interruption, you can quickly swap

into those nodes, and then once you stand
your new spot instance back up, then

you can swap back to that spot instance.

where we're headed.

that's what Q4 is gonna be working on
this year, is to be able to automate that

entire process to where you can float back
and forth between reserve capacity and

spot capacity to really save on those,
data analytics jobs, those large ML jobs.

We're not to the GPU side of things yet.

I'd love to get us to where we could
migrate GPU workloads 'cause that's where

the next big bottleneck is gonna be.

the hooks aren't there in the
Nvidia tool sets yet for the Cuda

tool sets in a lot of places, to be
able to get what we need for data.

we're figuring our way around that.

they tend to be much larger, so the
time taken to move them very expensive.

it might be 20 minutes to be able
to move a job from one to the other.

'cause it took 20 minutes to get
a startup up in the first place.

just due to the size of the models and
how much data you have to replicate.

we're starting to put some POC work
into the GPU side of things while we're

continuing on full steam with building
out the expansion of the feature set

of the CPG, and memory based workloads.

Alright.

Dan, I was curious if you've seen
on, the implementation side of this.

when we talk about.

the need to live, migrate a pod, whether
it's for maintenance then the almost

feel like the next level is the idea
of spot instances, I love that idea of

my infrastructure dynamically failing
and my applications can handle it.

is there a maturity level where
you see people start out it's hard

for me to imagine like on day one
someone's like, yeah, let's just put

it all on spot instances in yolo.

let's just fi we don't care.

It's all good.

Mi Live migration will solve it all.

Because obviously, there are
physics limits to the amount of

data we can transmit over the wire.

I'm imagining this scenario where you're.

accrediting certain workloads,
like this replica set is good

for spot because it's low data.

we don't need to transfer a hundred
giga data during a two minute outage

or a two minute notice of outage.

do you see that as a maturity
scale where you have to

Yeah.

I mean, it is absolutely
a maturity scale, right?

kind of going back to the references
we talked about, the early days of

VMware, nobody started doing vMotion
in production, everyone started it.

Oh, we've got this, five second
interruption, development and test

boxes can handle that all day long.

so it's the same concept really
that we're living in now.

We're going through that same evolution.

I agree with Phil.

I think we're doing it much faster
than VMware did in 2002, 2003.

I was around when that happened as well.

So I remember racking and
stacking all those boxes.

but yeah, it's very much the same thing.

container live migration is brand new,
we've just been GA for a month with it.

So we've had conversations at trade
shows and with customers and there's

a lot of excitement around it.

I think we're still trying to
figure out where it fits, what

are the exact workloads that it
makes the most sense to do this in.

And yeah, I think it's going
to be a process of adoption.

there's definitely a lot of use cases.

I think the spot is a very interesting
use case, especially the large data models

and things that we're processing today.

I'm working with a customer now that's
doing a lot of video processing in

Kubernetes and that's a very, you
know, CPN memory intensive job.

I mean, you know, we're talking a cluster
that scales from a D CPUs to 6,500

CPUs while they're processing this.

we're really trying to figure out
where it makes the most sense to

apply this type of technology.

no one wants to have that kind of
dynamic scale and then have to pay

for reserved instances for all of
that, like, worst case scenario.

that sounds like a billing nightmare.

and you don't want a job that
runs for, you know, hours that

cost you tons of money to fail 80%
through and have to restart it I

mean, that's just not efficient.

So, yeah, I think the ability to
really move this and allow those

workloads to finish is gonna be.

Huge for the market.

Alright, so we have been
talking a lot about the problem

and some of the solution.

we do have some slides that give
visualizations for those on YouTube.

this will turn into a podcast.

So audio listeners, we will give
you the alt text, version of it

while we're talking about it.

But, Philip, I'm Exactly what is
happening and the process of how a live

migration, like how does it kick off?

what's really going on in the
background when it starts,

Absolutely.

and we do have some
better demos other places.

I think on the website.

basically what we have is a live
migration controller looking

across all the workloads and nodes
that are lab migration enabled.

You don't necessarily have to
turn this on for everything.

You've got all your stateless
workloads, you don't need to live

migration, stateless workloads,
just treat them as normal.

You've got your stateful workloads
that you do want this to use for,

so you could set up a specific,
you know, node group for that.

that's gonna allow you to select what you
actually want to do live migration for.

You could use it for everything, but
it just eats up more network bandwidth

if you're using it for the stuff
that already tolerates being moved.

that controller's gonna be looking for
different signals within the cluster of

when something needs to be live, migrated.

instance, interruption is a good one.

being able to do bin packing, evicting
a node from the cluster because it's

underutilized, and then migrating those
workloads to another node in the cluster.

what we call rebalancing.

basically, rebuilding the
cluster with a new set of nodes.

And that could be because you're
doing a node upgrade, you're doing a

Kubernetes upgrade, you're doing an
OS upgrade, you're just trying to.

Get a more efficient set of nodes.

all of those are good reasons that you
would want to do your live migration.

So what's gonna happen in that process
is the two daemon sets on the source

node and the destination node are
gonna start talking to each other.

They're going to look at the pods on
the source node, start synchronizing

them over to the destination node.

So behind the scenes, all of that
kind of memory is being copied over

any disc states being copied over
any TCP/IP connection statuses are

being copied over and you're doing
all that prep work behind the scenes.

If you have, ephemeral storage, on
the node that'll start getting copied.

Obviously, depending on how much, it's
gonna depend on how long it takes.

Once the two nodes have identical
copies of the data, that's when

the live controller will say, it's
time to cut over it will cut the.

Connections from one, pause it and put
it into a pause state in containerd,

then it will unpause on the new node.

It'll come up with a new name.

Right now we call 'em
clone one, clone two.

We just add clone to 'em.

So you can tell which was the
before and which was the after.

when that clone one unpause,
then traffic will be going to it.

It'll have the same exact IP address that
it had while it was on the previous node.

All the traffic continues onto that node.

It picks up exactly where it left off,
and the old pod disappears, right?

The old pod gets shut down and torn down
if you have something like a PVC attached.

So if you've got an E-B-S-P-V-C
attached, there is a longer pause

because you have to do a detach reattach.

with the API calls, it works.

It just takes a little bit
longer for that pause state.

That's The downfall of
having to work with, APIs.

it takes time to do an unbind
rebind, to the new node.

but it works today.

If you're using NFS where you can
do a multitask, then it's instant,

it doesn't actually add any delay.

just NFS is a slower storage technology.

So does that sort of make
sense from a high level?

Yeah.

when we talk about Cast AI as a
solution, it do live migrations

based on certain criteria?

Is it making decisions around, if you
say I want a bin pack all the time,

it in the background, is it just like
doing live migrations on your behalf?

Or is this something where you're
largely doing it with humans clicking

buttons and controlling the chaos.

No, this goes back to what we
had talked about at the beginning

where, automation is key.

when a node is underutilized,
our bin packer is, probably the

most sophisticated on the market.

It analyzes and runs live tests on
every node in the cluster of whether

that node can be deleted, whether that
node doesn't need to be there anymore.

And it'll simulate all the pods being
redistributed throughout the cluster.

if the answer is we don't need this
node, it would automatically kick

off a live migration of all the
pods on that node Once it's empty,

it'll just get garbage collected.

once it's gone, all your pods
are running on the new nodes.

Everything's moved seamlessly.

You haven't seen any interruption.

the cluster keeps continuing as normal.

Most of our customers do
scheduled rebalances, so those

are just in the background.

It's evaluating how efficiently
designed the nodes in the cluster are.

If the nodes in the cluster are.

Not as efficient as they could be, and
different shapes and sizes would be

better for, that setup at that point
in the day, based on the mixture of

workloads there, it'll do a blue-green
deployment, set up new nodes live,

migrate the workloads to those new
nodes and tear down the old ones.

So everything that we're
talking about here can either

be scheduled or it's automatic.

It's running every few minutes on a cycle.

but yeah, no, it's entirely
seamless to the users.

Nice.

So in the technical details, we're
moving the IP address, I think you had a

diagram, showing the pod, on the nodes,
when we get down to the nitty gritty of

Kubernetes level stuff, pod is recreated,
but pod names have to be unique you

can't have the IP on both nodes at once.

And then there's the difference
between TCP and UDP and other, you

know, IP protocols and the, there's
a lot of little devils in the

details that I'm super interested in.

We won't have time to go into all
of it, but I do remember you showed

the replica, the pod that you're
creating, step one is we create a

pod and download an image, right?

this is all still going
through containerd.

So it's not, there's not like voodoo
magic in the background happening

outside of the purview of containerd.

maybe you can talk about that for a minute

Right, Exactly.

So being by changing the pod name,
you now have a. Placeholder for

your new pod information to go into.

And it does maintain the same IP address
when it moves over from one to the other.

So to your point, that's when that
switch has to kick in where the

old pod definition disappears.

And the new pod definition appears
in your, control plane with the

API calls up to the coop API.

that cutover is extremely important
because you can't have the same

pod living in the same place twice.

that's why we do have to change
a name when we switch it over.

there's certain services that, cause
some tricks because they have an

operator structure where they expect
there to be a certain pod name.

So when you move it and add the clone
suffix to it, we're working on finding

workarounds to that, in certain areas.

that is a little bit tricky on
certain workloads because you can't

have the same pod existing with the
same name in two different spots.

They have to be unique.

but yeah, definitely.

the IP is the same, but the pod
name is gonna have a clone dash

one or something like that on it

Yep.

yeah, so it starts with your pod
and then there's an event that

happens outside of the pod that is
talking into, containerd and you're

It's a second pod.

Correct.

It's adding that placeholder.

And because the placeholder pod is
actually still in a paused state.

It can have the same IP address.

it's not actually routing traffic to
it, 'cause it's not an active pod yet.

it'll be in a staged state.

So you stage it up with all
the information, but it has

to be named differently.

And then when you do the cutover,
that's when you switch it from

being inactive to active and
switch the old pod to be inactive.

and that's the final stage when,
the clone pod becomes the primary.

And because it's maintained the
exact, IP address within the

system, it's not losing any traffic.

So the networking system within
Kubernetes routes it to the new node

and the routing tables are updated and
the pod goes to the new destination.

Yeah, that does sound like the hard
part of like the old pod is shut

down so the IP can be released.

I assume the IP.

Can't be taken out while
that old pod is still active.

it's one of these things where it's like,
I understand it at the theory level, but

I have no idea what containerd and coup
proxy and, all these different things that

are binding to a virtual interface and the
order of the things that have to happen

in the exact right sequence in order for
you to first assign that IP to the new

node and then also replay all the packets.

it does seem like a very discreet
order of things that have to happen.

it has to go in a certain order or
they're just all fail it feels like.

so that's the part that took
us about a year to figure out.

there had been a lot of studies
and some research papers around,

the moving of memory and the
snapshotting of different workloads.

Like that part was a little bit
more straightforward because it

was really out of the vMotion
playbook days, from early on.

there were also some college
studies around using cryo to

replicate and migrate containers.

None of them had been able to
solve the IP side of things,

the connectivity side of things.

that's what Cast AI was able to solve for.

and it took a lot of research,
took a lot of, in-depth work.

We started on this, early 2024,
with a team of about five engineers,

deep kernel level Linux engineers,
Kubernetes engineers, people

very familiar with the, code.

they contributed to the Kubernetes
open source project, it was.

10 months before we had a demo
and that was using Calico.

before we could demo, we had to
have a custom AMI at that point in

time in AWS because everything was
kernel level at the a MI level.

we knew that was not feasible going
forward to production, but that

was the first demoable version Like
anything else, There's a lot of warts

and vaporware in the first version.

since then we were able to move
the logic up to a containerd plugin

makes it a lot more portable.

Now it can be applied to different clouds.

It's much less invasive.

You don't need a specific AMI.

and under the hood anymore, we were able
to move it to the A-W-S-V-P-C-C-N-I.

So you don't need the custom Calico CNI.

all of those were iterative steps to
build this and make it more, production

viable and adoptable by the industry.

Now it's a matter of, we've
got kind of two forks going on.

One is continuing to build out
additional platforms, figuring out

cilium, figuring out Azure, CNI,

The other is performance tuning the
existing migrations, reducing time to

migrate, being able to reduce the size
of the deltas down further and further

so we can migrate it faster and faster.

so we've got kind of those
two tracks right now.

the team's up to I think
10 or 12 engineers.

kind of working on those two paths,
and this is probably one of our most

heavily invested, areas in the company
is being able to further this technology.

'cause we see how much value it brings.

Yeah, I imagine it won't be very
long where, you know, this technology

is pretty advanced, but like others
eventually will probably if it's truly

the thing that we all are looking for.

And it sounds like, it feels like it is,
it feels like the kind of, tooling where

it's a hard problem to solve and we'll
maybe see other people attempt to do it.

I mean, the.

the research I had to do for the show.

'cause I was very curious.

I was like, what's the
history of all this?

And we, someone mentioned Sierra
IU, which I believe you're

using at least some of that.

that's a project that's been around
for quite some time over a decade.

And it's not a new idea, but like
a lot of these other technologies,

the devil's in the details.

we never really had an ability, to
capture and understand what a binary's

true dependencies were, whether
it's disc or, networking things.

until we had containers you mentioned
on here like it's in LXC, it's

in Docker, it's in pod man, like
this actually tool is used widely.

it's just maybe not, well known
to us end users because it's

packaged as a part of other tooling.

And I can sort of see a world where.

if this becomes more widespread, you're
gonna end up with haves and have nots

where my solution doesn't have live
migration or my solution does have live

migration at some point, maybe it's
ubiquitous, you're building functionality

on top of it, like your automation that
truly adds value around, spot instances

where my company maybe has never done spot
instances because it was too risky for

us and we didn't have the tooling to take
advantage of it without risking downtime.

I definitely have a couple of clients
that I've worked with over the last

couple of years that are like that,
where they're a hundred percent reserved

instances because they want the cheapest,
but they also need to guarantee uptime.

And they can't do that at a level
that live migrations would provide.

So they have to pay that extra
surcharge, for avoiding, ephemeral

instances and stuff like that.

to me, it gives me comfort
that the technology stack.

is part open source,
part community driven.

there's also the product, and private IP
side of this as well, but it's not like

you're reinventing the Linux kernel.

It wouldn't have been that long ago where
you had to actually throw in a Linux

module or kernel module rather, that
would only work on certain operating

system distributions of Linux and
that you would have to deploy a custom

ISO that wasn't that far in the past.

But now that we've got all these modern
things, I don't know if EBPF is involved

in this at all, but we've got more modern
abstractions that it feels like you can

just plug and play as long as you've
got the right networking components.

From an engineering perspective,
pretty awesome because it allows you

to build stuff on the stack like this.

the team's not on the call, but the
team that's developing this, good job,

Bravo, that's, some great engineering.

obviously anytime something is a year
long effort to crack a nut like this, I

feel sad for the people that had a six
month, like, no one's gonna see this

feature for a year and I'm gonna work all
year on it and I hope someone likes it.

So that's from a software development
perspective, that's the hard part.

That's the true engineering.

Um.

Yeah, absolutely.

we could talk about this forever,
but people have their jobs to do.

okay.

How do people get started do they just
go sign up for Cast AI and this is

like a feature outta the box that they
can implement in their clusters or.

Yep, absolutely.

it's in the UI now, so if people
want to sign up and onboard, we do

recommend having somebody on our sales
engineering team work with folks.

So reach out to us, We'll also reach
out when people sign up as well.

it's all straightforward.

There's no caveats.

It's helm charts to do the install
and then you set up the autoscaler.

we will be adding support
for Karpenter today.

It's using our autoscaler,
but we will support Karpenter.

around end of Q4 or early Q1,

Yeah, that's great.

my usual co-host is with AWS, so
they would greatly appreciate that.

I know that over the last
year, Karpenter's been

out, a little over a year.

We've had, a surprising number
of people on our Discord server.

For those of you watching, there's
a Discord server you can join.

There's a lot of people on Discord
talking about using Karpenter.

I'm really impressed with
the uptake on that project.

And in case you're wondering what
Karpenter is, it's with a K and it's

for Kubernetes, you can look on this
YouTube channel later because we did a

show on it and has had people talking
about it on the show and demoing it.

whenever it was released.

I think that was 2024.

I can't remember exactly.

alright, so everyone
knows how to get started.

Everyone now knows that they wish they
had live migrations and they currently

don't unless they're a Cast AI customer.

Where can we find you on the internet?

Where can people learn more
about what you're doing?

Are you gonna be at conferences soon?

I'm assuming, cast is probably
gonna have a booth at KubeCon again.

They always seemed to have a booth there.

we've got a big booth
at KubeCon this year.

I think we've got a 20 by 20.

We're gonna be doing demos and
presentations in the booth.

this is gonna be a big part of that.

we'll also be at reinvent in Vegas.

In later, early December, I
guess is first week of December.

so I'll be at both of those events.

I'm also really active on LinkedIn, so
if anybody wants to reach out to me on

LinkedIn, if you wanna set up a session
just to go into more detail, feel free

to ping me I post a lot of Kubernetes
content in general, best practices,

things that we see in the industry from
a Kubernetes evolution side of things,

and also obviously a bunch of cast stuff.

so, you know, feel free to follow or
connect Happy to share more information

Awesome.

well, I'm looking forward to hearing about
that continual proliferation of all things

live migration on every possible setup.

someday it'll be on FreeBSD with
some esoteric, Kubernetes variant.

It's, pretty cool to see
the evolution of this.

Well, thank you both for
being here, Philip and Dan.

see you all later.

Ciao.

Thanks for watching, and I'll
see you in the next episode.