DevOps and Docker Talk: Cloud Native Interviews and Tooling

The original title for this episode was AI
Killed the QA Star, which would make more

sense if you knew the eighties lore of
the very first music video played on MTV.

Um, that's a music television TV channel.

Back, back in the days, how you
watched videos before the internet.

That was in 1981 and it was called
Video Killed the Radio Star.

But I decided that a Deep cut title
was too obscure for this conversation.

Yet the question still remains, could
the increased velocity of shipping

AI generated code cause businesses
to leave human based QA behind,

presumably, because we're not gonna
hire any more of them, and we don't

want to grow those teams and operations
teams just because we created AI code.

And would we start relying more on
production observability to detect code

issues that affect user experience?

And that's the theory of today's
guest, Andrew Tunall, the President

and Chief product Officer at Embrace.

They're a mobile observability
platform company that I first

met at KubeCon London this year.

Their pitch was that mobile apps were
ready for the full observability stack

and that we now have SDKs to let mobile
dev teams integrate with the same

tools that we platform engineers and
DevOps people and operators have been

building and enjoying for years now.

I can tell you that we don't yet have
exactly a full picture on how AI will

affect those roles, but I can tell you
that business management is being told

that similar to software development,
they can expect gains from using AI to

assist or replace operators, testers,
build engineers, QA and DevOps.

That's not true, or at least not yet.

But it seems to be an expectation.

And that's not gonna stop us from trying
to integrate LLMs into our jobs more.

So I wanted to hear from observability
experts on how they think this

is all going to shake out.

So I hope you enjoyed this
conversation with Andrew of Embrace.

Hello.

Hey, Bret.

How are you doing?

Andrew Tunal is the president and
chief product officer of Embrace.

And if you've not heard of Embrace,
they are, I think your claim to me the

first time talked was, you're the first.

Mobile focused or mobile only
observability company in the

cloud native computing foundation.

Is that a correct statement?

I don't know if in,
probably in CNCF, right?

Because like we're, we were certainly
the first company that started going

all in on OpenTelemetry as the means
with which we published instrumentation.

obviously CNCF has a bunch of
observability vendors, some of

which do, mobile and web run,
but, completely focused on that.

Yeah.

I would say we're the first.

Yeah.

I mean, we can just, we can
say that until it's not true.

the internet will judge us harshly,
for whatever claims we make today.

Yeah.

well, and that just means
people are listening.

Andrew.

Okay, I told people on the internet that
we were going to talk about So, the idea

that you gave me that QA is at risk of
falling behind because if we're pretty

If we're producing more code, if we're
shipping more code because of AI, even

the pessimistic stuff about, you know,
we've seen some studies in the last

quarter around effective use of AI, it is
anywhere between negative 20% And positive

30 percent in productivity, depending
on the team and the organization.

So let's assume for a minute it's on
the positive side, and the AI has helped

the team produce more code, ship more
releases, have, even in the perfect world

that I see teams automating everything,
there's still usually, and almost always,

in fact, I will say in my experience,
100 percent of cases, there is a human at

some point during the deployment process.

Whether that's QA.

Whether that's a PR reviewer,
whether that's someone who spun

up the test instances to run
it in a staging environment.

There's always something.

So tell me a little bit about where
this idea came from and what you

think might be a solution to that.

Yeah.

I'll start with the, the belief that
I mean, AI is going to fundamentally

alter the productivity of software
engineering organizations.

I mean, I think the, CTOs I
talk to out there are making

a pretty big bet that it will.

you know, there's today and then
there's, okay, think about the, even

just the pace that, that AI has evolved
in, in the past, the past year and

you know, what it'll look like given
the investment that's flowing into it.

But if you start with that, the claim
that AI is going to kill QA is more

about the fact that, we built, Our
software development life cycle under

the assumption that software was slow to
build and relatively expensive to do so.

And so if those start to change,
right, like a lot of the, systems

and processes we put around, our
software development life cycle,

they probably need to change too.

because ultimately, If you say, okay,
we had 10 engineers and formerly we were

going to have to double that number to
go build the number of features we want.

And suddenly those engineers become, more
productive to go build the features and

capabilities you want inside your apps.

I find it really hard to believe that
those organizations are going to make

the same investment in QA organizations
and automated testing, etc. to keep

pace with that level of development.

The underlying hypothesis was like
productivity allows them to build

more software cheaper, right?

And like cheaper rarely correlates
with adding more humans to the loop.

yeah, the promise.

Like I make this joke and it's probably.

Not true in most cases, cause it's a
little old, but I used to say things

like, yeah, CIO magazine told them to
deploy Kubernetes or, you know, like

whatever, whatever executive VP level C,
CIO, CTO, and you're at that chief level.

So I'm making a joke about you, but,

funny.

but that's when you're not in
the engineering ranks and you

maybe have multiple levels of
management, you get that sort of.

overview where you're reading and you're
discussing things with other suits.

You're reading the suits magazines,
the CIO magazines, the uh, IT

pro magazines and all that stuff.

yeah, the point I'm making here is
that, people are being told that AI

is going to save the business money.

I, I think I've said this to several
podcasts already, but, you weren't here.

So I'm telling you that I was in a
media event in London, KubeCon, and,

they give me a media pass for some
reason, because I make a podcast.

So they act like I'm a journalist
standing around all real journalists

and pretending, but I was there and
I watched an analyst who's, from my

understanding, I don't actually know
the company, but they sound like the

Gartner of Europe or something like that.

and the words came out of the mouth
at an infrastructure conference,

the person said, my clients
are looking for humanless ops.

And I visibly, I think,
chuckled in the room because

I thought, well, that's great.

That's rich that you're
at a 12, 000 ops person.

Conference telling us that your
companies want none of this.

these are all humans here doing this work.

the premise of your argument about QA
is my exact same thoughts was nobody

is budgeting for more QA or more
operations personnel or more DevOps

personnel, just because the engineers
are able to produce more code with AI,

in the app teams and the feature teams.

And so we've all got to do better
and we've got to figure out where

AI can help us because if we don't.

They're going to, they're just
going to hire the person that

says they can, even though maybe
it's a little bit of a shit show.

my belief is that software
organizations need to change.

The way they work for an AI future, so
that might be cultural changes, it might

be role changes, it might be, the, you
know, the word, the words like human in

the loop get tossed around a lot when
it's about engineers interacting with

AI, and the question is, okay, what
does that actually look like, right?

Are we just reviewing AI's PRs and,
you know, kind of blindly saying, yep,

like they wrote the unit tests for
something that works, or is it like

we're actually doing critical thinking,
critical systems thinking, unique

thinking that allows us as owners of
the business and our user success?

to design and build better
software with AI as a tool.

and it's not just QA, it's kind of
like all along the software development

life cycle, how do we put the right
practices in place, and how do we build

an organization that actually allows us,
with this new AI driven future, you know,

whether it's agents, doing work on our
behalf, or whether it's us just with AI

assisted coding, to build better software.

And, yeah, I'm interested in what that
looks like over the next couple of years.

That's kind of the premise of my,
the new Adjetic DevOps podcast.

And also as I'm building out this new
GitHub Actions course, I'm realizing that

I'm having to make the best practices.

I'm having to figure out what is risky
and what's not because no one has really

figured this out yet in any great detail
and, in fact, at, KubeCon London in April,

which feels like a lifetime ago, there
was only one talk around using AI anywhere

in the DevOps and operations path.

To benefit us.

It was all about how to
run infrastructure for AI.

And granted, KubeCon is an infrastructure
conference for platform engineering

builders and all that, so it makes sense.

But the fact that really only one talk,
and it was a team from Cisco, which

I don't think of Cisco as like the
bleeding edge AI company, but it was a

team from Cisco, simply trying to get
And workflow or maybe you would call it

like an agentic workflow for PR review,
which in my case, in that case, I'm

presuming that humans are still writing
their code and AI is reviewing the code.

I'm actually, I was just yesterday, I
was on GitHub trying to figure out if

I, if there was a way to make branch
rules or, some sort of automation

rule that if the AI, wrote the PR,
the AI doesn't get to review the PR

Yeah, yeah, right.

we've got this coming
at us from both angles.

We've got AI in our IDEs.

We've got, multiple companies,
Devon and GitHub themselves.

They now have the GitHub
copilot code agent.

I have to make sure I
get that term, right.

GitHub copilot code agent,
which will write the PR and

the code to solve your issue.

And then they have a PR
code copilot reviewer.

Agent that will review the code.

It's the same models, different
context, but, it feels like that

doesn't, that's not a human in the loop.

So we're going to need these guardrails
and these checks in order to make sure

that code didn't end up in production
with literally no human eyeballs

ever in the path of that code being
created, reviewed, tested, and shipped.

cause we can do that now.

Like we did, we

Yeah, totally, you're totally good.

And I mean, you can easily perceive the
mistakes that could happen too, right?

I mean, before I, I took this role, at
Embrace, I was at New Relic for four and a

half years, and before that I was at AWS.

And so, obviously I've spent a
lot of time around CloudFormation

templates, Terraform, etc.

You can see a world where, you
know, AI builds your CloudFormation

template for you and selects an EC2
instance type because the information

it has about your workload is
optimal for this EC2 instance type.

But in the region you're running,
that instance type's not freely

available for you to autoscale to.

And pretty soon, you go try
to provision more instances,

and poof, you hit your cap.

Because, that instance type just
doesn't have availability in Singapore.

And, us as humans and operators, we learn
a lot about our operating environments, we

learn about our workloads, we learn about
the, the character, the peculiarities

of them that don't make sense to a
computer, but are, like, based on reality.

over time, maybe AI gets really
good at those things, right?

But the question is, how do we build
the culture into kind of be guiding our,

our, you know, army of assistance to
build software that really works for our

users, instead of just trusting it to go
do the right thing, because we view, you

know, everything as having a true and
pure result, which I don't think is true.

a lot of the tech we build is
for people who build consumer

mobile apps and websites.

I mean, that is what the
tech we build is for.

And you can easily see, you know, some
of our, our engineers have been playing

around with using AI assisted coding to
implement our SDK in a consumer mobile

app that, and it works quite well, right?

You can see situations where, an
engineer gets asked by somebody, a

product manager, a leader to say, Hey,
you know, we got a note from marketing.

They want to implement this new
attribution SDK that's going to go.

build up a profile of our users
and help us build more, you know,

customer friendly experiences.

You have the bots go do it, it tests
the code, everything works just fine.

And then, for some reason that SDK makes
a uncached request out to US West 2

for users globally that, you know, for
your users in Southeast Asia instead

of adding an additional six and a half
seconds to app startup because physics.

And, what do those users do?

if you start an app and it's.

sits there hanging for four, five,
six seconds, and you don't have to use

the app because it's you know, you're
waiting for your boarding pass to come

up and you're about to get on the plane.

You probably perceive it
as broken and abandoned.

And to me, that's like a reliability
problem that requires systems thinking

and cultural design around how you're
engineering organization to avoid.

one that I don't think is
immediately solved with AI.

right.

observability is only getting more complex
even, you know, it's a cat and mouse game.

I think in all, in all regards, and I
think everything we do has yin yang I'm

watching organizations that are full
steam ahead, aggressively using ai, and.

I'd love to see some stats.

I don't know if you have empirical
evidence or if you sort of anecdotal stuff

from your clients, but where you see them
accelerate with AI and then almost have a

pulling back effect because they realize
how easy it is for them to ship more bugs.

and then sure, we could have the AI
writing tests, but it can also write

really horrible tests, or it can
just delete tests because it doesn't

like them because they keep failing.

it's that to me multiple times.

In fact, I think, Oh, what's his name?

Gene, not, it wasn't Gene Kim.

There's a, a famous, it might've been
the guy who created Extreme Programming.

I think I heard him on a podcast talking
about he wished he could make certain, all

of his test files have to be read only.

Because he writes the tests, and then
he expects the AI to make them pass,

and the AI will eventually give up
and then want to rewrite his tests.

And he doesn't seem to be able to stop
the AI from doing that, other than just

denying, deny, or cancel, cancel, cancel.

And there's a scenario where
that can easily happen in

the automation of CI and CD.

Where, you know, suddenly it decided
that the tests failing were okay.

And then it's going to push
it to production anyway, or

whatever craziness ensues.

Do you see stuff happening
on the ground there?

That's

did you read that article about, the,
AI driven store, a fake store purveyor

that Anthropic created named Claudius?

That, when it got pushed back from
clo, the Anthropic employees around

what it was stocking, its fake fridge.

Its fake store with it called security
on the employees to try to displace them.

I mean, it's, I think what you're
pointing to is like the moral compass of

AI is not necessarily, it, like that's
a very complex thing to solve, right?

Is is this the right thing to do or not?

Because it's trying to solve the
problem however it can solve the

problem with whatever tools it has.

Not whether the problem
is the right one to solve.

And right is, I mean, obviously, you
know, even humans struggle with this one.

Right.

I've just been labeling it all as taste.

like the AI lacks taste

Yeah, that's

like in a certain team culture,
if you have a bad team culture,

deleting tests might be acceptable.

I used to work with a team
that would ignore linting.

So I would implement linting.

I'd give them the tools to customize
linting and then they would ignore

it and accept the PR anyway.

And did not care.

They basically followed the
linting rules of the file.

Now they were in a monolithic
repo with thousands and thousands

of a 10 year old Ruby app.

But they would just ignore the linting.

And I always saw that as a
culture problem for them.

That I as the consultant can't,
I would always tell the boss that

I was working for there that I
can't solve your culture problems.

I'm just a consultant here trying
to help you with your DevOps.

You need a full time person leading
teams to tell the rest of the team

that this is important and that these
things matter over the course of

years as you change out engineers.

You can't just go with,
well, this file is tab.

This file is, spaces.

And this one follows, especially
in places like Ruby and Python and

whatnot, where there's multiple, sort
of style guys and whatnot like that.

But I just call that a taste
issue or culture issue.

AI doesn't have culture or taste.

It.

It just follows the rules we give it.

we're not really good yet as
an industry at defining rules.

I think that's actually part
of the course I'm creating is

figuring out all the different
places where we can put AI rules.

We've seen them put in repos.

We've seen them put in
your UI and your IDEs.

Now I'm trying to figure out how do I
put that into CI and how do I put that

into even like the operations if you're
going to try to put AIs somewhere in your,

somewhere where they're going to help with
troubleshooting or visibility or discovery

of issues, like they also need to have.

Taste, which I want to change all
my rules files to taste files.

I guess that's my platform.

I'm standing.

Yeah.

Cause it,

Yeah, I mean, at the same time, you
probably don't want to be reviewing,

hundreds of PRs that an AI robot is
going through, just changing casing of

legacy code to, meet your, to the, Change
obviously introduces the opportunity

for failure, bugs, etc. And if it's,
you know, it's somewhat arbitrary

simply because you've given it a new
rule that this is what it has to do.

I mean, I had a former co worker
many years ago, like 15, 20 years

ago, who, was famous for just, going
through periodically and making

nothing but casing code changes
because that's what he preferred.

And it was just this, endless stream
of, trivialized changes on code we

hadn't touched in months or years.

That's it.

that would inevitably lead to
some sort of problem, right?

And because,

The toil of that.

I'm just like, yeah, that you're
giving me heartburn just by

telling me about that story.

yeah, it's, well, I've
gotten over the PTSD of that.

It's a long time ago.

So, okay.

So, what are you seeing on the ground?

do you have some examples of
how this is manifesting in apps?

I mean, you've already given me
a couple of things, but I'm just

curious if you've got some more,

I kind of alluded to it
at the very beginning.

obviously, in my role, I talk to a lot
of senior executives about directionally

where they're going with their engineering
organizations, because I think, the

software we build, it does a bunch of
tactical things, but at a broader sense,

it allows people to measure reliability as
expressed through, you know, our customers

staying engaged in your experience by
virtue of the fact that they otherwise

liking your software are having a better
technical experience with the apps you

build, that the front end and ultimately
measuring that and then thinking through

all of the underlying root causes is a
cultural change in these organizations

and how they think about reliability.

It's no longer just like are
the API endpoints at the edge of

our data center delivering their
payload, you know, responding in

a timely manner and error free.

And then the user experience is well
tested and, you know, we ship it to prod.

And.

That's enough.

it really isn't a shift in how people
think about it, but when I talk to them,

I mean, a lot of CTOs really are taking
a bet that AI is going to be the way

that productivity gains, I won't say make
their business more efficient, but they

do allow them to do more with less, right?

You know, like it or not, especially
over the past few years, I mean, in B2B,

we're in the B2B SaaS business, there's
been, you know, times have definitely

been tougher than they were in the early,
2020s, for kind of everyone, the, I think

there's a lot of pressure in consumer
tech with tariffs and everything to do

things more cost effectively, and first
and foremost on the grounds, this is a

change we are seeing, whether we like
it or not, and, you know, we can argue

about whether it's going to work and
how long it'll take, but the fact is

that, like you said, you mentioned that
the CIO magazines, leadership is taking

a bet this is going to happen, and I
think as we start to talk about that

with these executives, the question is
like, is the existing set of tools I have

to cope with that reality good enough?

And, yeah, I guess my underlying
hypothesis is it probably isn't

for most companies, right?

if you think about the, the world of,
you know, a web as an example, like

a lot of, a lot of companies that are
consumer facing tech companies will

measure their core web vitals, in
large part because it has SEO impact.

And then they'll put a, exception handler
in there, like Sentry that grabs, you

know, kind of JavaScript errors and
tries to give you some level of impact.

around, you know, how many are impacting
your users and whether it's high

severity and then you kind of have to
sort through them to figure out which

ones really matter for you to solve.

so take, existing user frustration
with, human efficiency of delivering

code and the pace, the existing pace.

users are already frustrated that
Core Web Vitals are hard for them to

quantify what that really means in
terms of user impact and whether a user

decides to stay on the site or not.

so, yeah.

and the fact that they're overwhelmed with
the number of JavaScript errors that could

be out there because it's really, I mean,
you go to any site, go to developer tools.

look at the number of JavaScript errors
you see, and then, you know, take your

human experienced, idea of how you're
interacting with the site, and chances

are most of those don't impact you, right?

They're just like, it's a, it's an
analytics pixel that failed to load, or

it's a library that's just barfing some
error, but it's otherwise working fine.

So take that and put it on steroids now
where you have a bunch of AI assisted

developers doubling or tripling the
number of things they're doing or

just driving more experimentation,
right, which I think a lot of

businesses have always wanted to do.

But again, software has been kind
of slow and expensive to build.

And so If it's slow and expensive
to build, my thirst for delivering

an experiment across three customer
cohorts when I can only, deliver one,

given my budget, just means that, I
only have to test for one variation.

We'll now triple that, or quadruple
it, or, and, you know, multiply that

by a number of, arbitrary user factors.

It just gets more challenging,
and I think we need to think about

how we measure things differently.

once the team has the tools in place to
manage multiple experiments at the same

time, that just ramps up exponentially
until the team can't handle it anymore.

But if AI is giving them a chance
to go further, then yeah, they're

just, they're going to do it.

I mean,

yeah, I mean, you're going to get
overwhelmed with support tickets

and bad app reviews and whatever it
is, which I think most people are.

Most business leaders would be
pretty upset if that's how they, are

responding to reliability issues.

I was just gonna say, we've already had
decades now of pressure at all levels

of engineering to reduce personnel.

you need to justify every new hire pretty
significantly unless you're funded and

you're just, you know, in an early stage
startup and they're just growing to the

point that they can burn all their cash.

having been around 30 years in
tech, I've watched operations

get, you know, Merged into DevOps.

I've watched DevOps teams get, which
we didn't traditionally call them that.

We might have called them sysadmins
or automation or build engineers

or CI engineers, and they get
merged into the teams themselves.

And the teams have to take
on that responsibility.

I mean, We've got this weird culture
that we go to this Kubernetes conference

all the time and one of the biggest
complaints is devs who don't want to

be operators, but it's saddled on them
because somehow the industry got the

word DevOps confused with Something.

And we all thought, Oh, that
means the developers can do ops.

That's not what the word meant, but we've,
we've worked, we've gotten to this world

where I'm getting hired as a consultant
to help teams deal with just the sheer

amount of ridiculous expectations a
single engineer is supposed to have,

that not the knowledge you're supposed
to have, the systems are supposed to

be able to run while making features.

And it's already, I feel like a
decade ago, it felt unsustainable.

So now here we are having to
give some of that work to AI.

When it's still doing random
hallucinations on a daily basis,

at least even in the best models.

I think I was just ranting yesterday
on a podcast that SWE bench, which is

like a engineering benchmark website
for AI models and how well they

solve GitHub issues, essentially,
and the best models in the world.

Can barely get two thirds of them.

Correct.

And that's, if you're paying the
premium bucks and you've got the premium

foundational models and you are on the
bleeding edge stuff, which most teams

are not because they have rules or
limitations on which model they can use,

or they can only use the ones in house,
or they can only use a particular one.

and it's just, it's one of
those things where I feel like

we're being pushed at all sides.

and at some point, it's amazing
that any of this even works.

It's amazing that apps
actually load on phones.

it's just, it just, it feels like
garbage on top of garbage, on top

of garbage, turtles all the way
down, whatever you want to call it.

So where are you coming in here to
help solve some of these problems?

yeah, that's fair, yeah.

I'll even add to that, All of that's
even discounting the fact that new tools

are coming that make it even simpler
to push out software that like barely

works without even the guided hands
of a software engineer who has any

professional experience writing that code.

some of the vibe coding tools, like
our product management team uses,

largely for rapid prototyping.

And I can write OK Python.

I used to write OK C sharp.

I have never been particularly
good at writing JavaScript.

I can read it OK, but like when
it comes to fixing a particular

problem, quickly get out of my depth.

That's not to say, I couldn't
be capable of doing it.

It's just not what I do every day,
nor do, I particularly have the energy

when I'm, you know, done with a full
workday to, go teach myself JavaScript.

And, I'll build an app with
one of the Vibe coding tools

as a means of communicating
how I expect something to work.

So if you have any questions, feel
free to reach out to me, and I'll

be happy to answer them, and I'll
be excited to hear back from you.

And for coming.

and I'm like, ah, it's better
to just delete the project

and start all over again.

And, you know, if you can make
it work, that doesn't necessarily

mean it'll work at scale.

It doesn't necessarily mean that there
aren't like a myriad of use cases

you haven't tested for as you click
through the app in the simulator.

and so, you know, I think the
question is, okay, you know, given

the fact that we have to accept more
software is going to make its way

into human beings hands, because we
build software for human beings, it's

going to get to more human beings.

How do we build a, a reliability
paradigm where we can measure

whether or not it's working?

And I think that stops focusing
on, kind of, I guess to go back

to the intentionally inflammatory
title of today's discussion, it

stops focusing on like a zero
bug paradigm where I test things.

I test every possible
pathway for my users.

I, you know, have a set of
requirements, again, around

trivialized performance and stuff.

And, trying to put up these kind of
barriers to getting code into human hands.

And I just accept the fact more code
is going to get into human hands faster

at a pace I can't possibly control.

And so, therefore, I have to put
measurements in the real world.

So, I use a lot of different tools around
my app so that I can be as responsive

as possible to resolving those issues
when I find them, which is I guess my,

you know, charity majors who co founded
Honeycomb, she, her and I were talking a

few months ago and big fan of stickers,
she shipped me an entire envelope of

stickers and they're all like, you
know, ship fast and break things, right?

I tested production, stuff like that.

And somehow I feel like, you know,
in our world, because we build

observability for front end and mobile
experiences, like web and mobile

experiences, I feel like that message
just hadn't gotten through historically.

part of it's because like release
cycles on mobile were really slow.

you had to wait days for
an app to get out there.

Part of it was software is expensive
to build and slow to build and so

getting feature flags out there where
you can operate in production was hard.

Part of it was just the observability
paradigm hadn't shifted, right?

Like the, the paradigm of measure
everything and then find root cause

had not made its way to frontend.

It was more like measure the known
things you look for, like web exceptions,

like core web vitals are on mobile,
look at crashes, and that's about it.

And the notion of, okay, measure
whether, users are starting the app

successfully, and when you see some
unknown outcomes start to occur or users

start to abandon, how can you then sift
through the data to find the root cause?

Hadn't really migrated its way.

And that's what we're trying to do.

we're trying to bring that paradigm
of how do you define the A, for

lack of a better term, the APIs.

The things that your humans interact
with, with your apps, the things they

do, you don't build an API in your app
for human beings, you build a login

screen, you build a checkout screen,
a cart experience, a product catalog.

How do we take those things and measure
the success of them and then try to,

attribute them to underlying technical
causes where your teams can have, better

have those socio technical conversations?

so that they can understand and then
resolve them, probably using AI, right?

as we grow.

but like it, it allows better,
system knowledge and the interplay

between real human activities and
the telemetry we're gathering.

Yeah.

Have you been, I'm just curious,
have you been playing with, having

AI look at observability data,
whether it's logs or metrics?

have you had any experience with that?

I'm asking that simply as a generic
question, because the conversations

I've had in the last few months, it
sounds like AI is much better at reading

logs than it is at reading metrics or
dashboards or anything that sort of

lacks context or, you know, it's not
like we're putting in alt image messages

for every single dashboard graph.

And that's probably all coming from
the providers just because if if

they're expecting AI to look at stuff.

they're gonna have to give more context,
but it sounds like it's not as easy

as just giving AI access to all those
systems and saying, yeah, go read the

website for my dashboard, Grafana,
and figure out what's the problem?

I've seen it deployed in two ways, one
of which I find really interesting and

something we're actively working on,
because I think it's just a high degree

of utility and it's, You know, given
the kind of state of LLMs, I think it's

probably something that's relatively easy
to get right, which is the, the notion of,

when you come into a product like ours,
or a Grafana, or, you know, a New Relic,

you probably have an objective in mind,
maybe you have a question you're trying

to ask, what's the health of this service?

Or, I got page on a particular issue.

I'm going to, I need to build a chart
that shows me like the interplay between

latency for this particular service
and the success rate of, you know,

some other type of thing, more like
database calls or something like that.

Today, like people broadly have to
manually create those queries and it

requires a lot of human knowledge around
the query language or schema of your data.

And I think there's a ton of opportunity
for us to simply ask a human question

of You know, show me a query of all
active sessions on this mobile app for

the latest iOS version and the number
of, traces, like startup traces that

took greater than a second and a half.

And have it just simply pull up your
dashboard and query language and build

the chart for you quite rapidly, which
is a massive time savings, right?

And.

It also just makes our tech, which
you're right, can get quite complex, more

approachable by your average engineer.

Which, you know, I'm a big believer that
if every engineer in your organization

understands how your systems work.

And the data around it, you're going
to build a lot better software,

especially as they use AI, right?

Because now they, they understand
how things work and they can better

provide instructions to the robots.

so I think that's really
a useful, interesting way.

And we've seen people start to roll that
type of assistant act, functionality out.

The second way I've seen deployed, I
see mixed results, which is I mean, an

incident, go look at every potential
signal that I can see related to this

incident and try to tell me what's
going on and get to the root cause, and

more often than not, I find it's just a
summarization of stuff that you, as an

experienced user, probably would come to
the exact conclusions on, I think there's

utility there, certainly, it gets you a
written summary quickly of what you see.

But I do also worry that, it doesn't
apply a high degree of critical thinking.

and you know, an example of where lacking
context, I mean, it wouldn't be very

smart, right, is you've probably seen
it, every traffic chart, around service

traffic, depending upon how it runs,
tends to be pretty lumpy with time of day.

Because Most companies don't
have equivalent distribution

of traffic across the globe.

Not every country across the globe
has an equivalent population.

And so you'd tend to see these spikes
of where, you know, you have a number

of service requests spiking during
daylight hours or the Monday of every

week because people come into the office
and suddenly start e commerce shopping.

And you see it taper throughout the
week, or you taper into the evening.

I think that's normal.

You understand that as an
operator of your services, because

it's unique to your business.

I think the AI would struggle, lacking
context around, your business to

understand, somewhat normal fluctuations,
or the fact that, You know, a marketing

dropped a campaign where there's no
data inside your observability system

to tell you that that campaign dropped.

It's not a release.

see your AI is lacking context.

It's lacking the historical context
that the humans already have implicitly.

Yeah,

and I mean that context might be in a
Slack channel where marketing said we

just dropped an email and you know an
email drop so expect an increased number

of requests to this endpoint as you know
people retrieve their, their special

offer token or whatever that will allow
them to use it in our checkout flow.

today like just giving, if we provided
that scope to a, an AI model within our

system we would lock that type of context.

Yeah, if I'm not creating any of
these apps, I'm sure they've already

thought of all of this, but, the
first thing that comes to mind is

well, the hack for me would be give
it access to our ops slack room,

Right.

we're probably all having those
conversations of oh, What's going on here?

And someone in someone's who had
happened to get reached out to for

marketing was like, well, you know,
yesterday we did send out a, you

know, a new coupon sale or whatever.

So, yeah, having it read all that
stuff might be, necessary for it to

understand the, because you're right.

it's not like we have a dashboard in
Grafana that's number of marketing

email, you know, the email sent per day
or the level of sale we're expecting,

based on in America, it's the 4th of
July sale, or, you know, some holiday

in a certain region of the world.

Or a social media influencer dropping
like some sort of link to your

product that suddenly, you know, that
it's a new green doll that people

attach to their designer handbags.

I don't know like anything about what
the kids are into these days, but

it seems kind of arbitrary and like
I, I would struggle to predict that.

Let me put it that way.

they're into everything old.

So it's into everything that I'm into.

Yes.

all right.

So if we're talking about this at a
high level, we talked a little bit

about before the show around how
embrace is thinking about, observability

and particularly on mobile, but
you know, anything front end there.

the tooling ecosystem for, engineers
on web and mobile is pretty rich.

but they all tend to be just like
hammers for a particular nail, right?

It's you know, How do we give you a
better craft reporter, better exception

handler, how do we go measure X or Y?

some of the stuff that we're thinking
about, which is really how we define

the objective of OB observability
for, the coming digital age, right?

Which is you know, as
creators of user experiences.

I think my opinion is that we
shouldn't just be measuring

like crash rate on an app.

We should be measuring, are users
staying engaged with our experience?

And when we see they are not, sometimes,
crashes, I mean, the obvious, the

answer is obviously they can't, right?

Because the app explodes.

But, I think, you know, I was
talking to a senior executive at

a, a massive food delivery app.

And it's listen, we know, anecdotally,
there's more than just crashes that make

our users throw their phone at the wall.

you're trying to You're trying to
order lunch at noon and something's

really slow or you keep running into
a, just a validation error because we

shipped you an experiment thinking it
worked and you can't order the item

you want on the two for one promotion.

you're enraged because you really want
the, you know, the spicy dry fried chicken

hangry.

and you want two of them because I
want to eat the other one tonight.

and you've already suckered me
into that offer, you've convinced

me I want it, and now I'm having
trouble, completing my objective.

And broadly speaking, the observability
ecosystem on the front end really

hasn't measured that, right?

We've used all sorts of proxy
measurements in, out of the data

center because the reliability story
has been really well told and evolved.

Over the past 10 to 15 years in the data
center world, but it just really hasn't

materially evolved in the front end.

And so, a lot of that's like shifting
the objective from how do I just measure

counts of things I already know are bad to
measuring what users engagement looks like

and whether I can attribute that to change
in my software or defects I've introduced.

So, that's kind of the take.

just about anybody who has ever built
a consumer mobile app has Firebase

Crashlytics in the app, which is
a free service provided by Google.

It was a company a long time ago that.

Crashlytics that got bought by Twitter
and then got reacquired by Google.

it basically gives you rough cut
performance metrics and crash reporting.

Right.

I would consider this like the
foundational requirement of any level

of, app quality, but to call this NSAID
app quality, I think, you know, our

opinion is that would be a misnomer.

So we're going to go kind of go
through what this looks like, right.

Which is it's giving you things.

You would expect to see, a number of
events that are crashes, et cetera, and,

you can do what you would expect here,
crashes are bad, so I need to solve a

crash, so I'm going to go into a crash,
view stack traces, get the information I

need to actually be able to resolve it.

and, you know, I think we see a lot
of customers before we talk to them

who are just like, well, I have crash
reporting and I have QA, that's enough.

there's a lot of products
that have other features.

So like Core Web Vital measurements
on a page level, this is Sentry.

It's a lot of data, but I don't
really know what to do with this.

beyond, okay, it probably
has some SEO impact.

There's a, you know, Core, bad Core
Web Vital or slow, you know, slow

something for a render on this page.

How do I actually go
figure out root cause?

But again, right, this is a single signal.

So this is kind of, you don't know that,
whether or not the P75 core web vital here

that is, considered scored badly by Google
is actually causing your users to bounce.

And I think that's important because I
was reading this article the other day

on this like notion of a performance
plateau, like there's empirical science

proving that like faster core web
vitals, especially with like content

paint and interaction to next paint.

et cetera, improve bounce rate
materially, like people are less likely

to bounce if the page loads really fast.

But at some point, like if it's long
enough, there's this massive long

tail of people who just have a rotten
experience and you kind of have to

figure out, I can't make everyone
globally have a great experience.

Where's this plateau where, I know
that I'm improving the experience for

people who I'm likely to retain and
improve their bounce rate versus I'm

just, you know, going to live with this.

And so we kind of have a different
take, which is like we wanted to

center our experience less on just
individual signals, and more on

like these flows, these tasks that
users are performing in your app.

So if you think about the key flows,
like I'm breaking these down into

the types of activities that I
actually built for my end users.

And I want to say, okay, how many
of them were successful versus

how many ended in an error, like
something went truly bad, right?

You just could not proceed.

Versus how many abandoned, and
when they abandoned, why, right?

Did they abandon because they, they
clicked on a product catalog screen, they

saw some stuff that they didn't like?

Or did they abandon because the product
catalog was so slow to load, and images

slow, like slow, so slow to hydrate?

That they perceived it as
broken, lost interest in the

experience and ended up leaving.

And so, the way you do that is you
basically take the telemetry we're

emitting from the app, the exhaust
we collect by default, and you create

these start and end events that allow
you to then, we post process the data.

We go through all of these sessions
we're collecting, which is basically

a play by play of like linear
events that users went through.

And we hydrate the flow to tell
you where people are dropping off.

and so you can see like their actual
completion rates over time, you know,

obviously it's a test app, so there's
not a ton of data there, but what gets

really cool is we start to, we start to
build out this notion of once you see.

the, issues happen, well how can I now
go look at all of the various attributes

of those populations under the hood
to try to specify which of the things

are most likely to be attributed to
the population suffering the issue?

So that could be an experiment.

it could be a particular mobile,
a particular version they're on.

It could be, an OS version, right?

You just shipped an experiment that
isn't supported in older OSs, and those

users start having a bad experience.

And then each of those gets you down to
what we call, this, user play by play

session timeline, where you basically
get a full Recreation of every part of

the exhaust stream that we're gathering
from you interacting with the app or

website just for reproduction purposes.

once you've distilled here,
you can say, okay, now let me

look at that cohort of users.

And so I can do pattern
recognition, which I think is pretty

Hmm.

So for the audio audience that didn't,
that didn't get to watch the video,

what are some of the key sort of, if
someone's in a mobile and front end

team and this is actually going back
to a conversation I had with one of

your Embrace team members at KubeCon in
London, what are some of the key changes

or things that they need to be doing?

If I guess if I back up and say the
premise here is that if I'm it's

there's almost like two archetypes.

What am I trying to say here?

There's two archetypes
that I'm thinking about.

I'm thinking about me, the DevOps
slash observability system maintainer.

I've probably set up.

Elk or, you know, I've got
the key names, the Loki's, the

Prometheus, the, the Grafana's.

I've got all these things
that I've implemented.

I've brought my
engineering teams on board.

They like these tools.

They tend to have, especially for
mobile, they tend to have other

tools that I don't deal with.

They might have platform, like the, the

Traditionally, right?

We're obviously trying to change that.

But yeah, traditionally, they
have five or six other tools that

don't play into the ecosystem,
observability ecosystem you have set up.

Yeah.

So, so we're on this journey to
try to centralize, bring them

into the observability world.

you know, traditional mobile
app developers might not even.

Be aware of what's going on in the
cloud native observability space and

we're bringing them on board here.

Now suddenly they get even more code
coming at them that's slightly less

reliable or maybe presents some unusual
problems that we didn't anticipate.

So now, you know, we're in a
world where suddenly what we have

in observability isn't enough.

yeah,

you're a potential solution.

What are you looking at for behaviors
that they need to change to things

that people can take home with them?

And

I mean, I guess the way I think about
it is, right, the way, the reason

observability became so widely adopted
in server side products was because in

an effort to more easily maintain our
software and to avoid widespread defects

of high blast radius, we shifted from a
paradigm of like monoliths deployed on

bare metal to virtualization, which was,
you know, various container schemes kind

of that has right now most widely been
around, Kubernetes and microservices

because you could scale them independently
and you could deploy them independently.

Right.

And that complexity of the deployment
scheme and the different apps and services

interplaying with each other necessitated
an x ray vision into your entire system

where you could understand system wide
impacts to your, the end of your world.

And the end of your world, from the
most part became your API surface

layer, the things that served
your web and mobile experiences.

And, you know, there are
businesses that just serve APIs.

Right, but broadly speaking, the brands
we interact with as human beings serve us

visual experiences that we interact with.

right.

It's the server team managing
the server analytics, not so much

the client device analytics that

right.

the world has gotten a lot more
complicated in what the front

end experience looks like.

And you could have a service that
consistently responds and has a nominal

increase in latency and is well within
your alert thresholds, but where the

SDK or library designed for your front
end experience suddenly starts retrying

a lot more frequently, delivering
Perceived latency to your end user.

And, and so I think the question
is, could you uncover that incident?

Because if users suffer perceived latency
and therefore abandon, what metrics do

you have to go measure whether or not
users are performing the actions you

care about them performing, whether
that's attributable to system change?

In most instances, I don't think
most observability systems have that.

and then the second question is, right,
so, and by the way, Bret, that's the

underlying supposition that in a real
observability scheme, mean time to detect

is important, is as important, if not
more so, than mean time to resolve.

the existing tooling ecosystem for
frontend and mobile has been set up to

optimize mean time to resolve for known
problems where I can basically just

count the instance and then alert you.

So, And, you know, the lack of
desire to be on call, like I've

heard this stupid saying that
there's no such thing as a front end

emergency, which is like ridiculous.

If I'm, you know, if I'm a major travel
website and I run a thousand different

experiments and a team in, Eastern
Europe drops an experiment that affects

users globally, 1 percent of users
globally in the middle of my night.

That, makes the calendar control
broken and that some segment of that

population can't book their flights.

That sounds a lot like a
production emergency to me.

I, that has material business
impact in terms of revenue.

Or the font color changes
and the font's not readable

yeah, I guess I am imploring the
world to shift to a paradigm where

they view like users willingness
and ability to interact with your

experiences as a reliability signal.

And I think the underlying supposition
is that this only becomes more acute of a

problem as the number of features we ship.

I guess I'm starting with the
belief that from what I hear, people

are doubling down on this, right?

They're saying we need to make software
cheaper to build and faster to build.

Because it is a competitive environment
and if we don't do it, somebody else will.

and as that world starts to
expand, the acuity of the

problem space only increases.

Yeah.

I think your theory here, matches
are well with some other like we're

all kind of in this little bit of
we've got a little bit of evidence.

We hear some things
and we've got theories.

We don't have years of facts of
exactly how AI is affecting a lot of

these things, but other compatible
theories I've heard recently on this,

actually with three guests on the show.

that I'm just coming to mind.

One of them is, because of the velocity
change and because of the, it is

this increase in velocity, it only
is going to increase the desire for

standardization in CI and deployment,
which those of us in, if you've been

living in this Kubernetes world, we've
all been trying to approach that.

Like we're all leaning into Argo CD
as one of the, as the number one way

on Kubernetes to deploy software.

you know, we've got.

This GitOps idea of how to standardize
change as we deliver in CI.

It's still completely wild, wild west.

You've got a thousand vendors
all in a thousand ways you

can create your pipelines.

And hence the reason I need to make
courses on it for people, because

there's a lot of art still to it.

We don't have a checkbox of
this is exactly how we do it.

And in that world, the theory is.

Right now that maybe AI is going to allow
us to act where it's going to force us

to standardize because we can't have a
thousand different workflows or pipelines

that are all slightly different for
different parts of our software stack,

because then when we get to production and
we have started having problems, or if the

AI is starting to take more control, it's
just going to get worse because the AI

Yeah, and at the end of the day,
right, standardization is more

for the mean than the outliers.

And I think a lot of people assume they're
like, oh, we can do it right because we

have hundreds of engineers working on like
our, you know, our automation and stuff.

It's do you know how many
companies there are out there

that are not technology companies?

They're a warehouse
company with technology.

Right.

And like standardization allows them
to more often than not build good

technology, to measure correctly, to
deploy things the right way, like we're

all going to interact with it in some
way, right, and as the pressure for more

velocity and them building technology
speeds up, the need for you know, a

base layer of doing things, where it's
consistently right, but only increases.

And I think that's, you know,
for the world it's a challenge,

for us it's an opportunity.

Yeah.

Awesome.

I think that's a perfect
place to wrap it up.

we've been going for a while now.

but, Andrew, io, right?

That's,

Yeah.

www.

embrace.

io.

let's just bring those up.

We didn't show a lot of
that stuff, but, the,

in case people didn't know, I made a short
a few months ago about Embrace or with one

of the Embrace team members at KubeCon.

which we had touched a little bit on
this show, but we talked about that.

Observability is here for your mobile
apps now and your front end apps and

that those people, those developers
can now join the rest of us in this,

world of modern metrics, collection
and consolidation of logging tools

and bringing it all together into
ideally one, one single pane of glass.

if you're advanced enough to figure
all that out, the tools are making

it a little bit easier nowadays,
but I still think there's a lot of.

A lot of effort in terms of
implementation engineering to get all

this stuff to work the way we hope.

But, it sounds like that you all
are making it easier for those

people, with your platform.

We're definitely trying.

Yeah.

I think it'd be a future state that would
be pretty cool would be the ability for

operators to look at a Grafana dashboard
or a, you know, dashboard or Chronosphere

or New Relic or whatever it is.

see that, you know, they see a disen a,
an engagement decrease on login on a web

property, and immediately open an incident
where they page in the front end team and

the core team servicing the off APIs in
a company and have them operating on data

where the front end team can be like.

We're seeing a number of retries
happening after we updated to the

new version of the API that you
serve for logging credentials.

even though they're all 200s, they're
all successful requests, what's going on?

And that backend team says, well, it
looks like our, you know, we were slow

rolling it out and the P90 latency
is actually 250 milliseconds longer.

Why would that impact you?

And they say, well, the SDK retries
after 500 milliseconds and our P50

latency before this was 300 milliseconds.

So, 10 percent of our users or
something are starting to retry

and that's why we're seeing this.

You know, the answer here is to increase.

The resource provisioning for the
auth service to get latency back

down and or change our SDK to have
a more permissive retry policy.

and, you know, have teams be able to
collaborate around the right design

of software for their end users and
understand the problem from both

perspectives, but be able to kick
off that incident because they saw

real people failing to disengage,
not just some server side metrics,

which I think would be pretty neat.

Yeah.

And I should mention you all,
if I remember correctly in cloud

native, you're a lead maintainers
on the mobile observability SDK.

Is that, am I getting that right?

I'm trying

we have engineers who are
provers, on Android and iOS.

we have the, from what I'm
aware of, the only production

React native, OpenTelemetry SDK.

we are also participants in a new
browser SIG, which is a, a subset

of the former JavaScript SDK.

So our OpenTelemetry SDK
for web properties is.

Basically a very slimmed down chunk of
instrumentation that's only relevant

for browser React implementations.

so yeah, working to advance the kind of
standards community in the cloud native

environment for what, Instrumenting
in real world runtimes where Swift,

Kotlin, JavaScript are executed.

Nice.

Andrew, thanks so much for being here.

So we'll see you soon.

Bye everybody.