DevOps and Docker Talk: Cloud Native Interviews and Tooling

Bret: This is DevOps and Docker
Talk, and I'm your host, Bret Fisher.

In this episode, we're gonna get
into the weeds with agents and CI.

I've got the team from Semaphore on the
show, Marcos and Marko, who joined the

show to really just talk about their
transformation over the last year.

Now, if you've been around a minute
with this show, I had Semaphore on

probably five years ago, four to
five years ago, and we talked about

sort of typical CI/CD stuff, and this
time it was completely different.

just in the last year, they have
completely transformed their product

and their tooling to be AI first,
and we get into what that really

means, and it's not… In this case,
I don't feel like it's a buzzword.

I think it truly is an optimized
way for interacting with your

CI from your agent harnesses.

And two, they open sourced their
platform, so you can now run it yourself

if you don't wanna pay them to run it.

So they offer the typical options of
running their open source CI or running

it on their platform, paying for runners
and having them run their, their runners.

They actually, just recently did a
benchmark comparing themselves to a lot

of the other CIs and saw that they were
cheaper and faster than the others.

we dug into exactly what it means to
just drive your CI completely from an

agent harness and not just sort of in
the contrived way that I've been doing

it, which is, you know, telling it, "Hey,
can you go check on that CI job? Hey, can

you make sure that that git push commit
thing happened?" And, like, all the…

Like, they've sort of rethought it from,
I would say, first principles of Why does

CI running, or why do tests or workflows
that I need to run around my code, like

linting, testing, and all the different
forms of that testing, security scanning,

like why do I need to wait on a git
push in order to kick those things off?

What would happen if my agent could know
everything about my CI just from starting

up, have all the tools and MCPs and skills
and command lines, have everything it

needs on every project, and then when I
start a feature build, it knows that it

can spin up testing and all the other
various workflows I might have without

needing to make a commit or pushing code.

And they have multiple features
that sort of come together in

something they call test boxes.

And so we get into a lot of that because
that, to me, has been my vision of where

I think CI/CD is going for agents, where
they are free to run CI anytime they want.

They don't necessarily have to run it
through the PR typical workflow just

to be able to run tests and run lints
and do scans and make sure that their

generated code is gonna work, and that
feels like what they've just built.

I went through their backlog of blogs,
and for the last six months or more, they

have been posting very regularly about
their updates to their open source CLI

tooling, to all the dozens of skills that
they've made for their CI, everything

from helping it automate the fixing of
flaky tests to essentially onboarding

yourself to their entire platform by just
installing the Claude Code plugin or the

Codex plugin and just saying, "Hey, set me
up on Semaphore, build me some workflows,

and keep working until they're green."

Which is a pretty, I don't know.

That sounds like a demo type of prompt,
but it seems like that's what they've

built and that they've designed this
whole platform and their skills and

their workflows and MCP tooling around
just letting the agent cook and not

having to worry about whether it's
figuring out CI or whether it needs to

go check CI again or whether it needs to
fix that thing that it pushed into CI.

And it just feels like it, they're
streamlining the whole process,

and I'm excited to get into it.

So let's jump in to Marcos and Marko
tell us their journey to building

an open source AI native CI/CD
platform all right, let's get the

titles and locations out of the way.

Marcos, what do you do
and where are you from?

Marcos: Well, I do software, but
I'm software engineer in Semaphore,

and I'm here from Natal, Brazil.

Bret: Nice.

Brazil.

Marko, same question

Marko: Yeah.

So I'm, responsible for the
product at Semaphore, and I'm

calling in from Slovenia, Europe

Bret: nice.

and you're now alumni of the DevOps
and Docker Talks show, so welcome back.

We talked a lot last time around
sort of how Semaphore was optimizing

CI/CD, and we were f- we were talking
a lot about, local testing back then.

I mean, this was pre-AI.

It was almost like, is there anything
relevant in that conversation anymore?

Because everything has changed,
especially when it comes to automation,

and I'm excited to get into it because
you all recently went, well, in the

last year, like the rest of us, you
went through a journey as a company

as well as individual developers and
figuring out what agents can do for you.

Marko, tell us about the journey
of, like, how did you start s- first

thinking about CI as something that
needed to sort of be reinvented?

I don't know if that's the ri- maybe
that's a strong word, reinventing.

But, I feel like there's so much
opportunity for changing the way that

automation and CI testing works now
that it… how did you come about this?

What's that origin story?

Oh, hey Thanks for listening.

And I got a couple of quick announcements
about stuff that I've created.

A lot of this is free stuff
you can get right now.

Just in case you've not seen some
of my other videos or stuff I've

been doing on my newsletter, I've
created a GitHub security scanner

that is an open source project.

It's called GASA, GitHub Actions
Security Assessment, and it's

expanded to even have more rules.

I think I'm about at now a dozen
rules on it that scans your repos,

including the entire organization,
and the focus of it isn't to replace

other linters like the GitHub Actions
Linter or Sysmor or other scanners

like Poutine that are all really great.

It's meant to supplement those and
solve my own problems as well as my

clients' problems, and I just thought
I would open source it to give it

to everyone, and I've spent a lot of
time trying to make it a solid tool.

Actually just spent some today, time
today in Fable Model building out

some better testing frameworks to
test it against real organization

repos as a part of its PR testing.

But the whole purpose of this is
to scan its personal repos or even

repos you might wanna use, or mostly
your organizational repos all at one

time to give you this report on your
stance of GitHub Action security.

And it looks not just at the
workflows, but also the repo

settings on the back end.

And this is based on my half decade
of consulting around GitHub Actions,

and there's no agenda there.

It's completely free.

I've actually been using it to implement
and upgrade the security of my own repos

to make sure that I'm not as vulnerable to
some of the recent supply chain attacks.

So the whole reason of doing it was people
were asking me to help them lock down

their supply chain against the attacks
we've seen over the last year on GitHub

Actions, particularly around open source
repos, or even using other people's

GitHub actions in a more secure way.

It will guide you through how you
can change and improve the security.

It won't actually change all the
settings for you, because I'm not

quite comfortable with it going in
blast radius of your entire org.

But go check that out.

It's on my GitHub, and the
link is in the show notes.

Next, I had in July a GitHub
Actions security workshop for free

because Chainguard sponsored me.

I was actually gonna charge for this
workshop 'cause it was hours worth

of hands-on learning around how to
use your own agent or your own hands

to actually lock down your GitHub to
make sure that you're not vulnerable.

So it's walking through what this GASA
tool does, but also why these settings

need to be locked down, why you need to
care about those particular settings,

and what happens when you maybe do one
thing in a workflow and another thing

in a security setting in your admin
settings, and those two things conflict

and cause a security potential risk that
GitHub isn't probably warning you about.

So it's meant to supplement all the
current stuff out there that really

just focuses on the workflows, but
don't actually also consider the

admin settings of your workflows
which are behind the scenes, right?

People can't see those settings, and
they're not usually easy to find.

So anyway, you can go look at that
workshop also free on my website.

Thanks to Chainguard for doing that
and making it- Free for everyone.

And finally, if you didn't know about
my Agentic DevOps Guild, we have been

meeting weekly since March, where we--
I think we're, like, twenty-five weeks

of weekly meetings, and we're now over
fifty teams of people in there solving

problems for their team on how we're gonna
manage agents, how we're gonna manage

the security and the sandboxing of our
AI for platform engineering use, DevOps,

anything related to cloud and Kubernetes.

If you're somewhere in an ops role,
this was the program designed for you.

It's not just meetups, but
I'm also creating courses.

I've got two courses in the works
right now where videos drop on a sort

of a random basis, but based on what
comes out of our meetings, I turn that

into lessons for everyone to adopt.

And also that comes with workshops
that we do, including this Gaza

workshop that was last month.

And those workshops are
meant for the guild.

The guild will always get
those for free, even if someone

doesn't sponsor the workshop.

But go check that out also on my
website, and I hope that helps solve

some of your AI adoption problems.

Back to the episode.

Marko: Yeah.

So, so, s- with all this automation
and, the, the CI is becoming even

more important for the folks,
especially that the whole development

cycle is, getting shorter and
there's a faster turnaround time.

So, we started thinking, how can we
enable our customers to kind of get

on that train even faster and leverage
the whole technology that is emerging?

And of course, when we started
thinking about it, the first

thing that we did, what is it?

Like la- last quarter of last
year, we created an MCP server

because that was the thing back
then, If, everybody was doing that.

Bret: Yeah, it's crazy how early MCP was.

Like, it was pre-skills, I
think pre-agent files really.

Yeah.

Marko: Yeah.

Yeah, for sure.

But that was the thing back then.

It's kind of like everybody was like,
you know, "Hey, that's the way how we

are going to connect to different tools."

And I think I believe it still is,
but as we were starting to develop our

internal processes around agents and
scale our agentic use, we have realized

that it's maybe something different,
and it's not one or the other thing,

but it's actually both things or all
things because all the developers,

they have a special preferences,
and they have their good reasons why

they prefer one, one or the other.

But working with our customers, what
became very obvious is that, we have this

new surface that the developers are using.

It's not it's not before you had like,
I don't know, five or six windows

that you were switching through in
order to do your work as a developer.

And nowadays it was becoming more,
more clear that either you upload

code or codecs or, pick a, a, a coding
agent, it's becoming your only surface.

And you want to be able to, to
talk to it and that everything

needs to be done, there.

And then we were fine looking into
"Hey, what is the way, how can we

interface with the coding agent of our
customer's choice?" Yeah, in a very

efficient way and give it everything
that, that the developer needs to do the

work, that the developer needs to do.

So, in essence, Semaphore is all
about developers' productivity.

And then for us this was like, "Hey,
okay, we need to get in like we need to

create an interface that feels native
for this new surface coding agent," and

that's how the whole Sem AI emerged.

And it's, yeah, Semaphore for AI.

The that is all about it.

And then, as we were developing our
own hands-on experience, we were

kind of thinking like, "Hey, how
do you connect efficiently with the

coding agent?" And then, Marcus was
our head engineer on that project.

He came up with the concept of "Hey,
let's create the CLI." But the CLI

is not only commands, but it's a
mix of the commands and MCP server

that is a local server, so that the
developers can choose their preference.

But also agents themselves can also like,
use whatever they prefer at the same, at

the time, because we also noticed that,
for certain tasks, developer, agents

have the preferences for some reason.

So it's that, that was one of, one
of, many, confirmations that it's

not one or the other, but it's both.

And then we try to make it as, as
optimal for the agents as possible.

So everything is you know, JSON
native and all the commands are very

well documented, not for the humans
to read that, but for the agents to

be able to use them and understand
the use case and a lot of examples.

But from the product perspective,
my kind of vision for this interface

was that, hey, let's enable people
to have the natural language in a

conversation with the CI and then
lower their bar- that barrier of entry.

Because in order to deploy a, a kind
of you know, put in production a

CI/CD tool as you… I can imagine all
the viewers know it's a lot of work.

You had to learn the platform,
develop a com- a, a configuration,

make sure to test everything.

Bret: Yeah.

Learn how the files work, yeah

Marko: Yeah, learn.

Marcos: The first try was SemaBot, right?

Marko: Exactly, yeah.

And yeah, so that, that was kind of
like also in the, in that during the

journey we had the OpenClaw became
a very big thing, and then we did

our own version of OpenClaw for CI.

But there are a couple of things
from the product perspective

that, that was very important.

First one is that, since we are
open source but we are also like

serving a lot of enterprises,
we are all very big on security.

And then if you're in an enterprise
environment, you cannot just like, put

in a AI and put your own LLM and say
to the customers, "Hey, this is going

to be you." That, that will not work.

So, so we said, "Hey, let's create a
solution where our, where the user, the

developer can select the LLM that is, in
the enterprise environment, it is approved

by the enterprise and they're paying
for subscription, all the legal work in

is worked out. And then just create an
interface that can, that will empower

that LLM to do more for developers."
So that, that was one thing that, that

was like a vision for the project.

The other thing was how can we
create, create interface for 100% of

functionality of Semaphore so that
the developer can complete all of

the time stay in that coding agent
of choice, let's say, Claude Code.

And then never leave and,
and do everything through it.

And then how can we create
experience where you… I don't

have to learn anything except
high-level things, of course.

Yeah, I need to conceptually
know what the CI and CD does.

But how… I don't have to
learn the configuration.

And that was described in a way,
hey, we know the CI/CD inside out.

We have a lot of engineers on
our team that are like, building

CI/CD tools for like a decade.

Let's put all that inter- expertise,
package it somehow so that the

other people can leverage it
instead of them becoming experts.

And that's what we have discovered
that the good combination of a right,

a CLI with the, as I said, commands
plus, MCP and set of skills and lot of

examples, are actually doing the thing.

And then actually, and getting us to
that, that, experience, that developer

experience that I can just naturally
in natural language say "Hey, this

is what… This is the outcome that
I want." And then the, then my coding

agent will be able to interact with
the, with Semaphore in this case and

do everything that needs to be done.

Bret: Yeah.

It's funny that in a world where we're
gonna end up with these ad- advanced

AIs that do a lot of the, grunt work
for us, that the sweet spot seems

right now like to stay at the CLI.

And if you'd asked me like- 10 years
ago or even five years ago, "Hey,

if you can imagine a future with AI,
do you think we'll be spending more

time or less time in the terminal?"

I would've said probably less, right?

We're gonna have AIs do all that, right?

But being such a, a terminal fan, like
one of… I think actually literally

the most popular page on my website
is my s- my shell setup page, which I

detail, like, all the different things.

It's a lot easier nowadays, but 10
years ago, for those of you kiddies

out there listening, we used to
have to do things like we have to,

we'd have to install specific fonts,
which, you kind of still have to do.

But we'd have to worry about
italics, like italicize didn't

work by default, so you'd have to
customize that stuff in your editor.

You'd have to do all sorts of very
particular settings and configurations

just to get a, a VIM to work right in a
shell with all of the settings you wanted.

We had to do a lot more a decade ago than
we did do nowadays to set up your shell.

But I still get a ton of traffic
just when they, they just wanna

know the fonts I'm using, the themes
I'm using, the, the GUI I used.

And then the, now my most popular
video this year is my video of talking

about leaving Ghostty Terminal, which
has been for a couple years the big

thing for terminals, leaving that for,
Supacode, which is a g- a lib Ghostty.

It's actually a Ghostty project, but it
adds a bunch of agent management features

so that you can manage your product.

So those of you watching, if you're
interested in agents in the shell, go look

at my YouTube later, but not right now.

We're talking about CI right now.

But I'm… It's hilarious that, the,
the terminal people are getting their,

they're getting everything right now.

All the best stuff right now I
feel like is in the terminal.

All the best coding agents.

I'm not… I'm trying to use the GUIs,
but I don't feel like they're as good.

And the fact that I can now… it took,
to me, the way I'm looking at this

is it took agents or it took AI for
us to start looking at platforms like

yours in a way of, we, a lot of, we
all had CLIs for these tools, but you

had to really memorize the commands.

I always feel like I don't know all
the commands, and I'm constantly

like, "Is there a command for this?

Am I searching for a command?" you
s- look at me look at the help.

Can I even do this in the… and
so I think as humans, we have this

ability where we learn the command
line tool once, let's say Docker.

But then Docker adds a bunch of
new features five years later,

and no one uses them in the CLI
because we all already learned it.

And when I talk to people about this, they
seem to agree that w- we don't feel like

we're really experts in the CLI, the full
power of the CLI of a lot of these tools.

And now I just feel like that's, that
problem has been completely erased.

Like y- like the way that I'm
looking at your documentation, I'm

realizing I, I could pick this up.

I could use the CI, and the only thing
I have to be good at is understanding

CI concepts like the, the basics
of CI and automation and how you

manage a Git repo with workflows.

But I feel like the only thing I
really have to understand is, how

to prompt my intent to the agent.

And I feel like your demos on your
website, the, the little ones that

say "Set up my CI for me," they… A
lot of websites, they feel contrived.

We used to have these sort of made-up
demos on the front page, but that's

literally probably what I would
type to the CI is, "Could you please

just set up Semaphore for me?"

Because I don't know the shell, I don't
know the CLI, and I'm not even familiar

yet with the GUI, so I don't know
what possible options are for the GUI.

And that's just a, it's a cool target
to, to approach of like, our onboarding

literally needs to be someone who's
switching from some other CI and

doesn't know anything about our
platform, but the agent can make them

extremely productive in the first hour.

And is that something that you are
constantly testing and building out?

Looks like it was, like,
a, a major feature for you.

Marko: Yeah, absolutely.

Absolutely.

that barrier of entry, how, b- this
is becoming extremely important.

Like with all the code being generated
by agents, like if you do not have a

great CI that can test and make sure
that you're like shipping something that,

that, has no regressions and, and it's
high quality, that's extremely important.

So it's very important that it's very
easy for people to get on board with

the CI/CD, and then also to give them
freedom to easily switch from one

to another, whatever is the reason.

And then, those barriers were
very difficult and hard before.

These were like in a really project
that DevOps organizations were planning

on them, they, like asking for budgets
and all like, and so it was very hard.

And now it's, it just we have a tool
that is like very good at reading

documentation, which we as humans were
really bad at, writing documentation,

especially reading the documentation.

Bret: it's our last

Marko: Exactly the last…

Bret: hope is documentation.

Yeah.

How far can I get without documentation?

Marko: Yeah.

And then nowadays we have s- we have a
tool that is very diligent in reading

the documentation, especially if
you format it in the right way, and

then it's, it's "Hey, I have this,
please, cop- look at this like Jenkins

configuration and make it, in Semaphore."

And then it starts to, to run
because it, it knows like a lot of

details about Jenkins and all the
Groovy script and everything, and

it knows exactly how to mimic that
in Semaphore and that's awesome.

And then you can really quickly s-
try it out and see what you can,

Marcos: It was even a point that
I was pestering Marko, because I

wanted to set up a project of mine
on Semaphore, only it starts when

I started working with the company.

But I didn't know how to do that.

And now we only-- we just have a comment
that you can talk with your agent

that can do anything that you want.

And you can set up a CI/CD from zero
to hero like we have, one series

about that in around ten minutes.

So it's very interesting.

Bret: Yeah.

in fact, I don't think I've thought
a lot about the fact that w- what

one of the things that agents are
letting us do is switch tooling.

I Not just switch tooling, but
now we're seeing, c- complete

projects being rewritten in Rust,

Marko: Yeah, that's very popular.

Bret: a- and if we can rewrite an
application in Rust with $10,000 of

tokens, like we can probably switch our
CIs much quicker with far fewer tokens.

But I haven't thought about, too much
about the fact… 'Cause we had, I mean,

we have an entire ecosystem, I feel
like, of tooling to migrate one workflow

file, you know, migrate everything
away from Jenkins, which I approve,

and then m- move from, we, a lot of us
had to move from Travis years ago when

Travis changed a lot of its free…

Remember f- a decade ago, Travis was
very free, and we were all addicted to

the free, and then they weren't so free,
and then we had to move to something

else, and I, and we have all these tools
at the ecosystem, or even some of the,

we, you probably, I don't know, maybe
made some of these converters yourself.

And it never went 100%.

It was never, it never really
worked on the first try.

It was more like it would convert…
It was kind of like my Dockerfile m-

making, conversion, converting things that
would try to figure out your Dockerfile.

It's like they get you to 80%, but
then the last 20% actually takes

you a l- a really long time and
hours of work to troubleshoot.

And some of the concepts are just
different, but I didn't really ever

think about the fact that this is
almost a solved problem if you can

give all the documentation and all
the right skills and all the right MCP

endpoints to your agents, like this
could be a very smooth, fluid, process.

But before, I want to get in that.

So we're gonna get into the details
of, like, how exactly people can use

an agent to access and manage a CI.

There's some really interesting things
I wanna talk to you about, because I've

been doing a deep dive into your blogs
over the last six months, and there's

some really interesting ideas there that
I like about, that you all are able to…

You're basically able to be a
little more nimble than some of the

cor- the giants out there of CI.

It seems like you're able to move quicker
and modify the platform to be more agent

native, than maybe a lot of the other
sort of incumbents and legacy platforms.

But first I wanna just talk
about you going open source.

'Cause I'm a cloud native ambassador.

I go to all the KubeCons, I'm a
part of the CNCF and the Linux

Foundation, and now you all are, too.

you're members of the Linux
Foundation, members of the Cloud

Native Computing Foundation.

You're actually listed, for those
of you out there that you know

about the landscape, the CNCF
landscape, they're listed on the

landscape, which is very cool, 'cause
there's a lot of CI options there.

But I always thought of Semaphore
as this closed source tool that is

hosted in Europe and is for teams that
are maybe burned by some of the big

incumbent players and are looking for
something a little more their style.

How did exactly, what,
how did this come about?

This seems like a really big
decision to open source the platform.

Can you tell me, can you walk
me through how that happened?

Marko: Yeah, so, it was a very big lift
for the company, in order to open source

your code, it kind of like there's
a lot of work that you need to do,

make sure that everything is buttoned
up from the security perspective.

So, so it was a big lift, but what we
wanted to do is really enable people and

to really meet them where they are, If
you are only a cloud provider, then there

are, there's… You're limited in a way.

Who's your customer?

And we also have some, some big
customers that are, you know, running

the whole, Semaphore ecosystem on-prem.

And we thought "Hey, let's open source
this, let's enable people to try the

Semaphore in the way that suits, suits
them the best so that, if you want to

run it on your own, you can. That we
can open up the platform to call for

contributions so that people can…
also developers can collaborate with us

much, much easier." For the developers
it's much easier to submit the PR or

add the comment than to hop on a call.

So, so, so it's also to get more
feedback, to get more visibility,

and to also make it possible for
the, all the Jenkins customers.

Sorry, sorry, Jenkins.

But all the Jenkins customers to
try something else, because they

are really in, in the corner.

It's a very, sticky platform for the,
in my opinion, for the wrong reasons.

So I think, these are all the, all
of, all the, motivations for us to

do an open source and we are, like,
starting to see benefits of it.

And then we as a company, we
believe that, that the, the

open source is the way to go.

Bret: Yeah.

Marcos, were you a part of that effort?

Marcos: No, I will-- I came later to
that, but I made some AI open source.

Bret: Yeah, 'cause that
sounds like a lot of work.

Marcos: I'm glad to be… didn't have
to be part of that work because it

was a-- Like, I was reading about
the documentation that they had.

It was a lot of things that we had to
solve before going public with the code

Bret: Yeah.

Yeah, y- it's, I mean, there's so
many factors, and I'm sitting here

thinking, as a consultant, I s- I
see a lot of different projects, and

a lot of the ones that are closed
source, if I was one of the major

people on that project, I would be very
nervous about opening, opening it up.

Not from just a security perspective,
but now suddenly all my code is v- it's

not … It's one thing when it's a dozen
people that are, critiquing my code.

It's another thing when it's thousands
of people critiquing my code.

And and sometimes I'm … When
I look back at my old code, I'm

often like, "Who wrote that?

Oh, that was me.

What was I s- what was I thinking?

That was dumb." so yeah, that's gotta be
… I mean, especially when you have a project

that's many years in the making, right?

It's we all develop differently now
than we did a decade ago, and … our,

our intelligence about a, a code
base is better than a decade ago.

I mean, everything's different.

So, when you have a long sort of a
monolith, maybe not monolith, but

something, anything older than a decade,
I feel like at this point is almost

like, legacy code if you don't constantly
maintain it, keep it up to date.

And I imagine one of the,

I'm gonna guess one of the things you
wanted to do before you open sourced

it was to make sure that it was, like,
on the latest libraries and it was

using some of the latest paradigms.

So all right, so you made it open source.

People can download this and run this
themselves, or they can just easy button

it with you on the hosted platform.

… Tell me about, like, the runners.

You provide … Or what do you call
those workers or runners or what

do you call those, those machines?

Marko: Yeah, runners.

Yeah, so we provide our own runners,
but people can also self-host.

And there are a lot of customers
choosing for one or the other

for their specific reasons.

And then the benefit of running on
our runners is that we do the, all

the maintenance of the runners.

The benefit of having your own
self-hosted runners is that, you can

create the custom environments that,
that are really bespoke for your needs.

Maybe they are resembling
100% production environment.

So it, it makes sense
in different use cases.

It makes sense to, to
use one or the other.

And we have a quite a lot of customers
that go hybrid, so some of the runners

are self-hosted and some of the
runners are, they're using our cloud.

So, kind of as I said, we like to, to,
to meet our customers where they are,

and then, you provide them with the best
possible service, that we can offer.

Bret: like I think any CI/CD team at
some point they're gonna end up needing

to run runners on their, i- whether it's
in their data centers or on their VPCs,

like they're gonna need something that's
closer to staging a production, especially

if they're doing CD inside there.

And the runner story is one of
those things where, until you're a

seasoned CI/CD veteran, I feel like
you don't… You may not look at a

platform to evaluate it and think,
let me spend a lot of time playing

around with runners, deploying runners,
figuring out how to maintain runners,

like getting the performance right.

Let me…" There is so much to
it that, I mean, there are entire

other CI/CD platforms where I d-
I don't even recommend they use

their runners, that they run their
own, or that they use an outsourced

service 'cause there's an entire
ecosystem of outsourcing your runners.

And you had an interesting blog post
this year, actually this month, talking

about benchmarks, and I thought this was
really interesting because a- as you're

probably saying here, the speed of runners
is actually one of the most critical

points to me in terms of evaluating a
CI platform because you're gonna spend

so much time waiting, and historically
that has been infuriating for those of

us that are managing these platforms.

Because as someone who manages CI/CD,
like, my goal is to get these things to

run as fast as possible, because I want
my developers to use it more and I am

also kind of s- I'm not only the CI/CD
custodian a lot of times, I feel like I am

the advocate for people automating more.

I'm always looking for toil to automate
in teams, not just testing their code, but

also all of the other maintenance tasks.

Let's throw those into CI as well, make
it a more universal automation platform.

But also I'm also security… I kind
of fit in that DevSecOps security

realm where I'm wanting them to do
more linting, I want them to do more

security evalu- I'm, like, suggesting
workflows all the time to them.

"Hey, we should add this to the code.
Hey, we should add this to code,"

even when I'm not a developer in the
team, because I feel like developers,

they spend so much time being told to
focus on features and code maintenance

of the actual platform that it's
one of those, what's the analogy?

The, the cobbler's children have
no shoes or something like that.

I always feel like our CI/CD platforms, if
there's not advocates for them, you end up

in this space, and Jenkins was the perfect
example 15, t- 10 years ago, where it

just became this hodgepodge of fragility,
and it ended up being that one person

in the team that knew how to maintain
it and no one else could touch it.

A- and good luck with the backups
and restores of that mess, right?

Like it was… There was always that
precious Jenkins box under someone's

desk that could not be restored
successfully, so you could not touch it.

And I hope for everyone that those years
are gone and that we are at least using

better maintained and supported platforms.

But you do have this- a blog post
talking about how much faster, the

runners build times, the cost of jobs.

I'm a big advocate of AMD and
moving everything you can in your

coding platforms to AMD because
it's so much a better value on the

big clouds, and it's just cheaper.

And then, and talking about some of
the competitors and how, whether or

not they're faster or slower, they
also tend to be more expensive.

So I… For those listening, do everything
you can to avoid running your own runners.

Just because it's not a solved problem,
and you're probably not better at making

them faster than the professionals
running these platforms that their

whole job is to make them faster.

Marko: Yeah, there's a whole team
of full-time employed engineers

that are making sure that the, it
runs, everything is up to date.

I don't know if you have already addressed
that in your, with in your podcast before,

but in last, I would say three to five
months, especially in the last three

months, there's like a huge amount of CVEs
out there that are popping up every day.

Typically, you would do some maintenance
every month, and that would be fine.

But here, we have every day
there are new things popping up.

I'm glad that, probably we will end
up with a much more secure software

in the end, but at the moment,
it's so overwhelming to, to make

sure that all the CVEs and that
your environment is fully patched.

Because in the-- if agents can find
security holes, that means that other

agents can also exploit your thing.

So everything needs to be patched
up, and buttoned up really quickly.

And having internal teams working
on that, I think it's very

expensive at, twenty twenty-six.

So it's much better to, to do
that on scale, as we are doing as

example for all of our customers.

Bret: Yeah.

We have a question, "Are you guys taking
contributors? I'd love to contribute.

If yes, how can I get started?"

Marko: Yeah, of course.

We, we take contributors.

It's very easy.

you submit the PR and then some of
our colleagues is going to review it

and, getting the communication with
with the person submitting a, a PR.

And then if, if everything is
looks good, then it passes all

the C- CI, then it's merged and
it becomes a part of the product.

Marcus can maybe talk about when we
open… When, we started with Sem

AI, for example, immediately as open
source and as we presented it to a

couple of our customers, they were
starting to submit PRs like week two.

So it was very fun to, to watch how
people are thinking about it and what were

their usage scenarios and their comments.

Marcos: Yeah.

We are always accepting new things
and also improvements that people

see that we probably don't see.

And you can just go to the repository
that we have and follow the

contributing documentation, and we
will take a look on the pull request.

Bret: Awesome.

We got 37 contributors, so pretty cool.

This is a question I
didn't think to talk about.

Do you, are you thinking about
contributing to the CNCF?

Are you gonna keep it for yourself?

I don't know how to ask that question.

Are you thinking about contributing
it, or are, because it's your platform,

I guess it's a little, gonna be
a little hard to do that, right?

'Cause this isn't just a,
a separate product, right?

Marko: Exactly.

It's a very big decision to make.

So, so yeah, that's still under
consideration, I would say.

Yeah.

Still discussion, yeah.

Bret: it's one thing to do open core, but
it's another thing when it's the whole

platform and it's something like this.

But yeah, we're all kind of, all
over the map with CI, and I was

gonna talk about earlier that I
don't remember the last time…

It-- I'd say 50% of the time I walk
in as a consultant, I w- especially

if, we're talking enterprise here,
big companies, hundreds of developers

if not thousands, it is very rare
for me to see them have one CI.

And one of the reasons for that,
getting back to our agent story,

one of the reasons for that is that
everyone's trying to get off Jenkins.

For five years or more, everybody's
been trying to get off Jenkins.

And the challenge always ends up, it's
like an 80/20 thing where they can get

that first 80% off in a reasonable amount
of months, and then there's this 20%

left that are these really complicated,
multifaceted jobs that are insanely

complex and should have never been created
that way in the beginning to begin with.

But they end up leaving them there because
they just, they burn out on the migration.

And so then now you have
N plus one problems.

And the, one of the biggest projects I
was on in the last five years, it was a,

it was an enterprise security company,
and they had three different CIs.

And they had the old CI, the CI
they tried to migrate to, but

then it didn't do everything.

I think that was Drone.

I'll pick on Drone for a second.

and this was five years ago or
four years ago, so this was pre-AI.

And Drone didn't quite do everything
they wanted to do, and so then they

decided they had to migra- migrate off of
that, and then they were still migrating

off of Jenkins, and then there was the
new platform, and then it was just…

talking about the agent migration story,
we, I feel like we could do a whole

ep- episode and then demo on it, but
it would be, it would be amazing for

me to walk in and do a CI/CD migration
project in a month and just, end to

end, cut it off, and now we're new and
improved all in a very short timeline.

That, that almost feels like something
I need to start selling as a product.

I will migrate your CI for
you in less than a month.

Marko: Absolutely.

Yeah.

Yeah, that would be a very desirable
service out there, I'm quite sure.

But it's especially, migrating from
the work, such a customizable tool

like, like Jenkins, I don't think
it's possible to really migrate only

kind of like a one-on-one, because
of that level of customization.

I was involved in like maybe
at least like 10 projects in my

career that were all on Jenkins.

We were never able to upgrade
from Jenkins to Jenkins.

Like from one version to another.

Even not moving to another tool, it's
just keep staying like, you know, up to

date and current was not possible because
at a certain moment somebody decided

to use the plugin rightfully, but then
that plugin was not available anymore or

not supported and then at that version.

So it's, we ended up like, really, sitting
down and rethinking some of our flows,

and some of the out- the pipelines.

So I think that is necessary, but I also
think that's an opportunity and good

thing to think through because as you
said, what we knew and what we thought

it was right five years ago or even
three years ago, maybe it's not right

now because of like a lot of reasons.

And then it's good to see, take
some time and say "Huh, okay.

If I have a blank slate, how would I
like this to work?" And now with agents,

if I can describe that, like in, in
natural language and create like a very

comprehensive like in documentation,
again with the hel- help of agent, that

can be configured in I think 10, 15,
20 minutes, with the, with the agent.

So, so, yeah, I think it's a
great time for engineering.

Before we were spending a lot of time
in that grind of typing code, learning

configuration, clicking buttons.

Here we have a lot of time to
really think through some of our

decisions, take a lot of things in
consideration, try out hypothesis.

So, so, for our team internally,
that's really a blessing, where we

can spend more time thinking through
and, and discussing and then the, the

groundwork is handed over to the machines

Marcos: Actually, one of the test
cases of Sem AI was migrating public

repositories from Jenkins and GitHub
Actions to Semaphore, just like a

benchmark for the usefulness of the tool.

Bret: Yeah.

it's almost like those are like evals
to test whether or not the agent

can handle all that work, right?

Marcos, I'm really, I really wanna
dig into, the work that y'all

have been doing to sort of make
everything agent harness first.

But I wanted to first talk about You
c- you've prov- you've created a plugin

system for Claude Code and Codex.

Can you talk about that real quick,
about everything that's included in

there and why the decision was made
to do things like commands still, even

though that feels almost like, w- is
that still cool anymore to do commands?

I thought we were just vibe-
vibing the skills at this point.

So, can you walk me through all that?

'Cause it looked like you have
a lot of s- rigor there that is

more than just someone simply
publishing a free skill, for example.

seems like a lot more than that.

Marcos: Yeah.

Initially the SemAI was just
to be a CLI, so a simple CLI.

We have the same CLI.

It's a old one that we have, and we wanted
to develop something that was agent first.

So naturally, we started with a tool
that was only communicating in JSON,

so the agent could run it and pipe
it for… to anything that it wanted.

But then we discovered that MCP
was a thing, was booming, and we

wanted to support MCP as well.

But one rigor that we took, one
choice that we took was we wanted

to have the same capabilities
of the CLI on the MCP server.

So basically we are using Viper
and Cobra on the CLI, and we are

basically using the commands that
we have for the CLI on the MCP.

It's a little bit of a hacky thing, but
it's fully supported by the libraries,

and you can check it on the repository.

But you'll be a- always have the support
that you have on the CLI on the MCP.

And this is important because as Marko
talked about on the start of the podcast,

some tasks are easier to do using a CLI
and the others are easy to do using MCP.

For example, your agent may decide to
pipe the output of SemAI into a Python

script to extract something that it wants.

This is something that
I see daily right now.

And for some easy tasks, just asking
a question or something for the, to

the platform, it normally uses MCP.

And we decide to document and make
it easy for the agent to do the

right thing, not do the wrong thing.

So we have included descriptions,
examples, and skills.

Each one has a, a, a place, right?

So basically the descriptions is basically
what the surface level knowledge that

the agent should have about that command.

Each command has examples, so if the
agent is, doesn't know how to do the,

the command, it can look through,
into the examples and basically

figure it out from the examples.

And the skills that most people
are kind of abandoning, the complex

ones at least, is where we have
the cheat sheet for the agent.

So basically it will be able
to know what is all of the step

that it needs to do for one task.

So for example, if I want to migrate from
Jenkins to Semaphore, it has a com- a

skill that it can load, and it will have
all of the thi- all of the things that

it needs to do, and also will have the
input from the developers of SemAI on how

to do that, because we encounter a lot,
lots of particularities with that flow.

And then we are creating skills.

So basically when you install the plugin,
you will always have the skills available

for you, and your agent can choose if it
will use the MCP or the S- CLI itself.

And one nice feature of the plugin is
that it auto-updates, so you always have

the latest version when you install it.

Bret: Wow.

That's a lot.

What about the agent
makes it self-healing?

I'm looking at this blog post on
your journey to making the SemAI.

This blog post was what pulled me in a
little bit because there was a couple

of things that I thought were different.

One, yeah, you're doing the slash
commands and wha- and it talks about why

those matter and basically determinism.

Embedding CI/CD best practices into
the agents, which, I mean, for other

CIs, I had to build that myself.

Like, as a part of my courses I offered
because, 'cause it's not really built

into a lot of these tools to, to really…

Like to me, best practices are an opinion.

I used to say better practices because
I always felt like, b-a- my best, I

don't know what if I have is best.

I mean, it's better.

And I feel like the more opinionated you
can get, like the better that might be.

And so sometimes best practices
can not always work for everyone.

But it looks like you've got all of
these different areas that you focused

on with like the caching, 'cause that's
a thing that like people often will

screw up caching or they don't understand
caching, so they're like leaving,

they're leaving minutes on the table
because they're not getting the most

optimum performance out of their CI.

And I've always felt like CIs are
always trying to optimize for that and

like they either go too far to make it
too automated with the caching setup,

and then it becomes limiting sometimes
because it's, you have to do it this way

in order to use the cache or it doesn't
work, or you have to become like an

expert in how their caching works and
how your tools would need caching and

which tools you have that need caching,
and how to avoid cache poisoning.

And like there's just, even just
in that world, I feel like there's

someone's job dedicated to that, right?

And y- to be an expert.

But you have this other list of like
the, the testing reports, the failure

diagnosis, the artifact management, like
how do we get these artifacts out of this

build and into where they need to live?

And so you've got all that
built in there into the skills.

Then you've got self-healing pipelines.

Tell me about… I mean, you literally
have this quote in here of, "Tell the

agent, 'Work until the pipeline is
green.'" Which I will point out is

not the thing that I would normally
actually type, because that is an

outcome, and I'm often dumb with agents.

I need like agent therapy.

I'm often dumb, and I'm like giving
them the detailed step-by-step because

that's what we had to do two years ago,
versus I'm, this is an outcome, right?

This is like what I would tell the
junior engineer who understands CI, but

they're like, "What is my job here?"

I'm like, "Your job is to work
until the pipeline is green."

And, and so I'm guessing
you test this, right?

Like this isn't just a demo.

This is something that you verify.

Tell me about that.

Marcos: Yeah, basically the comments,
the comments, the skills that we

put is basically we compile all
of the knowledge that we have.

We got-- We have lots of customers and we,
years of experience on the platform, and

we try to make it available for the agent.

But the, this, about this quote
of just working until it's green

is something that I do daily.

And yes, I test that because it's
basically SemAI has tools, commands

that the agent can run that will watch
the pipeline until it has a result.

It will extract the errors to
the agent, and we will also be--

you also can basically go back
in time and see errors as well.

So for example, I can start my day just
asking my agent how the pipeline, how the

CI is going, and can go over all of the,
the recent pull requests that we have.

It can extract the errors, and this
part of extracting the error is the

difference between my agent gaming
the system and just deleting the

test and it actually fixing the test.

So in these skills, we talk about
adversarial reviews of the agents, and

how to use SemAI to extract the error.

And in general, it's a little bit of
magic as we, we are having with, agents.

So it's, it just works most of the time,
ninety-nine percent of the time, because

it will only be able to see the error.

It will have the error, clearly
labeled on its front, like it

will just so see the error.

And most of the time, it will just
look because it tries, it can try on

the CI itself, it can push a commit.

But if you want to test it,
you can test it locally.

And also, if you are not able to test
it locally because the test is too

big or because it needs something that
only the CI has, we have the test box.

So it's basically a job that you can
spawn, and you can SSH into it, and

you can give it that machine because
it's ephemeral to the agent to, for

it to do anything that it wants there.

So it can try lots of different
solutions and then just try to get

that to the point that the CI is green

Bret: Let's talk about that.

let's dig into that 'cause we're,
we're running a little long, but

I think the Test Box scenario and
having the agents doing that work is,

to me, that's a really cool feature.

I have been envisioning for a
while, and I don't make CI/CDs,

I just use them constantly.

And I just, at the end of the day, I
kinda just want my agent to constantly be

running CI/CD, whatever that looks like.

Whether that's just the testing workflows
or maybe just the linting plus testing.

But I kinda want things to constantly be
happening without me having to tell it,

"Okay, every time you make a change, make
a commit and push, make a commit and push,

make a commit and push." This just feels
like a, That mindset around m- waiting

until the git com- commit and push to do
automation, it has its place, but it feels

like in this world we need this whole
new thing, and that thing is before that.

And s- a lot of people, I
mean, Dagger's been trying to

solve this problem for years.

Shout-out to Solomon Hykes.

there's all these tools that are
trying to make it easy for us to run

tests locally and in, in CI, and it's
always been a challenge to recreate

the CI-like environment on a lo- on a
random local machine that's a th- of one

of three different operating systems.

And that's, I think that's
always been challenging for CI.

I mean, a lot of CIs do it still, but
this idea that my agent can just do

this on my behalf, it can do it in
the background, it can do it on my CI

machines, but it has nothing really to
do with me committing code, like, that

is pretty cool, and I wanna hear it.

And what, I don't know anything
about how you do it, so I wanna

dive in for maybe our last topic.

Marko: So this is also one of the
transformation in the agentic way of

working, because previously when you
were like, typing the code, then you

know, you make a change, you run a couple
of tests on your own machine, and then

you're, "Aha, okay, I'm happy with it.

Let's push this and let's run
around the whole CI." But nowadays,

that's not exactly… Nowadays,
it's not your job anymore.

Your job is to orchestrate two,
three, five, four, five, how many,

like what, how, like depending on
the context that you can keep in your

head, agents that are doing things.

And then how do you mimic that thing?

It's very hard to have a
local machine that is powerful

enough to run all those things.

So here there's like a really nice way
to scale it, to use your… A benefit

is that you're using a CI box, so that
it's exactly the same as the CI box.

That's an additional benefit, but
it's, it becomes scalable and it keeps

you in that same loop as, as before
ju- just on a different scale because

you, you have like a different you
know, automation tool at your disposal

Bret: All right.

So you're checking out a Git repo
and like I think a lot of us that

are getting a little more advanced
in agent-based workflows, you have a

local clone and then you use either
your harness or maybe yourself, you

create gert- Git worktrees for, d- th-
the each one of the agents has its own

safe space essentially locally, right?

And then you're, you have this plugin,
you have the Semaphore plugin in

your Claude Code or in your Codex.

I think those are the two
you support right now, right?

And in that includes, does that, that
plugin includes MCP skills, commands,

and a CLI, or I guess it's using
the CLI to do some of the stuff?

Okay.

I wanna make sure I, include
everything it is, it has.

So it has all this stuff and I guess
those skills are essentially like

providing it the documentation as
well as the best practices and then

how do you get it to behave this way?

Are you prompting it
specifically around Semaphore?

Or when you ask it to do a feature,
are you just saying, "I'd like

you to work on this feature.

I need you to implement this thing.

Maybe I'm telling you about it or
maybe I'm pointing you to my ticket

or wherever, where the, where
you need to get the information.

And then while you're doing it, I
need you to do these CI things."

Like how exactly, what does
the prompting look like there?

I'm thinking about our audio listeners
and like they're trying to imagine working

through this workflow and how do they get
Semaphore to do the coolest stuff here.

Marcos: Yeah.

For Semaphore doing the cool stuff
is just install SemAI because it has

already the prompts and everything
that you for your agent to do.

So for example, if you are using
SemAI, SemAI, you can just ask the

agent to create a entire organization
or project, set up the Git connection

to you so it can do everything there.

And there is my part is like I have
some documentation on some CLAUDE.md

and other documentation on my
repository to instruct it on how to

do stuff and how to, basically give
one machine like one machine to one

agent, and it can do anything there.

I can be pretty relaxed to, some point
with the sandboxing there because I am

just letting that sub-agent reuse that
machine so my machine is not impacted.

And also, from the side of Semaphore,
I can put secrets on the CI and for

example, I can create two, two projects
and one be production and the other

one being just a staging, for example.

And I can give each agent access
to some project because I can

create multiple service accounts.

This is the way that Semaphore
does the, the things, and I can

put different roles and different
permissions to each service account.

So I, I also had, one month ago,
persistent agents that could do things

for me, and they were working with
the CI, but because they had specific

permissions, they could not go outside
of the sandbox that they had on the CI

analysis that they were doing for me.

So for example, I just ask it
for the SemAI to give me the,

the status of the project.

So I have a 33% pass rate because of IQ.

and it's telling me some flaky tests
that it detected and what I can do.

So for example, I also have
two requests that are red.

I can ask it to run and fix this stuff.

So while we are talking,
please fix the red stuff.

And this is it.

Bret: So in your initial prompt for
the feature, did you just add into the,

what presumably was the big prompt on
get started on this particular feature,

you just said, "And while you're at
it, run Semaphore test boxes until

everything's green and then commit"?

Is that kind of how you would do it?

Marcos: Yes, I would do that.

And because the plugin is installed,
it will probably just use the SemAI.

It may be sometimes, it will request
for permission, because of the

auto-classifier and stuff like that.

But once you get permission
to it, it will just use.

And also, on the CLI and on the MCP,
I don't see many people doing that,

but we put some flags that you have to
pass so that the auto-classifier will

be more likely to prevent that command
if you didn't give permission to it.

So basically that, "Hey, I want-- I
really want to delete this stuff." So

normally from the test that I did, the
auto-classifier holds the agent back

if I don't really tell it to delete it.

So we are trying to improve the
tool every day because we use that.

Bret: Do you update your Claude file
or your agents file in the repo to

say something like, every time you
make an edit run, run a Semaphore test

box to validate so that, that way you
don't even… it's just sort of part

of the initial prompt initially so
that it always defaults to testing.

'Cause that feels, this feels like the
kind of thing that I'd always want running

for every feature that my app is building.

I would just-- I don't wanna ever have
to say, "You also need to remember to run

some testing before we make a commit."

Marcos: So yes, I have put that on
the claude.md, so just use SemAI.

This is like for SemAI is
the only rule that I have.

So use SemAI to test
stuff and r- sees the CI.

And the other one is use sub-agents
to not pollute the main context.

Just basically that.

Bret: Yeah.

I mean, I'm putting this in perspective
to what I currently have to do, right?

'Cause I'm not using Semaphore yet,
and I'm… I have to do things like I

want it to, I want it to at least lint,
or like you said, basic unit testing.

And so then I have to make Makefiles
or Docker Compose files or something

else that's gonna run locally, and
then I have to put in my agents

file for each project a bunch of…

a list of things.

"Please always run these linters
before," or, "Please determine

which are the best linters for this
language, and please always run them.

And also run this thing to make s- if you
edit ever a g- a, a workflow file, go run

these linters on this thing." I have to…

I mean, I have a list of things
that I always want it to do on every

change, and those are all really just
CI things that I can't run locally,

so I have to give it… I have to
make sure those tools are installed.

I have to give it permission to
install those tools if I think they're

not gonna be installed, which means
my machine has to be very custom.

It's just not… It is not ideal.

I feel like I'm kind of working
around the limitations of my own CI.

And, a- and I was trying to… And
I, and this is sort of me asking

you all the questions of the things
that I really wish my CI did.

So,

Marcos: Yeah

Bret: okay.

So we've got this thing r-
creating these test boxes.

are… I'm just curious, Are you able
to… I'm assuming Docker is on there,

so then I can spin up Kubernetes clusters
and I can do all those things, right?

That's all n- very normal runner stuff.

Marko: Absolutely.

Yep.

Yeah, and then everything that you
described, Bret, it's like what this

is what we typically see that our
customers put in their pipeline.

Bret: Yeah

Marko: And it's even better than
it, when it is part of the pi-

pipeline, Because if you tell it to
the agent, then it's it will do it,

but it's not really algorithmic.

When you put it in a pl-
pipeline, it has to be done

because then that's algorithmic.

That's not probabilistic
like a, like an LLM.

They are great, getting better
every day, but still, this

is, for sure it will happen

Marcos: we said, most of the s-
stuff that is, it's doing on its

own is described on the skills.

We have the guide on how to debug
pipelines, on how to deploy,

fixing flake tests, even migrating
from GitHub actions to Semaphore.

So we have lots of skills of
curated procedures that we do.

So on Semaphore, we deploy like at least
three times a day to production, and we

have some spiky days, some bursty days
that is more than 20 deploys on that day.

So we have some knowledge
on how to do stuff on that.

So for example, you can read the skills.

I, invite the audience to also go and
view what is going on with the skills.

We have a list of commands and steps
that you should do, for the agent to do

and how to get the result that you want.

So that's why you can just ask
it to do stuff because it already

has the documentation, the
step-by-step on how to do that

Bret: Yeah.

And so for those not watching, the,
I mean, we're staring at the open

source SemAI repo, and there's what
looks like dozens of skills there.

I mean, for my GitHub Actions, I
only have one giant skill, so now I'm

realizing I'm already doing it wrong.

Because this is like,
these are very specific.

They're very detailed, but they're
also very specific, and why not?

Like, why shove it all into one skill?

Why

Marcos: Yeah.

And we also have the golden
goose here, is the sem-ai watch.

So this is what enables the loop.

basically it's a comb that
will check your pipeline.

It's simple.

It's very simple, but sometimes
the simple thing is what we miss.

So basically it will
watch your pipeline run.

When it's green, it will get back
to your agent, so that way he

can loop without needing to have
the AI harness loop it by itself

Bret: So in this scenario, the main
agent would probably start a sub-agent

with this watch and the command in the
background if it's doing it right, and

then it-- this would allow my agent
to work on consecutive features one

after the other, or decide stopping
points, I guess, if I gave it some more

detailed instruction to stop halfway
or whatever and test that particular

function, run it through the unit test.

Yeah, that's kinda how I imagine
that we would get these agents to do

is it's just this automatic thing.

It's happening in the background.

I'm not fully aware of how many
times it's run tests or whether I

need to go look at a page and stare
at it waiting for the green button.

And I am curious, a quick question.

You talked about the profiles.

This is a subtle little
security conversation.

you talked about the patches that
you're creating for each… What's that?

Marcos: Roles, the service accounts
basically on the Semaphore.

Bret: Yeah, these different accounts
essentially, like how does that

work in your harness specifically?

Okay, because I feel like this
is really important for people to

understand that ideally, we wanna
isolate our, and sandbox our agents

and give them only the keys they need.

And right now, one of my biggest
frustrations is we didn't… Like the

personal access tokens and the, and all
the ways we provide our local CLIs and

tooling to access ser- services on the
internet, like it wasn't really designed

for a world where we have disposable,
always running agents, and like we need

these things to be ephemeral, and they
need-- They might need to be short-lived,

or they need to be managed in a way
where I can make more than one, right?

'Cause I remember when Docker it was
a long time before Docker Hub had

anything other than your password, right?

And then when `they made it, when they
created PATs, they were very limited.

They have read or read-write
across everything.

It's sort of like a giant blast radius.

And now, when we're in this agent world, I
think what you just described is probably

what, what I'm imagining is I really
want this agent on this work tree to

really have the smallest access possible.

I don't want them to
have admin on my repo.

I only maybe need them to be able to git
and git commit and push, and then run CI.

And I only want them to run the
CI for that particular repo, and

I need to give them that key, but
how do I give, how do I do that?

Or how do you do that, I guess,
specifically, 'cause this is

maybe not a Semaphore thing.

It's maybe more how you work.

How do you do that without
it being, like how the other

agents on your machine get that?

Are you just doing environment
variables per shell?

Is that kind of how you're doing it?

Or how do you approach that?

Marcos: Yeah.

What I would say, is that you should,
for example, start way more restrict

than what I'm doing currently, because
this is in scenario, I have lots of

tools, tool calls that I have already
whitelisted for my agents, right?

So you should start small,
don't whitelist anything or

just do some basis, basic stuff.

On Semaphore, you can create service
accounts and you can define roles for each

service account on each project, right?

I can put it as a reader or
as a admin or anything that I

want on my specific project.

You can basically control more,
most of this stuff this way.

So if you want a agent that
just wants to read the stuff

on Semaphore, you can put it.

Most of the time you are okay with
that because you just want to read the,

the pipeline status and then you, if
you are using GitOps, you are going

to commit another thing on GitHub
and it will spawn a-another pipeline.

And you can basically, like the
idea here is you can mix and match

permissions and each service account
will have a token, and that token

is used to authenticate on SemAI.

what I currently have, right, is basically
I have one token that has less privilege

than-- it cannot basically hit anything
that would damage my account or my

Semaphore, organization in this case.

because it was like with time and I put
more restrict-re-restrictions on it.

but basically I just let it run,
because it cannot do anything.

It cannot deploy to production.

It cannot mess with any
production related projects.

It can only do stuff on staging.

And for example, staging is something
that I think is more important in this

case because I can let it have more
reign over the staging environment.

It can do more stuff, but I am
always the gate to production when

we is talking about deployment.

Bret: Yeah.

Is this coming in through an
environment variable on your machine?

Is that how this is in your shell?

Marcos: This one, yeah.

This, this whole shell is
just one environment variable

of the token, basically.

Bret: Yeah.

I just had a live show last week.

We talked about nono, nono.sh, and
that's a sandboxing technology, and I

had one of the founders on the show.

And I'm a big fan of that.

So I'm just, for those listening, like
if you're trying to think about how

to isolate your tokens as well as your
agents, and Claude Code has this sandbox

thing built in, but the, the one of the
cool things that nono can do is you can

actually build out profiles that could
include these kind of secrets, these

kind of paths for things like Semaphore,
as well as egress rules that say like

you can only access package managers and
GitHub and then Semaphore, and that these

are only things you can do or whatever.

And then you, when you start
up Claude Code, this is kind

of the thing I'm wanting to do.

I haven't actually set it up yet 'cause
I'm using nono today, but to wrap

my Claude Code and my OpenCode and
my Py, my three favorite harnesses.

And I have found that Claude Code's
built-in sandboxing, I find it lacking.

So, the nono that wraps around it allows
me to give it file paths and environment

variables and secrets securely outside
of my harness, and then I can just sort

of feed all that in so that I don't
have to copy and paste things into

each shell every time I'm running them.

So I don't know if that's a pro tip, but
you can use nono in this way to probably

make a lot of this CI/CD stuff easier.

Like for me, it's like my default
GitHub command line has my key, which

has every repo that I have access to,
and I don't want my agents to do that.

Similar for like any of my
artifact storage or my CI.

Like I kinda want, especially as
a consultant, like I have access

to lots of different things, and I
don't want the blast radius there.

So I get nervous, and I end up using
nono to try to basically it removes

a lot of those secrets and removes
access to those things, so it doesn't

have access to my default tokens.

Shout out to the Agentic DevOps Guild
that I'm running because they're

the ones that we kinda worked out
a lot of these ideas around how to

use these for DevOps specifically.

Like, how do I avoi- avoid
Terraform, the agent having access

to GitHub, Terraform, Docker Hub,

Semaphore, AWS, and every key that
I have for the company, how do I

avoid it from nuking everything
every time I spin up Claude Code?

'Cause I know a lot of us are
dangerously skipping permissions.

I know I am, and that's bec- that's
because we're tired of hitting the

Yes key and, the, the, the Enter key
for every yes and request it has.

So if we're all doing that, like my pro
tip, if you haven't heard that episode

before this one, is to go check out nono.

Add that to Semaphore.

Now you're gonna, now you're
gonna level up your DevOps.

You're gonna… You're now DevSecOps
because you're using a sandbox.

That's maybe not the entire requirements
for being DevSecOps, but that's an

important step that I think a lot
of us aren't comfortable with yet.

But nono's really easy to use.

I feel like I'm doing an ad for
them in the middle of this show.

And they'll work great with Semaphore
for isolating some of these workflows.

Very cool.

Marcos: You can also do something
interesting with Semaphore because you

can put the secrets on the project itself.

So for example, we can put the GitHub
access tokens and other platforms

that you are using on the Semaphore.

And you can, for example, if you really
want to be very, very secure about that,

you can create one project per agent
or per activity that you are doing.

So you can basically put
the secrets on Semaphore.

It will only have access to those
secrets because you can restrict it to

only working on the ephemeral machine
and it can only have access to that.

And that with the service account will
be basically a really good sandbox to

keep your agent in check basically.

And also for deployments, you
can put a lot of s- specific,

preconditions on Semaphore.

So if you are deploying from Semaphore,
you can prevent rogue agents from

deploying stuff that it, they shouldn't.

Bret: Very cool.

Yeah, CI/CD is turning out to be
a really, a really good place for

attackers to find weak spots in all
of our, in all of our automation.

And I feel like, prompt injection,
between prompt injection and just general

supply chain attacks, I feel like we're,
our CI is constantly under threat.

So anytime I can level up the
security or run, like security audits.

Is Yeah, that's cool.

I mean, The fact that
it's built in is nice.

The fact that I don't have to go
find a different solution to plug

into my CI that makes this kind of
possible is probably where everyone

else is gonna go eventually too.

I think this is, as, as… I
just wanna stay in my harness.

I don't wanna have to wait any more than,
than the prompts require me to wr- wait.

I wanna reduce the number of steps.

I feel like in a lot of ways,
agents are helping us untangle the

last 10 to 15 years of complexity.

As someone in the cloud native ecosystem
and has been teaching Docker and

Kubernetes for a decade, to me, those
tools were, especially Kubernetes,

was always primarily for operators.

It wasn't meant to be
a tool for developers.

And also, a lot of the teams I would
work with would have a couple people

that were well-versed in CI/CD, but not
everyone needed to be a CI/CD expert.

But something happened over the last
decade where, I don't know if we got

distracted by the name full stack
developer or if the, if somehow people

got confused about what DevOps really
is, but somehow at some point, the

industry decided that developers are
responsible for everything, which I feel

like is a sad state of things where you
can't just focus on being an excellent

front-end developer or an excellent
architect of applications, but now you

have to know the CI, all the sa- CI/CD
paradigms, all of the operations, all

of the cloud infrastructure management.

You have to know Terraform, you have
to know Kubernetes and Docker, and

oh yeah, and you also gotta be this
great developer, and I always felt

like that was completely unrealistic.

And now what I'm seeing is, as me and my
friends talk about how we talk about a

couple of things that aren't necessarily
great, like one, when was the last time

we actually looked at code besides this?

When was the last time I
actually wrote some code?

It's been a while.

And two, when was the last time
I actually learned something

new without the agent involved?

Like, when was the last time
that I learned a new tool just by

hand, or learned a new language,
or learned a new platform?

It's actually, we were talking about
it, that we're actually worried

that these agents are making us so
nimble and it's real- It's, to me,

I think it's just a sign that we all
love this tech, but it was too much.

There was too much we had to know.

There was too much we
all felt we had to learn.

And this is allowing us to do things
like be an expert in CI/CD, be an

expert in automation, be an expert
in Kubernetes without having to

know the Kubernetes command line.

And I don't know where this is all going.

I don't know if in three years we're
gonna wish that we hadn't done all

this and, we- we're all Luddites in our
understanding of command line tools, but

I am enjoying very much not having to
remember all the command line nuances

of the dozens of command line tools I
think I have to use in a given week.

It's just, it's gotten crazy.

Like 15, 20 years ago
we had very few tools.

We were all in one editor.

On Windows side, we
barely even had terminals.

So I'm just sort of soaking, basking
in the amazement of this moment where

agents are allowing us to implement
these things very quickly and to

get very productive pretty quickly.

I just hope they don't go away because…
Or that we don't lose access to them.

While we were talking before
the show about, you gotta have

a backup subscription now.

Your agents have to be redundant,
so Claude Code isn't… We had

a Claude Code outage yesterday.

Claude Code can't be
your only subscription.

You have to have some other backup
plan, whether that's OpenRouter or

ChatGPT or, yeah, or Kimi or whatever
your favorite subscription is.

You gotta have something there.

Okay.

This has been really cool to dive in.

I've been really interested in how
this all works, and I do think between

your advanced plugin model and the
agent, having all those different

skills and all the different commands,
as well as the fact that it is sort

of empowered with the test boxes, I
think that's a really cool advantage.

'Cause someone earlier was asking,
"What are your differences between

you and other CI/CD platforms?" And
you're welcome to answer that question.

We always get the comparison
question in a show about a product.

But I will just answer that question
myself and say, as someone who operates

CI/CDs, those two things, like the
plugin model for those harnesses and

the test boxes to me with, in just
one hour of looking at Semaphore are,

to me, the advantages there, as well
as it just being open source and I

can run it myself if I feel like it.

l- like that, that seems like a,
th- the true futuristic agent n-

agent native way to operate a CI.

So I don't know if you
can say it better, but

Marko: No, you, no, you did a great job.

Thank you so much.

Yeah

Bret: But yeah, you can hire me anytime.

I'll be your DevRel.

but yeah, this is r- it's amazing
to see how far you all have come

in a year since you open sourced.

This is just a lot that you've
done, and it's pretty impressive.

So what's next?

What are you working on?

What are the hot tips for
the rest of this year?

Marko: Yeah.

So, so now that we have this interface to
the agents, we want to get to the basics.

And it's again, focused on, like really
doubling down on developers' productivity

and what does that mean for us?

First of all, if something fails in
my workflow, in my pipeline, I want

to be able to give as much as possible
information, the right information

to the agent so that the agent can
figure out what it is all about,

and hopefully fix it on its own.

So that I don't, as a developer,
I don't have to intervene.

And there is like a lot of nuances
like, what are, what's the format?

What's the amount of information
that you need to deliver to the

agent in order to get this right?

So that's the number one thing,
because I want to get to green state

as Marcus showed as soon as possible.

The other thing is that if everything is
green and everything is okay, I want to be

able to monitor how is my pipeline working
over the time and give enough data to my

agent so that it can optimize my pipeline,
because that's also very important.

We all know that as your test suite
grows, the duration of your pipeline

grows as well, and it's always healthy
to refactor or optimize your pipeline in

order to keep it under 10 minutes, under
15 minutes, whatever is your target.

And then last but not least is
like, enterprises, they need a

lot of governance and security.

We want to introduce some additional
services that can complement

the SLSA compliance, that's
the word that I'm looking for.

So that, that is also, these things can
be done with Semaphore right now, but

we want to make them out of the, like
working out of the box without additional

configuration and thinking through.

So it's how can we make developers more
productive with the focus on CI/CD.

And there are like s-
a lot of small things.

For example, like a next thing that we are
going to roll out is ability to just run,

rerun a single job in the whole pipeline.

Because a lot of the times, currently
you can rerun only the whole block.

But for a lot of the
times, only one job fails.

And you really want to be able to really
effectively just rerun that job to s- to

make sure that, hey, maybe it was like a
little bit of flakiness or something like

that, but you want to be able to push it
forward quickly and keep on developing.

That's, that's the goal.

Bret: Especially if that was
the f- the two-minute run

versus the, the 20-minute run

Marko: Absolutely.

Absolutely

Bret: You mentioned SLSA.

This is what you're talking about, right?

S-L-S-A?

Marko: Mm-hmm.

Yes.

Bret: Yeah.

So for those of you listening that are
not aware, I'm actually a big fan of SLSA.

we actually talked about it recently.

I've mentioned it more than a few times
over the years because I was really

paying attention to it even before version
one came out, and this is, this to me

is, the supply chain checklist bible.

I look at this as if I am not sure
around whether my supply chain is up

to snuff in terms of its security and
me doing all the right things as a

DevOps professional, it's not like we
have formal education where I come out

of school and I know the exact minimum
things that I should be doing in an

enterprise to keep my software safe.

And so SLSA to me is one of the
closest things to help me with that.

It will give me very prescriptive
step-by-step approaches to make sure

that I'm building securely, that I'm
providing providence with my code,

essentially built into the artifacts,
that I'm building the artifacts with

all the details that I need in order to
understand how and where they were built,

and when they were built, and by who,
and with all the right, all the different

details that I need to put in there.

And then making sure that I'm sort
of doing the best practices in the

industry, like linting code, or I'm not
sure if linting is actually on there.

But like doing some of the minimum sort
of… So the, the stuff that honestly,

quite frankly, if we, if every open
source repo in particular was doing

that over the last year, we probably
wouldn't had near as many attacks on

repos that we've seen in the headlines
over the last year, because almost

all of that is due to a lack of teams
properly securing their workflows

and their behaviors and approaching
things from a DevSecOps mindset.

And if they had been following the
SLSA, which has levels, so, you can

start at a sort of beginner level, and
you can sort of promote and graduate as

you get better and more strict, 'cause
some of these things are easy and some

of these things, the steps by, steps
that it makes you do are gonna require

your, more than just your own effort.

And so I'm always trying to s- on,
especially on the re- open source

repos where they're public and I care
more about them, I'm always trying

to, level up my SLSA, from w- I can't
remember the levels or whatever.

But level one, level two, I'm not sure
if they're numbered, but I'm always

wanting to try to level those up to get
my software a little more hardened, a

little more, a little safer essentially.

And so go check that out.

That's S-L-S-A.dev, and they are part of
the OpenSSF, which is also a great thing

for everyone to check out for security,
and they have other stuff beside…

All, all the software security stuff is
over at OpenSSF, so go check that out.

That's another part of the cloud native
ecosystem, so I feel like I have to

plug it ' cause it's, it's cool stuff.

awesome.

Well, I'm looking forward to it.

How can people get started?

Okay, so you have the repo.

they can go over to,

Marko: Just go over to semaphore.ai.

Yeah.

And yeah, .io, exactly.

And, and, there is it's a simple command
that, they can run and install it.

it gets installed and then they can
say, "Hey, create my account." It will

prompt you to create your organization.

So the whole onboarding
is also part of the SemAI.

Bret: Right.

yeah, 'cause on your getting started
guide, it's funny, this is this is just

the indication of the cloud native or the,
agent native world that we live in now.

it's not download our CLI and
create an account and go do this

and read our repo for, or read our
documentation for how to use a CLI.

It's pl- install this plugin into Claude
Code or Codex and then ask it to get

started with Semaphore or whatever
the com- whatever the prompt is.

And, I am so… I feel like this is one
of my last resistances against the a-

the agents, is I'm so not used to that
workflow of just get the tool, install

it, create my account, do the things.

I don't want it… And
tell me when it's all done.

I'm just, I d- I s- I still am so
programmed to look for the command in

Brew, find which Brew package I need to
install, read the getting started guide,

create my account and all that stuff.

So it's cool that you're making--
You're not just making the use

of a CI AI native, you're making
the setup and c- account stuff AI

native too, so that's really cool.

All right, Marcos, Marko, thank
you so much for being here.

Marko: Yeah.

Thank you for having

Bret: you around the, we'll see
you around the agents, around…

We'll be chatting with our agents
are just gonna do this from now on.

Actually at some point this podcast is
just gonna be a bunch of agents talk.

Your agents will join my agents.

We will, they will live stream and
tell everybody about it, and then

our jobs will be to sit around
the pool and drink margaritas

Marko: Yeah.

Well, okay.

Sounds good.

Bret: All right.

Sounds good.

That's the utopia future.

The other future I don't wanna talk about.

Bret (2): Thanks for watching, and
I'll see you in the next episode