Search
Home
Login
Sign Up
Not Just Faster Horses
Seth Rubenstein from Pew Research
Sep 7, 2026
Seth Rubenstein from Pew Research
15
30
00:00
40:48
Listen on
YouTube
YouTube
Transcript
0:00
In any given month, our average is about 4,000 to $5,000 in token costs for six engineers and a couple of designers. I'm not hiring a developer for 60 grand. No.
0:07
And in the DC metro market, I'm not hiring a junior for 60 grand. I'm certainly not hiring a senior for 60 grand. We, we had this whole plan for the year, and we're pretty much wrapped up.
0:16
In just a few months, we took out a whole year's worth of, of work. That is the struggle is like we can build anything. What should you build?
0:24
This is Not Just Faster Horses, conversations about putting AI to work inside media companies across people, product, and engineering. Not the strategy, but the reality. I'm Ben May, founder of The Code Company.
0:37
Today, I'm speaking with Seth Rubinstein, head of engineering at Pew Research. Pew is one of the most regularly cited research organizations in the US.
0:46
So Seth, you've spent 20 years in WordPress and around a decade at Pew. Um, where is the engineering team today? Paint us a bit of a picture for today. We are in a completely different era, obviously.
0:58
Um, we are about six of us now, um, and with a team of a additional five designers and a few kind of producers and, and editorial people.
1:08
And, uh, day in, day out, uh, we're working entirely on WordPress projects, uh, and this digital news platform that we call PRC Platform that is open source and available for anyone else to download and try.
1:21
Um, and our day-to-day is just chatting with agents now. [laughs] Absolutely, which we're gonna dig into.
1:27
So the broader organization, you, we were talking earlier today about, um, the digital team you sort of sit inside within design, engineering, um, under, I'm guessing, like, the web and all other digital products.
1:38
What does the editorial part of the business look like, the people who are using the stuff you're building, um, for people?
1:44
Yeah, so, so right now we're actually really in a very exciting kind of phase because we are in the middle of what we're calling our Word to WordPress project, which has blown up to actually become the everything to WordPress project, where we're collapsing a lot of our digital production workflows.
1:59
We are like a very print first heavy kind of focused organization. We produce PDFs before we actually produce web material. So our editors will go and will talk with researchers. They'll get their kind of ideas.
2:12
They'll go write a report in Microsoft Word. They'll take that, go make a PDF of the report with charts and stuff. The producers will take that PDF and then put that into WordPress. Guess what? Nobody reads the PDF. Yes.
2:25
[laughs] It's a lot of work.
2:25
So we're in a, we're in a interesting phase right now, actually, where we're starting to onboard the rest of the staff, the rest of the centers, the researchers and everything, and we're bringing them into WordPress to just start writing there now.
2:35
We're removing kind of the production and the editorial, editor assistants out of that, out of that flow, and they're kind of finishing up the final product at the end.
2:42
And we're hoping that that will kind of accelerate not just, you know, all the things that we've been doing with AI, but it'll accelerate the writing process, too, because you're in there, you're gonna see what the final product looks like for the, for the, the web product, which is the real, the real thing that we wanna deliver.
2:55
What people consume. Yeah, exactly. And so we've been working just a lot on what does that look like? You know, what are all the editorial tools you need inside of WordPress to, to kind of achieve that mission? Mm.
3:04
Things like, obviously, editor notes, real-time collaboration, track changes, uh, kind of pipeline workflows, you know, so that posts go from in review to, like, in number check.
3:16
That's a thing that we do at the Pew Research Center way too often. Um, et cetera, et cetera, et cetera.
3:23
And so we're in this really exciting phase where we're kind of going through this digital transformation of getting everybody out of other products- Mm...
3:30
Microsoft Word, Adobe InDesign, and we've brought that entirely into WordPress. Wow. So you mentioned before, uh, your days talking to agents. Uh, jumping into the engineering team, what does a regular day look like?
3:42
You've sort of... You know, you- you're quite prolific on Twitter, uh, or X, talking about, uh, you know, the, the, the, the woes and, and challenges and, and, and success, as well, of sort of y- a lot of the...
3:51
You and your team's journey, um, as you sort of double down. What does a normal day look like? What does the sort of life cycle of work look like right now where you are in this sort of transition?
4:00
Yeah, so right now, I mean, it is kind of changing every day, uh, 'cause we're trying out new things. Mm.
4:04
Um, but on any given day, we have a few automations that run, and we're running through Slack conversations from the day before, new Linear tickets, performance metrics from the day before, and we're kind of having new tickets be generated.
4:16
Like, "Oh, uh, we saw on this database page, you know, uh, transaction time was 1,000 milliseconds. Why did that happen?" This is just happening automatically now. Mm.
4:26
Um, these were things that, you know, we would have to dive into and kind of inspect the logs and go into New Relic and et cetera, bring it down locally and test that, et cetera, et cetera.
4:34
We just kind of collapsed that to just be all autonomous. And so now a lot of our day is spent managing the agents on, like, what are you gonna do next? Was that output good? Et cetera, et cetera, et cetera. Yeah.
4:45
And the, the hard work is now just being handled autonomously by kind of our, our whole system. It's... We've kind of set up, like, a whole self-healing system, right? Yeah.
4:56
How do you, when you're sort of designing what the new team looks like, think about how you sort of manage budgets, and, and even from a financial perspective?
5:03
[laughs] Um, because everyone at the moment, uh, is, is really excited. We're trying new things. We, you know, can, can go off to some deep ends pretty quickly. Um- Yeah... how do we...
5:11
E- especially 'cause it's sort of... You know, there's an element of this that is replacing headcount 'cause it's not just traditional effort hours anymore. It's, you know, this force multiplier stuff. Yeah.
5:20
I'll push back on this. I'll push back on this idea that, like, AI is replacing headcount. Um, I think for... I think we're, we've seen this play out over the last year, like big tech has had a lot of layoffs.
5:30
Lot of headlines, yeah. Right, and they're saying it's because of AI. Huh. Uh, but then we're finding out, like, oh, actually, they need to hire a bunch of people back. Yeah.
5:38
Conversely, you look at smaller organizations, like Pew Research Center- Yeah... where we don't have high headcount, we're using AI to augment the roles that we don't have. Yeah. Right?
5:46
We don't have an accessibility person checking- Right... every single thing. We have an agent checking that. Yeah.
5:50
Um, you know, all these sort of things that we don't ha- we don't have the, the, the financial resources to afford, well, you know, $100 in token credits is, is much less than- Fifty, $80,000 salary, uh, benefits, et cetera, et cetera.
6:05
Mm-hmm. Um, so I'll push back on that a little bit that I don't think that, I don't think that the AI replacing headcount thing is playing out.
6:13
Not, not, not how the doomsdays, uh- Yeah, not, not the way that people are kinda portraying it y- yet. I don't know where that might go. Yeah.
6:19
But at least right now I think it's, it's interesting because in the small organizations that I've talked to, it's, it's not, they're not talking about how, how can we reduce the headcount.
6:27
It's how can we augment what we don't have with AI. Yeah. So that's one thing.
6:32
On the costs, and I think this is probably the, the scariest part, is a lot of organizations I think I'm, I'm looking at, they're using Claude Ultra, Claude Max. They're not realizing the token costs- Yeah... at all.
6:44
We are. Um, I switched us over to token billing at the beginning of the year because I wanted an idea of like, hey, if, if all of my engineers are not hand coding anymore, what does that cost me? Um, and
6:57
I think it's reasonable. Uh, I'll, I'll tell you. Uh, in any given month our average is about 4,000 to $5,000 in token costs for six engineers and a couple designers using our, our Cursor team plan. Yep.
7:07
Um, and that has gone down in just the last couple months because of new models like Groq. Just token efficiency in general, yeah. And token efficiency.
7:13
I mean, the models are getting better, that they're also reducing costs, but there's more competition now I think for like that frontier model access. And so this last month we spent $3,000.
7:24
The month before, before we had access to Groq, we spent $6,000. So we- we've been doing a lot of experiments with like which models are, are we getting the most bang for our buck basically. Yeah. Yeah, great.
7:36
Um, so given stuff gets done a lot faster now, uh, the time from sort of conception to production can, is compressed dram- dramatically. Mm-hmm. Um, who is deciding in your team what, what's actually going to be built?
7:50
Um, that's where the problem seems to be shifting up towards. Now, now that the building is actually compressed and, and collapsed, it's now what do we actually build? What's the priority? What moves the needle?
7:58
How are you thinking about that sort of rubric? Yeah, well I mean, at, at always, I mean, w- what we're building is are we fulfilling our mission, right? Are we informing what we need to do for the American public?
8:08
Are we telling the story that we need to tell? But, you know, how we, how we do that, the ways that we do that has expanded now because of all this agentic engineering power.
8:17
Now we're having designers actually prototype out ideas way ahead of time, and they're kinda getting through all of the, the, the messy thoughts and the things that you don't think about when you're just in Figma.
8:27
Um, and that's accelerated kind of the, the breadth of products that we do offer. It was very limited before.
8:32
You know, we had a, we had a map product and we had a kind of a chart product and, and, and so on and so forth. But now we can try out all these sort of different things. So w- who decides what gets built?
8:41
It's actually pretty democratic now because anybody can open up a chat session and say, you know, "Let's go try this out." Whether it makes it to production or not, that still lands on our, my team. Yep.
8:53
Do you have a traditional sort of product owner, m- product manager function that sort of sees the digital product as a whole? Or how does that sort of work?
9:00
Historically no, uh, but because of how fast we are moving, we actually have hired like a product manager. Yep.
9:07
Um, and we are kind of changing all the way that some of our project managers work to think more product oriented. Yeah.
9:12
Because we can move so fast, because projects are, I won't say trivial, you know, s- there's still work to be done, but it's, it's, it's, it's so much quicker now so we have that, that headroom to think about like, "Well, how do we productize, how do we productize this?
9:25
How do we take that thing that we keep building over and over and over, and how do we make that something that we can make into a plugin or make into like a content type that's really easy for a layperson to use, a researcher or writer, not a designer or an engineer?"
9:40
Um, and so just all of the eng- agentic engineering powers have, have forced us to think about product in a different way. Yep.
9:49
And I, I imagine for Pew perhaps compared to some, some more traditional media companies that, uh, a north star's gonna be selling a subscription, ad tech. Like there's a lot more commercial imperatives.
9:59
I, I imagine your north star's probably somewhat different, uh- Yeah... for, for Pew. Yeah. Our, our north star is we tell the American public who they are. We hold up a mirror to them, you know? Yep.
10:09
What do you believe? What does your neighbor believe? What does the person in the next state believe about X, Y, or Z topic? Yep. That's the most important thing for us.
10:16
Uh, and so, you know, the cost on what it makes to build something, that's like not as important to us as like the mission. Yep.
10:24
So, so a lot of teams experimenting, uh, using AI in, in production and, and moving things a lot faster. The... I think what a lot of people are challenged with is how do you quantify this?
10:33
Um, I know you guys recently did a case study with WordPress VIP- Mm-hmm... which we can talk about now, the, uh, sort of, um, LLM readiness project- Yeah... I think it was called. Yeah.
10:42
Um, and if, if nothing else, it's an interesting story of, of some work which we can jump into because that's talking about AI almost from the other end of, uh, consumption of content with AI. Yeah, yeah.
10:50
But just on the media si- sorry, on the engineering side for the, for the minute, um, at least that looked like you had some quantifiable, um, measurements from, uh, what you're thinking in terms of time to build all of this stuff and how long have they done.
11:04
Lines of code is obviously a very bad measurement. Token usage, very bad measurement. But, um, yeah, do you wanna talk I guess from the, from the product and engineering side? Then we can...
11:12
or however we wanna flip through. Yeah. Yeah, sure. So yeah, I mean, beginning this year, uh, we actually got a grant from our parent organization, the Pew Charitable Trusts, to kind of dive into AI readiness.
11:20
What does it mean for like a small news publisher, especially a nonprofit news publisher, to be AI ready? Because our concerns are completely different from most news publishers. We don't have ads. That's not how...
11:33
We don't- You're not asking for money. Right. We're not asking for money. Um, and so if ChatGPT comes along and, you know, steals, indexes all of our content without asking us, that's not a big problem for us.
11:42
That's actually a good thing for us. We want to be everywhere that people are asking questions about these hot topics. So the kind of like
11:50
the, the concerns that we had are completely different than I think the rest of the industry. For us what we wanted to make sure is- Are we visible? Obviously. But are they getting it right? Mm.
12:00
Uh, that's the biggest concern. You know, if you ask, uh, ChatGPT, "What does South Dakota think about X?" And you give some completely wild number, that's just like not acceptable to us. That's the biggest risk to us.
12:12
It's not about losing kind of traffic to the site, it's about getting the information wrong. It's about informing- Which is really odd, because there's no- Yeah...
12:19
because you're trying to fix accuracy on a platform you can't control- Right... and gives no instruction on how to provide- And it's not deterministic... the data correctly. Yeah. Right?
12:26
There's no way to say it- you should always return this number. That's right. There's just no way to do that.
12:30
So a lot of, a lot of the beginning of the year was, like, just kind of benchmarking all of the different AI models. How are they seeing our information? How are they seeing our data?
12:38
Um, you know, are they making mistakes? Are they hallucinating? And, uh, well, the, we found out, yes, they were hallucinating quite, quite, uh, often.
12:47
Um, it'd be funny, we'd ask, you know, a question and you'd get back a URL. Like, oh, that URL structure looks correct. That slug- Mm... that looks like a post title we'd write. You click on it, 404. Oh.
12:56
'Cause that's not a real article. Oh. So it would just kind of invent, like, a URL like, yeah, sure, they would write this. So that, we were concerned about that, and so that's where this AI visibility then came in.
13:06
So we, we partner with the VIP through their, uh, FDE program, and we had an engineer from VIP kind of on board with our team. And in January we said, "Hey, here's our roadmap."
13:16
He sat down with us, "Here's our roadmap for the whole year." And over a couple, the next couple months, we worked together. He was kind of prototyping things with Claude. We'd take it on and go further and finish it up.
13:29
But a lot of, like, the AI functionality that we wanted to get really early in, done within 30 days. Yeah. Uh, things like converting all of our content to markdown.
13:39
Um, things like, uh, you know, exposing proper head, head tags and meta that AI indexers were keying off of. Um, that was, like, a lot of research that we had to do to kind of get to that.
13:51
Um, and then, uh, converting that content to, like, m- different content types so that we can be in more places.
13:59
Because one of the other things that we have found is that it's not just enough to say, "Hey, you know, ChatGPT, go to this webpage." I kinda call it flood the zone.
14:10
We need to be on as many signals as possible so that when you do ask a question and when ChatGPT is going and res- searching Pure Research, but it's also searching Twitter, and it's also searching Instagram, and this is also searching Wikipedia, and so on and so forth.
14:22
So how can we kind of repurpose our content to different formats like video or audio podcast or whatever? So that was the other second half of kind of what we worked with a VIP on. And that also just took a few months.
14:35
Um, I think the most exciting thing is just, like, we're at August now, we, we had this whole plan for the year, and we're pretty much wrapped up. Right now we're just kind of, like, we're, we're polishing everything.
14:47
Yep. But in just a few months, we took out a whole year's worth of, of work. Yeah.
14:53
Which, which, going back to our earlier point about, um, that the amount of human hours required, all of a sudden you can just do so much more with the same team- Yeah, yeah...
15:00
doesn't mean necessarily offloading team members 'cause all the work's done for the year. But actually all this time we've got three, four mon- months of runway, uh, to get stuff out.
15:08
So in that, in that case study, you talked about some, some numbers, I think, um, uh, you know, 1.5 million lines of code, which, you know- [laughs]... developers love knowing lines of code. Yeah, yeah, yeah.
15:16
That's a real indicator of anything. Um, but yeah, three to four times ahead of roadmap, so 10 times faster than conventional development.
15:23
I like that we're now calling that artisanal, uh, software engineering- [laughs]... as conventional development. Um, and then six months down to one month for that full, um, you know, build. Mm-hmm.
15:33
Uh, how did you sort of...
15:34
O- one of the things people are obviously trying to do is measure the efficiency outputs of AI, which is really tricky because, a- and this is like the whole name of this show, is like th- it's not just making everyone a little bit faster, it's, you know, trying to understand that the, the promise of AI in, in this sort of engineering context has been able to, you know, completely upend the process from, you know, brief to pushing something to production.
15:55
How do you think about, like, measuring that efficiency?
15:57
Were you, when you said it was a six-month conventional build, um, had you gone through that process of, well, if we were building this by hand, here's how we would sort of scope it out, and then measure it?
16:05
Because everyone's trying to figure out, like, what is our actual ROI or what's our, like, superpower now? Yeah, yeah. Uh, uh, we, we didn't, like, scope this out, like what would it be to kind of build this by hand.
16:16
[laughs] Yeah. Which is such a weird thing to say now- Yeah [laughs]... about code, right? Um, the ROI on it is the work that you're doing. Um, is your time best spent writing code? Yeah. I don't really think so.
16:29
I think your best time, you know, your, what you bring to the table is, like, your higher level thinking. Like, how does this fit into the organization? How does this fit, fit into the code base?
16:38
That's, like, still a question that I don't think that AI can answer really well, that, like, a human needs to kinda have that artisanal touch, I guess. Yeah.
16:46
Um, and how does it interplay with everything else, you know, with all of your other plugins, and how are you gonna use this thing?
16:54
The, the time, the time crunch down on that, now you're spent reviewing it, now you're spent thinking more broadly. Before, it would've been like, okay, we're coding, and you're testing along the same time.
17:06
And so now most of your time is spent on the backend of things, not the f- the front part of it. And I think that's actually been kind of liberating because now I can just...
17:15
I don't have to worry about, like, oh, you know, how are React effects, uh, playing with each other? Did I use too many effects? Did I minimalize this?
17:21
You know, that's like, takes like a whole day to just kind of write one really good React component and test it. Yeah.
17:27
And now I'm just spent time thinking about, like, well, where can I use that React component in other places, and, and so on and so forth.
17:32
It's just more about how can I utilize all this technology and stuff that we're building in a way that saves us all time from having to, like, keep repeating ourselves. Yeah.
17:44
Obviously preaching to the converted, but I guess the thing I am interested in, in pulling the thread on is when, when we talk about, like, so co- your costs weren't outrageous.
17:52
It's not like you're spending hundreds of thousands of dollars, but- Right...
17:54
but there is always gonna be a point where some bean counter will look at a profit and loss and be like, "Well, hang on, we're spending $60,000 a year on tokens." Yeah.
18:01
And if you're not necessarily saying, "Well- That $60,000 is contributing- equals, right... to 50% efficiency. Have you had that pushback? Has it come up at all?
18:09
Like, it's, I don't think anyone's got the right answer for this yet- Yeah... but it's interesting of, like, who's f- thinking about it. I don't know how you quantify it. Yeah. Right?
18:15
Because it's like, yeah, it's- it's- it's- it's at worst 60 grand a year for us. Okay. Yeah. Well, I'm not hiring a developer for 60 grand. No. In, in the DC metro market, I'm not hiring someone for 60 grand. Yeah.
18:27
I'm not hiring a junior for 60 grand. Yeah. I'm certainly not hiring a senior for 60 grand. So I think on that end of the, the kind of spectrum, like my bosses, you know, the, the- Yeah...
18:36
people that are paying the checks to, to Cursor and so on, um, they understand that, right?
18:41
They see that, like, we're a small team, we have big ambitions, we like to move fast, and you can either increase headcount to achieve that or there's this other option. Yeah.
18:50
And so that's, so that's what I'm saying about, like, the, the kind of AI and headcount equation of this, is this, I think that you need to invert it a little bit. Yeah.
18:56
It's not about how are you gonna take away, it's what are you gonna add that you don't have? Yeah.
19:02
And again, I think in digital that's historically been very hard to do, because we all agree hours are the worst un- economics because one hour of my time versus a junior's is two different things. Right. Yeah.
19:12
Lines of code mean nothing. Consumption of tokens mean nothing. What is a feature? What is a... You know, we've, we've invented these whole systems like Fibonacci to try and abstract from that.
19:22
[laughs] So we've, we've literally created a whole generation of getting to this point where we actually have no way of quantifying output without adding costs.
19:30
So it's a really weird, like, journey to sort of be navigating. 'Cause no one's gonna say, no one's gonna be like, "You know what? We'll just cut that 60K and go back to doing it all by hand." Like- Right.
19:38
Oh, no we're not... staff will be leaving. Like, you know. Yeah, yeah, yeah. Well- I'm not gonna write code by hand... well, and the other thing is, like, let's quantify the quality, right? Um,
19:46
I like to think of myself as, as quite of a good developer. I like to think of my team as good developers.
19:51
But there's certainly things where once, once kind of Opus 4.5 came, and once really things started to take off, you know, it's writing stuff, I'm like, oh, uh, it's definitely handling things like react effects and memoization way better than I was.
20:03
Yeah. You know? I think that the other har- other part of this is
20:08
for, for really small teams, you know, if you're not a big organization that has all these different skillsets available to you and all these different roles that are, like, checking for all these other performance things and accessibility and et cetera, et cetera, et cetera,
20:21
uh, our quality's improved. Mm. It's not gone down. It's actually improved. And, and if you're tracking the right things, outages, rever- like, you know, uh, merge reverts- Right. That tells you-... things like that...
20:29
yeah... those things are la- they're lagging indicators, but at least they will help validate- Yeah... that case of, well, you know, we're actually having less reverts on production because we're making...
20:37
things are getting more thoroughly tested and by more layers and- Yeah, yeah, yeah. Well, and one of the really interesting things has been kind of like this AI bu- let's call it bug bot, right?
20:47
Like, looking for bugs before you've actually... That's just, like, not something that I'm- Yeah... familiar with, right? Like, my bug finding process before was push the code up to a server, run through stuff- Yeah...
21:00
see what breaks. Quickly revert. Right. And maybe, maybe you write some tests. Yeah. I mean, writing tests, you know, we can go on all day of, like, what's the right thing there, whatever. Yeah.
21:08
But, uh, I didn't write a lot of tests before. Who did? [laughs] That's how I tested. Yeah. And now kind of we're testing in the code as it's getting written by the agent, and that's just, like, a different kind of
21:21
speed boost I just don't think people are, are, are understanding really. Yeah. The, the quality is better and the output because it has been tested as it's going, you know?
21:30
The agent is kind of thinking about more things as it's going, as it's writing a few lines of code. Back in the room. Yeah, exactly. Yeah.
21:36
So before we leave the LLM project you were talking about, obviously the, the accuracy is really important. How has that project sort of shipped out? What are the...
21:42
I, I know we're jumping out of engineering for a second, but how is that from a consumption of content from LLMs and things like that? Yeah.
21:48
And, and has it helped move the needle on accuracy from, again, a very hard to test and measure, um- The accuracy thing is really funny because when you go to ChatGPT, when you go to Gemini, when you go to Claude and you type in something, you're hitting the API.
22:04
But around that is hundreds of tools and different layers that Claude and ChatGPT have created to kind of scope your question and run it through some different checks, and when the answer comes back, make sure that that's kind of really valid.
22:18
That's gotten a lot better. But on the other hand, if you actually just make, like, an API call to Claude model, and you ask a question, that still isn't great.
22:27
And the, the strategy that we've kind of, that we've, we've taken for that is, like, well, let's, like I said, flood the field.
22:33
If we can get as much of our information into various different formats and outputs, then we're hoping that on the API thing, um, and when m- more broadly, that as indexers look at our information, they're seeing it in a variety of places, and that gives a signal to say, "This is the answer," you know?
22:51
Mm. Yep. So the accuracy thing, it's gotten better on the kind of consumer end. Um, what's been more interesting is the indexing part of, of, of, of, of the equation.
23:01
Um, so now we're seeing ChatGPT, Claude, et cetera, actually, like, index more of our content.
23:06
They're going deeper into our site map, um, which I think kind of shows that they're kind of understanding the structure a little bit better.
23:13
Um, and we're seeing in a variety of tools that we have, like Profound and kind of Toolbit and some, some nascent AI tracking tools that we're seeing a lot more.
23:21
You can actually see, uh, ChatGPT breaks out their indexer from, like, web search to their actual corpus indexer to, like, what's gonna get- They're learning, yeah... rolled up into the model.
23:29
And we're seeing, like, more of our content actually get rolled up into the main model's training data set. Right.
23:34
Which, once again, we'll see how that plays out, but I would think that that's gonna yield better results over the long term. And you would imagine it could then use that information for decisioning and, and- Yeah...
23:42
and summarizing something that may not necessarily reference the specific study or survey or, or bit of research. Right, right, right.
23:49
Uh- It's just part of its broader corpus of knowledge, which is, I mean, that's what we really, really want. You want it to have all the right information. That's really hard. Yeah.
23:56
That's a, that's a level of visibility that, like, we could not have gotten before. Yeah. Sure, New York Times can kind of get that level of visibility, but not us.
24:04
And so now when you go into ChatGPT and you ask a question
24:08
We're competing obviously with a lot of different sources, but I think it's just a little bit more of an even playing field in a way because there are no SEO hacks here. There are no tricks to get you to elevate more.
24:17
Mm. It's just is your content answering the qu- the question? Yeah. Um, so the, the so the name, the theory behind this- this- this interview is around sort of this whole workflow changing from just being made faster.
24:30
Um, what- what is sort of, um, a complete change from how you're working a year ago in terms of, you know, someone wants to make the button blue, or they wanna add a new menu item or whatever else?
24:40
How does that look now from a idea or, or, you know, right through to getting put onto production? It's more structured- Okay... in a way.
24:48
The, the whole AI race has actually made us more professional in a way, um, I guess because it's, it's kind of- It's been a reinforcing function. Yeah.
24:55
Man, it's like it's freed up time so that you can focus on those pipeline tasks, right? Like, your actual workflow 'cause you're not having to code anymore. You're not having to do the work.
25:04
You're thinking about the higher level stuff. Mm. So now it's actually more structured. You know, like you go create a ticket.
25:09
Before it might have been like we had a conversation in the hall, and, uh, I might prototype this real quick in Slack, and it would be back and forth, and this might take days, and it might get lost. [laughs] Whatever.
25:19
Um, but- De-prioritized. Yeah, exactly. But now it- it- it stays on task. You go make a linear ticket. We go assign that to an agent. It runs through. It provides all of its validation.
25:29
It provides screenshots and screen recordings to- to prove that what it did is correct, and now our stakeholders kind of can engage with that without having to directly go through us.
25:38
So it's, like, more visibility for them. It's democratizing [laughs] uh, the work- Yeah... more broadly across the organization. Yeah, okay.
25:49
So I mean, it sounds like you are, you know, for want of a better term, living the dream, uh, at- at the forefront of this, and I imagine that, you know, an organization of six is a lot easier to sort of do this pivot- Oh, yeah...
25:58
uh, from end to end.
25:59
What do you think's worked really well for- for you and your team because of the size that you are or, you know, for- for larger organizations that are still really grappling with this, um, that are- are just still, you know, uh, some people are- are still copying and pasting, like, snippets of code out of ChatGPT into their IDE, and- and that's still...
26:14
That's AI engineering through to sort of being able to, like, convert a Slack thread into a ticket which gets provisioned and- and off you go.
26:21
Um, what do you think the- the sort of learning lessons or if you were starting again or in a larger organization, um, sort of trying to make this journey that everyone's sort of in the middle of right now?
26:30
Uh, well, I think that because we are small, we've been able to move, like, really, really fast. You know, there's just less, um, red tape that we have to go through.
26:36
Um, if we decide to change a model next week that we're using, that's not a problem. No one's kind of watching us to tell us what to do, and no one's tracking our costs to kind of let the minutiae of- of- of tokens.
26:48
Um, so it's just really let us move quickly and experiment a lot. Um, if I had to start everything over again, um, I would probably start with, like... I would...
27:01
When we started really agentic engineering, my idea was, like, okay, throw the most intelligence at the problem. Um, use as mo- m- as many tokens as possible, right?
27:10
Because in my head I was like, "That's how you're gonna get the best result." Um, and I think what we have found is, like, that's not actually true.
27:17
I guess if I were starting over again, I would just have a better understanding of the kind of intelligence of models and how much effort you should really apply for what you're trying to achieve.
27:26
At each piece of the journey. Yeah, exactly. Like, a button change- Yeah... you don't need Claude Opus 4.7- Yeah... extra high fast, right? Yeah. [laughs] You know?
27:34
But that's what I was doing because, like, I didn't want to make any mistakes, and I wanted to have the kind of full context of the code base- Yeah... when I was doing things.
27:39
But now that we've, now that we've kind of fully into it, we have a bunch of agent skills.
27:44
We have this whole harness around just not only the agent engineering but our code base so that when an agent goes into it, whether it's a new model or a- a new platform provider at all, we're kinda getting the same outputs across Claude, Cursor, Groq, Gen- Mm...
27:58
GPT, et cetera. From a change management perspective, did you need to get a lot of buy-in from both your team, which I think engineers are more li-...
28:05
Most engineers I think are really excited by this, so probably not gonna lot of resistance, but I guess from your managers, from your- your higher-ups, um, worries, concerns? Uh- There were some worries. Yeah.
28:16
Uh, you know, uh, keep in mind we're what? Five, six years now into this, to- to the AI era. Yeah. Um, you know, when- when ChatGPT first came out, we were very, very, very concerned. Mm.
28:28
Um, not from the developers using it perspective but from once again the, like, there's this new platform that we need to be on, and they're providing no visibility into how our content's performing or how do we even rank in there or et cetera, et cetera, et cetera.
28:44
So there was, like, a lot of concern there for many years, and
28:49
it's really just in the last 12 months I think that that executive le- leadership and kind of our parent organization has really gotten comfortable with it because we now see that it's not going anywhere, um, and that a lot of our original fears
29:02
w- it wasn't a doomsday, right? Like, it- it- it's not entirely not factually correct. Yeah. It's not entirely spitting out bullshit all the time. Um, and so that really kind of made everyone at ease. Like, "Okay, y-
29:15
you didn't code that all by hand? That- that's- that's okay, I think." [laughs] Yeah. You can sleep at night. Yeah. But, you know, it... What's the right word here? Uh, trust but verify, I guess. Yeah.
29:24
Um, that's just kind of been our approach, and as an organization, we're really... We- we love innovation. We love experimenting. Yeah. So w- there's never been a, like, "Don't do that," you know? Yeah.
29:33
That's never been an- an edict from on high. Yeah, great. Uh, how do you think about bringing more junior talent in?
29:39
I know that's another one that, again, a lot of people we're talking to at the moment of- That's a tough one... yeah, is, like, where does that r-... And I think it's partly because we're all figuring it out still.
29:46
Um, where do you, yeah, bring people into the organization? Are they more AI native most likely than- than- than, um, seniors who may be, you know, uh, the opposite?
29:54
But yeah, how are you thinking about solving that- that problem? Yeah, I mean, that let's- It's a newer problem. Um, we haven't had, like, a lot of juniors that we've hired in just the last couple years.
30:03
We have one junior that we've hired in the last 24 months, um, and they came from, like, a very computer science background. Um, they had done the work beforehand.
30:12
Um, so, you know, when I'm thinking about onboarding new people, I think with, with this, I still want them to know the fundamentals. I'm not gonna hire a vibe coder to be, you know, on my team.
30:23
Um, I still want them to have a proven track re- record of being able to, like, think through the problems, write out the code by hand. Doesn't have to be the best code. Mm-hmm.
30:30
You know, because now, now that's not a problem. But to, so that they can still solve the problems manually. Um, so we're in Phoenix at the moment for, for WordCamp US, uh, which is obviously a WordPress...
30:41
It's the, the biggest WordPress, uh, conference in North America. Um, where do you see WordPress as a stack?
30:47
And I guess, like, this philosophical thing of where software, what exists in the, you know, the build, borrow, buy paradigm.
30:54
Um, you know, again, everyone, everyone has got existential, uh, thoughts, uh, uh, you know, from engineers to software to, to writer. Every- everyone's got that.
31:02
So where do you think about, like, you're a, a pretty strong advocate of WordPress. You're very involved in the community.
31:07
Um, where do you see platforms like WordPress, both their strengths and, and, you know, how they remain competitive- Mm-hmm...
31:16
uh, and, and, you know, not, not existential when you can, you know, vibe code a CMS in an afternoon, you know? Yeah. Well, um, I don't think there are a lot of weaknesses.
31:23
I, I know that, you know, in the WordPress community, everyone loves to say the sky is falling. I, I don't think that's true, um, because, yes, you can vibe code a CMS. Now you gotta support it. Yeah.
31:33
Um, and I think that's where we're gonna kinda see a lot of things fall flat over the next couple years, is a lot of people have decided, "Hey, I don't need X, Y, and Z. I can do it myself."
31:43
And you're gonna find out actually you can't do it yourself. There's a lot of things you're not thinking about, about maintaining that product.
31:48
I think WordPress is, like, really well positioned because it has this broad base of users, of developers. It has 20-plus years of examples and, um- What not to do... what not to do. What to do, yeah. Yeah, it's, it...
32:01
There, there are a lot of... You know, you can kinda do whatever you want with WordPress, but there are a lot of opinionated, uh, kinda development paradigms for WordPress. Yeah.
32:09
And so those are really well established and really well documented.
32:11
And so I think that, you know, if you were to start, like, a n- a Next.js project today, completely vibe code that, there's no guardrails around that- Right... really. Um, that's, one, that's freeing, sure.
32:22
But that's also, like, really terrifying for me as, like, a engineering leader. If you're, if you're taking it over as well. If someone else built it and left the organization. Right. Right. Yeah.
32:30
Um, and so I, I think that actually WordPress is really positioned well for that because there's always gonna be this kind of ongoing maintenance. We know that WordPress has done really well at backwards compatibility.
32:40
Mm-hmm. So there's just this, these fundamentals, I think, that you can't vibe code. Yeah. Yeah, and I mean, we've seen that as well, and it's sort of the same problem that's always existed of building your own CMS.
32:50
But when it would take six months, now it takes six minutes. Right.
32:53
You've still got the maintenance problem, the, you, you know, you know, benefiting from, like, an opinionated set of standards to follow to, to build so other people can plug in, or other agents to plug in and, and continue to build on.
33:04
And I think there's something to say about, like, developing in isolation, too, right?
33:06
Like, if you go and develop your own CMS and now you're on the hook for maintaining that, you're also on the hook for all the f- the new functionality you wanna add down the road. Yeah.
33:14
Um, and there might be things that you're just not thinking about, um, or that you don't have kind of the, the skill set to think about. Let's take real-time collaboration as an example. Mm-hmm.
33:25
Um, could someone vibe code that for WordPress? Maybe, but there's, like, so much to that. There's infrastructure. Yeah. There's the arc- underlying architecture of WordPress. 100 edge cases. [laughs] Yeah.
33:34
Yeah, exactly. 100,000 edge cases. Exactly. Uh, and so you're gonna be on the hook for everything, and I think that you still want something that's extremely flexible. Mm. WordPress is.
33:44
You can do whatever you want with it. But there is that backing at the end of the day, like, new functionality, new ideas are coming from all over the place. Yeah.
33:52
And that can influence your, your agentic programming with, with WordPress, you know? Yeah. How do you... Do you lose sleep, uh, at night with security and AI?
34:00
Uh, there's two schools of thought, uh, that, that this is the end of, uh, open software and all the bad guys are out there using it to find all these, these back doors.
34:08
Or where I think I sit more is that i- this is, i- it's happening in the public is much better. It's gonna happen much faster. We're gonna have a bumpy ride for a little while. Yeah.
34:16
There's gonna be a lot of old stuff, but any open source project. Like, they did this with Firefox. You know, they, they ran it through- Right... Git or sort of whatever. Right.
34:22
And they found, like, 200 bugs that had been there for, like, 50 year- well, not 50 years, but, you know, had been there forever. Um, so we're in the, like, shaking the, the cobwebs off phase. Yeah.
34:32
Um, but I mean, that, I, I see it as a net positive that, that you're gonna have so much more. And if your alternative is, like, the vibe coded CMS that no one even knows what it is- Right...
34:42
you know, you're getting the core of your platform and architecture, you know, so heavily scrutinized with so many eyes and so quickly now. It's just gonna be a lot of maintenance releases in the, in the near term.
34:52
[laughs] That's all. Yeah, well, and you know, there's also, like, the broader ecosystem around security and WordPress, you know.
34:58
Um, WP Scan, all, all these other kind of players that are always looking for security vulnerabilities, patching them. But also your hosting providers that are doing the architectural infrastructure work to protect you.
35:09
Um, once again, if you vibe code all this, you're not probably, you probably don't have that. Um, I don't lose a lot of sleep on the security stuff. I actually feel more empowered because of AI for security stuff.
35:19
I feel like I can do broader, deeper reviews of our code with AI for security issues. Um... Yeah, you're not paying $10,000 to do a pen test every two months now or whatever. Right, right.
35:30
And you know- You're getting enterpri- like, incredibly enterprise. Exactly. You can build an agent and be like, "This would've been a six-figure security contract-" Yeah...
35:37
"uh, with some company in the past that we can now do much better ourselves." But also with, like, automations and kind of this idea of, like, a self-healing application. Mm-hmm.
35:44
I, I also worry less about the security b-... implications because as they approach, we can quickly fix them, right? Um, and we always have backups, right? [laughs] Yeah. Always have backups.
35:54
[laughs] Always have backups, that's right. Um, with open source, like you contributed a lot of stuff, and then open sourced a lot of your, your, your internal stuff.
36:02
A discussion I was having last night was around, you know, s- and maybe, maybe this is more on the product and, and agency side. Everyone's gonna have all these same tools. We're all gonna have access to the same models.
36:12
Uh, a lot of knowledge is now gonna be, you know, codified in, in skills and, and all kinds of, you know, marked down file somewhere.
36:18
[laughs] Um, has like the amount of stuff you're creating sort of changed or evolved your thinking of like what you release, what you don't release?
36:25
What's sort of baked into these, these, you know, whether it's a, um, you know, a skill or an agent or a task or a, a plugin where you're sort of baking in more of your IP, um, verse, you know.
36:37
And again, like we've, we've both benefited from like WordPress and, and everything being shared, but a lot more of that stuff's coming together.
36:42
I guess a lot of stuff that's lived in people's heads forever is now being codified. That's the, the promise we have to, to live up to. Um, so yeah. Has, has that contributed to how you think about...
36:51
Uh, if anything, I think it's m- made us more open source. Mm. Um, because it's made it easier to like make those plugins more ready for broader usage. Yep.
37:01
Um, before I think the problem that we had, and, and o- once again, I... Like we're very committed to open source.
37:07
Um, just as like a little tangent, I, I think it's like really important, um, not only for WordPress, but like selfishly for us. Mm-hmm. Um, I like to say that we don't just use WordPress, we help make it.
37:17
Um, there's a reason for that. Uh, we have tabs shipping, the tabs block in 7.1, which I'm really happy about. Amazing. Um, we contributed that, and then other contributors took that to the next level. Um, so it's...
37:31
For, for open source, it's not just about like, oh, are we all kind of creating the same stuff, and are we duplicative, and, um, you know, what are the security implications of that?
37:40
Um, I really think of open source as, hey, I need that. I need this in WordPress either because I don't have it or I don't wanna maintain it anymore, and I think that's still the promise of open source.
37:52
Um, even with agentic coding, like we just said, you still have to maintain all of this stuff. Yeah.
37:57
And a lot of it is autonomous, sure, but there is still like a lot of higher level thinking of, okay, well, WordPress 7.2 is in three months. What do... How do these features and functionality play with that? Yeah.
38:08
What do we need to take away? What's gonna be added? So on and so forth. And without open source, how do you know? Yeah. How do you know what's coming to your platform? Yeah.
38:17
So I still think open source is like extremely important, and I think if anything, we're gonna see maybe even- Become more valuable... become more valuable- Yeah...
38:23
because you have these building blocks that your agents can build with that are open source that- Yeah... once again, you don't have to maintain. It's been battle tested. All, all the things that we know that are great.
38:32
Exactly. So it hasn't been too, uh, um, pessimistic of a conversation, thankfully. But what are you, what are you most excited about? Like it, it, you know, again, I think we're both software engineers by trade.
38:42
Uh, you know, if you started doing this 10, 20 years ago and to now, it's a, a unimaginable difference to where we are now. Yeah. But, um, where, where, where are you excited where this is going?
38:51
What's, what's, you know, in your roadmap of things that sort of get you out of bed in the morning right now? Uh, where I'm excited about where this is going for us is just kind of 100% owning our business operations.
39:02
Like I said, you know, we've, we're replacing Microsoft Word. We're, we're replacing InDesign. If you had said to me four or five years ago that we were gonna be able to do that, I would've laughed.
39:12
There's no way that a six-person team- Well, you need to 10, 10X your engineering team. Yeah... right. There's no way that a six-person team could replace Microsoft Word- Yeah... or Adobe InDesign.
39:20
And so we've created... We, we, we just finished this PDF generation product. It takes blocks. It goes and generates a nice PDF out of, out of your templates and stuff.
39:28
Um, that's just stuff that would not have been possible. That's the stuff that really excites me is what couldn't we do that we can now do- Yeah...
39:35
that we can own and like customize and tailor exactly to our business needs. I think that's where we're gonna see some really interesting things play out over the next couple years.
39:46
Yeah, you do need big companies making kind of these building blocks, but there's a lot of stuff that's out there that does 10% of what your business need. You're paying $10,000. Yeah. Salesforce. Yeah.
39:58
[laughs] Um, you know, things like this where- Yeah... your organization might be paying out the nose for- Right... limited functionality. Yeah. And you're kind of locked in.
40:08
I'm excited about the opportunities to kind of break away from those, those behemoths, those monopolists. It's gonna be a huge re-equalization of like- Yeah... where, where economics of, of SAS and- Mm-hmm...
40:17
yeah, all that sort of stuff. And then at the other end, am I gonna go build a whole Calendly replacement to save $49 a year? And that, that's the struggle now.
40:24
[laughs] That is the struggle is like we can build anything. What should you build? Yeah. It's... You've really got to find that threshold of, yeah, I mean, DocuSign or these kinds of things that are- Right...
40:32
you're so, so locked in on and paying outrageous amounts of money on. Um, yeah, of where you can apply your efforts to, to solve all these problems. Uh, great. Well, thank you for joining me today.
40:41
It's been great chatting, and- Thank you... thank you.
