How to Redesign Work for Real Business Value from Agentic AI

Allan Schweyer: MIT recently came out and said only 5% of AI initiatives result in a real ROI or P&L result, and that's because when you bolt AI onto existing processes, great, people are learning about AI. They're getting comfortable with it. They're building pilots and prototypes. That's fantastic. But they're not creating ROI for the company.

It's only when you redesign work and redesign workflows that you can expect AI to really take off.

Steve Odland: In today's conversation, we're gonna talk about agentic AI. Now, we've talked a lot about many forms of AI as it's developed over the years, but agentic AI is special, and many leaders treat agentic AI and skills-based work as separate agendas, but they're really becoming one agenda where the work needs to be deconstructed into tasks and skills, and redesigning work at scale needs AI.

So same decisions govern both. Joining me today to dive into all of this is Allan Schweyer, Principal Researcher in the Human Capital Center at The Conference Board. Allan, welcome.

Allan Schweyer: Thanks, Steve. Glad to be here.

Steve Odland: And Allan, welcome and congratulations on this brand-new report which you've just issued on agentic AI. Why don't you tell our listeners where they can find the report? And you've got an additional tool accompanying that report.

Allan Schweyer: Yeah. The tool is very interesting. It will be released in a couple of weeks, but the paper is available now, and to find that, you simply go to The Conference Board's homepage. You can go into the Artificial Intelligence Hub if you like or search on agentic AI but it is published and available.

Steve Odland: All right. Terrific. So that's tcb.org, and click on the AI Hub. Great. So let's talk about your report because your report focuses on agentic AI. Now, a lot of organizations, most organizations would say, "Oh yeah, we do agentic AI because we've got some chatbots." Is that the same thing, or how do you define agentic AI?

Allan Schweyer: Yeah, I think it's really an important distinction because some of what organizations are doing with chat is exceptional: building pilots and prototypes, things that might create a daily newsletter that lands on your desk every morning on subjects that you're interested in so that you don't have to look through all of the newspapers and sources that you generally do.

That's important and that's useful, but it's not risky. An AI agent, the ones that we looked at and really studied in this report, are the types of agents that go out and do things on your behalf. So you give them a fair bit of independence and autonomy, not just to create a newsletter, but maybe to go into your HRIS and look up employee records or your financial systems, your compensation, your customer records.

When they start doing things like that, then you need to look at governance. So we really separated in this research the type of agent that really doesn't carry much risk, whether it creates a newsletter or does something like that, and looked at the types of agents that can give you oversized returns but can also bring a fair amount of risk if you don't do it right.

Steve Odland: So technically when people are using chatbots or things that are a little bit simpler, yeah, it is agentic AI, but what you're talking about is the more sophisticated stuff, which is stuff that's happening really more autonomously and almost like a separate employee, you know, somebody doing work on a specific task, right?

Allan Schweyer: Yeah, and I think that's the right analogy. No one's gonna say an AI agent is the same as a worker, but the analogy is apt, and it's a good way of thinking of it. If you hired someone, let's say an entry-level person, today's AI agents are doing a lot of the repetitive and routine work that entry-level workers typically do.

But you wouldn't hire someone and just say, "Go for it." You would give them a job description. You would give them what kinds of decisions they can make and what decisions they need to elevate to a manager. You would give them some boundaries. You can access this database but you don't have access to everybody's compensation, things like that.

In exactly the same way, you would give an agent very careful and detailed instructions and very careful and detailed boundaries so it doesn't go out and do things that you never intended it to do.

Steve Odland: Yeah, it reminds me of when people were installing ERP systems, typically SAP or Oracle, and they thought we'll just drop a new ERP system in place and away we go.

It's just software. What's the big deal? And in fact it's a huge deal because they learned that they needed to revise a lot of their work processes to fit the model. Now, agentic AI-- and you talk about this in your paper-- it's very similar to that in the sense that you're implementing a new tool, but you can't just follow all the same work processes. You talk about how companies need to revise or think through their work processes.

Allan Schweyer: Yeah, and I think that is the single most important takeaway in this research. Personally, I think there has not been an invention nearly as transformative as AI since electrification. So I would take it all the way back to the 1890s when, I think it was Nikola Tesla, invented the first practical electric engine, and that electric engine was miraculous.

So factory owners thought, "Hey, let's bolt this onto our steam-era infrastructure," similar to what you were talking about processes. And they got a little bit of a gain, but not very much. It wasn't until the 1920s that factories were completely redesigned around electrification and the massive gains came.

And I would say the same thing is happening here and what we talk about in the report is if you bolt AI, whether that be a ChatGPT license or Copilot, onto existing workflows and work processes, you'll get a little bit of gain. You'll get things done a little faster and most employees would say their work is more efficient.

But MIT recently came out and said only 5% of AI initiatives result in a real ROI or P&L result, and that's because when you bolt AI onto existing processes, great, people are learning about AI. They're getting comfortable with it. They're building pilots and prototypes. That's fantastic. But they're not creating ROI for the company.

It's only when you redesign work and redesign workflows that you can expect AI to really take off and start to produce those real returns that the CFO will sign off on.

Steve Odland: Yeah, and that sounds scary when you think about, oh, I have to redesign everything I'm doing. Yeah, but, we've done this over the past 30 to 50 years routinely.

There was re-engineering, it goes on and on. There were different eras of it and flavors of this as you go. But essentially in order to get the kind of savings that you want, get the ROI for the AI investment, you have to have productivity gains, either in terms of cost savings or in terms of revenue gains.

And in order to do that, you really have to modify the way you're working around the tool and with the tool to have maximum effect. That's what I hear you saying. Did I get it right?

Allan Schweyer: Yeah, absolutely. We have a new technology. As I say, it's transformative. It's not like adding Excel or Word. It's a transformative technology. And so to get the most of it, you need to design around it. Not, again, not to say that you let the technology lead you, but that you maximize that technology by making sure your work processes fit it well. And I think as we talk about how you do that, I think it'll become clear.

Steve Odland: Yeah. Now, just to simplify things-- and I know the AI experts will guffaw at my oversimplification. But at the end of the day, AI is software, right? It's ones and zeros that you overlay. Yeah, you have to buy a bunch of hardware to do it or use it, da- da-da, but it's software.

It's software interaction with your constituents, your customers, your employees, your owners, the environment, the community. You can't just let her rip, as you say. It has to be done, at the end of the day, by and for people. I think that's what you're saying. You really have to map through all constituents.

Allan Schweyer: Yeah. So for example, let's suppose you wanted to improve something that you thought doing so would create a real advantage for you. One of the examples we use, and it comes from a real case study in our research, is a company was losing people it didn't wanna lose.

So we call that unwanted turnover, people whose skills are gonna be needed to execute the business objectives and plans. They wanted to stop or at least slow that down or reduce it. So they figured if they could do that, that would have an outsized impact on the business. Now, there's a lot of things you can do to stem or even stop unwanted turnover.

You could just pay people twice as much, but they wanted to keep it within reason. So what they did was they looked at a series of different workflows, and then they looked at the workflows that they thought today's AI agents could have a significant impact on, meaning they could do a lot of that workflow.

So two things: A, outsized business impact. Two, AI agents are capable of it. And they settled on the first thing that they would try was the exit interview. So only then did they invest the time in breaking down, you mentioned this at the beginning, breaking down that workflow into tasks and sub-tasks and looking at the skills that go into it.

And once they did that, then they had the information they needed to say, "Here's the agents we need, because we're gonna give this amount of work to the agents. We're gonna give this portion of the work to people maybe working closely with AI, and the rest is gonna be human or worker only." And then when they do that, you can see that we're not saying you need to boil the ocean and break down and reinvent all your work all at once, redesign everything all at once, but we're saying do it where it makes the most business sense first.

Do it where you at least suspect that today's AI agents can have a really big impact, and then divide the work between AI and workers. Steve, in that sense too, you're working with the workforce. They're helping you to break it down, the people who do the work, but you're also building that trust.

You should be building that trust. Exercising those things that we might call change management, right? Where we're gonna relieve you of this routine and repetitive work. Agents are gonna do it so that we can focus you on the things that we know bring a little more value.

Steve Odland: You're an expert in all things human capital. You've been researching and writing about this for years. You've talked a lot about the need for collaboration between people and departments and groups, and how you do that to drive innovation and so forth. What you're really talking about here is collaboration between people and AI agents.

Allan Schweyer: Just let's stick with the exit interview example. So now you have an agent that is going into your HRIS to find out who's leaving in the next month, and then sorting out, if possible, if your HRIS has this data, who's leaving voluntarily and who isn't, and then potentially it is accessing those employees' emails to invite them to an exit interview, their calendars to schedule it.

It's going back into their history to build a very specific script for that exit interview. It could even conduct the exit interview if you want it to using natural language processing. But it's doing a lot of things that, you know, tap into systems of record that if it went wrong, then someone's compensation data could get out there or someone's private information otherwise could get out there.

So when you say AI and humans working together, I think even when you're saying, "I have an agent that can do these six things, and I'm gonna let it act autonomously and independently," you still have to realize that the accountability rests with a person. And that person is responsible for making sure that the AI agent is doing what we expected it and set it up to do, to keep monitoring it, to review its outputs and decisions, and to basically govern it.

I don't see a time, at least in the next several years, where it won't always be humans working with AI agents, even if those agents are more or less autonomous.

Steve Odland: Yeah, I guess you could think about it as all employees need or require some level of leadership or supervision. You're saying the same thing here too. Call it an automated employee and it needs supervision. You can't just simply let her rip without any human interaction. I think that's your point in the paper. And so there's got to be quality control. You've got to make sure that what it's doing is consistent with the values of the company, what your business intent is, the objectives, you know all of that, right? So you don't just turn it on and walk away.

Allan Schweyer: Exactly. You brought up values and culture. Even if an agent is capable of doing something, it doesn't mean that it should do something, and that is where it's different for every organization. One organization might say, "Absolutely, we want our AI agents to conduct that exit interview."

Another might say, "No way. We'll always have a human conducting an actual interview with a person." But that could extend to any decision. So yeah, I think it's different for every organization, but the constant is that AI agents can never be the ultimate source of accountability.

Steve Odland: Yeah, and so whether you use the metaphor that they require supervision or some other metaphor, it's still the same thing because you can't have software or an agent that is automated be accountable for business results or any kind of outcomes. It's that accountability that you're seeking, right?

Allan Schweyer: It is, absolutely, and it's also the risk that even if you've constructed an agent well and that and you've tested it in a sandbox before you've put it into production, we've seen to some degree that agents can do things beyond what you hope.

So you've got to keep an eye on them. You don't want them, I hate using the term going rogue, but that's what we've seen in some of these cases that have made the headlines. So yeah it's supervision, it's management, it's oversight. Of course, you're not gonna spend all your time doing that or you may as well not have put the agent in.

You've got to save some time or else there's no point in using the agent. So it's really more of a analogy, like you said, of I'm supervising and I'm managing it, but I'm not looking over its shoulder 24/7.

Steve Odland: So when you say governance, is that what you mean? Governance is a big topic. That's what boards do. It's what you talk about, corporate governance, all of that. So the term is a big formal term, but when we talk about AI governance in the paper, are you really talking about the supervision and quality control?

Allan Schweyer: Yeah, that's a big part of it. The other part of it might be cost and I just alluded to that. We, again, coming back to the headlines, we keep seeing organizations that wanted to get people to use AI more, so they put up leaderboards, some even gave incentives and then they blew through their token budget.

Tokens are the currency of use. So the more you use an AI model, the more tokens, roughly you spend. And they blew through their budgets in a month or two, and then they shut it down. So yes, to your questions about governance, but cost is also part of governance, and so you need to control the token spend.

But even more, you need to quantify the spend of the rest of the governance. So if you are spending two hours a week managing an agent, that counts towards the cost. If you've got a process where someone else looks at the outputs and reviews them and checks them off as good and they spend an hour or two, that's part of the cost. Everything you mentioned plus costs goes into governance.

Steve Odland: Yeah, and it is an interesting thing too that you talk about accountability, supervision, and so forth. With other software you buy a license, right? And you use it.

You buy a Microsoft Office license, and you have unlimited use of it. This is a different model and it's probably pretty smart on the part of the AI folks. But a token model, think of it as a toll booth. Every time you go through it, cha-ching, and you're racking up.

So you really also need to train people on how to use it efficiently and how to put these agents to work efficiently as part of this whole governance process, right?

Allan Schweyer: You really do. And most people, if you give them access to the latest, greatest frontier model of Claude or ChatGPT or whatever it is, will just use it full time, all the time because it's the smartest, give or take, right?

But it's also the most expensive, generally speaking. Giving people a little bit of guidance around using a non-frontier model, something that was maybe released six months ago as opposed to yesterday, because those things that were released six months ago are perfectly capable of doing about 95% of what you want to do anyway, and then maybe reserving the expensive latest and greatest model for really intensive work that you may want to do.

Steve Odland: These industries evolve, and it's probably going to evolve to a non-token model at some point for some base use. You see that with stock trades. It used to be, what, 3% or something per stock trade, and then dropped down, and now it's essentially free. It's built into the base price. This model will evolve over time, but this is just where we are in the whole evolution of the digital world, right?

Allan Schweyer: We are, yeah. And to be frank, look, you can download an open-weights model to your own server. That open-weights model, they tend to be Chinese models now and we have some security risk depending on the type of company that you operate. But there are also native US open-weights model. They're not as good as the latest and greatest, again, but they will do 85% of what the typical employee needs, and they're absolutely free in terms of tokens.

You might run into some server costs and costs of operating that server behind your firewall. But yeah, essentially you could do that now. I know TCB has done a little bit of that too. There are different ways that it is evolving.

Steve Odland: But you got to govern it 'cause your IP, if you're in a model run by another country or whatever, you could lose control of your IP. That's part of the governance here. Another part of the governance you've talked about in the paper is connecting to sensitive systems, whether they be customer systems with credit card data, human capital systems with sensitive data. How do you put the controls into place so that these agents don't misuse or get into the wrong things?

Allan Schweyer: Yeah. We had someone in our focus groups talk about this a little bit. They put an agent into production and they had a business case for doing so, and the business case was approved and they let it out there.

But what they forgot to put into the business case was some of the things we're talking about governance. And again, those can be the most time and end cost intensive because it's not just HR obviously, it's finance, it's IT of course, it could be legal in some cases.

Bring these people together to talk about what you wanna do. So if you wanna do something that is gonna get into customer data or compensation data or other sensitive data, then having the right people in the room is important. It's also costly. But then you're gonna look at the risks and you're gonna say what could go wrong if the agent mishandles information about our customers or about compensation?

And then assess the amount of governance and controls you need accordingly. For some, you might say, I need minimal controls, major autonomy for the agent. For others, you're gonna say maximum control and far less autonomy for that agent. But it just depends. It depends on the organization and its values and culture, as we said, but it also depends on the severity of the risk for what you wanna do.

Steve Odland: Yeah. Now I wanna loop back to change management because you've written about this over the years. You're an expert on change management, and we talk about this whenever you want to implement a new process or a new product or service, and it touches a lot of different places within a company, and you have to do change management processes to teach people and work.

This is no different though, and when I talk to CEOs about this, and you obviously have done the research and you've got your paper, this is where we find that people are falling down. They're putting this out there, they're rolling it out. IT says, "Okay, it's implemented. Move on." That's just the beginning because you have to revise your work processes. We've talked about that. But the whole change management process in order to get it deeply in the culture and to make sure that everybody knows what they're doing is really important. And there are tools out there like RACI and others but this is really important.

Allan Schweyer: It's very. I can't remember-- and I'm 62 years old. I've been in the field for well over 30 years-- and I can't remember a technology that stoked quite as much anxiety as this one, where something like 65% to 70% of Americans are what they call kind of anti-AI because they fear for their jobs essentially.

There's some other existential things on the margins. But people are worried about it, so you have a workforce that is afraid that as you roll out AI, and especially tools like agentic AI, that their jobs could be at risk. And we've already seen that it's more or less true, at least at the entry level.

There's been some significant impact on jobs. So the first change management, I'm gonna say imperative, is trust, and that is being transparent about why you're thinking about AI agents as you begin the process, what you expect them to do, what's in it for the worker, because it can relieve a lot of the drudgery and why it's important for the organization maybe from a competitive standpoint to be going down this path.

So building trust is probably the first pillar in that change management process. And then you got to bring in your workers themselves, and that's important because when you do redesign work and break it down, they're the ones who have been doing it. You not only need your HR experts and maybe your finance and other experts, but you need your workers who have been doing that.

So but that helps. And you bring them in and they're getting involved in doing that. You also give them access. I think lean towards the side of giving everyone in your organization access to good AI, not something that came out a year ago but good AI, in a very safe place, what we call a sandbox, so that they can practice with it.

They can learn how to build with it. They can even build prototypes and pilots. And those prototypes and pilots might eventually go through a process in the organization where they become put into production to help the company. So you're involving your workforce, and then you're also keeping an eye on two other things that came out of the research.

One is when you do relieve people of routine, repetitive, drudgery-type work, you're also relieving them of work that allowed them to shut down the cognitive part of their work a little bit, right? They're just doing kind of things they could do in their sleep, and you're asking them to take on, with those hours saved, you're asking them to take on higher-end work.

That higher-end work is usually high-cognitive work. They're working on the exceptional difficult cases in some ways or other things, and they may burn out. We already have a wellness factor that most CHROs and CEOs are concerned about. We could make that even worse if we simply replace repetitive work with higher-cognitive work.

So that's another thing. That's a second thing, I think, build the trust, then work with your teams to make sure you're not overburdening people with that higher-value work.

The third thing is do look at the entry level. We do have evidence here. It comes from The Conference Board's economics team. It comes from other economists and labor market experts who say there is a severe drop or a significant drop in opportunities for 22 to 25-year-olds, people coming out of college. Not so much for the trades, that's a whole different conversation, but for people who are going to go into offices and what might be called white collar work.

So look at that. If you don't hire at the entry level, three to five years from now you might not have anyone to promote, you might not have supervisors and managers to run your organization. So that's the third thing I think in this whole change management. There's probably more that you need to take into consideration.

Steve Odland: Yeah, really good points. And, you know, we have to be clear that in the aggregate, more jobs are being created by the introduction of AI than are being lost. But it's this churn and you do have to take care of these people who are being affected at at the entry level, as you said, but also, the low skill set levels.

Because their jobs are really being upgraded into something more. This is a great paper. People can find this at tcb.org in the AI Hub, as we've said before. Allan, final thoughts?

Allan Schweyer: Yeah, just we will also make that simulator available, and what I think is interesting about that is it takes that exit interview case study that came from the research, and it lets you get hands-on with it. It lets you make decisions about how you will choose and govern your agent to do various elements of workforce planning, and it also lets you calculate as you're making your choices the potential return on investment in doing so.

Steve Odland: All right. We're gonna leave it there. Allan Schweyer, Principal Researcher at The Conference Board's Human Capital Center, thanks for joining us today.

Allan Schweyer: Thank you.