Todd Schiller - Machine Learning

Co-pilots, not chatbots: highlights from a TaskUs Forward webinar

2024-09-02T00:00:00-04:00

I joined Manish Pandya (SVP of Digital at TaskUs) and host Alp Uguray (Masters of Automation) for a TaskUs Forward webinar on operationalizing generative AI in customer experience. A few moments from the conversation worth pulling out — the full transcript is here.

On AI looking like the early days of the internet

Todd: I think in a lot of ways AI looks like the early days of the internet. It's subsidized by an ample wave of investment from the venture capitalists. You have providers like Google, Amazon, and Facebook fighting to create this excess amount of infrastructure and capacity due to competitive pressure. We also see a very low barrier to entry to build prototypes. So ultimately I think AI is the future, but definitely not all the ideas and the companies and the headlines that you see are going to stick.

On human-driven AI

Manish: At TaskUs we believe in the power of human-driven AI, or human-in-the-loop, where the technology is amplifying the capabilities of the human as they deliver — not necessarily replacing them. This has potential to deliver empathetic, personalized, and nuanced responses that only a human can deliver, and not necessarily something that an AI can replicate, although AI can assist.

On chatbots vs. the natural flow of work

Todd: Chatbots are the fastest way to deliver AI to everyone at your company — and if you remember, ChatGPT was actually the fastest-growing consumer product of all time — but there's actually little consistency with outcomes when you just roll out a chatbot. […] What we're really seeing is companies figuring out how to embed AI into the natural flow of work, and as Manish said, it's really about making AI a true co-pilot for teammates.

On a true co-pilot, not just chatbots and search

Manish: What we're seeing is that clients are expecting the generative AI solutions to complement our teams so they can be freed up to perform high-value tasks. Essentially what they're looking for is a true co-pilot for our teams, not just chatbots and search.

On systems thinking and second-order effects

Todd: We've seen a lot of companies deploying, for example, customer-facing chatbots, and then running around with their deflection rate metrics saying, "hey, we did a great job," but then they see drops in their customer loyalty and lifetime customer value. So for me the key is to think holistically about your company and value chain — try to apply what a lot of people call systems thinking — and really try to understand what is unique about AI as a technology and therefore where it can best be applied.

On agentic AI and the next few quarters

Manish: Rather than talking about the next few years, let's talk about the next few months and next few quarters. […] You will also see that not just one-shot question and response with some follow-up queries, but being able to form a chain — which is what is called agentic AI — will emerge, where you provide a task and the generative AI solution is able to string together multiple automation capabilities together.

On building trust between humans and AI

Alp: We're introducing more automation to remove the mundane tasks that people hate to do, while bringing explainable AI — why AI does certain things and what the impact is, the transparency in execution. […] It's going to build a new trust between humans and AI. That trust is going to be really important and has its own requirements, and of course it has to bring business value — measuring the ROI where it needs to be, while we are driving meaningful work of the future.

The full webinar is on YouTube. A lightly edited transcript is on this site.

AI frees humans to be more human: highlights from Success League Radio

2023-09-19T00:00:00-04:00

I joined Kristen Hayer on Innovations in Leadership, the Success League Radio podcast, to talk through where generative AI fits in customer success. A few moments from the conversation worth pulling out — the full transcript is here.

On AI freeing humans to be more human

We think that AI frees humans to be more human. There are three areas. First, with AI you can automate a lot of the low-value activities… The second area is that AI can also be your personal trainer or coach… And then beyond that, there's the pure power to have AI upskill workers. It's a bit akin to that scene in The Matrix where Keanu plugs in, connects to the desk, and then suddenly downloads all the kung-fu knowledge.

On treating ChatGPT like an intern

The way I like to frame it is: let's say you were to hire an intern. What would you have them do? I say "intern" because interns require instruction. They're not experts in customer support. And they're not always perfect — sometimes you're going to have to check their work, or you might have to make some corrections.

On defusing hallucinations

Instead of saying "ChatGPT, go generate a response for me," what you instead do is you give it a list of things to choose from, or you give it some content and you say "summarize this content." When you're giving it those sorts of tasks, it's much better able to identify that it's only supposed to be using that as the source material, versus giving it all human knowledge and saying "what's relevant out of this entire space of human knowledge?"

On capabilities, not maturity

A maturity model is a model that looks at what sort of level of tools or processes you have in place — level one, two, three, or four. A capability model looks at specific capabilities and outcomes. When an agent picks up a ticket, do they have the context of prior interactions? Do we have alerting in place to make sure we're actually able to meet our SLAs about response times, even when someone's on vacation?

On the disclosure bright line

If it's fully automated, you should definitely be disclosing — not even just for ethical reasons, but for expectation-setting and maintaining trust with your customer. And then really, it's the assistant that becomes more of a gray area. We wouldn't disclose if I was using grammar check to write an email. I wouldn't necessarily disclose if I was using Google Translate to help translate a message to you. So that's kind of my bright line: if there is a human involved, I don't think you necessarily need to disclose. But if it's fully automated, it's probably both ethical and also just best practice to disclose there.

The full episode is on Spotify and Buzzsprout. A lightly edited transcript is on this site.

Markov logic networks for fun and profit: NBA playoffs edition

2016-04-19T00:00:00-04:00

Markov Logic Networks (MLNs) are a tool for capturing your beliefs about the world and then calculating the likelihood of outcomes based on those beliefs. Since it's the NBA playoffs, let's use MLNs and our beliefs about the NBA to predict the 2016 NBA championship.

Specific Beliefs (Predicates)

To get us started, I know for a fact that the Golden State Warriors won the 2015 championship. In MLN-speak [1], we write this fact as:

champion(2015, Warriors)

Here, champion(...) is a called a predicate. A predicate is either true or false depending on its arguments (in this case, the year and the team).

With MLNs, we can also state beliefs that aren't certain. For example, I'm 60% sure the Warriors will win the 2016 championship. In MLN-speak, we write this belief by assigning a probability to the predicate:

0.6 champion(2016, Warriors)

General Beliefs (Rules)

In addition to statements about specific things, we can use MLNs to capture rules about how the world works. For example, in the NBA, we know there's exactly one NBA champion each year. We write this in MLN-speak with two statements:

champion(year, team!)
EXIST team champion(year, team).

The team! in the first statement says that, by definition, there is at most one team for which the predicate champion(year, team) holds. The second statement says that, by definition, there exists at least one championship team each year. Together, these two statements encode that there's exactly one championship team per year.

In reality (and fantasy sports), we also have beliefs that are generally true, but are not hard-and-fast rules, e.g.:

A team that wins the championship will also win the next year
A team with an injured starter won't win the championship

In MLN-speak, we write these rules as:

??? champion(year1, team), year2 = year1 + 1 => champion(year2, team)
??? hasInjuredStarter(year, team) => !champion(year, team)

The => symbol indicates that the predicates on the left imply the predicates on the right. That is, if the left side is true, then the right side is also true (but not necessarily vice versa). The ! symbol in the second statement is negating the predicate champion — that team is NOT the champion for year.

But what probabilities should we assign these rules? If we did have probabilities, what would the relative importance of the rules be? Are the Golden State Warriors more likely to win even if Stephen Curry is out with an injured ankle? For rules, instead of assigning a probabilities, we assign them relative weights. Relative weights allow the MLN to handle multiple, possibly conflicting, rules.

Learning Rule Importance (Weights)

If you have good historic evidence (data), a good way to assign the weights is to have the MLN learn the weights automatically from the evidence. The MLN will pick the weights that make the historical outcomes most likely. In our case, it's going to pick weights that made the previous NBA champions the most likely champions according to our rules (upsets be damned).

If the NBA just had 4 teams, the Heat, the Spurs, Warriors, and the Cavaliers, the historic evidence for 2015 would be:

hasInjuredStarter(2015, Cavaliers)
!hasInjuredStarter(2015, Heat)
hasInjuredStarter(2015, Spurs)
!hasInjuredStarter(2015, Warriors)
!champion(2015, Cavaliers)
!champion(2015, Heat)
!champion(2015, Spurs)
champion(2015, Warriors)

Learning the weights based on the 2013-2015 seasons, the MLN finds the following weights:

-3.7 champion(year1, team), year2 = year1 + 1 => champion(year2, team)
4.2 hasInjuredStarter(year, team) => !champion(year, team)

Here the negative weight for the first rule indicates that our intuition was incorrect — based on the evidence, winning the championship makes a team less likely to win the next year! Indeed, there was no back-to-back champion in the 2013-2015 seasons (the Heat won in 2012 and 2013). The size of the weights indicate the two rules are roughly equally important, but in opposite directions.

Predicting the 2016 Championship (Inference)

Once we've assigned weights to the rules in our model, we can have the MLN infer (estimate) the probability of uncertain predicates. In our case, we want to know the probability of each team winning the 2016 championship. This is called our query:

champion(2016, team)

Before we query our model, though, we need to provide our beliefs about the likelihood of injury for each team in the 2016 playoffs. We provide these beliefs by adding the predicates, with their probabilities, to the evidence:

0.9 hasInjuredStarter(2016, Warriors)
0.8 hasInjuredStarter(2016, Heat)
0.4 hasInjuredStarter(2016, Spurs)
0.2 hasInjuredStarter(2016, Cavaliers)

The MLN can then infer the probability of each team winning the championship. The inferred probabilities will be consistent with the rules we captured, our beliefs about the probability of injury, and the historical evidence:

0.3500 champion(2016, Cavaliers)
0.3400 champion(2016, Spurs)
0.3000 champion(2016, Heat)
0.0100 champion(2016, Warriors)

The probabilities sum to 100%, which is a good sign. Our model is bearish on the Warriors, giving them only a 1% chance of winning the championship. The low probability is due to fact they won in 2015 and we assigned them a high likelihood of injury this year.

Next Steps

There's a lot of different directions we could go to improve our model. We could capture our beliefs about anything from match ups to home court advantage to backup players to mascot popularity. In general you'll want to target the areas that will likely have the largest impact. These will be beliefs that either you haven't accounted for yet, or that your current model is most sensitive to (i.e., rules with large weights).

Summary

Markov Logic Networks (MLNs) are a tool for capturing your beliefs and inferring the likelihood of events based on those beliefs. In this post, we used an MLN to capture our beliefs about the NBA playoffs. We had the MLN learn the relative importance of general rules based on historic evidence, and then inferred the probability of each team winning the 2016 championship.

Footnotes

[1] This post uses the Tuffy syntax for Markov Logic Networks.