Support Stack E17: Did Fin Just Leak Our Internal Feature Flag? with Dawn Perrott (Fin)
R&D reached out over Slack with a conversation that looked bad: a customer had used one of Fin’s internal feature flag names, and nobody could immediately say where they’d got it. That’s a horrible question to answer by hand, because the information could have come from a help article, a snippet, an internal doc, a data connector, a procedure step, a guidance rule — or from the customer themselves.
Dawn Perrott, AI Knowledge Manager at Fin, cleared it up in under ten minutes using Operator. In this episode she walks through the actual prompts: first asking whether the term appears anywhere in the content at all, public or internal; then ruling out the other four places it could have originated; then sharing the conversation link and asking Operator to pinpoint the language; then widening the search to every conversation from that company. Operator turned out to be a thought partner as much as a search box — it suggested checking whether a CSM had shared the name on a call, which is exactly what had happened. Dawn also explains why her instinct is to fix the content or add a terminology rule to Operator’s memory, rather than reaching for a guidance rule every time something goes wrong.
We also get into simulations: how you can hand Operator a conversation that went badly and ask it to build a simulation from it, so the same broken procedure fails loudly next time instead of quietly. If you own Fin content, run a help centre, or you’re an Intercom admin trying to lift answer quality, this is a practical look at using Operator to investigate what Fin actually did — and at making your next mistake easier to catch.
🔗 Resources mentioned:
Run Simulations for Fin Procedures
AI knowledge management workflow — from setup to content readiness
NPI: How to ensure your agent is ready every time you ship
Support Stack E15: How Fin’s AI Knowledge Manager Uses Operator Memory with Dawn Perrott
Want daily, practical tips on getting the most out of Fin and Intercom? Subscribe to my daily email list.
Episode transcript
Conor Pendergrast (00:00)
Hello, I’m Conor Pendergrast and welcome to episode seventeen of Support Stack.
Conor Pendergrast (00:11)
So, Dawn Perrott, we’re back with a third episode. I am very privileged to have you. You are the AI Knowledge Manager at Fin. Formerly Intercom, now colloquially known as Fintercom. How are you doing today?
Dawn (00:23)
I’m very good, Conor. How you doing?
Conor Pendergrast (00:25)
I’m grand. Just firing all the banter at you to start off. But so we’re talking — this is the third in a series of episodes about Operator, and I’m saying it’s the third so that future Conor doesn’t change the order and mess things up. But we get to talk today about, like, using — so we talked about the knowledge management side in terms of updating content, and in terms of the content readiness list and workflow and checklist, which has been really helpful. Today we get to look at Operator in the context of when something goes wrong in a conversation. So tell us, set the scene for us, Dawn. What happened?
Dawn (01:05)
So, yes, I want to tell a story about how Operator was used as a detective really, to find conversations where — I suppose I actually should backtrack a bit, because the start of it was R&D reached out to us over Slack. They were sharing a conversation with us where a customer mentioned an internal feature flag, and they were worried that Fin AI Agent gave internal information publicly.
So what I was able to do in a matter of minutes was confirm that we don’t actually have that feature flag in any public content. And I was able to check all of the conversations for that company, so different users within that company. I was able to see whether they had actually referenced this internal feature flag before. And what we discovered was that it was shared by their CSM. So it wasn’t public-facing content. It was content that was shared over a call that the customer remembered and later shared in a Fin conversation.
Conor Pendergrast (02:23)
Yeah. So the general — maybe the general problem, or the general job that people are trying to solve, or that you would try to solve for, is: hey, there’s this information that came up with this customer. Where on earth did this come from? And that’s a really hard, hard, hard question to answer outside of having an LLM available to look through everything that you have access to and all of the customer conversations. So you used Operator for this.
Dawn (02:54)
Yeah, I used Operator. I mean, if I had to manually read through — I think it was searching through over 20 conversations. If I had to read through 20 conversations, it would be extremely manual. It would have taken my entire day. Whereas within five, 10 minutes, I actually was able to pinpoint where the information came from. So it’s very scalable for these kind of use cases.
If something is being sh— if the customer is talking about something, you’re able to pinpoint where it came from. If there is some issue with Fin’s answering, you can say Fin answer— you can share a conversation link within Operator and say, where did this language come from? Can you pinpoint this within the knowledge base? And you can identify within a matter of minutes, this is where the issue is.
Conor Pendergrast (03:48)
That’s perfect. So we’re looking at the Operator thread here. Okay. And so this is not the real Operator thread that you worked through, because that’s in your real Fin workspace. Whereas this is just your demo workspace. So, first thing to know. But this is — is this how you started off? You just said like, hey — and this can be anything, any feature flag or any piece of information — is this referenced anywhere in our public knowledge base?
Dawn (04:14)
Yes. So to simulate this here today, I actually created a few different conversations from the same fake company, essentially. And what I was trying to do was share an example of the workflow I did in my normal working workspace.
So I initially started off asking, do we have this feature flag? So I just gave it this name. Referenced anywhere in the public knowledge base, articles, snippets, internal content. If Fin used this term, it would have had to come from somewhere in our content.
So then Operator has looked for the feature flag name and confirmed it doesn’t appear anywhere in the knowledge base. It searched all of the content, public and internal. And then it — if Fin used this term in conversation, it didn’t come from your content. A few other places it could have originated is a data connector, procedure step, the customer’s own message, a guidance rule. So I asked it to check these as well, and it confirmed that the feature flag name wasn’t in any of those.
I thought it was really interesting that Operator was being a thought partner here and was like, okay, we checked the knowledge base, but have you thought about these sections?
Conor Pendergrast (05:32)
Yeah, because there are so many, so many places you can write these things down and add these things in. Just as someone who — I spend a lot of my time working in procedures, and you can end up, like in some cases, codifying product knowledge somewhat in procedures to make sure that the procedure goes well. And that can end up out of date sometimes, and so that’s another valuable use for Operator, is to go through and say like —
Dawn (06:00)
Yeah, okay, the content is important to update, but also data connectors, descriptions in there, guidance — absolutely, guidance is critical — and procedures as well. So I think this is a really good example of that too. It’s definitely made it a lot more scalable to be able to maintain all of those facets of the product via Operator. You’re no longer having to have it all within your head. You kind of can rely on Operator to take a bit of the toll, so it’s not as much cognitive load.
So then I prompted Operator again and I asked it — what I actually did in this case was I shared the conversation link where it was mentioned. So this is just a fabri— this is just a generated response I tested with.
And basically what Operator is doing here, it’s looking at the conversation. So it’s calling out initially — but this is the same as what happened internally — so, Fin did not reveal Smart Deadline Beta. The customer Jamie introduced the term themselves in their very first message. Fin was just mirroring what the customer said.
So then it’s giving an overview of what Fin did wrong. We don’t really have to pay attention to that right now. And then it’s giving a recommendation at the end. Adding a guidance rule could prevent this in future if you don’t want Fin using feature flag terminology. I’ve actually added this as a terminology rule to the Operator memory. So it doesn’t even get into the knowledge base.
I’d be hesitant to add too much. I would be hesitant to add guidance here. My inclination would be to fix the content or add it to the Operator’s memory, rather than creating guidance for something specific to this. Guidance works best when you don’t have hundreds of different guidance scenarios — it becomes a bit of a whack-a-mole situation. It’s much better if you have very specific guidance for very specific customers, I suppose.
Conor Pendergrast (08:02)
Yeah. But if there was, I guess, higher risk involved in it, and if you were — if you, dear viewer, for example, are using this to investigate something that is a bit higher risk than just leaking a feature flag name, then guidance could be a way of increasing the— or decreasing the risk of that as well, which might be helpful for people to think about.
Dawn (08:27)
Yeah, that’s true. So—
Conor Pendergrast (08:32)
Because just because it came from outside of Fin doesn’t mean that Fin can’t learn better from it as well.
Dawn (08:40)
For sure. The second recommendation was to investigate how Jamie obtained the feature flag name. So this is actually how it — it was almost rubber ducking with Operator, that when this was reported to us by R&D, it got me thinking like, was it a CSM or someone in sales that actually mentioned this to the customer? And that’s where they got the feature flag name.
So Operator acts as a thought partner as well when you’re going through these detective cases. And this is the example where I was asking Operator, was this mentioned in any other conversation? And it was pulling up the other conversations where the feature flag was named. And specifically to that company as well.
I was asking it — so this is the dummy company name I gave to these test conversations. And I asked it, was anyone from this company talking about this also? And it was able to highlight those conversations. So this is just an example of how it looks for this scenario. And it really did speed up the process of figuring out, okay, this feature flag wasn’t named anywhere in the knowledge base. It was shared by a CSM.
Conor Pendergrast (09:57)
Yeah. That’s super. Yeah, I think this is a really good example of using Operator as well to do things that would just be very time consuming for us as mere humans to do.
If anyone’s interested, another example that I would have of using Operator is something that I’ve talked about on my daily email list as well recently, which is accounting for my notorious laziness. Which is — so I spend a lot of time working in procedures. And there’s a lot of value there, but you also have to make sure that they’re working correctly.
And one way to make sure that they’re working correctly — just one of the ways, and you can’t rely on it exclusively — is what are called simulations in Fin, which is you set up a particular scenario and then you can run that through the simulation and see what the outcome — excuse me, run the simulation through the procedure, see what the outcome is, and then you can also set criteria for whether it’s a pass or fail. And so it’s really helpful. I’m just a little bit too lazy to, sometimes.
What I will find in a similar sort of situation is if I end up looking at a conversation that didn’t go through a procedure correctly — so maybe it didn’t start the procedure, or maybe the procedure didn’t work correctly. The one I was looking at today for a client, for example, I hadn’t accounted for the conditions correctly in the procedure. I broke it last Friday, and because I didn’t have a simulation set up, I didn’t notice that I broke it until a customer conversation went the wrong way. And so one of the client colleagues flagged it with me, which was really helpful.
And so what I’ve done is fixed the procedure, and I used — I’m just going to share my screen. I’m not exactly showing— no, I’ve pressed the wrong button. So this is from a test environment rather than the actual client’s environment, but I said, use this conversation to create a simulation in the procedure. And then it’s got the test procedure name with the conversation link.
And I also said, use the exact responses from the data connector in that conversation. This is really important to include as well, because otherwise the simulation will just use the default data connector responses, which is most of the time not correct — for my procedures, which have lovely conditionals and sub-procedures in them.
And so this is just an example of another way to use Operator, in this case to make it vastly more efficient. You can also create other procedures that way as well and run them this way. But it’s just really helpful as a way of accounting for my laziness, as I said. And then it also makes it less likely that I will make that mistake in the future — or when I do make mistakes in the future with that procedure, I rerun all the simulations and then it says, hey, you made a mistake here, because this simulation is suddenly failing. Because the outcomes that are down the bottom here, the success criteria, one of them will fail because of the fact that it doesn’t run correctly.
So I wanted to throw in — it’s not about debugging. Well, I guess it is actually about debugging a Fin conversation as well with Operator. But this one is more to do with procedures than content. But I think it’s another really good example of using Operator. You know, it gives you a little superpower that once again I will say accounts very well for my laziness.
Dawn (13:15)
Operator has multiple superpowers. It’s applicable to all roles. So I use it for content. The conversation designers use it for procedures as well, probably similar to how your use cases are. Managers are using it as a teammate coaching tool. It has access to monitors, it has access to reporting data.
It’s very helpful if something is happening — if we’re noticing a dip in Fin’s resolution rate, we can prompt Operator, what happened this day that the metrics went down. So it really is applicable to a lot of different use cases.
Conor Pendergrast (13:55)
Absolutely. I could not agree more. Well, lovely. Is there anything else you want to share about Operator?
Dawn (14:00)
Just how it has changed my day-to-day work. I’m able to work — I think it’s like having five extra people on the team, to be honest. It’s massively helpful.
Conor Pendergrast (14:11)
That’s fantastic. Yes, I can imagine that Fin’s knowledge base is pretty giant. I have no idea how you handle it.
Dawn (14:19)
It’s very giant, and we’re shipping — we’re shipping most, like, everyday something new. So to keep on top of that, I don’t think I would have been able to do it without Operator.
Dawn (14:30)
There will also be a blog post coming from the new product introduction manager at some point. So that’s going over our process when it comes to releasing features. Maybe that will be helpful for people as well.
Conor Pendergrast (14:43)
Super, yes. And I imagine that you might be posting about that and sharing that. If people want to find that blog post in the future, Dawn, where should they find it?
Dawn (14:54)
So if anyone wants to follow me on LinkedIn, they can find tips and tricks there.
Conor Pendergrast (14:59)
Goody, and that’s Dawn Perrott on LinkedIn. And scroll down, click the link in there and follow Dawn, and she will share that future blog post. Although by the time you’re watching this, dear guest, maybe through the magic of time travel and editing, that’s actually the present.
Perfect, Dawn. This has been really, really useful. I really appreciate you coming on, sharing your experience using Operator at Fin. There’s definitely something very meta about that. And hopefully it’s going to give a lot of good ideas for how to make the most out of Fin and the most out of Operator, and get great customer support experiences as a result of it.
Find Dawn on LinkedIn. You can also find me on LinkedIn if you want to. But you can more importantly find future episodes of Support Stack right here — just gently press that subscribe button, and then gently go down and feed my ravenous ego by giving me a nice little like. You can also find my daily email list at customersuccess.cx/daily.
Other than that, have a lovely day. Thank you, Dawn, and goodbye.
Conor Pendergrast (16:01)
Like, subscribe.
Dawn (16:03)
I can subscribe.