Support Stack Solo Ep 06: Data Connector, Procedure or Fin Skill? How to Pick (and Prove It)

You’ve built a data connector that pulls a customer’s subscriptions. Do you wrap it in a Procedure so you can test and simulate it, let Fin call it directly, or put it inside a Fin Skill? That’s the question I got on Fin’s community office hours, and in this short solo episode I answer it properly.

I walk through it in my own Intercom workspace. Start with the data connector on its own: set it to call directly, tighten the prompt with the Optimize button, and test it in Fin preview. Roll it out, read the logs, and fix most misreads with clearer field names and a better prompt. Then, if you want to go further, put the connector inside a Fin Skill with prioritised content and guidance, save it to a Release, run evals with mocked data connector responses, and A/B test the Skill against the standalone connector.

The throughline: skipping the Procedure doesn’t mean skipping the testing, and you don’t have to guess whether a Skill is worth the extra complexity, because you can prove it. If you own Fin content, run a help centre, or you’re an Intercom admin trying to lift answer quality, this is a practical look at choosing between data connectors, Procedures and Fin Skills.

🔗 Resources mentioned:

Data connectors quick start guide

Getting started with Fin Skills

Getting started with Releases

How to run and review evals inside a release

Go deeper on Fin Skills with Brian Branca (Support Stack Ep. 21)

Want daily, practical tips on getting the most out of Fin and Intercom? Subscribe to my daily email list.

Episode transcript

Conor Pendergrast (00:43)

Okay. Hey, I’m Conor Pendergrast, and this is just a short episode of Support Stack Solo.

Recently I was on the Fin community office hours — specifically, it’s the data connectors and Procedures one. I got this question from Ali. I’ve lightly edited this question just to make it anonymised, but it’s about saying:

“Okay, I built a data connector that fetches a customer’s subscriptions. We’re advised to use a Procedure because you can test and simulate it. But Fin doesn’t need to follow steps here; it just needs the data to hand when it’s answering a question. Do I let Fin use the data connector directly and lose the testing, the simulations? Or is this what Fin Skills are for?”

So I had a couple of suggestions for Ali based on this question.

First and foremost is: yeah, you absolutely can just use this as a standalone data connector. So when you’re editing the data connector, you can use this option down the bottom, just called “directly”, and that lets Fin use it based on this prompt that you’re showing here. You can use the Optimize button here to improve the quality of the prompt immediately — it just gets Intercom to do the feedback on it.

And then the other thing to do, Ali, as I said on the call, is use the Fin preview button and test out some options there. So that’s the first thing you can do. That’s actually kind of the starting point. That’s where I would suggest you start: just straight away start testing it.

Testing this live, you could roll it out to a specific audience, or to everyone it’s relevant to. But you can then look at the logs for this and see how it was used — see where Fin is using content alongside this data connector, and where it’s not using that content correctly.

If you get to the point where you see a couple of patterns where Fin isn’t appropriately answering the question, first of all, try — just in this data section — giving Fin very specific information on what each of these means. You can do that, for example, by giving it the correct terms. So if your Growth tier, for example, is now called your Enhanced tier, or something along those lines, you can rename it in there.

And then the second thing, as well, is obviously in the prompt: give it more information about what to do with the information, how to interpret the information, how to apply the information. And then also make sure your content works really well, because content is the foundation of all that.

However, if you want to move to the next step of it, you can then use a Skill. So in the Skill you can load in your specific data connector, and that tells Fin to use the data connector alongside this Skill. You can also add in tools, and you can prioritise content. So if you have specific content that you want Fin to reference when it’s talking about customer subscriptions, when it’s talking about features that are available, for example, inside this Skill you can reference that content. And then you can also guide Fin on how to do this.

Once you’ve got that, instead of just setting it straight live, save it to a release. And then you’ve got your release set up here. So the release shows you everything that has changed. So in this case, I’m saying, like, okay, I’ve modified a Procedure and some content, and I’ve also added a new Skill here. You can do all of these changes, then on the right-hand side you can preview all of the changes at once.

But the other thing is, you’re not losing out on the Procedure’s testability. You’ve still got evals that can run. You can set them all running here — you can just click them running like that. Nice. These can run across everything that’s changed in the release; you can run these evals against it.

So you can set up an eval that’s based on a couple of different customer situations — the simulations within the eval. You can also set it up so that you have data connector responses, so you could edit some data connector responses and give it different information for different scenarios. And then you can also set the outcomes for each of the simulations within an eval, and it gives you exactly what you’re looking for. So then you can just continue to iterate on it.

And once you’re happy with the testing with the preview, once you’re happy with the evals, then you can move on to the rollout. So you can do an A/B test, and that means you’ve still got the data connector independently set live. But in this version, in this rollout, maybe you set it so that that data connector is not available independently, and only available through the Skill.

So this A/B test then could say: okay, well, does Fin do better with a standalone data connector? Or does Fin do better with the Skill housing the data connector, where Fin can’t use the data connector independently outside of it? This gives you the ultimate test of: can I improve the Skill to make it so that it works better than the data connector? Or was it working just as well, and it’s not worth the additional complexity?

So that’s how you can do it. You can also judge it based on resolution rate or CX Score. Once this is rolling, it’ll start gathering data, and then it’ll eventually give you a conclusive result, if it’s able to. It does need a certain threshold for that, though.

Hope that helps to answer the question, Ali. And yeah, let me know — you and anyone else reading this — if you have any other questions in the future.

Bye bye.

Next
Next

Support Stack E21: Procedures Are Recipes. Fin Skills Are a Kitchen. with Brian Branca (Fin)