Qubot: Engineering GitHub’s Internal AI Data Analytics Agent

GitHub developed an internal AI tool called Qubot to help employees navigate complex data warehouses using natural language. This Copilot-powered agent enables users to perform self-service analytics by translating plain English questions into technical queries across multiple data engines. The system relies on a robust context layer that organizes documentation and business rules, ensuring the AI provides accurate and relevant insights. By integrating with Slack and VS Code, the tool makes data exploration accessible to both technical and non-technical staff. Since its deployment, the company has observed a significant decrease in routine support requests for the data team, fostering a more autonomous decision-making culture. Ultimately, the project demonstrates how structured metadata and automated evaluation frameworks are essential for building reliable AI-driven engineering tools.

Note:  This podcast was AI-generated, and sometimes AI can make mistakes.  Please double-check any critical information.

Sponsored by Embersilk LLC

Transcript

SPEAKER_00 0:00

Okay, so picture this. You were staring at this uh absolutely massive, horribly complex spreadsheet.

SPEAKER_01 0:08

Oh, I know the feeling.

SPEAKER_00 0:09

Right. And you are literally begging the little cells to just like speak English and hand over that one single metric you need for a meeting that starts in you know exactly four minutes.

SPEAKER_01 0:19

The panic is very real.

SPEAKER_00 0:21

It is so stressful. But uh that universal frustration is exactly what we are jumping into for today's deep dive. Yeah. Because GitHub actually solved this with something they call Quebot.

SPEAKER_01 0:31

Yeah, they built an internal AI data anal analytics agent.

SPEAKER_00 0:35

Exactly. And it is so exciting because AI is finally making this kind of complex data just totally accessible to everyone. It is really empowering.

SPEAKER_01 0:43

It really is.

SPEAKER_00 0:43

Oh, and by the way, if you are feeling inspired to build your own AI automation or you know you need help figuring out where agents could make a massive impact for your business, you should definitely check out EmberSilk at embersilk.com.

SPEAKER_01 0:55

They are fantastic for all your AI integration needs.

SPEAKER_00 0:58

Yeah, for sure. Yeah. But okay, back to the data bottleneck. How bad was it before QA?

SPEAKER_01 1:02

Well, I mean, historically it was a huge productivity drain. If you wanted a specific insight, you couldn't just get it yourself.

SPEAKER_00 1:09

You had to file a ticket, right?

SPEAKER_01 1:10

Right, exactly. You had to file a ticket, wait for an analyst to figure out the right data model, write a custom SQL query, and then validate everything.

SPEAKER_00 1:18

Aaron Powell, which takes forever.

SPEAKER_01 1:20

It does. That ticketing bottleneck became so expensive and slow that GitHub realized they just had to bypass the analysts entirely. So that that is how Kubot was born.

SPEAKER_00 1:31

Okay. So what exactly does it do for their employees?

SPEAKER_01 1:34

Aaron Powell It allows their employees, they actually call them hubbers, to ask plain language questions right in their usual tools.

SPEAKER_00 1:41

Like Slack.

SPEAKER_01 1:42

Yeah, Slack VS Code or the Copilot CLI. And they get accurate answers back in seconds.

SPEAKER_00 1:48

That is wild. It is literally like having a brilliant, I don't know, a tireless data scientist just living inside your Slack channel.

SPEAKER_01 1:54

That is a really great way to put it.

SPEAKER_00 1:56

Just ready to instantly answer, like, hey, what product moved this metric the most? But wait, I have to push back here. With all that vast, varied data, how does the bot not just get totally overwhelmed or like queried the wrong database entirely?

SPEAKER_01 2:10

Ah, so that is the secret sauce. They built this really clever federated context layer.

SPEAKER_00 2:15

Okay, what does that mean?

SPEAKER_01 2:16

So the data is structured by curation level. You have bronze data, which is just the raw telemetry, then silver, which is curated facts.

SPEAKER_00 2:24

Okay.

SPEAKER_01 2:24

And finally gold, which holds the strict business rules.

SPEAKER_00 2:27

Oh, I see. So to use the librarian analogy, bronze is like throwing you into the raw archives.

SPEAKER_01 2:33

Yes. And gold is the librarian handing the AI a meticulously verified executive summary.

SPEAKER_00 2:39

Okay, so that constrains the hallucinations. But even with that, how does the engine handle the actual search without a massive cloud computing bill?

SPEAKER_01 2:47

Well, they use a highly efficient dual query engine. It automatically acts as a traffic router.

SPEAKER_00 2:52

Wait, so it decides on the fly?

SPEAKER_01 2:53

Yeah, it does. If you ask about fast recent events like site traffic over the past hour, it routes your query to Kusto.

SPEAKER_00 3:01

Kusto got it.

SPEAKER_01 3:01

Right, which is built for real-time telemetry. But for complex historical joins across different sources, it automatically switches to Trino.

SPEAKER_00 3:09

That makes so much sense for efficiency.

SPEAKER_01 3:11

Yeah.

SPEAKER_00 3:11

But with different product teams defining these gold metrics, how do they ensure a new update doesn't just break the bot?

SPEAKER_01 3:18

They use a rigorous offline evaluation framework.

SPEAKER_00 3:21

Oh, so they test it behind the scenes.

SPEAKER_01 3:23

Exactly. Every time someone updates that context layer, the system automatically tests the AI against hundreds of known correct prompts.

SPEAKER_00 3:31

Aaron Powell But how does testing it offline make the bot faster for the user in real time?

SPEAKER_01 3:37

Because by benchmarking those queries offline, the system basically learns the shortcuts to the data.

SPEAKER_00 3:42

Oh wow.

SPEAKER_01 3:43

Yeah. By the time you ask a question in Slack, Cobot has already cached the optimized route, making it up to three times faster.

SPEAKER_00 3:50

Three times.

SPEAKER_01 3:51

Yeah.

SPEAKER_00 3:51

That is incredible. So it really is this amazing hub and spoke model.

SPEAKER_01 3:54

It really is. It dramatically reduced the data team support tickets.

SPEAKER_00 3:58

While giving all the employees the confidence to just explore the data autonomously, it shifts data from being a locked vault to just like turning on a tap.

SPEAKER_01 4:07

It proves that our tools are moving toward a highly empowering, frictionless state. We are actively solving the human bottleneck.

SPEAKER_00 4:16

So people could just focus on actual problem solving. It really is a game changer for human curiosity.

SPEAKER_01 4:22

Absolutely.

SPEAKER_00 4:23

It makes you wonder, you know, once AI can map our data perfectly and answer any question instantly, how long until it stops waiting for us to ask and just starts proactively pointing out the brilliant insights we didn't even know to look for.

SPEAKER_01 4:36

That is a very inspiring future to look forward to.

SPEAKER_00 4:38

Truly. Well, if you enjoyed this deep dive, please subscribe to the show. And hey, leave us a five star review if you can. It really does help get the word out. Thanks for tuning in, and here is to an incredible future.