Bypassing Big Tech: How social intelligence professionals are building their own tools

The social intelligence tech (SITech) space has exploded in recent years as many tech companies are trying to capitalise on the growth of social data. We’ve seen this clearly in our own research for our annual SITech Landscape report: in two years, the number of tech providers in this space increased by more than 20% to over 600. That’s even once we factor in mergers and closures of existing companies.

In many ways, this is a great thing for social intelligence professionals. It gives them more choice than ever to find the right tool for the use cases they're working on. Particularly as many of the new technologies that enter the market are becoming more specialised in niche areas of social data analysis. 

But, alongside this rise in commercial technology is a growing number of SI professionals developing their own tools for social data analysis. As far back as 2023, we asked in our annual State of Social Listening survey how many people were planning to invest in developing their own internal social data technologies. Of the 200+ respondents, 20% were, with 30% of agencies planning to invest. This makes sense as a way for agencies to stand out from their competition by offering bespoke services to clients. But 18% of brands were also planning to invest. And companies, such as Microsoft, have already been developing their own analytical models outside of their commercial social listening platforms. With the rise of generative AI and easy access to commercial LLMs, it’s becoming easier to build bespoke analytics tools and AI agents.

So could this be an industry trend to watch? And should SITech vendors be concerned? 

We spoke to three SI professionals who have developed their own tools or AI agents, to understand what the challenges and benefits are.

‍

Why are SI professionals building their own tools?

Despite the diversity of commercial SITech avilable, there are several reasons why companies are interested in developing bespoke tools. 

Firstly, because of the type of data that can be accessed (or rather, not accessed) through commercial social media platforms. As Alessia Clusini, Co-founder of Trybes Agency explained, “We are called the ‘tailors of research’ and clients typically come to us with a ‘mission impossible’. As a result, we develop ad-hoc research strategies that sometimes imply gathering particular datasets and types of insights.” If a traditional social listening tool can’t provide the data needed, SI professionals don’t have a choice but to build it themselves. 

Michael Williams, Director of Market and Social Intelligence at Jellyfish had a similar reason to develop their GenAI agent. “Most commercial social listening platforms don’t allow for deep audience-level interrogation, largely due to the absence of usable audience data. We aimed to plug that gap and provide both internal teams and clients with something more insightful and actionable.” 

The functionality limitations of commercial tools is another reason why people explore building their own. As Roi Perez, Senior Data and Digital Strategist at Unlimited Group explains, “over the past 18 months, we’ve seen a clear shift in how clients want to leverage social data. The demand is moving away from traditional, numbers-led metrics like Share of Voice, and towards richer, more human insights - narratives, communities, and the motivations behind them…This shift has fundamentally changed how planners approach their work. We’re no longer planning by channel. Instead, we ask: Where are the tribes? What do they care about? And how can brands show up in ways that genuinely meet their needs?” 

They’ve built a GenAI-powered agent that ingests data from social platforms and news sources, and identifies and clusters people into tribes. It can then surface the key narratives within each tribe, helping them to understand not just what’s being said, but who’s saying it and why. This is an example of where, to get accurate analysis for the specific use case you’re working on, it is often better to build a bespoke tool.

Another driving factor is cost vs value. Commercial social listening tools are often expensive, particularly for smaller brands or agencies who only need them for specific, one-off projects. For small projects that don’t require large-scale data access, it’s often easier and cheaper to build your own tool. As Alessia explains, “Most [commercial] social listening tools offer insights that we can surface by querying the social media directly, while in the case of specific needs (e.g. cultural influencers, deep understanding of tribes and subcultures), we need to use an ad-hoc tech stack and different research methodologies (also classic qual and quant).” In these cases, it doesn’t make financial sense to buy a third-party tool.

‍

Is it better to build your own SI tool?

Developing your own tool might seem like the perfect solution for projects where niche data sources or particular research methodologies are needed. Especially as the barrier to entry has been lowered thanks to generative AI - we can all be software developers now and vibe code ourselves a tool. 

In all seriousness, it requires a certain level of knowledge. Both technical, and of the problem you’re trying to solve. For Alessia, “we always start from the business problem and translate it into an actionable outcome. For example, years ago, we built the first tool in the world to analyse Facebook communities. The main challenges were technical and strategic: from data gathering and visualisation to weighing the influence of the members and the cultural impact of the content shared.”

And it can take a lot of iteration to get to a point where your tool can provide useful and reliable insights. As Roi explains, “We started small, using an AI coding platform to build a proof-of-concept. That allowed us to stay nimble, iterate quickly, and validate whether the tool was delivering what the business needed. Once we saw the impact, we scaled it into an enterprise-grade suite of features.”

So is it worth it?

Yes and no. Creating a bespoke tool can help you get to the specific insights you need quicker and more easily. However, there are limitations, particularly around scalability. 

For Alessia, it resulted in the first research on Burning Man and Transformational Festivals ecosystem. However, “the tools we build are born to be handled internally, and not necessarily ready to scale to millions of users. This makes an important difference, in terms of servers, legal protocols and UX.” 

According to Michael, “While the outputs are interpretive rather than absolute, they provide a strong foundation for hypothesis-building and strategic thinking, especially when paired with other data sources for validation.” So, in both cases there are compromises that need to be made.

And, while bespoke tools can solve a specific challenge, there’s still a place for commercial tools too. “We continue to use commercial social listening platforms for tasks like brand monitoring, topic tracking, and share of voice analysis,” explains Michael. And according to Roi, “we’re not trying to compete with VC-backed startups. Our goal was to test and evolve an idea that made sense for our clients and our way of working. And it’s worked. [But] We still use commercial tools.”

‍

Advice for building your own social listening tool

So, if, after reading this, you’ve been convinced to build your own social listening tool, what are the key things to consider to set yourself up for success?

For Alessia, she would advise first-timers not to be too ambitious. “Keep it straight to the point and focused on your customers’ goals. I.e, don’t add fancy metrics that they are never going to use. Co-create with your customers and offer less, better.” She also recommends making it scientifically solid. “Think thoroughly whether the tool is going to depict all metrics accurately, especially the qualitative ones, or if they are going to be approximate/misinterpreted. Better nothing than wrong answers.”

According to Roi, you should “follow a proper innovation framework. We used the Six Is of Innovation, which helped us stay focused on the business need and kept the team accountable and engaged throughout.” 

And for Michael, it’s all about testing. “We invested significant time in refining prompt design (against various data sets) to ensure the most valuable and relevant outputs possible. Testing wasn’t just a phase, it was a continuous process that shaped the agent and the output's effectiveness.”

It’s also important to weigh up the ROI of developing something in-house vs looking for a ‘plug and play’ tool from a trusted SITech vendor. Unless you have technical expertise in-house, a unique use case that keeps cropping up, or you’re an agency looking to differentiate yourself with a bespoke service (that actually works), it probably doesn’t make sense to reinvent the wheel. 

‍

Should SITech vendors be concerned?

The rise of in-house tech development probably isn’t a major cause of concern for SITech vendors. It’s still a relatively small percentage of social data users that are considering developing their own tools, and an even smaller proportion of those who actually follow through. And, in most cases, the need for proprietary technology is because of specific, less typical, use cases. 

However, as it becomes easier for people to develop their own technology to solve their social data problems, scrappy as it may be, this could be a good time for commercial SITech providers to think about their own business models. Particularly around how accessible their product is. 

As Alessia points out, most are targeting large clients with big budgets, pricing out smaller players that have the same social data needs. Her advice: “Lower the entry barrier. Give companies, teams and researchers the chance to buy project-by-project. Try more contemporary business models, for example, based on credits as opposed to yearly plans. The future of research is more fractional. Be transparent with pricing! Make your tool easy to buy as opposed to a journey into salesy demos.” 

Even if commercial SITech is made more accessible, it’s likely that people will still develop their own tools and perhaps blend them with commercial ones. And that’s ok. Good, even, for the industry. Because, social listening isn’t technology. It’s about what technology enables you to do, the insights it can help you generate and the meaning you extract from it. And there isn’t one way to get there. So it makes sense that people will find their own ways to turn data into meaningful insights, whether that’s with their own technology, a commercial tool or a blend of both. 

‍