Rumination 2023 @Crunch
Thoughts, learnings, mistakes, zoom-out-perspectives, plans and a pulse overall
What a year it’s been for everyone here at Crunch!
The first emotion I deeply carry as 2023 sets down is gratitude. Given everything shaping up in the world of AI, a lot of what made sense a couple of years back is no longer relevant. We’re better informed, accounting for factors that were not close to being obvious a couple of years back. It’s easy to feel that we’re more challenged, but speaking to peers who started in the past 3-4 years makes it clear how better placed we are today to avoid certain pitfalls.
I have grown up admiring year-end letters by legendary founders, often shared publicly. I recently learnt that Warren Buffet would always write his shareholder letters addressing them to his sister, and then before sending them out, he would replace the word sister with shareholders. This helped him be transparent and candid. That is probably what makes them brilliant. This is my first attempt at the same. In this series, you will find broadly three things — a recap of everything that went down, today's status of what we’re attempting with Crunch and a few learnings that we have never broadcasted before.
From serving a decentralized world of anonymous folks to building for analysts, today, we were on a roller coaster throughout. Looking back, I am reminded of the days we would look at companies that pivoted to completely unrelated domains and shake our heads in disbelief. Today, we’re living that. Let’s get into the depths of everything that has happened and where we stand.
A quick look back at why we pivoted
DAOs are decentralized, and a group of people, equally powerful, not only move slowly with complex interpersonal dynamics but as a group, they always also optimize for short-term gains only. We had a location disadvantage, which to some extent started getting solved as we sponsored a bunch of events, but overall, we still found ourselves largely ambiguous around monetization. As an extension of Dao Manager, Robin, our AI assistant for DAO folks, gained a lot of limelight, but because of the broader ‘hand-it-over-to-us-free’ signal from the DAOs around DAO Manager, we went ahead and sunset it.
When we decided to kill the core thing we had developed to be the best amongst peer offerings, an unavoidable feeling of ‘Are we moving slowly?’ kicked in. We changed gears and cut all tracks within the company, leaving only two: Product and GTM. In retrospect, this was one of the best things we did. There were significant organizational revamps, and a series of experiments happened over a span of roughly 75 days. We had given ourselves 90 days to come up with a plan.
It feels as if this was ages ago, but a few experiments eventually carved the paths for what Crunch is today. We played with an IRL event experience platform (Bumpp), a hackathon management platform (Hatched), a community as GTM (Superfans), AI-based financial report creation (Fintegrity), New DAO protocol (encouraging voter participation by reducing gas fee), an AI-chatbot for your webpage to interact with your APIs for complex task as well as information retrieval (Chatcon), A tool creation engine for internal usage within enterprises (Mintools), a trading tools to help discover emerging assets backed by social signals (Marcat) and more.
We regained our momentum, and as we hopped on from one idea to another, changing all variables every other week, one thing became clear. We had a team ready to stick with us through the thick and thin. In the process of experiments, some of you helped re-focus only on ideas that could grow substantially big. Everyone in the team started working outside of their comfort zone. In the course of experiments, we realized how most of the thin wrappers around AI would eventually die anyway.
The Inception of Crunch
Crunch was born as we tried to get AI models to understand interfaces to perform actions. We realized if we could understand interfaces, we could tag them. And if we could tag them, we could answer questions about them, and as such, product analytics would become super easy. This and the fact that we as a team were naturally inquisitive, looking at dashboards and asking 20 follow-up questions, shaped the rest. We realized that it was incredibly frustrating having a question in our heads and not getting an answer to it immediately. It was time to test if the problem resonated with other people.
We researched, and here is what we learnt —
The setup was a massive pain point for everyone. Teams spent up to 6 months just setting things up, and it still never ended.
Interacting with filters was not how we as humans think. Imagine a question like extracting the list of people who have not logged in more than once every week or whose session times lately have been down by 95% and have raised three support tickets that haven’t been answered within 24 hours. Fairly easy to imagine but incredibly tough to plot if you play around with filters.
Non-analyst product folks and executives ask questions and often have follow-up questions. Analysts who can query datasets relatively cope better, but the leaders are often left hanging on analysts to turn around, and when they do, there are new questions that pop up.
B2B companies had many more data warehouses than B2C companies. The kind of analysis a B2B analyst was doing was more complex, had less volume of data (increasing our margin) and was likely to be more appreciated. B2C has a lower ACV, so user or event-based pricing would be trickier. Lastly, B2C teams have analytics as a core function in their business, and the threshold (in terms of company growth) by which an in-house team would pop up was significantly low.
For all these three points, we had a solution: Auto-track took 5 mins to deploy, and theoretically, every event could be seamlessly captured. If we deployed AI models on top, using NLP, we could convert statements in English into queries, and it would be a very happy world.
On top, We introduced a whimsical-like canvas, which would help everyone plot multiple hypotheses through a tree of widgets. The visualization would also mean people could collaborate over data; the story became clearer as a tree diagram was plotted, and for the curious few folks, we would give the option of how the graph was made (bringing trust in the query as opposed to being a magic black box generating results that you trust as much as you trust yourself not to blush if Elon Musk called you to wish you a happy birthday).
Lastly, we planned on starting with product analytics and eventually expanding to other types of datasets.
The first mistake — what we did not learn above?
The setup was tricky, but if someone had gone through a setup of 6 months, they had no good reason to switch for a delta of less than 500%. Surfacing a delta of 500% over a tool like Amplitude is a very long battle. Our mistake was not realizing the setup was a big deal, but only for people getting started.
Coming onto people who were getting started, constraints were a massive problem. Some early deployments saw customers feeding a particular type of dataset and asking for an altogether different question. Something we had hoped would not be a problem. Introducing constraints in the UX naturally looked like the right next step, but this solved the problem for people who were just getting started. Our mistake was assuming that the text bar would solve the problem for everyone and be the default way of querying data.
To cater to companies as they grew with a target of retaining them, we would have to be very aggressive in being accurate with our responses. When you depend on LLMs to generate some functions, you run into scenarios where unforeseen issues would pop up. On top of that, everyone had a unique UI framework and a poor nomenclature discipline, making things exponentially challenging for us and leaving our auto-tracker itself crippled. This was a solvable problem, but it was clear we had made the mistake of underestimating efforts. The engineering was much more challenging than we had anticipated, and what we would gain from maybe six months of engineering would still fall short on some fronts.
AI models hallucinate once in a while despite training. When that happens, anyone would immediately lose trust in the system. It was here that we realized that executives preferred calling up analysts directly to cross-check. Further, because of that fear, most leaders are okay with waiting an extra day for their analysed insight to reach them. Our mistake was assuming that our system would be accurate enough to completely eradicate analysts as far as a leader goes, thereby defeating the purpose of making bosses independent — at least for the near future. Also, the time to answer was not as big of a deal.
Modification in our approach
If we have been good at one thing as a team, then that has been being fast. We immediately changed course. We paused further deployments (despite having a very extensive waitlist). In parallel, the product team started working on making AI more accurate, and the GTM function went back to the drawing board with new questions that needed answers. Below is a detailed summary of what all we have changed and learned.
#1) Selling results instead of software
Chat with customers, and then some of you also indicated that selling results directly instead of software that could be used to derive results would be a better fit. From our conversation with customers, it had become more evident that businesses viewed problems in two phases
Acquisition / Expansion (dependent on targeting, upsell and monetization)
Engagement & Retention (dependent on product and customer’s experience)
We picked the problem of retention because of two factors. First, every customer we had spoken to indicated fixing retention as an area for which they would have a budget. Secondly, we had already seen a bunch of offerings struggling with monetization. So, the GTM function started evaluating whether a pre-packed insight on retention makes sense to companies out there.
We soon realized retention largely depended on factors that were out of control and subjective. An enterprise could do everything wrong and still have a renewal. A smaller company could do everything right but still have a churn. That meant retention is a problem that can’t be solved entirely because of a product, and to whatever extent it can be, the risk is entirely on the product. This means people prefer a model where they give us a cut of the money we help retain, and as such, it might be a bad business. This means the customer wins if we win; the customer has nothing to lose if we lose.
Further, no one has shown a ‘hell yeah!’ reaction to insights as an offering. People have shown mild interest. However, no one has returned with a ‘this-specific-insight- will-change-my-life’ reaction. We believe that’s because medium-sized businesses don’t have a perfect segregation of who is handling retention vs engagement. They have a very blurry idea of what their growth team should do, and we have mostly talked to B2B folks so far, which makes it even more ambiguous.
#2) Refocusing our Target customers
It became clear that our assumption of the ideal customer needed some revamping. Three things changed here:
Unless an organization has product-led motion, B2B companies look at data only for their internal prioritisation, which gets pretty well served by existing solutions, and if it does not, no one is yet crying for help. So, we started talking to B2C and B2B companies with a PLG motion.
As far as the company size goes in both categories, smaller companies don’t have a prioritisation problem (as in, they are okay with sending a resurrection notification to everyone instead of people who need to be resurrected). Further, smaller companies have low data, and an out-of-the-box solution likely fails because the model never gets adequately trained. Enterprises, on the other hand, have a human factor, and everyone feels AI is not yet there to replace humans completely (they may be in for a surprise, but it’s risky chasing this path because you have insufficient tech). That leaves medium-sized businesses and small to medium enterprises.
The stage at which a company would benefit from our offering became clearer. We talked to companies of all sizes. Most people like the idea of consuming multiple data warehouses, but they don’t know what they would do with such an offering. So, they lean on analysts today. These start from Pre-Series B folks and go up to enterprises. However, out of this target market, it’s usually around the time a company scales post PMF that they see breaking structures, issues with data mapping, new tools getting deployed but with inaccessible insights, growth targets on their heads and a sense of cluelessness. This meant targeting people before their Series B round.
#3) Moving away from product analytics to business analytics
Saying we compete with Mixpanel steers the conversation in the wrong direction. People immediately evaluate us from the lens of the cost of migrating as opposed to a new way of analyzing data that goes way beyond the product. We started building on product data as a starting point and not an end outcome. We, however, made the following realizations:
We realised that the setup of product analytic tools couldn’t be made easy beyond a certain point because of the intricacies of sending custom events in the correct sequence whenever a product action gets triggered. That makes the ‘we-make-your-setup-easy’ value proposition itself weak.
The whimsical-like canvas view is excellent, but UX can’t be the differentiator.
Product Analytics tools have scope for improvement, such as supporting analyst-friendly code workflows. However, with PostHog also adding notebook support recently, chances are whatever scope remains with Mixpanel and Amplitude is already on track to get covered. Probably sooner than we would if we started today.
When we become the product analytics provider, we get into solving problems with very high competition and a setup which makes the time to value longer. Working on top of Mixpanel / amplitude data seemed much more sensible.
Mixpanel and Amplitude, as can be absorbed with most of the incumbents out there as well, are adding AI workflows. AI itself will not be a differentiator anymore for anyone in the industry.
Positioning ourselves as an insight layer on top of existing product analytics tools is also a hollow route to pursue. That’s mainly because, in this age of budget cutting, an extra tool that would increase costs for someone initially becomes a tough sell and not something anyone is very keen on. Insight is not something anyone is looking for every other day. It can be argued that it all depends on the kind of insights that we generate. However, having spoken to users and buyers, the insights that can not wait are insights that immediately disrupt revenue flow. In these scenarios, a frequency of even two monthly alerts is a no-brainer. But, we’re yet to identify if there are enough of these insights that can be packaged into a product.
Instead, we realized by targeting analytics overall; we could directly work on silos of data lying around. However, this approach must account for all our learnings, such as the occasional inaccurate AI results.
#4) Targeting Analysts as opposed to Non-Analysts
When we look at the cross-warehouse analysis, there are stakeholders from the product, there are stakeholders from the business team, and then there are analysts. Here is what we learned about each of these segments:
Customer Success Managers and similar folks prefer built dashboards, and they are not people who will ask questions a lot. Very few people would. Success Managers prefer working against tasks and lists. They live and breathe in tools like Gainsight, and when a problem occurs, they switch to a Zoom call with the client. Also, CSMs are present only in SMBs and enterprises. The ones in enterprises again have nothing to gain with data because it’s all about relationships, and as far as cross-function alignments go, tools like Hubspot and Gainsight do the job. The closest we have come to someone citing this as useful from an established org is a use case like “My inbound leads are not enough to meet our monthly quota. Could you analyse the website traffic and help me with a sequential causation analysis”? Even this PoC agrees that business teams will not have the right mindset or analytical thought process. Plus, getting the budget for an analytical tool from a CS team will always be tricky because it’s not their KPI.
Product Leaders are not capable enough to examine code themselves. Instead of working towards the vision of an analyst-free world, we realized getting buy-in from analysts was much more sensible. Further, digging deep into data may only be an analyst job and not that of a leader. Does an interface like Google Gemini, plus a revelation of logic behind the answer that the system came up with, plus the ability to add context based on experience, help? Probably yes, but other than the few curious CEOs, most leaders ask a question when someone else above or a customer asks them a question. And even when they ask a question, they prefer primarily leaning on an analyst because they have the luxury of time.
Analysts have the technical acumen to prompt the system. We spent hours evaluating analyst workflows across our target companies. Today, people lean on analysts even with AI because they don’t trust their data-analyzing capabilities. Analysts don’t exist in tiny companies. On the other hand, analysts in enterprises have the luxury of time to build a custom solution using open AI. For medium-sized company analysts, time is short, data is growing, and tons of low-hanging products exist beyond just doing in-depth analysis. For such an analyst, building an in-house platform is not a logically intelligent move. These analysts spend their time doing the following:
getting the need to analyze (sometimes ad-hoc, sometimes because of deviation in standard metrics, sometimes because a stakeholder has an ask and sometimes when a new feature or campaign is being launched)
gathering information about sources that need to be queried (business context) and getting an alignment on the same
extracting data from all warehouses and cleaning it
querying data until insight is found (can involve parallel standard queries with new variables or a series of queries in case of open-ended requests). Sometimes, it’s about a bullet list of probable causes
Visualizing the data and presenting it
For an analyst, open-ended requests consume 20% of their time, and standard queries take the remaining 80%, but overall, 99% of their day-to-day is spent just running queries. Most analysts always have a series of questions they need to provide answers for, which are queued all the time. Most analysts have an expectation set in terms of wait time before the bosses get back an answer, but that leaves room for a question. Is that because the bosses know there is no better alternative to fast-track the process without affecting the quality of the results?
Regarding the exact tools used for analysis, today’s analysts use SQL, Excel Sheets, Jupiter Notebook, Metabse, Python and, in some cases, Hex /Deepnote /Einblick. Both Hex and Deepnote thought about the same thing around the same time four years back. In that sense, they are both innovators. Thoughtspot has acquired tools like Mode that might have inspired Hex. Thoughtspot might be getting more aligned towards replacing Tableau, the company. It’s essential to highlight why all these details matter. It ultimately helps us narrow further down to the segment of analysts we are catering to, for example, the ones using code vs. those using no code.
Thoughts On Code vs No-Code
Chances are that the way things have evolved in the last 20 years will not even remotely be close to what will happen in the next 20 years. For example, over the last decade, both developers and Webflow users have grown. People use Webflow because they don’t want to code. Webflow, however, has limitations, and devs have always managed to exist. In 5 years, maybe Webflow will be capable of replacing devs who build websites altogether! So, we can’t take the last 20 years as a reference. In 5 years, maybe no code will become the winner. Many developers also prefer using Webflow, so they don’t have to start from scratch.
Now, let’s focus on analytics. Hex vs Tableau (not precisely the company but the idea like Tableau the company, Looker, etc). Hex is the code version of Tableau. Tableau is old age, with a no-code platform to talk to multiple databases. If no code is going to become more mainstream, Hex might not be needed eventually. But Hex has an intelligent move. Being the dominant no-code tableau of the future is a dicey play, and chances are that someone might be just early in their attempt. That’s probably why Hex is building a code-friendly version to lure developers while gradually adding no-code features. That way, with testimonials from developers, Hex can lure no-code developers as well.
Also, on the other hand, if anyone today thinks of replacing Tableau (the idea), which is mainly used as a visualization layer and not as an analysis layer, chances are that they will feel the need to build something like Hex’s support for code because it helps them cater to analysts folks who want to use the product but are comfortable with code.
There is a lot that needs to be answered. We’re actively working on this.
Continuing on the point before the no-code bit, our hypothesis primarily leans on the fact that since analysts are buried under a pipeline of pending queries, making them more efficient is of value both to them and their bosses (being validated separately). The way to do this would be to help analysts on the steps (step ‘b’ to step ‘e’ above) from extraction to cleaning to analyzing to visualizing (both regular and open-ended stuff) with an AI assistant that saves them 90% of the time in a notebook-like interface. The benefit of this is the following:
Analysts, being technical people, can modify queries. So, while they save time, they can also instruct the AI to give perfect results.
Analysts will still write queries, but they will write 50 queries where they were earlier writing 500. We have seen eyes glitter while presenting our idea.
Eventually, analysts can set things up in a way that leaders can also interact with data directly without having to cross-check everything with them.
Some people may argue that this is ultimately like asking analysts to support a tool that will replace them eventually. Whether we do it or not, it will happen anyway, and analysts know it. The future likely will see everyone being as informed (not just data-rich) as analysts and analysts having super-powers because of AI.
#5) Exploring Telematic Data Analytics
One another point frequently being raised by the users we have been talking to who have spent time working with machines, hardware and data such as IoT, RFID, GPS, Energy, etc, is that they all feel these particular industries are underserved on the analytics front. We’re yet to get enough clarity on whether this is a pain point and in need of a painkiller that we can develop. Two points that make this worth exploring are:
Plants have machines and lots of them. Plants don’t have analysts. Every machine needs a custom model that, when powered by AI, can look at all the recorded logs and alert the floor manager whenever something goes wrong, indicating what that might be. Today, a human being is involved, whereas if you imagine a machine making weird sounds, they can, with experience, try diagnosing the problem. However, a human can’t be placed next to each machine.
In the world of SAAS, there are Amplitudes, Power BIs, Looker and Tableaus of the world. All of them are adding AI capabilities faster than they have ever before. However, we have discovered no tool yet that tackles the same level of analytics for non-software data.
A lot remains to be answered here, but if this were to become a niche, it would be a brilliant one. We’re working on getting clarity here.
Next Steps
Ultimately, it boils down to getting more and more clarity on the following. So, this is and will likely remain a section that will evolve.
Who are you replacing?
This has to be the ‘team of analysts’ in the long run.Who is paying?
Chief Data Officers, Head of Analytics, and if these functions don’t exist, product.What kind of companies are we targeting?
B2C, B2B, but 100% PLG. Medium-sized companies (starting Pre-Series B) to small and medium enterprises. Ideally, the team has at least three PMs and two business or data analysts. More segmentation will gradually become clearer as to which specific industries and specific analyst profiles. We know that people who respond to us usually have an employee count between 51-500.What is the moat or the advantage in our favour?
For now, if we bring about a behaviour change (such as no need for a human analyst), we become the first mover. This is not much of a moat, but it’s an advantage.What is that unsolved problem which people are excited about that we can solve in a way that we will be the best in 2-3 years?
It's unclear for now.
What do we know people like in whatever we have tested so far?
Analysis spanning multiple data sources
The fact that analysts can verify if things are correct
People love that AI can do causation analysis, but at the same time, the causation can be modified.
People loved the whimsical tree diagram, though that is not a moat and more of a UX
Questions that we’re pondering over as we move to the next year (at the back of our heads)
How can we get to a bottoms-up distribution with as small a time to value as possible? This is given that connecting warehouses often becomes tricky.
How accurately can we help analysts with issues in data-stream, such as an event misfiring? This is not needed immediately, but gradually, as we envision an analyst-free world, this would be a very obvious concern.
How can we get to a spot where no one feels hesitant because they are the first people trying us? How do we get some case studies?
How do we close the first five design partners?
What is the right time to start with an on-ground presence in the US?
How can we demonstrate proof with newer types of datasets?
As AI evolves, what new UI patterns will evolve? How can we automate part of workflows without having a text input?
What is not going to change? What is going to change in the world? As models change, what will become not much of a big deal anymore? What is going to be obvious to be built in-house?
A look at our team
We have become a very lean team. In 2024, we’ll have 17 people to start with. We also recently got our ISO certification, and GDPR is coming. We’re completing month 2 of the three-month evaluation cycle for SOC2 as well.
Engineers: 3 backend, 3 frontend, 1 QA and Apoorv
GTM: 1 Marketing, 1 HR + Ops and Vikram
Product: 1 Product Lead / CoS, 2 product designers
1 Finance Lead + 2 Support Staff (Pantry and Operations)
Small Gestures that went a long way
I am super grateful for some special folks amongst you who have constantly touched base with us, checked in on us and nudged us. I am grateful when I realize how we ended up having your backing. It’s not easy, and we never assumed it would be, but it still feels good to check in with each other occasionally. If you have been in touch, thank you :)
Where do we need help?
Help us close design partners. I can’t insist on this more.
Help us with an introduction to the right communities, events and gatherings. Bay area introductions are just as welcome.
Thoughts on the Future of Startups and Entrepreneurship in General
As we have been building and breathing in AI for the past year, we have developed some views. These views are the basis of our daily decisions, and we continuously think about these as principles.
In this age of evolving baseline, moving slowly might be an advantage unless you’re solving a technically very, very, very tough problem. GTM, growth loops, and product experience will not matter as much in this coming decade. What will matter is true innovation — something most of us have not felt the need to attempt in a while. We’re all students of the game — all of us. Being verticalized might be an excellent first step towards moving in this direction.
At this point, most of the startup attempts look stochastic. Established players who have grown in 2023 will also struggle by the end of 2024. For example, if you’re making an AI voice thingy, the player likely to win is the video creation platform because innovation will become incredibly cheap. Most services will have to be end-to-end to thrive in an AI-first era.
Most enterprises will likely seek services with fractional experts coming in to eat the market share that tools have commanded. They will do it not only because it’s going to get easy or become more secure to do so, but with the pace of technological development, the market will expect more for less $ of expense. Because everyone will offer the same piece of technology easily, there will be price wars. Enterprises will be forced to do things in-house. Chances are that more companies will be forced to price based on outcomes or RoI instead of seats or feature usage.
Most of the LLMs struggle with data pipelines. The data that can be fed into an LLM is not the data we see all around. There will be efforts on data pipeline services. Also, by nature, everyone would interact with more than one foundational LLM. Someone will end the battle of facilitating who takes care of what.
Foundational models on their own are no longer of significance. Heavily small models can achieve the same level of accuracy, and after a point in time, most model improvements don’t impact the masses. Models like Open AI will likely have some benefit because of being the first mover, but that too because of the advantage that comes from developing marketplaces and not because of being the first mover.
We will see more things change in the next five years than we have seen things change in the last 25. That means everyone’s baseline is probably off. You can’t be prepared enough.
In Analytics, existing tools are going to target newer markets now. For example, Looker and Power BI may have appealed only to analysts so far because of how tough they are to use. With Gen AI capabilities and the support of giants like Google and Microsoft, they will attract a newer set of companies. Whether they will succeed or not is to be seen.
This year saw us say forced goodbye to several phenomenal folks as we pivoted. This year was spent in our new office as we adopted a more aggressive work-from-office policy. This year saw both the founders get married within the same week, and about 20 days before the scheduled wedding, we decided to pivot. This year saw us scratching our heads more than we have ever before. It feels like this is in the distance, but we saw the SVB fiasco unfold and realized why they say, bank with the best. This year saw us feeling as if Black Mirror episodes would come real!
I read somewhere that becoming spring means accepting the risk of winter. To become present means, accepting the risk of absence. In that tune, to become successful would mean accepting and embracing our failures.
We will march on in 2024 with more vigour, more patience, more energy, and more determination than ever before. On a closing note, gratitude and more gratitude.
Onwards and Upwards 🚀