Why Your AI Search is Barely Scratching the Surface

And why ChatGPT can't give you a complete spreadsheet of competitors, employees, or even jobs

Imagine spending hours browsing dozens of websites every day just to find people to hire or companies that have raised funding. Many professionals still do this tedious work manually.

The clever ones turn to ChatGPT or Claude for help, but these tools end up paraphrasing the top few popular search results.

But what if you could simply ask: "Full-stack engineers in SF that are great at design, and have worked at an AI startup" and get a perfectly formatted spreadsheet in return?

Or as a startup founder understanding the competitive landscape: "Find all companies developing AI products in the United States that have fewer than 100 employees and were acquired between January 2023 and the present."

These aren't just the few popular results reformatted into a spreadsheet the way generic models do. These are comprehensive spreadsheets with hundreds of unique rows and all the columns of information you actually need.

The answer to these isn't generic AI, but specialized AI agents that crawl and index a large part of the internet for you. I will first explain why generic AI falls short.

Why Generic AI Falls Short

ChatGPT and Claude will attempt to create these spreadsheets for you. They'll search the web, format some results, and deliver what looks like a comprehensive answer. But, the results often include only the top few results or well-known players.

For a typical competitive landscape analysis, Generic AI tools typically return 5-15 well-known companies or recent news results. They miss the smaller players, international companies, and niche players that might be exactly what you need.

Generic AI tools like ChatGPT and Claude aren't designed for comprehensive data collection. Here's what they actually do:

1. Take your question and run a few web searches

2. Read the first several pages of results

3. Reformat and summarize what they found

This approach works great for general questions, but fails when you need exhaustive, structured data from across the entire web. Now let me explain how these specialized AI agents work differently.

How Specialized AI Agents Work Differently

Specialized web intelligence tools take a fundamentally different approach:

1. Massive Data Indexing: They continuously crawl and index millions of websites, not just search a few pages on demand. They literally build a database of a large portion of the internet.

2. Smart Data Structuring: They convert this information into formats that AI can efficiently search and analyze.

3. Deep Query Processing: They can run complex searches across their entire database to find comprehensive results.

Think of it like the difference between asking a librarian to quickly check a few popular books versus having access to a complete digital catalog of every book ever written.

The Spreadsheets These Tools Built for Me

So I have primarily used two tools: Exa AI Websets and Parallel AI. Thanks to Exa for a generous free trial of websets, which I ended up using extensively.

Use Case 1: Getting names of prospects with their emails.

One of the biggest use cases is market research. But, not just to know the companies in your addressable market but also their contact details.

One of the queries I used was: “Investment management firms, hedge funds or asset management with less than 20 employees with focus on biotech and pharma or also have that as a focus area.”

This query yielded me around 600 rows of data and a total of 10 columns. I further enriched the dataset with emails of key personnel.

I would be careful about trusting the emails though. Don’t directly use them to start an apollo sequence. I have had my domain reputation harmed because of this. Do use email verification tools before using these emails completely. When I had gotten them through exa, some emails were plain wrong, hallucinated or don’t exist anymore.

But, I haven’t yet tried the emails retrieved using parallel AI. In the case of parallel AI, I noticed there tended to be lesser emails that were plain wrong. Like exa had some emails like xxx@<company name>.com and so on.

Use Case 2 Market research:

I queried for 100 companies that are building AI agents for ad making. I was curious about the existing ad maker space. I thought maybe instead of spending time learning how to make a great launch video and use 2 brain cells, I can throw 20$ at some ad maker and get it done. But, even otherwise you can use this for market research too.

Use Case 3: Finding niche angel investors

You can find angel investors or even VC’s who have experience or invest in the space you are interested in. You can get creative and find people who have worked at certain companies/areas or already invested in certain areas and so on.

Use Case 4: Finding companies hiring software engineers and key personnel’s email

If you are looking for roles, you know alpha isn’t where everyone else is looking but across the web. Most job portals don't necessarily list every single role posted across all companies or even the major companies.

Use Case 5: Finding events

I am super passionate about finance and love attending relevant events. Finding finance events is hard, there is no Cerebral valley or Luma where majority of the events are collated in one place. I am sure there are tech events you could find too using these tools that are not covered well.

So what’s better than AI agents that could crawl the web and find them for you :)

The above shows most to be in Eventbrite, but the majority of them aren't on Eventbrite, they're scattered across the web.

Use Case 6: Finding hacker houses in SF :P

There are apparently so many different hacker houses in SF, so I thought let me find all of them at once :)

So these are some use cases I have used them for. In fact, I used to run a site called noleaf.app that was primarily using these tools. These use cases barely scratch the surface of what's possible when you have access to agents that crawl most of the internet and provide your desired spreadsheet.

Now that you've come this far, on a lighter note you might have noticed I'm looking to talk to people working in private equity, hedge funds, or other investment professionals, I'd love to connect with you. So instead of having me spend $$$ to find you, reach out to me at kamathhrishi [at] Google's mailing address :)

Also if you are an angel investor or VC who fund pre-seed rounds.

Hit the like or subscribe button if you liked what I wrote.