Clay (Free): to scrape paginated databases online and enrich results
Before we begin
Say goodbye to copy-pasting listings by hand. Now Clay automates it for you.
This Claybook shows you how to scrape data from public databases like Psychology Today, automatically generate profile URLs for each record using Clay’s integration with Google Search, and enrich those records with emails, phone numbers, or other data.
Whether you’re building prospect lists or researching a niche segment, you can handle the entire workflow without leaving Clay.
How it works:
- Copy the template to your Clay workspace
- Identify and add the URL of the public database you want to scrape
- Use a Clay formula to generate paginated URLs
- Scrape each profile into a Clay table
- Use Google Search to find personal or professional profile URLs
- Further enrich results with work emails, phone numbers, and more
Steps
Copy this template
First, copy this template in Clay.
If you’re already a user, you’ll be directed to your workspace. If not, you’ll be prompted to create a free Clay account.
The interactive demos on the right will walk you through each step of this Claybook.
Step 2:
Find the URL of the database you want to scrape
Next, we need to identify the URLs of the public database you want Clay to scrape into your table. In this example, we’re targeting New York–based therapists listed in Psychology Today’s online directory.
First, locate the URL for the New York therapist listings on Psychology Today. Then, use a Clay formula to generate URLs for each page of that directory. Clay’s scraper will use these URLs to automatically pull in the data.
Step 3:
Find user profile via Google Search
Next, after the table automatically writes each individual therapist listing to a new table, using Clay’s native integration with Google Search, find the user profile URL of every therapist. Or in your case, the direct profile URL of every result in the database.
From here, you can further enrich results with work emails, phone numbers, and more.
Copy the template
Start scraping listings from public databases, directly in Clay.
Simply copy this template, follow the steps above, and let Clay work its magic.
Learn with our resources
Check out Clay University, read our GTM blog, or try out our top templates to transform your growth ideas into outreach in minutes.
Clay University
Master Clay in a few core lessons. Uplevel your data enrichment, craft better outbound messages, and book more meetings.
Go to market blog
Our best content on growing your GTM motion, from outbound sales automation to AI lead generation.
Integrations
Browse through all the integrations you can connect to Clay in seconds and discover what you can do!
Clay Experts
Hire a Clay Enterprise expert or find a Clay partner to help with your next project
Clay's Chrome extension
Clay’s Chrome Extension makes it wicked easy to find leads almost anywhere on the internet