ScraperCity + Pipedream

Log in to ScraperCity and go to app.scrapercity.com/dashboard/api-docs to copy your API key. Then in Pipedream, navigate to Settings > Environment Variables and create a new secret variable named SCRAPERCITY_API_KEY. Storing the key as an environment variable keeps it out of your workflow code and prevents accidental exposure in Pipedream's execution logs.

Create a new workflow inside a Pipedream project. Choose the trigger type that matches your use case:

• Cron Schedule - runs the scrape automatically on a fixed interval. Use the "Every" option (e.g. every 1 day) or a custom cron expression like 0 8 * * 1-5 to run at 8 AM on weekdays.
• HTTP / Webhook - fires when you send a POST request to the workflow's unique URL. Useful for triggering a scrape from an external app, a form submission, or another automation tool.
• Manual - click "Run Now" in the Inspector. Good for testing before enabling the schedule.

Note: Pipedream manages the servers for scheduled workflows, so there is no server or cron daemon to operate yourself.

Add a step and select HTTP Request (or use a Node.js code step with axios for more control). Configure the request as shown below. This example scrapes Apollo for Director-level contacts in SaaS companies with a verified email address.

Method: GET
URL: https://app.scrapercity.com/api/v1/database/leads
Headers:
  Authorization: Bearer YOUR_SCRAPERCITY_KEY
Query Parameters:
  title: Director of Sales
  industry: computer software
  hasEmail: true
  limit: 100
  page: 1

Replace YOUR_SCRAPERCITY_KEY with {{process.env.SCRAPERCITY_API_KEY}} when using Pipedream's object explorer, or reference process.env.SCRAPERCITY_API_KEY inside a code step. The Lead Database endpoint requires the $649/mo plan. All other scraper endpoints work on any plan.

The ScraperCity API paginates at 100 leads per page with a maximum of 100,000 leads per day. Add a Node.js code step to loop through all pages and return a flat array of leads for downstream steps:

export default defineComponent({
  async run({ steps, $ }) {
    const axios = require("axios");
    const allLeads = [];
    let page = 1;
    let totalPages = 1;

    do {
      const response = await axios.get(
        "https://app.scrapercity.com/api/v1/database/leads",
        {
          headers: {
            Authorization: "Bearer " + process.env.SCRAPERCITY_API_KEY,
          },
          params: {
            title: "Director of Sales",
            industry: "computer software",
            hasEmail: "true",
            limit: "100",
            page: String(page),
          },
        }
      );
      allLeads.push(...response.data.data);
      totalPages = response.data.pagination.totalPages;
      page++;
    } while (page <= totalPages);

    return allLeads;
  },
})

Using axios (rather than $.send.http()) is recommended here because you need to read the response body to extract pagination metadata and pass the full lead array to the next step. Pipedream makes any value you return from a code step available to all downstream steps via the steps object.

Add an optional email validation step after the pagination loop. For each lead returned, POST the email address to the ScraperCity Email Validator at $0.0036/email. This filters catch-all addresses and undeliverable contacts before they reach your CRM or outreach tool - keeping your sender reputation clean.

export default defineComponent({
  async run({ steps, $ }) {
    const axios = require("axios");
    const leads = steps.fetch_leads.$return_value; // from previous step
    const validatedLeads = [];

    for (const lead of leads) {
      if (!lead.email) continue;
      try {
        const res = await axios.post(
          "https://app.scrapercity.com/api/v1/email-validator",
          { email: lead.email },
          {
            headers: {
              Authorization: "Bearer " + process.env.SCRAPERCITY_API_KEY,
              "Content-Type": "application/json",
            },
          }
        );
        if (res.data.deliverable === true) {
          validatedLeads.push(lead);
        }
      } catch (err) {
        console.log("Validation error for " + lead.email, err.message);
      }
    }

    return validatedLeads;
  },
})

Add a destination step after your data is collected and optionally validated. Pipedream has pre-built actions for common destinations - click the + button and search by app name:

• Google Sheets - use the "Add Single Row" action to append each lead. Map fields from the ScraperCity response (name, email, title, company, phone) to your sheet columns.
• HubSpot - use "Create Contact" to push validated leads into your CRM pipeline with custom properties.
• Pipedrive - create Persons and Deals from scraped contacts with a single action step.
• Airtable - insert leads into a base for review and tagging before outreach.
• Slack - post a summary message to a channel when a scrape completes, with lead count and a link to the results sheet.
• Instantly / Smartlead / Lemlist - POST validated leads directly to your cold email tool's API using an HTTP Request step.