This article relates to the legacy Apify Crawler product, which is being retired in favor of the apify/legacy-phantomjs-crawler actor. All the information in this article is still valid, and applies to both the legacy Crawler product and the new actor. For more information, please read this blog post.
For new projects, we recommend using the newer apify/web-scraper actor that is based on the modern headless Chrome browser.
With Zapier integration you can connect Apifier to hundreds of apps, including Slack, Trello, Google Spreadsheets, Dropbox, Salesforce, and loads more. Your Zapier workflows can fetch crawler results from Apifier, start a crawler running, compare results, or get existing results from previous runs. You can also use the integration to trigger a workflow whenever a crawler finishes running.
In this tutorial we'll set up a Zapier workflow which will upload results to a Google spreadsheet when the crawler finishes. This way we can, for example, monitor prices on an e-commerce site daily.
We need two things before we start with an integration:
- Crawler on Apifier - we'll use the 6pm community crawler that extracts basic info about products on an e-commerce website
- Google spreadsheet where products will be stored (we'll use this one, which is publicly shared)
The results of the Apifier crawler look like this:
The spreadsheet initially looks as follows (we'll upload just the main attributes and add the date when each product was scraped):
Setting up a trigger
Once you sign up for Zapier (you can use a free plan for up to 100 tasks per month), click the "Make a Zap" button. Since we want to trigger our workflow when the crawler finishes, we'll choose Apifier as the trigger app:
The only Apifier trigger is called "Crawler run finished", so let's choose that:
Now we need to connect our Apifier account to Zapier. For that we'll need User ID and Manage Acts Token. You can be found on the Apifier account page.
After a successful login, we can choose the crawler from the dropdown list. Every time this crawler finishes, our Zap will be triggered (unless you hit your monthly task limit on Zapier).
On the next page we will click "Connect & Continue" and after a successful connection we'll click on "View your crawler run". At this point, Zapier fetches the last run of the crawler and shows its results:
Setting up an action
Now that we have our trigger configured, we can set up an action that will be performed when crawler finishes. Let's choose Google Sheets from the app menu:
On the next page we'll choose Create Spreadsheet Row(s) action, which is available under the "Show less common options" link:
After that we have to connect our Google account and grant write access for Zapier. Let's hit the Test button to see if Zapier has access:
And now the main part - we have to map results from Apifier to columns in our spreadsheet. At first, we'll choose the Google spreadsheet document and the sheet we have prepared for this use case. Then we select an attribute from crawler results for each row in a spreadsheet. We can easily do this using a drop-down list filled with the results of the last crawler run. The entire mapping form looks as follows:
Now we're ready to hit the "Create & Continue" button on the next page:
Now the Zap creates a test run of our action and we can see our data uploaded to our spreadsheet:
Looks great, right? Finally, we'll give a name to our new Zap and set it to "on":
Once our Zap is on, it will be started every time our crawler finishes. Actually, you can see the internal Zapier webhook which is set in the Advanced settings of the crawler configuration:
This was set automatically during our Zap setup. Keep in mind that removing this webhook will disable the Zap. Also note that this use case of appending rows to a spreadsheet is limited to a couple of hundred rows per one Zap run. If you want to upload more than that every run, you will have to create a new file in Google Drive, Dropbox or another service that can be connected with Zapier.
And that's it. We’ll be sharing a few ideas over the next few weeks, and we’d love to hear from you if you’ve found a great use case. Just let us know at firstname.lastname@example.org
And remember - Zapier makes you happier!