Apify actors such as Web Scraper (apify/web-scraper), Puppeteer Scraper (apify/puppeteer-scraper) and Instagram Scraper (jaroslavhejlek/instagram-scraper) enable you to crawl websites that require a login.
To do this:
- Add a cookie editor like EditThisCookie to your browser.
- Log in to the website you want to crawl.
- Export your cookies. Click the EditThisCookie button next to your URL and click Export.
- Paste the cookies into the Initial cookies field in the Proxy and browser configuration tab in your selected scraper's Input section.
If this does not work, it means the cookies are short-lived. In this case, you will need to implement login in your code.
See the full tutorial in our documentation.