<html><head></head><body><div class="content educate_content"><section class="section section__article">
    <div class="breadcrumb" dir="ltr">
  <div class="link__arrow o__ltr">
    <a href="https://help.apify.com/en/">All Collections</a>
  </div>

      <div class="link__arrow o__ltr">
        <a href="https://help.apify.com/en/collections/1669746-extracting-data">Extracting data</a>
      </div>

  <div class="link__arrow o__ltr">How to scrape a website using sitemap.xml and Apify SDK</div>
</div>

  <div class="paper paper__large">
  <div class="content content__narrow">
    <div class="article intercom-force-break">
      <div class="article__meta" dir="ltr">
          <h1 class="t__h1">How to scrape a website using sitemap.xml and Apify SDK</h1>
          <div class="article__desc">
            Website with proper sitemap.xml = free jackpot for every web scraper
          </div>
          <div class="avatar">
  <div class="avatar__photo o__ltr">
      <img src="https://static.intercomassets.com/avatars/1227910/square_128/picture-3-1580399945.jpeg?1580399945" alt="Marek Trunk&#xE1;t avatar" class="avatar__image">

  </div>
  <div class="avatar__info" style="vertical-align: top;">
    <div>
      Written by <span class="c__darker"> Marek Trunk&#xE1;t</span>
        </div>
  </div>
</div>

      </div>
      <article dir="ltr"><p class="no-margin"><i>Check out our <a href="https://apify.com/vaclavrut/sitemap-sniffer" rel="nofollow noopener noreferrer" target="_blank">Sitemap Sniffer</a> &#x2013; it can discover sitemaps in hidden locations.</i></p><p class="no-margin"></p><p class="no-margin">A website&apos;s sitemap contains a list of all of its pages in XML format. We can use it to generate a list of pages to crawl. </p><ol><li>Locate the sitemap, using either the <code>/sitemap.xml</code> path or with the <a href="https://apify.com/vaclavrut/sitemap-sniffer" rel="nofollow noopener noreferrer" target="_blank">Sitemap Sniffer</a>. </li><li>Identify the URLs for the pages you want to scrape and <a href="https://docs.apify.com/tutorials/scrape-websites-using-the-sitemap" rel="nofollow noopener noreferrer" target="_blank">create a regular expression</a> to capture them.</li><li><a href="https://docs.apify.com/tutorials/scrape-websites-using-the-sitemap" rel="nofollow noopener noreferrer" target="_blank">Import</a> the URLs into the Apify SDK&apos;s <a href="https://sdk.apify.com/docs/api/request-list#docsNav" rel="nofollow noopener noreferrer" target="_blank">RequestList</a>.</li><li>Use the created RequestList in <a href="https://sdk.apify.com/docs/api/puppeteer-crawler#docsNav" rel="nofollow noopener noreferrer" target="_blank">PuppeteerCrawler</a> and save the results to your dataset.</li></ol><p class="no-margin"><a href="https://docs.apify.com/tutorials/scrape-websites-using-the-sitemap" rel="nofollow noopener noreferrer" target="_blank">See the full tutorial and code examples</a> in our documentation.</p><p class="no-margin"></p></article>
    </div>
  </div>
    <div class="intercom-reaction-picker" dir="ltr">
  <div class="intercom-reaction-prompt">Did this answer your question?</div>

    <button class="intercom-reaction" data-reaction-text="disappointed" tabindex="0" aria-label="Disappointed Reaction">
      <span data-emoji="disappointed" title="Disappointed">
      </span>
    </button>
    <button class="intercom-reaction" data-reaction-text="neutral_face" tabindex="0" aria-label="Neutral face Reaction">
      <span data-emoji="neutral_face" title="Neutral face">
      </span>
    </button>
    <button class="intercom-reaction" data-reaction-text="smiley" tabindex="0" aria-label="Smiley Reaction">
      <span data-emoji="smiley" title="Smiley">
      </span>
    </button>
</div>

</div>

</section>
</div>
    </body></html>

Article for topic How to scrape a website using sitemap.xml and Apify SDK

Viewed Article

intercom:trackingEvent

https://www.googletagmanager.com/gtag/js?id=UA-67003981-4


        window.dataLayer = window.dataLayer || [];
        function gtag(){dataLayer.push(arguments);}
        gtag('js', new Date());
        gtag('config', 'UA-67003981-4');



    window.intercomSettings = {"app_id":"kod1r788","article_id":"1865554"};



    window.alexandriaArticleId = "1865554";
    window.alexandriaArticleContentId = "338772";


https://static.intercomassets.com/alexandria/assets/application-84072e7a0216dad14cca0527188cde7d5e6872f4bf9ca3efd917ca692507e42b.js