Article Summary
Import content directly from websites into AutoRFP by entering URLs. Choose to scrape specific pages or let AI automatically crawl entire websites to add publicly accessible web content to your knowledge base.
Estimated Time
1-2 minutes (scraping time varies by website size)
Prerequisites
Content Manager or Admin permissions
Website URL(s) you want to import
Website must be publicly accessible (no login or password required)
Step-by-Step Instructions
Step 1: Select Content Type
1. From the Dashboard, click Add Content.
2. Select Documentation as your Content Type.
3. Click Next.
4. On the "Where are you importing from?" screen, click Website.
Step 2: Select Web Scraping Mode
1. Choose your scraping mode:
Specific Pages
Extracts content from exact pages you enter
You control which pages are imported
Maximum 25 pages per import
Entire Site
Automatically navigates through your site following links
Gathers content from all accessible pages
Use for comprehensive site imports
Step 3: Enter Website URLs
1. Enter the full URL (e.g., https://example.com/about).
2. Press Enter to Add to add the URL to the list.
3. If using Entire Site mode, enter only the main website URL - AutoRFP will
automatically crawl through all child pages.
4. Click Next to continue.
Step 4: Categorize Your Content
1. Select or create a Location (folder) for your website content.
2. To create a new location, click Create New Location and enter a descriptive name (e.g., "Company Website Content").
3. To use an existing location, select it from the dropdown list.
4. Click Import to begin scraping.
5. AutoRFP will process the website(s). This typically takes 1-3 minutes for specific pages, or 5-15 minutes for entire sites.
Step 5: Tag Your Content (Optional)
1. After import completes, you'll see the tagging screen.
2. Leave as "All" if the website content applies universally (recommended for most company websites).
3. OR assign Tags if website content is specific to:
Geographic regions
Product lines
Industries
Markets
4. Click Complete Import.
💡Tips & Best Practices
Always include the full URL with https://
Test URLs in your browser first to ensure they're accessible
For Entire Site mode, start with the homepage URL
Avoid password-protected or login-required pages
✋🏼 Common Mistakes to Avoid
Entering incomplete URLs (missing https://)
Trying to import password-protected pages
Forgetting that website imports can't be manually edited later
Selecting wrong Content Type (must use Documentation for websites)
Need Help?
💬 Live Chat: Available in-app
📧 Email: [email protected] or contact your Success Manager directly for urgent support.
📚 Learning Centre: learn.autorfp.ai/en