DataHen documentation

Table of Contents

• DataHen Platform
  • Getting Started
    • Install DataHen Command Line Interface using rubygems
    • Get your access token
    • Set environment variable of your access token
    • Create the scraper
    • Deploying the scraper
    • Run the scraper
    • Viewing the Job Stats
    • Viewing the Job Pages
    • Viewing a Job Page Content
    • Viewing a Global Page Content
    • View the scraper output
    • View the scraper logs
• High Level Concepts
  • Scrapers
    • Available commands
  • Global Pages
    • Available Commands
  • Jobs
    • Available Commands
    • Paused Jobs
  • Job Workers
  • Job Pages
    • Available Commands
    • Search pages
  • Job Outputs
    • Available Commands
  • Job Exports
    • Available Commands
  • Job Stats
    • Available Commands
  • Job Error Logs
    • Available Commands
  • Parsers
    • Reserved words or methods in parser scripts:
    • Available Commands
  • Seeder
    • Reserved words or methods in seeder scripts:
    • Available Commands
  • Finisher
    • Reserved words or methods in finisher scripts:
    • Available Commands
  • Exporters
    • Available Exporter Commands
    • Available Export Commands
    • Automatically Start Exporters
    • JSON Exporter
      • JSON Write Modes
    • CSV Exporter
      • CSV Fields
        • Scientific Notation
        • Forcing Double Quotes Enclosure
    • Content Exporter
      • Exporting Failed Contents
      • Customizing the File Names
  • Job Task
    • Available Commands
    • Task list Options
    • Task show Options
      • Job
  • Resource Settings
    • Available Commands
    • List Options
  • Schemas
    • 1. Create the json schema file
    • 2. Create the schema config file and list the schema file that will be used to validate the collection(s)
    • 3. Update your config.yaml file to include the schema config file.
• User Access
• Scraper Development workflow
• Scraper Maintenance workflow
• Coding Tutorials
  • Simple Ebay scraper (Ruby)
    • Exercise 1: Create a base scraper locally
    • Exercise 2: Run the scraper on DataHen
    • Exercise 3. Create parser script
    • Exercise 4: Enqueue more pages, and pass variables to the next pages
    • Exercise 5: Exporting outputs
    • Exercise 6: Create a finisher script
• Advanced Tutorials
  • How to write a QA script to ingest and parse outputs from multiple scrapers
    • Step 1: Setup
    • Step 2: Seeding
    • Step 3: QA Validation Rules
    • Additional Validation Rules
    • Group Validations
    • Thresholds
    • External sources
  • How to handle duplicate pages from different categories
• How-Tos
  • Setting scraper’s mode
    • Single job mode
    • Multiple jobs mode
  • Setting a scraper’s scheduler
    • Prevent scraper’s scheduler from cancel an existing job
  • Enabling global page cache
  • Changing a Scraper’s or a Job’s or a JobPage’s Proxy Type
  • Changing a Scraper’s or a Job’s Profiles
  • Account deploy key
  • Setting a specific ruby version
  • Setting a specific Ruby Gem
  • Changing a Scraper’s Standard workers
  • Changing a Scraper’s Browser worker count
  • Changing an existing scrape job’s worker count
  • Enqueueing a page to Browser Fetcher’s queue
  • Setting fetch priority to a Job Page
  • Setting a user-agent-type of a Job Page
    • Disable user agent override
  • Setting the request method of a Job Page
  • Setting the request headers of a Job Page
    • Disable default request headers
    • Use custom headers
  • Setting the request body of a Job Page
  • Setting the page_type of a Job Page
  • Reset a Job Page
  • Handling cookies
    • Low level cookie handling using Request/Response Headers
    • High level cookie handling using the Cookie Jar
  • Force Fetching a specific unfresh page
  • Handling JavaScript
  • Max page size
  • Browser display
  • Browser interaction
    • Enqueue same page twice with different code
    • Generate GID without Enqueue a page
    • Intercept request
    • Executing puppeteer code before fetch is done
    • Sharing data between pre_code and code
    • Enabling browser images
    • Disabling browser Adblocker feature
    • Change browser fetch behavior
    • Dealing with responsive designs
    • Dealing with infinite load timeouts
    • Distinction of pages() vs newPage()
  • Taking screenshots
    • Screenshot options
    • Screenshots and browser interaction
  • Doing dry-run of your script locally
  • Executing your script locally, and uploading to DataHen
  • Querying scraper outputs
    • Querying basics
    • Querying from another Job or Scraper
    • Paginations and Performance Considerations
    • Logical operations
    • Comparison operations
    • Ordering Outputs by field(s)
  • Restart a scraping job
  • Setting Variables and Secrets to your Account, Scrapers, and Jobs
    • Setting Environment Variables and Secrets on your account.
      • 1. Set the environment variable or secrets on your account.
      • 2. Change your config.yaml to use the variables or secrets.
      • 3. Access the environment variables and secrets in your script.
    • Setting Input Variables and Secrets on your scraper and scrape job.
      • 1. Start a new job with input vars.
      • 2. Set the input variable or secrets on your scraper.
      • 3. Change your config.yaml to use the variables or secrets.
      • 4. Access the input variables and secrets in your script.
  • Using a custom docker image for the scraper
    • Using browser fetcher custom images
  • How to use shared code libraries from other Git repositories using Git Submodule
  • How to debug page fetch
    • no_url_encode: true
    • http2: true
    • response headers and request headers are different
    • bzip compression headers
  • Advanced Usage
    • Parsing Failed Responses
    • Exclude pages from a job