About the Human job-data crawler
If you found this page in your server logs, a request from our job-data crawler reached your site. This page explains who we are, what the crawler does, and how to slow it down, stop it, or have a posting removed.
The crawler identifies itself in every request with the User-Agent token jobdata-crawler
jobdata-crawler/0.1 (+https://meetthehuman.com/crawler; contact support@meetthehuman.com)Who operates it
The crawler is operated by Human (HumanHQ LLC), the company behind meetthehuman.com. Human is a hiring platform that helps candidates find open roles and helps employers hire.
Questions, complaints, and requests about the crawler go to support@meetthehuman.com
What it collects
The crawler collects publicly available job postings so that candidates on Human can discover open roles, including roles that offer relocation or visa support.
- It only reads job postings that are already public on the open web, from public applicant-tracking-system job boards and their public APIs (for example Greenhouse, Lever, Ashby, Workable and Recruitee) and from licensed job-data APIs.
- It does not collect personal data of candidates or applicants. It never reads application forms, applicant profiles, or anything behind a login.
- It does not create accounts, log in, submit forms, or execute JavaScript. It sends plain HTTP GET requests only.
- Each posting is stored as published, so we can tell when a role opens and when it closes. That history is what keeps our listings accurate.
How it behaves
- It fetches robots.txt with its own User-Agent, caches it per site for 24 hours, and honors it. A URL your robots.txt disallows for this crawler is never requested.
- It respects Crawl-delay. Whenever your robots.txt asks for a slower pace than our default, the slower value wins.
- It sends at most one request every two seconds to any single domain. Requests to the same site are spaced out, never sent in bursts.
- If your robots.txt cannot be retrieved because of a server error or a timeout, it does not crawl your site. Silence is not treated as consent.
- It always identifies itself with the User-Agent shown above and never impersonates a browser or another crawler.
How to opt out
To block the crawler from your entire site, add these two lines to your robots.txt:
User-agent: jobdata-crawler
Disallow: /The crawler re-reads robots.txt at most every 24 hours, so a new rule takes effect by the next collection run after that.
To keep the crawler but make it slower, set a Crawl-delay in seconds instead. For example, one request every ten seconds:
User-agent: jobdata-crawler
Crawl-delay: 10You can also email support@meetthehuman.com with your domain and we will exclude it by hand. Messages about the crawler are routed to an engineer, not to a canned reply.
Request removal of a posting
To have a specific posting removed from Human, email support@meetthehuman.com with the URL of the posting. We will remove it from our listings and stop collecting it.
Removal requests from the employer that published the posting, or from the operator of the site it was collected from, are handled first. Please mention which one you are.
