An end-to-end automation that collects job listings from 300+ career sites, filters duplicates, reads each posting with AI, and files complete records into a database. The client's team once did every step by hand. Two months after launch, monthly output had more than quadrupled.
Four chores, done entirely by hand for every batch. Here is what the same work looks like now.
Open 300+ career sites one by one, sometimes with a VPN, and copy paste every link into a sheet.
Search the database for every collected link. Most turn out to be repeats.
Open every surviving link and copy 10+ fields per job, one field at a time.
Hunt the exact address in a masterlist, job by job. The most tedious step of all.
One Google Sheet is the control panel. Every step below starts from its menu.
One click collects every job link from all 300+ career sites, allowed-state jobs only.
The sheet flags repeats by colour, then the Main Filter lets only truly new links through.
AI opens each job page and fills in every field, one record every 30 seconds.
The exact address fills itself from a 95,000-row reference database.
Clean, complete records land in the database. Slack pings me if anything fails.
The most painful chores were automated first. Three months in, the Link Collector closed the final gap.
Opening each career site by hand, sometimes with a VPN, and copy pasting every link one by one. Each company has its own rules about which US states are allowed.
300+ sites means 300+ small puzzles: different platforms, pagination, bot protection, region locks. Each one gets its own handler without touching the rest of the pipeline.
The seven-step journey from a company career page to a clean, address-verified record in Airtable.
One click gathers every job link from 300+ career sites, allowed-state jobs only.
Colour-codes duplicate links, gives each row a unique tracking ID, and sends the clean batch to the next step.
Every link is checked against the full database. Only truly new ones join the queue.
Tries three ways to load each page in order, from free to paid, escalating automatically only when needed.
AI pulls out the title, pay, location, and requirements. Low-pay postings are skipped before AI even runs.
Partial addresses are completed from the reference database.
The AI's output is double-checked, then the clean record saves to Airtable.
A 90-second run: duplicate flagging in the sheet, the filters and scraper in n8n, and records landing in Airtable.
The guardrails built into the pipeline.
Every link gets a permanent tracking ID, so updates always land on the right record.
Repeats are caught in the sheet first, then against the full masterlist. No duplicate reaches the queue.
Taleo, ADP, UKG, Dayforce, iCIMS, Greenhouse, Workable and more, each fetched its own way.
The client can bump any company to the front of the next run and watch progress live in the sheet.
Each page is tried the free way first. Paid tools step in only when a page blocks them.
AI reads the posting and fills the fields. A second step checks the result before it saves.
Partial addresses are completed automatically from a 95,000-row reference database.
Obviously low-pay postings are skipped before the AI runs, so it never spends on a dead end.
A new run exits if the previous one is still going, so nothing is processed twice.
Screenshots of all workflows, showing every step, branch, and error path in the canvas.
From the sheet menu, it walks all 300+ career sites and collects every job link in one run, even pages listing 999+ openings.
Close to 40 site types, each fetched its own way. Bot-protected sites go through a scraping service, replacing the manual VPN.
Only allowed-state jobs are kept, new links land in the sheet, and each company's visit date is stamped automatically.
Four colours mark four kinds of repeat, from already-processed links to same-day resubmissions, so every row's state is visible at a glance.
Every new row gets a permanent tracking number the rest of the system uses to follow it.
One click sends all pending links to the next step and re-sorts the sheet, oldest first.
Reads the submitted links, strips tracking junk from each URL, and de-duplicates the batch.
Each link is checked against the full job database. If it was submitted before, the sheet row is updated to show that. If it is new, it moves forward.
New links join the processing queue and the row is marked Transferred. Any error marks the row and pings Slack.
Checks that no previous run is still going, then rewrites platform links into a directly fetchable format.
A standard request first, a rendering tool if the page needs JavaScript, and a paid tool only as a last resort.
Obviously low-pay postings are skipped at no cost. Everything else is read by AI, field by field.
Extracted salaries are sanity-checked. Records below the pay threshold are marked skipped, not qualified.
Each record saves as Qualified, Salary Issue, or Error. Incomplete addresses are flagged for the Find Address workflow.
Every skipped record sends a Slack message with the reason. Qualified records save silently.
Finds all records in Airtable that the scraper flagged as having an incomplete address and processes them one by one.
Breaks the partial address into pieces and searches the 95,000-row database for companies that match by name, returning the top candidates to score.
Candidates are ranked by closeness of match, with a backup lookup if nothing hits. Worst case, the original partial address is kept.
Reads the master address sheet and prepares every row for upload.
Uploads in batches. Existing rows are skipped, so re-running never creates duplicates.
Reports how many rows were added and how many skipped. Run it whenever the masterlist changes.
Reads the companies list in the sheet and fetches the matching records from the shared database, so the two can be compared in one pass.
Compares the two and updates only the records that actually changed, keeping the sync fast and avoiding unnecessary writes.
A small companion to the Link Collector: the client's team always sees fresh last-updated dates for every company, with no manual logging.
Every tool in the pipeline, and exactly where AI fits in.
In the pipeline: an AI model (DeepSeek, via OpenRouter) reads job pages and fills in the fields. Every AI output passes rule checks before it is saved.
In the build: I used Claude (Anthropic) as an assistant for drafting code and copy. The architecture, decisions, testing, and client work are mine, and I review everything the AI touches before it ships.
Completed job links are the team's core monthly output number. The automation went live at the start of May.
The scraping service behind the collector was quietly billing every request at its most expensive tier. An audit caught it and cut scraping costs by 42%, with no change to coverage or results.
Open to automation projects, contract work, and full-time positions.