The purpose of this project is to build a remote job board that serves my needs. I am currently a software engineer at a company where I enjoy working but not long ago, I used to struggle finding remote job opportunities. I used to browse a few job boards in the past, but I found them to be quite limited in terms of the type of jobs they offer, especially how they treat remote jobs based on the location limitations like “only US based jobs” or “only jobs in Europe” or “visa required”. So, I am building this job board that makes the process of finding remote jobs easier.

Sources (as of now)

1. otta.com
2. 80000hours
3. remote.com
4. weworkremotely.com
5. workingnomads.co

Setup

Database: PostgreSQL
Backend: Python, Django
API: Django REST Framework
Frontend: Vue.js
Scraping: BeautifulSoup, requests, selenium Proxy IP: Webshare Queue: Redis
Worker: Celery

Data Collection

Every midnnight, the sources mentioned above are scraped and the data is stored in the database. To avoid being rate limited, the scraping is done with rotating proxy IP addresses. I have used Webshare for this purpose. It is a cheap option and I pay only $3/month for 100 rotating proxy IP addresses. It provides me with an API which I can use to rotate the IP addresses on each request.

• Use of database triggers to scrape jobs:

Periodic Jobs

I have used Celery to run periodic jobs. I have a few periodic jobs that I run every day.

• Scraping jobs from sources
• Cleaning up the database