MassTimes Church Directory Scraper
Overview
A Python scraper that extracts church information and mass timings from masstimes.org. It iterates through US ZIP codes to work around the site's missing search, runs in parallel for speed, and stores results as JSON files synced to Google Drive. Outcome: Produced a complete, structured dataset of US churches and mass times that powers the client's downstream directory product.
Architecture & Pipeline
flowchart LR
n0["US ZIP CodesSearch substitute"]
n1["masstimes.orgSource site"]
n2["Parallel ScrapersPython · Requests"]
n3["Rotating ProxiesRate-limit safety"]
n4["JSON per ChurchMass times + metadata"]
n5["Google Drive SyncFinal dataset"]
n0 --> n1
n1 --> n2
n2 --> n3
n3 --> n4
n4 --> n5
classDef step0 fill:#f1f5f9,stroke:#64748b,color:#1e293b,stroke-width:2px,rx:10,ry:10;
classDef step1 fill:#ecfeff,stroke:#06b6d4,color:#1e293b,stroke-width:2px,rx:10,ry:10;
classDef step2 fill:#f0fdfa,stroke:#0d9488,color:#1e293b,stroke-width:2px,rx:10,ry:10;
classDef step3 fill:#ecfdf5,stroke:#10b981,color:#1e293b,stroke-width:2px,rx:10,ry:10;
classDef step4 fill:#fffbeb,stroke:#f59e0b,color:#1e293b,stroke-width:2px,rx:10,ry:10;
class n0 step0;
class n1 step1;
class n2 step2;
class n3 step2;
class n4 step3;
class n5 step4;
End-to-end flow derived from this project's scope and tech stack. Tap View Fullscreen for a larger view, or scroll horizontally on small screens.
Key Features
- Search driven by ZIP codes to work around the missing site search
- Parallel processing for fast nationwide coverage
- JSON output with Google Drive sync
- Rotating proxies to avoid rate-limiting
- Scheduled runs from an Ubuntu server
- Tech Stack: Python, Requests, BeautifulSoup, Linux