Ask whether collection is necessary
Suppose you want to compare publicly listed course schedules across several training providers. Start by checking whether an official API, downloadable dataset, licensed feed, or permission-based export already exists. A supported interface is often more stable and clearer about permitted reuse than parsing pages intended for people.
Write the purpose, required fields, update frequency, retention period, and intended publication. Collect only what the analysis needs. Course title, start date, and public price may be enough; instructor biographies and personal contact details may add privacy and copyright concerns without improving the answer. Public accessibility is not the same as unrestricted permission to collect, retain, or redistribute information.
Review access rules and reuse conditions
Read the site's terms, relevant licenses, API conditions, and robots.txt rules before implementing a collector. If the intended use is unclear or sensitive, request permission or obtain appropriate legal guidance. Do not bypass authentication, paywalls, CAPTCHAs, or explicit access restrictions to complete a learning exercise.
The Robots Exclusion Protocol is standardized in RFC 9309, which explicitly states that robots rules are not access authorization. [1](https://www.rfc-editor.org/rfc/rfc9309.html) An allowed path therefore does not grant a license to republish its content, while an absent robots file does not erase other obligations. Treat technical crawling instructions and legal permission as separate checks.
Scope the collector conservatively
Create an allowlist of domains and URL patterns. For the course project, restrict collection to schedule pages rather than recursively following every link on each site. Set a maximum number of pages, run duration, response size, and redirects. Use an identifiable user agent and a contact method when appropriate.
Google's robots documentation explains that rules apply to the host and location where the file is served and describes the basic rule structure. [2](https://developers.google.com/crawling/docs/robots-txt/create-robots-txt) Use a standards-aware parser instead of improvised string matching. Recheck the applicable rules when needed, and stop for review if the site changes behavior or explicitly objects to the collection.
Respect capacity and error signals
Start with low request concurrency and a conservative interval between requests. Cache previously fetched responses and avoid downloading unchanged material unnecessarily. Respect documented rate limits and applicable Retry-After guidance. A server returning rate-limit or access-denied responses is not an invitation to rotate identities until it answers.
Set bounded retries with backoff for transient failures and a clear stop condition for repeated errors. Never create an unbounded loop around a failing URL. Save enough diagnostics to understand what happened without retaining unnecessary page content. Test against a local fixture first so parsing mistakes do not require repeated requests to a live service during development.
Validate extracted values and preserve provenance
Store the source URL, retrieval time, extraction version, and relevant context with each record. A price may exclude tax, apply only to one region, or be valid for a particular enrollment date. Preserve those qualifications rather than converting every visible number into a supposedly comparable price column.
Build parser tests using small permitted fixtures and include missing dates, alternate currency formats, and changed page structures. Quarantine ambiguous records for review. Do not silently fill missing values with guesses. If you publish an analysis, distinguish the observed page content from conclusions you derived, and link to sources where appropriate without reproducing more copyrighted material than your rights permit.
Plan correction, retention, and shutdown
Define how you will respond if a source requests removal, changes its terms, or corrects a published value. Apply a retention schedule to raw responses and derived records. If personal information is involved, review the applicable privacy requirements and individual-rights processes before collection begins rather than after a complaint.
Publish the methodology, collection dates, coverage gaps, and known extraction limitations with the project. Do not imply that your sample represents every course or the latest available price. A responsible collector is easy to stop, narrow in scope, and auditable. Completing the dataset is less important than respecting the boundaries under which the information was made available.
Sources & further reading
Last editorial review: 27 October 2026. Examples are educational, not live employer data or guaranteed outcomes.