ETL & Web Scraping

A supporting Python pipeline collects official FIS competition results, converts them into a consistent structure, and loads them into PostgreSQL for analysis.

The pipeline is intentionally small and focused. Its role is to build a reliable dataset for the SQL statistics rather than serve as a separate data platform. It can be run from the terminal through main.py, while the required PostgreSQL credentials are documented in .env.example.

Pipeline overview

Processing a range

The scraper works through an inclusive range of FIS race identifiers configured in config.py:

# first race ID to process (inclusive)
START_RACE_ID = 7720

# last race ID to process (inclusive)
END_RACE_ID = 7720

# pause between requests to avoid rate limiting
DELAY_SECONDS = 1

Keeping the range in version control provides a simple record of the last processed identifier and makes it easy to resume collection for the next season. A delay is applied between pages to limit the request rate.

For every identifier, Selenium opens the corresponding result page:

url = (
    "https://www.fis-ski.com/DB/general/results.html"
    f"?sectorcode=JP&raceid={race_id}#details"
)

The result table is loaded dynamically after the initial HTML document. Before parsing the page, the scraper therefore waits until the loading placeholder is no longer visible:

wait.until(
    EC.invisibility_of_element_located(
        (By.CSS_SELECTOR, ".loading-placeholder")
    )
)

Extract, transform, load

Extract

A Selenium WebDriver runs Chrome in headless mode and retrieves the rendered FIS page. The requested race_id is first classified so that missing, cancelled, or out-of-scope events can be skipped without treating them as pipeline failures.

Transform

Beautiful Soup parses competition metadata and the round-by-round result table. The extracted values are normalized and checked against their expected types before they are represented as Competition, Jump, and JumpResult objects.

The filter retains only men’s individual World Cup competitions and their rated rounds. Women’s events, team competitions, other competition series, qualifying, training, and trial rounds do not enter the analytical dataset.

Load

SQLAlchemy uses the PostgreSQL dialect to insert the competition, any previously unknown jumpers, and their jump results. All records belonging to one competition are written in a single transaction, preventing a failed load from leaving a partially inserted event.

The pipeline expects controlled, non-overlapping identifier ranges. Reprocessing an already loaded competition is therefore left to PostgreSQL’s unique constraint to reject rather than being handled with an upsert.

Skips and failures

Not every race_id is expected to produce data. The script distinguishes ordinary gaps in the FIS sequence from failures that could compromise the dataset.

Every processed identifier and its status are printed to the terminal. When an unexpected error stops the run, the log identifies the exact race_id that needs to be investigated.

An exceptional competition

race_id = 6931 is the only three-round World Cup competition in the collected range. FIS did not expose its results through the standard HTML table, which would normally leave a gap in the dataset.

For this single case, the results were extracted from the official PDF with AI assistance and stored explicitly as JSON in the project. Keeping the exception as data rather than adding special parsing rules preserves the standard pipeline for every other competition and makes the workaround easy to locate.

Database as code

The database lives in the same repository as the collection pipeline. Flyway migrations version its schema, while saved analytical queries and a PostgreSQL dump make the database structure and core analysis reproducible.

The complete source is available on GitHub.