IESO Coincident Peak Prediction — Data Acquisition & Exploratory Analysis

This notebook fetches 15 years of IESO hourly demand data and Open-Meteo historical weather for Toronto, merges the datasets, and performs comprehensive exploratory analysis of Ontario's peak demand patterns.

Data sources:

Load Local ICI Demand CSVs (2022–2025)

These files cover ICI base periods (May 1 – April 30) and have 3 metadata header rows starting with \\. Format: Date, Hour, Ontario Demand with hour-ending convention (1–24 EST).

Fetch IESO Historical Demand (2010–2021)

The IESO publishes calendar-year hourly demand at: https://reports-public.ieso.ca/public/Demand/PUB_Demand_[YYYY].csv

Format: Date, Hour, Market Demand, Ontario Demand with 3 metadata header rows.

Combine into Unified Demand Dataset

Merge local ICI demand (2022–2025) with remote IESO demand (2010–2021) into a single time-indexed dataframe. Create a proper datetime index in EST using IESO's hour-ending convention (Hour 1 = midnight to 1 AM).

Load Ground Truth Peak Data

Read the user's spreadsheet files containing:

  1. Top-10 peaks for 13 base periods (2012–2025) — Date, Hour, Demand (MW)
  2. Top-5 peaks with system-wide consumption for 15 base periods (2010–2025) — includes AQEW

Fetch Open-Meteo Historical Weather

Fetch hourly weather data for Toronto (43.65°N, -79.38°W) from the Open-Meteo historical archive API. Variables: temperature_2m, relative_humidity_2m, dewpoint_2m, wind_speed_10m, shortwave_radiation.

The API accepts date ranges up to ~1 year per request, so we batch by year.

Timestamp Alignment & Merge

Critical alignment: IESO uses hour-ending convention (Hour 1 = 00:00–01:00 EST, timestamp at 01:00). Open-Meteo uses hour-beginning convention in local timezone (hour 0 = 00:00–01:00, timestamp at 00:00).

To align: IESO Hour 1 at 01:00 → Open-Meteo hour 0 at 00:00. So we shift IESO's datetime back by 1 hour, or equivalently, shift Open-Meteo forward by 1 hour.

EDA: Annual Peak Demand Trend (2010–2025)

How has Ontario's peak system demand evolved over 15 years? The trend reflects embedded generation growth, conservation programs, and structural demand shifts.

EDA: Monthly Demand Box Plots

Demand follows a strong seasonal pattern — summer cooling load drives the highest demands, with a secondary winter heating peak in some years.

EDA: Hourly Demand Heatmap

Average demand by hour-of-day and month reveals the afternoon summer peak cluster where coincident peaks concentrate.

EDA: Peak Analysis

Distribution of actual top-5 coincident peaks by month, day-of-week, and hour-of-day across all available base periods.

EDA: Weather-Demand Relationship

The physical basis for prediction: daily maximum temperature vs. daily maximum demand. Above approximately 22°C, AC cooling load drives demand nonlinearly upward.

Save Cleaned Dataset

Export the merged demand + weather dataset as parquet for downstream notebooks. Also save the ground truth peak labels separately.