P1 case test split

Practice scenario (written by Spine, not an IB case study):

Larkford is a small city with a community bike-share called LarkBikes. It has 40 docking stations, each with between 10 and 20 docks. Each dock has a sensor that detects whether a bike is locked in it and reads the bike's ID tag. Every station sends its sensor readings to a central server over the mobile phone network every 5 minutes.

Riders register in the LarkBikes app with their name, email address, home postcode and date of birth. When a rider unlocks a bike, a trip starts. The server keeps riders, bikes, stations and trips in a relational database. Each trip record holds the rider, the bike, the start station, the end station, and the start and end times. The database now holds two years of trips.

Every evening, a machine learning model is trained on the trip records together with the hour of the day, the day of the week, whether it is a school holiday, and the weather. For each station, it predicts how many bikes will be taken in each hour of the next day. A van crew uses these predictions to move bikes to stations that are expected to run out.

The 28 stations in the city centre are busy. The 12 stations in the outer districts were added last year and record far fewer trips.

A data analyst trains the model on all two years of trips. She then measures its error on the same trips and reports that its predictions are wrong by only 0.3 bikes per hour on average.

(a) Explain why this figure does not show how well the model will predict tomorrow's demand. [2]

(b) A colleague suggests holding back the most recent two months of trips, training on the rest, and measuring the error on those two months. Explain why this is a better test for this model than holding back a random selection of hours (each station's count for those hours). [2]

[4]