The Basic Concept:
The true rankings adjust times across festival in an attempt to provide standardization in terms of conditions, boat speeds, and all other extraneous factors that are not related to team performance. Regression analysis is used to calculate the average change in times from one festival to the next, using all teams that competed in both as the sample. This average difference in times between the festivals is then removed – thus providing comparability - as both times are now adjusted to reflect the conditions of only one festival and times are now considered to be representative of “true” team performance. All festival times are adjusted to a master festival in this manner and teams are ranked based on their performance for all races across all the festivals included in the rankings.
In essence, the ranking tries to capture which teams, for an average race, under identical conditions, will be fastest.
How it works:
Let’s suppose we wanted to adjust the Pickering festival results in order to be in line with Toronto Island. Firstly, teams that competed in both festivals are identified and their average times in both are calculated. These teams will be used in our regression. As can be seen there is a clear relation in terms of team performance.
Unlike scientific regressions, we can’t expect (nor do we need) a near perfect R2 value as some teams will likely perform better or worse in one festival when compared to the other. Regardless, 93.3% of the Pickering times can be explained by the times of the same crews in Toronto. The line in this regression represents how much slower or faster a festival might be. If the slope of the line is 45 degrees and the intercept zero then the two festivals are identical in terms of average conditions. Times could be taken from either festival and considered equally representative of team performance.
Now to calculate the adjustment. We now perform the regression of Toronto times, vs. the percentage change of Pickering times (for example, if team A posts an average time of 2:10.0 in Toronto and 2:12.0 in Pickering, then the percentage adjustment is calculated as 98.5% - ie. Toronto was 1.5% faster for team A). This regression looks like this:
The Y-axis consists of the percentage change between Toronto and Pickering times. The X-axis is the Pickering times. The equation of this line (using a second order polynomial - as a curve was found to better fit the data) can now be used to calculate the adjustment for changing Pickering based times into Toronto based times.
(Pic time)*[-458014.518851399*(Pic time)2+1418.8797653647*(Pic time)–.1075239332]
So, using are above example: Team A post an average time of 2:10.0 in Toronto and 2:12.0 in Pickering. Now let’s make both times adjusted to Toronto. The Pickering time becomes:
(2:12)*[-458014.518851399*(2:12)2+1418.8797653647*(2:12)–.1075239332]= 2:10.9
As a result, the true ranking system suggests that team A’s Pickering performance would have been 0.9 seconds slower if they had run their Pickering race under Toronto conditions.
This change in performance can be observed in the above curve. For teams whose speed was roughly 2:12, the average adjustment (i.e. the point of intersection between the 2:12 mark and the curve) is roughly 99%. Since Team A performed 98.5% slower, while their competition performed only 99% slower, then the change in time assessed with performance, and not conditions, is a 0.5%. Correspondingly, the other 1.0% change in time is associated with a change in conditions or boat speed, thus Pickering has slower conditions for boats of this calibre.
There’s the basics. In the actual ranking however, the adjustment is applied to each individual Pickering race (not the average). The second order polynomial was used in place of a liner regression as of 2006, as a curve was found to be a better fit for adjusting the data.
Special Adjustments:
Cross Division Adjustment- Because it is easier to adjust times to the bigger festivals (larger sample size). It made sense to adjust the data in two divisions. The West-Central division, Toronto, Pickering, Sudbury, Welland, were all adjusted to Toronto times. The East-Central division, Montreal, Ottawa, and Lachine, were all adjusted to Montreal times. This was done using the process outlined above. A cross division adjustment was then applied in order to adjust the East Division to the West, once this adjustment is applied, then all times become based on Toronto. All further festivals are incorporated and adjusted directly to these Toronto based times.
Race Distance Adjustment- Because some festivals have races that are different lengths, for example in Welland there are 200m, 500m and 1000m, an adjustment is made to make all races of the same length (in most cases we adjust to be equivalent to 500m times). This was done by using basic regression technique explained above, i.e. comparing the average 200m times to the average 500m times by the same teams.
Boat Adjustment Montreal- Because of the large (admitted) discrepancy in boat times between the two sets of boats used in the 2005 Montreal festival, an adjustment was made in order to make all times based on one set of boats. The same basic process was used comparing the average team times in the fast boat versus the average team times in the slow boat. The fast/slow boats were identified using actual pictures from the festival. In 2006, the boats used by the Montreal festival, though physically different, did not produce any differences in time and so no adjustments were made.
Dropped Race Off Races- Because 200m races can be run more quickly than the more standard 500m race, a 200m based festival can log many more races than a typical festival. As a result, teams that competed in a knock out festival such as Lachine had their final results skewed by the higher proportion of races. As a result, the number of races for a 200m based festival are discounted, though the times for all races are fully included.
The races are weighted by the following:
1-2 actual races = 1-2 true ranking races
3 actual races = 2 true ranking races
4-5 actual races = 3 true ranking races
6 actual races = 4 true ranking races
For example, Team A races 6 races:
Team A 00:53.4
Team A 00:52.7
Team A 00:53.5
Team A 00:52.8
Team A 00:53.6
Team A 00:53.7
Their average time is calculated 53.3. Under the rule for 200m based festivals, the number of races counted towards the true rankings is reduced to four. These times would then be adjusted to Montreal, and later Toronto.
It would be as if Team A actually raced:
Team A 00:53.3
Team A 00:53.3
Team A 00:53.3
Team A 00:53.3
Dropped Team Races- It was established that teams with times that exceeded 3 standard deviations of the average standard deviations for teams between festivals could be reviewed for possible exclusion if it was believed that the team that raced was not actually reflective of the team that would typically be represented. To date this rule has rarely come into play, at most likely once per season or not at all.
Issues:
Sample Size- The largest weakness of the true rankings is sample size. Festivals should have roughly10 identifiable teams that have competed in another ranked festival. In the event that the sample size is deemed to be insufficient, the festival is not included as a true rankings festival.
Conditions- Conditions should be captured to the extent that all teams face the same conditions for a given time slot (for example, a strong headwind all morning) – this will be captured by our adjustment regression. In addition, condition biases are likely to wash out over a large number of races (since all races count, teams that enjoy better conditions in some races are also likely to face poor conditions in others). Conditions are more likely to have an impact for a team that only does a few races in conditions that are highly variable.
Boats- Different physical boat speeds of Dragon Boats and resulting spreads should be captured in the adjustment regressions.
Lanes- Because all races are included in the true rankings, it is very unlikely that teams will significantly benefit from lane bias. This is particularly true for teams that log a large number of races. In the event that a festival believed to be particularly biased, depending on it’s significance, it may be excluded from the true rankings.
General Team Improvement- No bias is introduced to the ranking if in general teams improve throughout the season at a roughly similar rate. The rankings are in a sense relative, and you need to improve as much as the next team in order to produce comparable times. However, if for example, teams in the East-Central region improve faster than the teams in the West or vice-versa, then the adjustments will be slightly skewed since they are based on cross festival team performance. It is unlikely that this is a significant issue.
Wash Riding and Stacking- Teams that achieve a faster time by wash riding or stacking will be ranked faster. However, if stacking results in a non-representative crew, then the three standard deviation rule described above may come into play.
Dogging it- Teams that dog it in standard races and then ramp it up for the final will be ranked lower as all races are included in the ranking.
Closing the Gap- Towards the end of the season, it has often been found that teams that rank towards the middle of the rankings often close the gap with the top teams by improving at a relatively faster rate. This may introduce some bias by making the top crews appear relatively slower. However, the introduction of the polynomial regression (using a curve), as described above, seems to have greatly reduced this effect.
Timing- As a result of the methodology, the ranking captures average annual performance for teams, adjusted to the time, boats, average lanes, and conditions of the Toronto Island Festival. Thus any improvement in teams performance is a relative to these variables.
International or out of Area Teams:
International teams and teams outside of Ontario and Quebec will not be ranked unless one of the following two conditions applies.
1) The team has competed in a “true ranking” festival for two consecutive years.
2) The team has competed in two or more “true ranking” festival in a given year.
This provision was put in place to prevent teams that show up periodically from being ranked, adding to the stability of the rankings from year to year.
Questions, Comments, of Info:
Any questions regarding the rankings can be addressed to:
If your team is not ranked in the top 250 and you would like their results please feel free to email. Any teams that did not paddle with normal crew and are likely 3 standard deviations off their other times should bring it to my attention.
Tyler Minty