Consistency in 2.1 engine

This forum should be use to discust players ratings. / Ce forum devrait être utilisé par discuté des côtes des joueurs.
hscommish
New in Town / Le Ptit Nouveau
Posts: 35
Joined: Sat Nov 07, 2020 2:41 pm

Re: Consistency in 2.1 engine

Post by hscommish »

Very interesting thread! Our board has decided to look into creating our own ratings for engine 2.1, would it be possible to get in on some of this research? We have no idea what we’re doing haha
Savard
The Accomplished One / L'Accompli
Posts: 119
Joined: Fri May 24, 2019 1:07 am

Re: Consistency in 2.1 engine

Post by Savard »

I finished my 2.1 ratings, and we will be using them in the UHL to see how they do in a "real" league (as opposed to the test environment). The tests were very promising, also when I first tested them on our league's rosters.

So I think 2.1 can provide great (as in realistic) results, it comes however with a few catches:

- it is very labour intensive to create those ratings. I didn't count the hours, but it's probably hundreds. I am not sure if I will be doing this again next season (even though much of the research is already done, the Excel sheets and R scripts are there)

- the "inverted" SK and the way the PH, PA and SC interact is a tough pill to swallow for some GMs. Ratings aren't as easily readable as those for other engines. Jornan Martinook for example ended up with a 97 in PA. That high rating was necessary to somewhat model the fact that he has a lot more assists than goals. He will not have more assists than Pastrnak (85 PA or so), his low SC and PH will ensure that, but it's nor very intuitive.

- as a result of the above, some OV were completely skewed. I ended up making my own OV formula in Excel (more complex than what you can do in STHS), put that result in the PO column (which we don't use anyway), and use that as the OV.

- Some types of players are impossible to model. For example Ovechkin who had a lot more goals than assists, or on a lower level Dominik Kubalik. It was the same for some of those extreme playmakers. In the end I had to modify my goal and assist targets for some players. That may be in part because of the averaging method I used, but I think it is imposible to reliably recreate very high (or very low) goal to assist ratios.

I would love to give out my 2.1 ratings for you guys to test, but some of the ratings (FG, ST etc.) I simply took from the BRHL 2.0 ratings and those are payware.
Président ANHS
The Addict / Le Drogué
Posts: 508
Joined: Tue Nov 29, 2005 2:30 am
Location: Summerside, PEI, Canada
Contact:

Re: Consistency in 2.1 engine

Post by Président ANHS »

The problem Commishes are still not understanding is the fact that no matter how high the ratings are for certain attributes, it does not guarantee a "number of stats".

PA and SC ratings will determine the rate at which the player will do the action of passing or shooting one over the other.

If your player has 85 PA and 85 SC, he will have a hard time figuring out what he does on the ice because when he has a decision of shooting or passing, he has a 50/50 chance of doing either.

If your player has 85 PA and 50 SC, you can be sure he will most definitely have a lot more assists than goals.

A player with 40 PA, 60 SC and 80 PH has just about the same amount of chances to end up the season with a player who has 65 PA, 85 SC and 80 PH. I've has players with ratings like those end up with the same number of goals at the end of the season...
Image
Steff Thouin
Président ANHS Commish
www.anhs.qc.ca

21 ans / 30 saisons de simulation
21 years / 30 sim seasons
Savard
The Accomplished One / L'Accompli
Posts: 119
Joined: Fri May 24, 2019 1:07 am

Re: Consistency in 2.1 engine

Post by Savard »

What Steff writes is not wrong, but it goes even beyond that. In my experience SC does influence assists a lot and PA influences goals a lot, which is why it is so hard to recreate Ovechkin. His high SC will result in too many assist (possibly from rebounds).
36Henry
The Accomplished One / L'Accompli
Posts: 118
Joined: Wed Dec 15, 2010 1:58 am

Re: Consistency in 2.1 engine

Post by 36Henry »

At some point several years ago while I still had time to spend on this wonderful stuff I dug deep into the play-by-play data the Sim provides. The aim was to create essentially a visual simulation of the games, sort of similar to this:

Image

Where all events would be logged and located on the ice. The locations would be based on actual NHL data providing tendencies for areas where for example Ovechkin shoots from. Was a pretty cool project that I sadly never got around to finish.

Another example would be Patric Hornqvist who does most of his work in front of the net. Compiling all data of his shooting, hitting, scoring and shot blocking into a huge database would be the base upon which the application would build. Whenever Hornqvist registered a shot on goal in the Sim, the application would randomly place the shot somewhere on the ice, within the areas where Hornqvist most frequently shoots the puck in real life. The idea being to bring the simulation to life in a more vivid way than just reading the score from the boxscore.

It also would provide data for advanced stats like Corsi and so on, since all the information needed to track these stats are available in the play-by-play data. You just need to process that information to extract what you need to provide accurate advanced stats of pretty much every kind. Wish I had time to finish that project lol.

Anyway, back on topic. What that project working on the play-by-play data did was highlight how goals are scored. Many of them are scored from sustained pressure and off of rebounds. Now, this may certainly be down to a combination of factors such as defensemen in my ratings being too bad at clearing the front of the crease or goalies being too good at stopping the initial shot but too bad at giving up rebounds, although I did calculate rebound percentages for all goalies both in the NHL and in the Sim and they were quite similar. Rebound percentage in the NHL was one of the main contributing stats for creating the Rebound control rating for goalies, naturally.

As for the relationship between SK, PH, PA and SC it has many layers to it. Essentially the add up to a number you could refer to as a skill-stat (maximum skill-value would be 396).

We can have a conversation about this using two players (bear in mind it's been years since I actually played around with the Sim, so my theories may be slightly off lol):

Player A: 70 PA, 90 PH, 90 SC and 30 SK has 280 skill-points.
Player B: 70 PA, 70 PH, 70 SC and 70 SK has 280 skill-points.

While we can all recognize that Player A is a much more skilled player, they have the same amount of skill-points. Given an equal amount of ice-time, they are therefore likely to produce about the same amount of points.

While high SK makes the player more prone to just hold on to the puck instead of advancing play through creativity, my impression of the simulation is that he will still factor into the play and be fairly effective given enough ice-time due to a high total value of skill-points.

Essentially I believe points are awarded based on a lottery of sorts. The players on the scoring team are awarded points randomly, and the higher the skill-points in relation to his linemates, the greater the chance of that player getting a point.

When creating ratings, one thing to look at is the distribution of goals between forwards and defensemen. If memory serves this typically is around 16-17% in the NHL, so that is a target to aim for when creating ratings. This is done easily in all engines available in the STHS, with the 2.1 being no more difficult than the others. And to do this you do not need to reduce players down to 40-50 SC. If you are doing that, you are doing everything wrong.

Typically in the ratings I used to make I would have the top goal scoring forwards at 99 SC and then the rest would follow on a sliding scale down to around 70-72 for the lowest scoring forwards (think John Scott or Justin Johnson, remember his NHL debut?). This essentially means all regular NHL forwards are rated somewhere in the 80-99 range. For regular defensemen the range typically sits from about 70-78, so a very narrow range. That narrowness is key for the 2.1 engine. If the distance between star players and the plebs is too great, the top players will score huge amounts of points while the rest wont get a sniff.

PH is a very useful tool in creating separation between top players and lower level guys, and SK also has a big room for tweaking things, such as for example getting Ovechkin to score more goals than his peers. Every year when I went through rigorous testing (hundreds of hours is indeed a very accurate measurement) I would always have teams like Washington frustrate me since they would be terrible due to Ovechkin not scoring enough goals when being rated using only the formulas. I would have to go in and manually boost his SK and PH in order to get him to produce at the level he is supposed to. Going through and tweaking individual players like this on every team is time consuming but ensures a very good set of ratings.

This highlights the power of SK and the nuances of it. While the framework of it being an inverse value of Creativity certainly works and is a very good model for how the 2.1 engine functions, you still have to be very methodical and detailed when creating the ratings in order for them to be as good as they can be. When I was testing my ratings I used NHL rosters and accurate lines (including special teams) to run hundreds of season with tweaks of individual players between each session. Changing SC on one player from 85 to 86 can have a huge impact on team performance, so it is a slow and lovely process which I often miss.

Savard is absolutely right in the connection between SC and PA. These can have a massive impact on performance when adjusted. PH and SK are much more forgiving and can be very useful to tweak individual performance.

Sorry for the rant :lol:
Image
hscommish
New in Town / Le Ptit Nouveau
Posts: 35
Joined: Sat Nov 07, 2020 2:41 pm

Re: Consistency in 2.1 engine

Post by hscommish »

36Henry wrote: Sat Mar 06, 2021 5:16 am At some point several years ago while I still had time to spend on this wonderful stuff I dug deep into the play-by-play data the Sim provides. The aim was to create essentially a visual simulation of the games, sort of similar to this:

Image

Where all events would be logged and located on the ice. The locations would be based on actual NHL data providing tendencies for areas where for example Ovechkin shoots from. Was a pretty cool project that I sadly never got around to finish.

Another example would be Patric Hornqvist who does most of his work in front of the net. Compiling all data of his shooting, hitting, scoring and shot blocking into a huge database would be the base upon which the application would build. Whenever Hornqvist registered a shot on goal in the Sim, the application would randomly place the shot somewhere on the ice, within the areas where Hornqvist most frequently shoots the puck in real life. The idea being to bring the simulation to life in a more vivid way than just reading the score from the boxscore.

It also would provide data for advanced stats like Corsi and so on, since all the information needed to track these stats are available in the play-by-play data. You just need to process that information to extract what you need to provide accurate advanced stats of pretty much every kind. Wish I had time to finish that project lol.

Anyway, back on topic. What that project working on the play-by-play data did was highlight how goals are scored. Many of them are scored from sustained pressure and off of rebounds. Now, this may certainly be down to a combination of factors such as defensemen in my ratings being too bad at clearing the front of the crease or goalies being too good at stopping the initial shot but too bad at giving up rebounds, although I did calculate rebound percentages for all goalies both in the NHL and in the Sim and they were quite similar. Rebound percentage in the NHL was one of the main contributing stats for creating the Rebound control rating for goalies, naturally.

As for the relationship between SK, PH, PA and SC it has many layers to it. Essentially the add up to a number you could refer to as a skill-stat (maximum skill-value would be 396).

We can have a conversation about this using two players (bear in mind it's been years since I actually played around with the Sim, so my theories may be slightly off lol):

Player A: 70 PA, 90 PH, 90 SC and 30 SK has 280 skill-points.
Player B: 70 PA, 70 PH, 70 SC and 70 SK has 280 skill-points.

While we can all recognize that Player A is a much more skilled player, they have the same amount of skill-points. Given an equal amount of ice-time, they are therefore likely to produce about the same amount of points.

While high SK makes the player more prone to just hold on to the puck instead of advancing play through creativity, my impression of the simulation is that he will still factor into the play and be fairly effective given enough ice-time due to a high total value of skill-points.

Essentially I believe points are awarded based on a lottery of sorts. The players on the scoring team are awarded points randomly, and the higher the skill-points in relation to his linemates, the greater the chance of that player getting a point.

When creating ratings, one thing to look at is the distribution of goals between forwards and defensemen. If memory serves this typically is around 16-17% in the NHL, so that is a target to aim for when creating ratings. This is done easily in all engines available in the STHS, with the 2.1 being no more difficult than the others. And to do this you do not need to reduce players down to 40-50 SC. If you are doing that, you are doing everything wrong.

Typically in the ratings I used to make I would have the top goal scoring forwards at 99 SC and then the rest would follow on a sliding scale down to around 70-72 for the lowest scoring forwards (think John Scott or Justin Johnson, remember his NHL debut?). This essentially means all regular NHL forwards are rated somewhere in the 80-99 range. For regular defensemen the range typically sits from about 70-78, so a very narrow range. That narrowness is key for the 2.1 engine. If the distance between star players and the plebs is too great, the top players will score huge amounts of points while the rest wont get a sniff.

PH is a very useful tool in creating separation between top players and lower level guys, and SK also has a big room for tweaking things, such as for example getting Ovechkin to score more goals than his peers. Every year when I went through rigorous testing (hundreds of hours is indeed a very accurate measurement) I would always have teams like Washington frustrate me since they would be terrible due to Ovechkin not scoring enough goals when being rated using only the formulas. I would have to go in and manually boost his SK and PH in order to get him to produce at the level he is supposed to. Going through and tweaking individual players like this on every team is time consuming but ensures a very good set of ratings.

This highlights the power of SK and the nuances of it. While the framework of it being an inverse value of Creativity certainly works and is a very good model for how the 2.1 engine functions, you still have to be very methodical and detailed when creating the ratings in order for them to be as good as they can be. When I was testing my ratings I used NHL rosters and accurate lines (including special teams) to run hundreds of season with tweaks of individual players between each session. Changing SC on one player from 85 to 86 can have a huge impact on team performance, so it is a slow and lovely process which I often miss.

Savard is absolutely right in the connection between SC and PA. These can have a massive impact on performance when adjusted. PH and SK are much more forgiving and can be very useful to tweak individual performance.

Sorry for the rant :lol:
Don't be sorry! I wish you would rant more! We are still in the process of creating our own ratings for 2.1 and very much appreciate the insight.

The biggest thing I think differs our concept from everyone else's is the ST and CK stats. Basing ST off of height and weight is absolutely mind boggling to me, so we aren't doing that. Same with CK, we're looking to have more than just hits influence that stat. Our focus is less on hitting accurate goal and assist totals, but hitting accurate shot attempt and possession totals. We feel like there's less reliance on luck that way, since there are less factors at play in generating a successful shot on net vs a successful goal.

We would love to work collaboratively with someone to develop something that would benefit every user of engine 2.1 if there is interest.
36Henry
The Accomplished One / L'Accompli
Posts: 118
Joined: Wed Dec 15, 2010 1:58 am

Re: Consistency in 2.1 engine

Post by 36Henry »

Totally agree on the ST-rating.

If I remember correctly I used a bunch of different factors for that. Height and weight was part of it, but I also had fights and hitting influence the setting as well. Essentially Tie Domi likely being a fair bit stronger than other players his size (and also quite a bit stronger than players bigger than him).
Image
hscommish
New in Town / Le Ptit Nouveau
Posts: 35
Joined: Sat Nov 07, 2020 2:41 pm

Re: Consistency in 2.1 engine

Post by hscommish »

36Henry wrote: Tue Mar 09, 2021 12:37 pm Totally agree on the ST-rating.

If I remember correctly I used a bunch of different factors for that. Height and weight was part of it, but I also had fights and hitting influence the setting as well. Essentially Tie Domi likely being a fair bit stronger than other players his size (and also quite a bit stronger than players bigger than him).
My thought process is that since, according to the manual, the ST rating affects how often a player gets hit, that the only things that should influence this rating are take-aways, give-aways, and hits taken (because the higher it is, the less chance a player has of being hit, which means the less chance of turning the puck over). All these stats are publicly available via Natural Stat Trick and easily downloadable by CSV right from their website.

In the NHL, there isn't really any correlation between a player's height, weight, and fights and their ability to maintain puck possession when they are hit that I have seen, and there is another rating that influences how often a player makes a hit attempt, so for me there is no reason these stats should influence the ST rating. It puts small, agile players (Ehlers, Gaudreau, Girard) who are very good at a disadvantage that they don't have in the NHL, as well as giving big, heavy players an advantage they wouldn't otherwise have (from what I have noticed during my time as a GM for the past ~6 years).

It could also be the reason it is so difficult to make a formula for 2.1. If the ST stat is off, the player could be turning pucks over (either positively or negatively) more often than in the NHL, which leads to more/less possession for their team, which could skew results. This is all entirely theoretical though, haha, I have not had the time to dedicate to develop the ratings to a point that they would be ready to be tested.
Savard
The Accomplished One / L'Accompli
Posts: 119
Joined: Fri May 24, 2019 1:07 am

Re: Consistency in 2.1 engine

Post by Savard »

Thanks guys for picking up this topic again. I don't know why, but this still fascinates me. We are about a third into our season, and I think I am quite happy with the results so far. Far from perfect, but a good starting point. You can check the scores and stats http://ultimatehockeyleague.net if you like: Except for my own team (Hellcats) the standings seem reasonable as are most of the scoring leaders. McDavid is scoring at a high pace, but he is mostly playing on a line with MacKinnon, so I'm fine with that I guess. Not quite sure though, why the Slapshots and Knights forwards are scoring at such a high pace.
36Henry wrote: Sat Mar 06, 2021 5:16 am Essentially I believe points are awarded based on a lottery of sorts. The players on the scoring team are awarded points randomly, and the higher the skill-points in relation to his linemates, the greater the chance of that player getting a point.
Mmmh. So you mean the sim actually spreads the points first in a "lottery" and then the play-by-play is built around that? I'm not sure that's true. When I created my ratings I did check the correlation between points production and the total of the "skill points", and while it's not completely uncorrelated, the correlation is very small. At least for my ratings. John Tavares had the same skill point total as Dillon Dube, but Tavares has twice as many points. Mmmh.
36Henry wrote: Sat Mar 06, 2021 5:16 am PH is a very useful tool in creating separation between top players and lower level guys, and SK also has a big room for tweaking things, such as for example getting Ovechkin to score more goals than his peers. Every year when I went through rigorous testing (hundreds of hours is indeed a very accurate measurement) I would always have teams like Washington frustrate me since they would be terrible due to Ovechkin not scoring enough goals when being rated using only the formulas. I would have to go in and manually boost his SK and PH in order to get him to produce at the level he is supposed to. Going through and tweaking individual players like this on every team is time consuming but ensures a very good set of ratings.
So did you leave the SC and PA as they were and tweaked PH and SK only? I tweaked all four ratings, mostly PA, SC and PH I guess. I found it very hard to really see patterns. Raising PH increased both G and A, too low of a SC and the player wouldn't score at all, but it still all seemed difficult to get a grip on.
Savard
The Accomplished One / L'Accompli
Posts: 119
Joined: Fri May 24, 2019 1:07 am

Re: Consistency in 2.1 engine

Post by Savard »

hscommish wrote: Tue Mar 09, 2021 10:00 am The biggest thing I think differs our concept from everyone else's is the ST and CK stats. Basing ST off of height and weight is absolutely mind boggling to me, so we aren't doing that. Same with CK, we're looking to have more than just hits influence that stat. Our focus is less on hitting accurate goal and assist totals, but hitting accurate shot attempt and possession totals. We feel like there's less reliance on luck that way, since there are less factors at play in generating a successful shot on net vs a successful goal.
That's a very interesting approach! I focused on G and A, because that is what GMs would look at the most. With your approach, did you also focus on shooting accuracy? How were the G and A stats in your tests, were they somewhat accurate? I imagine it would be hard to create a playmaking forwards, where you can't really model his play on the number of shots he takes, is that where possession comes into play?
hscommish wrote: Wed Mar 10, 2021 9:22 am My thought process is that since, according to the manual, the ST rating affects how often a player gets hit, that the only things that should influence this rating are take-aways, give-aways, and hits taken (because the higher it is, the less chance a player has of being hit, which means the less chance of turning the puck over). All these stats are publicly available via Natural Stat Trick and easily downloadable by CSV right from their website.
Mmmh. Not sure I concur with that. Yes, ST influences if a player gets hit, but so do PH and SK. Keeping the GA low for lighter player by making him stronger, doesn't seem right, at least at first glance. Shouldn't he simply be better at SK and PH, just like in real life?

But if that's the only way you can keep possession numbers realistic for those light players...
hscommish wrote: Wed Mar 10, 2021 9:22 am It could also be the reason it is so difficult to make a formula for 2.1. If the ST stat is off, the player could be turning pucks over (either positively or negatively) more often than in the NHL, which leads to more/less possession for their team, which could skew results. This is all entirely theoretical though, haha, I have not had the time to dedicate to develop the ratings to a point that they would be ready to be tested.
That makes sense!
hscommish wrote: Tue Mar 09, 2021 10:00 am We would love to work collaboratively with someone to develop something that would benefit every user of engine 2.1 if there is interest.
I would love to collaborate with you guys! Maybe we some others here are willing as well?
Savard
The Accomplished One / L'Accompli
Posts: 119
Joined: Fri May 24, 2019 1:07 am

Re: Consistency in 2.1 engine

Post by Savard »

hscommish wrote: Wed Mar 10, 2021 9:22 am [cording to the manual, the ST rating affects how often a player gets hit, that the only things that should influence this rating are take-aways, give-aways, and hits taken (because the higher it is, the less chance a player has of being hit, which means the less chance of turning the puck over).
First of all, are you sure "Take Aways" is what you mean it is? I believe it is how many times a player has taken away the puck from another, and not how often it was taken away from him...

I tried to verify your theories using the data I have from this season, but failed, however I only had a small sample size. As usual: I don't really see a pattern when trying to compare possession %, GA, TA etc to ratings... if you want the excel file I used, let me know.
ynohtna
The Addict / Le Drogué
Posts: 770
Joined: Mon Aug 14, 2006 5:15 pm
Location: Vancouver, BC
Contact:

Re: Consistency in 2.1 engine

Post by ynohtna »

And I'll buy them! lol... just need it to be stable.
I really like the idea and playability of v2.1. Just need to reign in the edge players and goalies from being so super.
BFHL Admin/Commish
http://www.thebfhl.ca/bfhl
Now in Season 15, using BRHL Player v2.1 Ratings modified.
hscommish
New in Town / Le Ptit Nouveau
Posts: 35
Joined: Sat Nov 07, 2020 2:41 pm

Re: Consistency in 2.1 engine

Post by hscommish »

Savard wrote: Tue Mar 16, 2021 9:48 am
hscommish wrote: Wed Mar 10, 2021 9:22 am [cording to the manual, the ST rating affects how often a player gets hit, that the only things that should influence this rating are take-aways, give-aways, and hits taken (because the higher it is, the less chance a player has of being hit, which means the less chance of turning the puck over).
First of all, are you sure "Take Aways" is what you mean it is? I believe it is how many times a player has taken away the puck from another, and not how often it was taken away from him...

I tried to verify your theories using the data I have from this season, but failed, however I only had a small sample size. As usual: I don't really see a pattern when trying to compare possession %, GA, TA etc to ratings... if you want the excel file I used, let me know.
I would love to have a look at that!

Yes, that is my definition of "take aways" as well, though maybe they don't need to influence the ST rating after all. We are still in the process of finding the right stats to influence each rating, haha.
Post Reply