Explaining Goals - How much does shot quality, location and distance actually add?

As the shot quality argument rages on in football, I have been running numbers to see what all the fuss is about. Expected Goal (xG) believers and skeptics have been back and forth on the topic for ages.

I will admit that I am skeptical of the degree to which shot location, distance and quality impact goals. Admittedly, much of this doubt comes from the work done in hockey but also James Grayson and Ben Pugsley contributed to my lack of 'buy in'.

Even with this doubt, I thought that given the popularity of xG's it would better explain Goals For/Against compared to raw shot metrics.

I would like to thank Michael Caley* and Paul Riley** for making their xG results public. This piece is not directed at them or anyone else in particular. I was curious and it just happens that they do not hide behind a curtain of secrecy, which I really respect.

Explaining Goals For

Testing Errors

2014-15 EPL 

For a description of the work I did please go to the bottom of the page. 

When looking at the errors, the lower the number the better.

Shots on Target seems to be the forgotten/ignored metric in the shot quality argument. Without the influence of location they fared much better than the 2 xG's models.

Danger Zone Shots are the elephant in the room. They did a worse job than Total Shots For in explaining goals this past season.

For the 2014-15 season, we see little evidence that location made a significant impact. Total Shots > SiB > Danger Zone.

We Need More Data

Now I know you are thinking that this is only one season of one league. I tried to find past season xG's results for both Caley and Riley's (or any) models but was unable to find it on the web to test more seasons of the EPL (If you care to share, I would love to test the results).

I did find xG data from Michael Caley's site for the 2013-14 Bundesliga, La Liga and Serie A seasons.

Here are the results from the continent:

Bundesliga 2013-14

In Germany, we see that location has little impact on Goals For. Shots on Target come out the best. 

La Liga 2013-14

Shots on Target continue the trend of being the best predictor of Goals For.  Location? Not adding a lot. 

Serie A 2013-14

And finally in Italia we see that SoT leads the way.  Again, Location not really doing what I had expected.

What are xGF Measuring?

I ran the r^2 of xGF vs all the raw shot metrics to look for correlation and these were the best matches:

EPL:

Riley xGF vs Shots on Target For: .92

Caley xGF vs Shots Inside the Box For: .87

Bundesliga:

Caley xGF vs Shots Inside the Box For: .95

La Liga:

Caley xGF vs Shots Inside the Box For: .94

Serie A:

Caley xGF vs Shots Inside the Box For: .88

We see that Caley's model heavily measures Shots Inside the Box, whilst Riley's model closely resembles Shots on Target (As I understand it his model is mostly based on Shots on Target)

Those are strong relationships to basic shots counts.

Goals For: Does Location Matter?

From what I can tell there is not a lot that location adds to the story of explaining Goals For.

Most intriguing here is that Danger Zone Shots are less reliable than counting all Shots inside the Box in every league. 

Shot quality may exist in attack but its impact is negligible given all the factors that go into scoring a goal.

The lack of publicly available data to properly test the various models is the most frustrating thing. The secrecy and proprietary element of xG's is a hindrance to moving forward. Again, it is a breath of fresh air that Paul and Michael release their results (even if their formulas and algorithms remain a secret).

Explaining Goals Against

Testing Errors

2014-15 EPL

First thing to point out is that Goals Against are tougher to explain as shown by the lower r^2 values.

We see here that Caley's model does a better job on the defensive side of things. Shots on Target and Shots inside the Box trail only slightly in explaining Goals Against. 

Given all the different factors that go into a goal against, are these marginal gains reason to claim location or quality of shot is significant?

Total Shots, Danger Zone and Shots inside the Box are all pretty close based on the errors.  Again it’s tough to call the differences significant but Shots inside the Box just edge Danger Zone Shots in terms of explanatory power.

Here are the results from the continent:

Bundesliga 2013-14

All metrics are in close proximity based on errors. Shots on Target with the marginal edge.

La Liga 2013-14

In Spain, things are close again with Caley xGA edging out Shots on Target. 

Serie A 2013-14

Finally, in Serie A, it is to close to call.

What are xGA Measuring?

I ran the r^2 of xGA's vs all the raw shot metrics to look for correlation and these are what I found were the best matches:

EPL:

Riley xGA vs Shots on Target Ag: .89

Caley xGA vs Danger Zone Shots Ag: .88

Bundesliga:

Caley xGA vs Shots Inside the Box Ag: .95

La Liga:

Caley xGA vs Shots Inside the Box Ag: .95

Serie A:

Caley xGA vs Shots Inside the Box Ag: .94

Again, we see strong links to other non-complex shots counts.

Goals Against: Does Location Matter?

As with Goals For, I am struggling to differentiate shot quality from non-shot quality based metrics based on the errors.  xGA did a better job than their xGF counterparts but we only see slight differences in the explanatory power vs non-specific location based metrics.

Conclusion

Despite my initial skepticism, I came into this wondering 'how far ahead are xG's models?'  Given their widespread acceptance, I assumed that it would be obvious that adjusting for location/quality makes a significant difference.

Instead, the numbers show that shot distance/location/quality is just a tiny portion of shooting (both for and against). The added complexity seems to be unnecessary, at least in explaining goals. It tends to not add anything significant compared to non-location based metrics, specifically Shots on Target.

I have often seen analysis comparing xG to TSR, I think I have shown it’s more valuable to start comparing xG's to Shots on Target instead of Total Shots.

Danger Zone Shots need more rigorous testing too. Repeatability and predictability tests needs to be done so we can determine if this is more than a fancy name. Maybe that is harsh but I am not impressed with how they performed vs. how they are presented in the public domain.

I get that this will not be a popular post but I hope it creates a conversation and leads to more rigorous public testing of xG's models as well as the inputs that go into these models.  I guess in the end I want more transparency. The point of analytics are to find truths and debunk myths. The results are important to moving the conversation forward.

 

Lastly, I remain a skeptic on shot quality. I do believe it is in there somewhere but more work needs to be done.

Cheers,

Clarke
@footyinthecloud

*Michael Caley's data and methodology can be found here and he is on twitter: @MC_of_A

**Paul Riley's blog and methodology is found here and he is also on twitter: @footballfactman

Explanation

What I have done is use the shot data for the various leagues from Michael Caley's* site to find slope equations for Shots For, Shots on Target For, Shots Inside the Box For, Danger Zone Shots For vs. Actual Goals For and then compared the results to both Caley* & Riley** xGF.  I ran a bunch of error tests to help us see more than just the r^2.


For example here is Goals For vs Total Shots For:

We take the slope equation and plug the actual Total Shots For into x and this gives us our 'xGF based on Total Shots For' for the 2014-15 season.

We do this for each shot metric and compare to Caley and Riley xG model's by testing the errors of each metric in explaining Goals For.  If you want to know the meaning of the errors, I would suggest google :)

End of Season Look at Schedule Adjusted Total Shots

This is a method designed by Micah McCurdy (@IneffectiveMath) for weighting schedule strength in the National Hockey League (NHL) using Total Shots (Corsi).  The method is described in full detail in the following link.  https://hockeygraphsdotcom.files.wordpress.com/2014/11/scheduleadjust.pdf

I was interested in how it would look using raw Total Shots For/Against from the Premier League.

End of season notes:

With the Premiership playing a balanced schedule (unlike North American sports leagues), I didn't know if Schedule Adjusted TSR would be that much different from raw TSR. Turns out, that in the end, Schedule Adjusting did little to alter raw TSR.

It is a weird year for in-season repeatability of raw TSR. R-squared of '1st half vs 2nd half' is way down to ~0.40 this year. A vast decline of the previous 5 season avg (~0.75).  TSR still maintained a high correlation to Goal Difference (r squared of ~.71).

Schedule Adjusted TSR did poorer in predicting itself than raw TSR. R-squared of ~0.37. While, versus Goal Difference, it slightly edges raw TSR, with a r squared of ~.72.

Given these results, I will probably not track Sched Adj Total Shots next season. Given the small value it adds to explaining goal difference I do not think it is worth the time I spend inputing the data.

I am glad I did it but I will appreciate using the extra 1 1/2 hrs per week it took to track and organize, on other things :)

Cheers,
Clarke

Here are the Schedule Adjusted TSR results (compiled using www.whoscored.com):

Season Totals (Matches 1-38)

Team	Total Shots For	Total Shots Against	TSR%	Schedule Adjusted Shots For	Schedule Adjusted Shots Against	Sched Adj TSR%	Dif
Arsenal	608	406	60.0%	600	410	59.4%	-0.6%
Aston Villa	415	481	46.3%	410	475	46.3%	-0.0%
Burnley	430	591	42.1%	439	590	42.7%	0.5%
Chelsea	564	415	57.6%	552	421	56.7%	-0.9%
Crystal Palace	440	526	45.5%	446	525	45.9%	0.4%
Everton	483	503	49.0%	482	499	49.1%	0.2%
Hull	434	503	46.3%	436	496	46.8%	0.4%
Leicester	457	557	45.1%	459	554	45.3%	0.2%
Liverpool	584	419	58.2%	581	430	57.5%	-0.8%
Man City	670	384	63.6%	658	392	62.7%	-0.9%
Man United	513	386	57.1%	506	387	56.6%	-0.4%
Newcastle	466	442	51.3%	470	448	51.2%	-0.1%
QPR	534	621	46.2%	545	633	46.2%	0.0%
Southampton	510	384	57.0%	505	386	56.7%	-0.3%
Stoke	503	455	52.5%	501	454	52.4%	-0.1%
Sunderland	409	625	39.6%	420	612	40.7%	1.1%
Swansea	427	551	43.7%	431	542	44.3%	0.7%
Tottenham	526	489	51.8%	530	495	51.7%	-0.1%
West Brom	413	546	43.1%	414	534	43.7%	0.6%
West Ham	487	589	45.3%	490	589	45.4%	0.1%

2nd half of Season (Matches 20-38)

Team	Total Shots For	Total Shots Against	TSR%	Schedule Adjusted Shots For	Schedule Adjusted Shots Against	Sched Adj TSR%	Dif
Arsenal	289	214	57.5%	282	218	56.4%	-1.0%
Aston Villa	226	208	52.1%	224	206	52.1%	0.0%
Burnley	215	287	42.8%	222	286	43.7%	0.9%
Chelsea	255	222	53.5%	253	227	52.7%	-0.8%
Crystal Palace	236	270	46.6%	240	268	47.2%	0.6%
Everton	236	256	48.0%	234	253	48.1%	0.1%
Hull	240	217	52.5%	241	216	52.7%	0.2%
Leicester	246	256	49.0%	246	255	49.1%	0.1%
Liverpool	294	199	59.6%	285	203	58.4%	-1.2%
Man City	350	182	65.8%	335	184	64.6%	-1.2%
Man United	262	177	59.7%	255	181	58.5%	-1.2%
Newcastle	230	209	52.4%	232	213	52.2%	-0.2%
QPR	264	328	44.6%	271	336	44.6%	0.0%
Southampton	256	211	54.8%	257	214	54.6%	-0.2%
Stoke	249	211	54.1%	252	216	53.9%	-0.3%
Sunderland	195	347	36.0%	202	335	37.6%	1.6%
Swansea	218	281	43.7%	222	275	44.6%	0.9%
Tottenham	274	251	52.2%	277	255	52.0%	-0.2%
West Brom	180	308	36.9%	182	296	38.1%	1.2%
West Ham	228	309	42.5%	232	307	43.0%	0.5%

1st Half of Season (Matches 1-19)

Team	Total Shots For	Total Shots Against	TSR%	Schedule Adjusted Shots For	Schedule Adjusted Shot Against	Sched Adj TSR%	Dif
Arsenal	319	192	62.4%	317	191	62.4%	-0.0%
Aston Villa	189	273	40.9%	186	269	40.9%	-0.0%
Burnley	215	304	41.4%	217	305	41.6%	0.2%
Chelsea	309	193	61.6%	299	196	60.4%	-1.2%
Crystal Palace	204	256	44.3%	204	257	44.3%	-0.1%
Everton	247	247	50.0%	250	246	50.4%	0.4%
Hull	189	286	39.8%	188	280	40.2%	0.4%
Leicester	211	301	41.2%	215	300	41.7%	0.5%
Liverpool	293	215	57.7%	301	221	57.6%	-0.1%
Man City	320	205	61.0%	320	209	60.6%	-0.4%
Man United	246	209	54.1%	246	206	54.4%	0.4%
Newcastle	236	233	50.3%	235	233	50.2%	-0.1%
QPR	269	293	47.9%	273	298	47.7%	-0.1%
Southampton	253	173	59.4%	248	173	58.9%	-0.5%
Stoke	254	244	51.0%	250	242	50.8%	-0.2%
Sunderland	214	277	43.6%	218	276	44.2%	0.6%
Swansea	209	269	43.7%	213	265	44.6%	0.8%
Tottenham	252	238	51.4%	250	242	50.8%	-0.6%
West Brom	233	238	49.5%	232	237	49.5%	-0.0%
West Ham	259	275	48.5%	258	274	48.5%	0.0%

14-15 EPL 11v11 Shot Metric Table

Team	Mins	TSR%	SoTR	Sh%	Sv%	PDO
Arsenal	2813	59.0%	62.2%	33.7%	72.2%	105.9%
Aston Villa	2798	45.1%	42.2%	24.5%	69.4%	93.9%
Burnley	2891	43.0%	40.8%	23.1%	67.5%	90.7%
Chelsea	2681	57.2%	63.3%	35.9%	77.3%	113.2%
Crystal Palace	2896	47.3%	50.0%	35.0%	65.0%	100.0%
Everton	2870	47.7%	54.7%	29.6%	62.5%	92.1%
Hull	2633	45.4%	46.2%	26.5%	67.2%	93.7%
Leicester	2815	42.9%	41.1%	30.6%	68.6%	99.2%
Liverpool	2798	58.4%	59.0%	27.8%	69.1%	96.9%
Man City	2768	63.5%	62.7%	34.7%	67.3%	102.0%
Man United	2785	57.3%	56.6%	37.9%	74.8%	112.7%
Newcastle	2918	50.6%	47.2%	28.8%	60.6%	89.4%
QPR	2851	47.0%	41.6%	27.8%	68.4%	96.1%
Southampton	2871	56.2%	61.3%	31.2%	75.3%	106.5%
Stoke	2876	51.7%	45.5%	36.0%	66.7%	102.7%
Sunderland	2761	39.9%	38.4%	24.5%	70.7%	95.2%
Swansea	2829	45.0%	45.3%	29.0%	73.3%	102.4%
Tottenham	2842	51.8%	50.3%	32.5%	69.1%	101.6%
West Brom	2747	47.7%	45.6%	28.4%	67.7%	96.1%
West Ham	2818	44.5%	45.4%	31.3%	72.7%	104.0%

What are we looking at?

What are we trying to find by looking at on pitch statistics at an individual player level?

Shots vs. Goals

Goals

Their is a tendency for the coaches/media/fans to judge individuals on goals/assists for and against. The problem with using goals as a performance indicator is the fact that 'luck', or lack thereof, plays a massive part in scoring/conceding goals. Read these 2 links on PDO. Link1 Link2

It is my experience, from hockey analytics, that players and teams with a high PDO get a lot more 'praise' than players/teams with a low PDO. 

High or low on pitch shooting %/Save% are not sustainable and will regress toward the average.

Shots

The more repeatable stat for determining performance is using total shots. Here is a great article on shot quantity vs. quality. If we can find out which players help increase shot differential, we can then increase the teams chance of scoring/saving a goal by playing them more or subbing them at the right times, therefore, leading to more points in the league table.

What I am looking for, are players who help increase total shot differential, of their team, while on the pitch. The higher the TSR% the more we know what part of the pitch the game was played in while a player is on the pitch.  Raw on pitch TSR% is okay but does not provide context.  What we see is that good teams have players with high TSR% and bad teams have players with lower TSR%.

The following stat allows us to differentiate players on the same team.


Relative Total Shot Number per 90 (TSR Rel/90) = Total Shot Number of player - Total Number of Team when player not on pitch

TSR Rel/90 begins to tell us who 'drives play' on his particular team.  Helping create a shot for or stopping a shot against are equally important. Over the course of the season we will be able to see who has impacted shot ratios. Obviously, the higher the number the better.

What does a TSR Rel/90 of 1.25 mean? It is saying that for every 90 minutes that a player plays 11v11 versus every 90 minutes that he doesn't play, his team will be better off by a total shot differential of 1.25 per 90.  The higher the number the better.


In Closing

It is by no means a 'perfect' stat. As always, their are question marks.  With 11 players on the pitch their is a lot of noise. The quality of the competition faced by each player, while he is on the pitch, is not accounted for. Game states or score effects are not taken into account. I just do not have the data and time integrate these into the analysis.

The general idea is to find a new way of analyzing players by removing goals from the equation. If we find a player who is very high or low we can look into his performances by watching film and looking at the quality of competition, quality of teammates, game states etc.




Clarke Ruehlen
Follow @footyinthecloud