Evaluating goalkeepers: Save Percentage vs Goals Against Average / by Bill Reno

By Bill Reno (@letsallsoccer)

It’s nice to see MLS goalkeepers starting to pick it up. After a rough first couple months, we have finally compiled enough saves to have an honest Save of the Week competition.

2015 Goalkeeper Expected Goals

Keeper Team Min SOG GA xGA G - xG

(AKA GmxG)

Bill Hamid DCU 1150 59 10 16.47 -6.47
Josh Saunders NYC 1245 66 17 20.82 -3.82
Clinton Irwin COL 1344 52 12 15.13 -3.13
David Bingham SJ 1344 61 14 17.11 -3.11
Andrew Dykstra DCU 347 24 5 8.09 -3.09
David Ousted VAN 1543 56 15 17.88 -2.88
Nick Rimando RSL 1252 43 12 14.86 -2.86
Chris Konopka TOR 574 23 5 7.60 -2.60
Stefan Frei SEA 1332 66 11 13.38 -2.38
Jeff Attinella RSL 191 19 5 6.86 -1.86
Tim Melia SKC 572 17 3 4.77 -1.77
Bobby Shuttleworth NE 1339 50 16 17.77 -1.77
Evan Bush MTL 856 52 13 13.89 -0.89
Jaime Penedo LA 1149 55 14 14.82 -0.82
Adam Larsen Kwarasey POR 1434 51 14 14.47 -0.47
Brian Sylvestre PHI 572 24 7 7.31 -0.31
Jon Busch CHI 577 24 9 9.09 -0.09
Travis Worra DCU 39 1 0 0.04 -0.04
Ryan Meara NYC 95 7 1 1.04 -0.04
Eric Kronberg MTL 97 3 2 1.93 0.07
Tyler Deric HOU 1450 59 18 17.91 0.09
Tally Hall ORL 381 17 5 4.49 0.51
Rais Mbolhi PHI 486 19 9 8.33 0.67
Sean Johnson CHI 668 25 9 8.23 0.77
Steve Clark CLB 1331 69 21 19.99 1.01
Brian Rowe LA 381 9 4 2.54 1.46
Brad Knighton NE 97 6 4 2.44 1.56
Chris Seitz FCD 672 27 10 8.15 1.85
Dan Kennedy FCD 667 23 9 7.04 1.96
John McCarthy PHI 474 19 9 6.88 2.12
Donovan Ricketts ORL 961 34 14 11.10 2.90
Joe Bendik TOR 574 36 11 8.07 2.93
Luis Marin SKC 771 30 11 7.95 3.05
Luis Robles NYRB 1237 45 17 13.76 3.24

SOG: Shots on goal faced, GA: Goals allowed (excluding own goals), xGA: Expected goals allowed based on our model, G - xG: The difference between GA and xGA (negative numbers are good)

So because of the sudden upswing in production, now is a great time to take a look at some stats instead of walking through every goalkeeper’s worse game this season. This month, we’re going to take a look at the stats Goals Against Average and Save Percentage to find out which one is worse. Or perhaps which stat is better, if you’re more optimistically inclined. It’s no secret that both stats are rather useless when gauging goalkeepers. There’s a reason why no one is bragging about being in the top ten GAA: it’s not that stellar of a group to be in. Sure we’ve got some of the all-time greats in there, but… Josh Saunders and Bouna Coundoul are in the list? Jimmy Nielsen is not only number one but he’s significantly ahead of his peers. That seems incorrect.

Even before the advancement in the soccer stat community, did anyone really feel more confident that one goalkeeper was better than another because his GAA was one tenth lower? It was more of a binary test. If a goalkeeper’s GAA was lower than 2-ish, “Yeah he’s pretty good. Maybe one of the best in the league.” Otherwise he was tagged with “shaky”. But is save percentage any more efficient in gauging goalkeepers?

I scoured ASA goalkeeper data and took the top sixty-five most involved goalkeepers from 2011-2014 (counting goals + saves). The cut off was when the numbers started getting significantly more erratic. (It turned out ninety-six shots on goal was the line.) Of course some goalkeepers repeated on the list with multiple years while others are only on there once. I compared GAA and S% versus varying stats and the ones that showed nothing were not of any real surprise: saves and games. As in, no matter how many saves or games a goalkeeper had in a season, it didn’t really affect either stat.

For our first graph, let’s see how GAA and save percentage line up against one another. Again, this is just a season’s culmination of stats, not a career.

There seems to be some correlation between saving more shots and conceding less goals, albeit not entirely hand-in-hand. Hmmm, let’s continue…

While I expected to find something here, the difference in the difficulty of shot across goalkeepers was so small and erratic it makes sense that there aren’t any defenses in MLS that would routinely give up impossible shot after impossible shot. It’s going to even out at some point.

Not a huge surprise that S% is not as tight as GAA. But what about expected goals?

Remember, this is counting the expected goals from the shots taken that have gone on frame, regardless of if the goalkeeper saved the ball or not. So should we expect the graph to point one way or another? Think back to the difficulty of shot graphs, shouldn’t the line average out with a large enough sample size, like S% does? But GAA seems to be a little more dependent on the xG. Is that good or bad? This is confusing so let’s add more information to see if that makes it better or worse.

The goals minus expected goals... number. (We really need to come up with a better name for this stat.) Here, as you can see, S% kicks GAA’s ass. Since expected goals are based off of shooting percentages, S% does have an advantage here but S% has essentially lapped GAA in the race here. Let’s look at two more sets of graphs before we draw any conclusions.


(Okay maybe not the best title for the graphs but you get the idea.) I counted the number of games in the season that the shots a goalkeeper faced were less than .5 expected goals, essentially having a fairly easy day in the office. Our reigning champion S% has almost no trend while GAA says the more easy games a goalkeeper has, the lower the GAA. Even more incriminating, take a look at games where goalkeepers are very busy.


GAA’s lawyer would argue, “Well of course GAA is affected by the toughness of shots he faces. It makes sense that if there are more shots against a goalkeeper, his GAA would go up!” Yes but what is the point of a personal statistic? It is to gauge the ability of a specific player, with some statistics being more reliable than others, of course. The GAA does not do this. Compare the xGD graphs and this last pair (G>2xG). GAA is more dependent on the opposition’s chance creation (or defensive inability) than a goalkeeper’s ability. GAA is a team stat, not an individual. Think of a pitcher in baseball, what if we used unearned run average instead of an earned run average (ERA) to gauge how good of a pitcher they were? It would just turn into a Runs Allowed team stat. Or what if in basketball, we used Points Allowed per Game to see how good the center (the big, tall guy) was? It does not make sense.

At best, you would have to use a +/- system if you still wanted to use GAA but even then you would have to get around 100 shots on goal (a large enough sample size) for both when the keeper was on the field and off, which isn’t even possible in a normal MLS season.  However our main man, S%, does the job a little better. It’s not perfect, but it consistent for what it is saying. While GAA claims to be an all-encompassing goalkeeper stat, S% only looks at shots on goal. It is not impacted by the difficulty of shot, but by the goalkeeper’s ability.

Looking in the scope of one game, a goalkeeper doesn’t have any favorable percentage stats. He simply has the raw number of two saves, three punches, etc. Two saves out of three shots on goal isn’t an accurate number and, really, xG doesn’t always work out on a one game basis. So while over time S% comes into play, in the short term we only have the eye test to choose from “good game” or “absolute awful”. But at the end of the season, we should be steering clear of the GAA stat.

So with that in mind, let’s turn our focus to the completely subjective MLS Goalkeeper of the Year Power Ranking Shortlist. Last month we went down to nine names and once again we’re going down to eight numbers.

1.    David Ousted - Coming up on their halfway point of the season and the Vancouver keeper distanced himself even further from the crowd with absurd save after absurd save.

2.    Stefan Frei - Besides his hair, when’s the last time he made a mistake? Sitting in first place on the MLS table definitely helps his cause.

3.    Nick Rimando - Up and down. He continues to gamble on plays and while I wrote him off early for not being a legitimate GOTY contender, he’s hanging in there.

4.    Bill Hamid - Climbing back up the ladder but will the Gold Cup absence could upset his run.

5.    Luis Robles - This was awesome and with Gold Cup on the horizon Robles may get extra games to shine that Rimando and Hamid will miss. Still a fairly quiet season thus far.

6.    Steve Clark - Noticeably playing better, specifically his timing in 1v1 situations is looking for in tune. Unfortunately for his awards cabinet, several other keepers are starting to hit their stride at the same time.

7.    Bobby Shuttleworth - Silently working his way off the list. He’s not doing horribly, but he’s not separating himself from the rest of the league’s keepers.

8.    Chris Konopka  / Tim Melia - The surprise contenders have both played half of their team’s so they combine for one spot. The journeymen are not only earning a starting spot this year, but most likely a pay raise for next year by keeping a struggling team in the middle of the pack, instead of dwelling at the bottom.