It’s Thursday and I feel a hankerin’ to a ranterin’. There’s a weird thing that happens in a lot of threads on the MMO-Champion and Wowhead forums (I don’t read the official forums much) where the argument goes like this:
OP: What’s best, A or B?
Poster 1: A is best. Here is some evidence, some links to show how this evidence is widely attested and subjected to peer review, and some advice based on this evidence.
Poster 2: Nah, B is best. You can’t just reduce everything to maths and theorycraft.
These sorts of conversations just boggle my mind. The really ironic thing is that I’m one of those irritatingly post-modern types who believes in principles like the value of an explicitly subjective hermeneutic because of the inherent limitations of human epistemology. Or to use other words, we can only know what we can experience and we can only generate that knowledge through interpretation of that experience – so an approach to praxis that acknowledges and embraces these limitations is to be desired above an approach which pretends it knows everything.
So I believe pure “objectivity” is by definition unobtainable by humans, or indeed by anything that we can conceive of within the boundaries of our understanding of the Universe, because in nature everything is related to everything else and experiences the world in the context of these interrelations.
Anyway, I’m getting sidetracked.
(Sorry to the guy who inspired this post, this is not a rebuttal or personal attack on you, or even on Poster B type responses in general, it’s just an interesting topic that I wanted to think out loud about and this most recent thread is what brought the topic back to the front of my mind.)
If somebody asks me for advice about how to play their class in WoW, I generally assume they want to know what’s going to give them the best results in a given role. Defining that word “best” is a bit tricky, but for damage-dealers I generally assume “best” means “meeting all the requirements of the encounter vis-a-vis survival, resources, movement, utility use and placement et cetera, while outputting the highest amount of damage possible as fast as possible”. That is, doing maxx DPS.
WoW’s DPS game works on numbers – it’s basically a mathematical system where every spell and ability is assigned a range of values that govern behaviour. So it’s pretty easy to model parts of the game either in formulation (spreadsheets and stuff) or by simulation. These models are based on using in-game testing to discover the mathematical rules which I just talked about, writing the same rules into the tool, and then establishing an environment where you can generate meaningful comparisons. When “scientists” do experiments they have to carefully construct the experiment so that:
- As much as possible, only the factor under investigation will affect results and
- There is something to compare the results of the factor to
In pharmaceutical trials this usually means that you’ll give one group of people the actual drug you want to test and another a placebo. You then compare the health of the people who took the experimental drug with the health of the people who took the placebo, and then you use fancy statistical maths to work out whether any differences are “significant”.
To translate this into WoW, let’s say we want to test a glyph in 25-man raids. What we’d have to do is assemble a large number of raids with as similar a composition as possible and with similar player skills. We then split up these raids into two groups with equal representation of raid balance, gear level and player skill and assign a very strict list of strategies and spell usage to each raid which they are not permitted to deviate from. Then the shamans in one group of raids all get the glyph we’re testing and the shamans in the other group use a control glyph. We then send each group of raid groups off to do the same boss fight over and over and over again – because we need enough samples to minimize the effect of WoW’s RNG. At the end we evaluate the logs and determine which shamans did higher DPS. We then adjust for gear level and raid balance if necessary and work out if there’s a statistically significant difference between the two glyph groups. And, because glyphs generally contribute between 1-2% of our DPS, we have a very narrow margin to evaluate.
So we’d need a very large sample size with very strict controls and very accurate measuring, and very careful analysis of the results. At the end we could reach some tentative experimental conclusions which would be subject to verification or challenge by future experiments.
Maybe it’s just me, but I think that seems just a little impractical.
So what are the alternatives?
Models are great because they allow us to create artificially controlled conditions and, in the case of simulators, quickly generate vast numbers of samples (formulators don’t need vast sample sizes because every “test” using the same variables is identical). The problem with models it that they aren’t “real”, they’re just models. People can make mistakes while designing them or while using them, or there can just be inherent limitations in what they can do.
But, the people who make and use models know this and will deliberately try to minimize these flaws – their models will be subjected to “peer review” by other members of the community, results are reality checked against log data and even anecdotal evidence, and systematic limitations like the difficulty of modelling movement in a spreadsheet can be worked around by “best guess”-type approximations that fit within the model.
In other words, I think that our gameplay models are indeed explicitly and confessedly “subjective”, but they still take into consideration a vast matrix of factors and thus produce very useful results – probably at least as useful as the big impossible playtest trials I outlined above.
Two: Well I feel that…
The other alternative is just to go with what you feel is best. I’m not comfortable using the word “feel” because I don’t want to imply an emotional judgement (or that emotions or intuition is bad!), but if I use a word like “think” it implies a level of, y’know, thought. Let’s say that you do Deathbringer Saurfang one night using Glyph of Lava and you do 12k DPS. You feel pretty good about it but are a bit miffed because that sodding arcane mage who won DFO last week beat you by 200 DPS. Next week you change to Glyph of Flame Shock and are thrilled to discover you did 12.5k DPS while that laughably incompetent arcane mage only did 11k. Perhaps you take a look at your World of Logs report and see that 10% of your damage that fight came from Flame Shock. “10% is a lot”, you might think, “and I did 500 more DPS tonight so that glyph is clearly pretty great!”
The trouble is next week you might swap back to Glyph of Lava and do 13k. What do you conclude then? Alternatively what if you get beaten on DPS by a guy using Glyph of Chain Lightning and rocking 1k less gearscore than you?
Don’t you take my math away
I know this is a really convoluted and unnecessary way to get to this point, but really – if you take away the carefully constructed and rigorously tested models that we use to reach our conclusions about best practice in WoW, what is left to rely on? If someone asks for advice, how do you answer if you just ignore the maths? “Go with whatever you find fun” maybe, but most people find it fun to do the highest DPS they can. How is it more or less fun to use, say, Glyph of Flame Shock over Glyph of Lava?
When I answer someone who’s looking for advice, I base my answer on the evidence available to me, and I factor in so-called subjective factors like player comfort, raiding situations (e.g. what you can rely on in pug raiding is often very different to hardcore progression raiding) and even that nebulous and ephemeral idea of “fun”. But ultimately it’s based on interpretation of hard numbers because that seems the most reliable and accurate source of information.
If someone then replies that you can’t just base everything on maths and theorycraft, well, what can you base it on? What is left? Anecdotal evidence will give you as many different answers as there are anecdotes and is still a manifest example of using our brains to evaluate evidence. What’s different between that and “maths and theorycraft”?
Wherein I admit that I’m a noob
I am pretty up to date with elemental shaman theory. I use a spreadsheet to work out my best practice and compare the results with other community-generated tools, and evaluate my logs after difficult fights to see where I could do better. I am a total minmax beast.
For every single other class and spec I play I just sort of wing it😀
I ask friends who I respect for advice, I think about what aspects of the playstyle I’m good or bad at, what situations I want to do best in, all that sort of stuff. I read those excellent and accessible guides which litter the internets to get basic ideas of stat priorities, glyphing and talent choices, all that sort of stuff – but at the end of the day I take what’s available to me and do what I think will work best. I enjoy doing that just as much as I enjoy the in-depth analysis and theorycraft that I use for my raiding main.
But if you ask me for advice about how to play those classes or specs, I’d have to give you a very qualified and incomplete answer because I genuinely don’t know. I can talk about my experience, what I find fun, the strengths and weaknesses of my own choice of spec and gear and glyphs and whatnot, but I’d have to say to any serious inquirer that they should go find a really good in-depth guide and theorycraft tools and play around with those.
Certainly if someone gives advice based on such evidence I would not flatly contradict it based on my own noobish messing about. I might say something like “I think Blood is overrated as a tank spec as compared to Unholy and Frost” and give a specific argument as to why, but without those reasons – even if they’re bad reasons – there’s no way for anyone to evaluate whether what I’m saying has any merit, and my advice is pretty much useless.
I don’t agree with the given conclusions from every piece of theorycraft I read, nor do I advocate what I produce here as some sort of shaman gospel, but I can’t think of any other way to go about discovering best practice than to use a carefully considered combination of tools, experience and analysis. And if someone doesn’t want to use those tools, that’s fine by me! It is, after all, a game that we play to have fun and enjoy ourselves (though, admittedly, with other people). But if I’m not permitted to base my own conclusions and advice on all this lovingly-crafted theory stuff then what can I base them on?