Is Quality Really a Goal?

meleu By meleu.

Contents


“We build computer systems the way we build our cities: over time, without a plan, on top of ruins.” – Ellen Ullman, computer programmer and author.

Intro

This article covers some parts of the RA history, focusing on the “quality” topic. How it was in the past and how we’re working on it today.

If you enjoy the RALore, keep reading.

How to know what is good?

Those who are with us since before 2017 will likely remember what RA was like when it came to quality. Achievements for starting a game weren’t rare. Also some other ones for having a “game over”. Many grindy, tedious, “impossible” ones…

The simple gut reaction is that this kind of thing happens due to incompetence. How wasn’t it just avoided in the first place?

Turns out that the opening quote opening to this article is not a sarcastic joke, like it may seem. What Mrs. Ullman says about computers, can also be applied to the way we build things here on RetroAchievements. And it’s not a joke, it’s the statement of a fact.

The point is: how to know what is good if we don’t know what is bad? We need ruins to build something better on top of it.

Why poor quality work is necessary

So, in order to do something good, one need to do a lot of poor quality work. Of course each person is different, both in terms of intellect and skills, but doing better with each iteration is a key factor for doing something good at some point.

There’s still a problem though: when you finish doing something you think is good, a short time later you are already thinking about how to make it even better. It may even reach a point where you won’t think it’s that good anymore… (for the hardcore nerds, this happens because of neuroplasticity).

Uh… but… Do we really need to see every single newcomer producing the same low quality work over and over again?

How to avoid repeating the same mistakes?

In the pre-historic days of RA there wasn’t any documentation. Then a member who was brave enough to open the Memory Inspector could find an address for “game over” and become excited. “Hooray!! I found an address meaning something! Let me create an achievement with this address real quick!”. As you can imagine, this led to the creation of a lot of low quality content.

Fortunately for us, RA fans, some ancient members (like Brian, cirellio and the RA creator Scott) probably made a lot of mistakes but were wise enough to write about their findings, and were generous enough to post them on the forums. Many achievement creation docs we have today, were built on top of their findings.

Thanks to their effort, today we can avoid repeating the same mistakes.

Experiment and iterate

Even with a lot of documentation, to fill the technical gaps, there are still a lot of non-technical mistakes, like bad achievement’s design, here and there producing some “basic” achievement sets.

Currently this is not a really serious problem, since we have Code Reviewers and QA Team preventing really poor achievement sets to be promoted as official sets. We have also the revision process, where people can improve the sets over time.

So, it’s ok to promote an “acceptable” set, and then enrich it in a future revision.

This way people can start creating achievements as soon as they can, and get better in the next iteration.

But well, it wasn’t so “smooth” in the past. And is surely still a source of discontent here and there.

A little of RALore

In the past, when the revision process didn’t exist. Changing an achievement set were very rare. Even if the set had poorly designed achievements like “start the game as player X”, demoting it could be considered a severe offense to some people (either the author or some players who had unlocked such achievement).

Somehow there was a belief in the collective unconscious that once a set is published it’s definitive. This “rule” wasn’t verbalized anywhere. We didn’t have docs neither a Code of Conduct, but that’s how it was in the past.

The problem with such scenario is that, regarding quality, it was very unfavorable.

Key problems:

  1. It doesn’t allow achievement sets to be improved
  2. It makes some creators who care a lot about RA deter newbies from start creating
  3. (consequence of point 2) It doesn’t allow creators to “learn to be good” (make something bad, and then improve it)

With the benefit with the hindsight, now I see that what we were unconsciously trying to avoid was just the complaints coming from the players. We were sacrificing our biggest asset (the achievements) to please people who just consume what we produce. Just because people want to “master” a game and then consider it as done.

That was causing a lot of tension between those who actually put a lot of effort producing content for the project.

Fortunately it’s now a problem of the past. But, again, we needed to face those problems in order to fix them.

Although we have left some wounds, over time we’re slowly maturing. Today the revision process is working reasonably well.

We even tried to open the voting to the whole community, and faced a sad reality: those who don’t put any effort to make RA better can’t vote with the same weight of those who spend a lot of their time working for RA.

Most of the players (not all of them) are here to brag about their site awards, and “quality” is relegated as secondary. After many heated dicussions we decided to close the revision voting only for achievement creators again. And this seems to be working fine so far.

So, is quality a goal?

Answering the question on the title of this article: No, quality is not a goal, it’s a road.

It is critical to understand that due to our malleable standards of evaluation, this process has no end, ever. We are never going to reach some utopian state where everything is perfectly satisfying. The fundamental reward of an improvement process is the experience of betterment, not some mythical destination.