FIFA TECH KNOWLEDGE #1

I posted a similar rant in the Runescape market thread. I'm a little bit more than a developer, more architecting scalable platforms. I'll leave it here as if anyone wants to read a bit more beyond "EA might, but in all likeliness don't use MySQL" ... here it is:

Original thread here: http://www.reddit.com/r/FIFA/comments/33nbnr/how_the_fifa_market_could_learn_from_the/

/u/TheFakeNepenthez - You're one of the smartest YouTubers/Streamers out there in terms of understanding EA's decisions/actions. I respect you, but your comments on the "server space" is just plain wrong.

The whole "it would take so much server space up" is a load of bullshit.

Several points I have:

  • Firstly, let's define "Server Space." I'm guessing you mean storing user specific data, which would be in a database. EA's Servers make requests to that database, which uses a bunch of resources, but the data is in the databases, which, in any setup of this scale, reside on other database-specific servers.

  • To clarify the above point, flat storage costs pennies, for instance, AWS (amazon web services, the biggest cloud provider) lists S3 (Simple Storage Service) pricing as $0.0275 per GB for 50TB+/month. This isn't accurate as there's a few things you have to take into account... it's not a file system, it's flat files only, it's not indexable etc. Databases servers use more resources than just disk space. Bandwidth, CPU cycles, memory (what most people think of as RAM) etc.

  • Also, there's no point trying to calculate how much data EA would need to store: If you've got one database server, and the hard drive fails, everything goes down. You'll have to get your backup (if it works, that is) and restore to a new database server, switch over DNS and have your app back up and running. That isn't what services at this scale do. We use replication. Everything gets replicated at least a few times in any horizontally scalable database (more on that below.) MongoDB uses sharding and doesn't have as much duplication, MySQL Master <-> Master replication copies everything, MySQL Slave -> Master replication has less redundancy and presents issues with slave replication latency. The point is that you can't add up the numbers and say that's only SIX kilobytes. You also need to think about global latency, this is a global product, so is all the data in US, or is there servers in different locations? If so, that might sound great, but now replication is a total bitch because of insanely high latency. If it's on the same fibre in the US it's lightning fast, even California -> Virginia replication, but overseas? Your latency goes up and up.

  • Also to take into account is operational complexity, in terms of maintaining and running a multi-region, highly available, distributed fault tolerant database.

  • EA doesn't do a great job of this. Any software engineer that's worked on a product of scale would be able to tell just from issues the average user runs into. Bad transaction handling -- think maintenance causing issues. Bad exception handling -- think any error throwing you back out of UT. Cards being "stuck" -- EA says its caused by bots overloading their servers, but it's caused by bad code. I'm sorry, but if your system starts fucking data up because too many requests came in, your implementation sucks. Maybe the service goes down -- think DDoS, but that your data gets fucked up? That's bad database transaction handling.

  • Seeing as most of EA's assets are document based, it's likely they use some NoSQL clustered database. I'd expect something like Mongo, but knowing EA and their inability to bring their engineering teams up to industry best practices, I wouldn't be surprised if they're still scaling MySQL, in a similar way that Facebook does (Facebook's data is much more relational and makes more sense than a document based NoSQL solution, which is more suitable for a product like FIFA.)

  • MySQL isn't as easy to scale horizontally (more servers instead of bigger servers (vertical) as vertical scaling gets to a point where there isn't a server that's bigger for you, but you'd be talking to Oracle if you're trying to vertically scale something like FIFA/FUT.)

  • Data storage is the easy, and cheap part. Unless EA are on the absolute edge of their ability to scale (highly unlikely) - I expect they find it difficult to reliably implement the feeding of that data into their databases.

  • The more data, the more replication. Replication gets harder/slower with more data, another reason why servers can be slow, read up on consistent writes/reads (required with a clustered database setup for a transactional write/read required app like FUT, a.k.a list a card twice quickly because you hit a different DB server than you first did and the data didn't replicate yet)

You're right on many things which I agree with, I'm sure you did talk to FIFA/UT devs, but having worked at many companies, engineers aren't the most likely to start telling you how they screwed up architecture from the start, because the higher ups couldn't give them an accurate spec, so they've dug themselves a hole they can't get out of without a full re-architecture, which would cost lots and get nixed by EA because they don't need to. The terminology "server space" alone kinda paints the picture that they were making up excuses.

I'd also like to say you're totally right, EA is an annual game, 16's scope was already specced out by 15's release, and not likely to change drastically. The underlying API is the same, and will be the same for 16, as such so will the database server images (NOT the exact same database servers/databases, as that's an easy way for EA to reset their scaling clock and patch some boxes! ;) ). The only thing they need to change is the API servers, NOT the game, the web app, the companion app. This is something that could be implemented alongside FIFA 16 Game client development, and would have very little impact on the 16 dev cycle... like a pop-up saying "Price ranges are a little different now!"

Final point I'd like to make is that EA's dev teams aren't following industry standards. I highly doubt they have a decent test suite in place, at best a few test harnesses, and as we know, live QA (quality assurance) people, who I'm guessing most of you assume is the only way to test software. These days, well engineered software is tested using fully automated test suites (code,) and then just QA'd as a "smoke test" to ensure we're good to go. This is why the FIFA development cycle (not the annual one, but changes to the current game) are SO slow and terribly glitchy. It's still all a big "OK, the QA guys said it works fine on staging, let's hope this works!"

/r/FIFA Thread