[Scummvm-devel] New data model for the website

[Matan Bareket]

The spreadsheet is the solution for data entry, data is in VCS under our
control, but really should just be synced.

We are at the point where flat files just don't work for us anymore and we
need a real database. For example to read a compatibility entry, we have to
glob the entire compat list, the entire game list, the entire engine list
and the entire company list. It's luckily not a huge amount of data and we
can (and should) leverage caching to just have it all in memory.

I consider this the "best of both" scenario - we have a structured way to
enter and validate data but still read and keep everything in VCS (in fact,
if we ever lose the spreadsheet, we can just recreate it from the data).
The alternative is to build a whole CMS system or leverage an existing one.
If anyone has a better idea I'm open to it.


On Tue, Sep 15, 2020 at 12:44 AM Will Herrmann <wjherrmann at gmail.com> wrote:

> Thanks for putting so much work into improving the quality of the data of
> the website!
>
> Question: is the spreadsheet intended to be a solution long-term? If it’s
> a temporary measure as we’re migrating data, that’s fine. But I think
> long-term, we should have the relevant data in VCS under our control,
> rather than externally hosted on Google. This will ensure that there will
> never be access issues and that changes are properly tracked (as well as
> having our project built by FOSS at all levels).
>
> On Sep 14, 2020, at 12:59 PM, Matan Bareket <mataniko at gmail.com> wrote:
>
> If you do not add data to the website (demos, compatibility, etc.) you can
> skip reading this mail.
>
> Over the last few months I've been working on improving the data behind
> our website culminating with
> https://github.com/scummvm/scummvm-web/pull/131
>
> From now on, most data is managed via a spreadsheet, with a script to pull
> data into the repository to convert to YAML. You can find the new sheet
> here:
> https://docs.google.com/spreadsheets/d/1QzwFleEKXOsE59cYMOcQB7C2f0Np48uAQOCG8kicX_s/edit?usp=sharing
>
> Data is structured similar to a relational database, with sheets for
> games, companies, versions, engines, game series and demos. (More to be
> added).
>
> When adding entries that reference other entities, please make sure you
> are not referencing a non-existing entity (sheets have some validation
> rules to help).
>
>
> Compatibility saw the most drastic change, since we now reuse old
> compatibility entries if they haven't changed. If you want to add a new
> entry:
> 1. Add a new line in the sheet
> 2. Fill up the id (game id), version the entry is relevant for, and
> supported platforms
> 3. Platforms are comma separated values, based on the platforms sheet
> 4. Optionally, add unsupported platforms
> 5. Optionally, add any additional notes - you DO NOT need to enter "game
> is supported, no problems" or "only dos version is supported" instead we
> will generate these messages based on the status and platforms
> 6. Once you're done - run the update script and commit the changes to the
> repository (composer update-data)
>
>
> Q&A
>
> Q: This looks just like a database, why not use a database
> A: I wanted to keep the data open, and not require us to build a whole
> setup to update the database.
>
> Q: Why do all this work?
> A: By using a "database" we can support more features in the site.
> Filtering, grouping, sorting, etc. It also keeps our data persistent across
> pages (demos, compat, eventually screenshots)
>
> Q: I don't have access to the spreadsheet
> A: Please reach out to me for access
> _______________________________________________
> Scummvm-devel mailing list
> Scummvm-devel at lists.scummvm.org
> https://lists.scummvm.org/listinfo/scummvm-devel
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.scummvm.org/pipermail/scummvm-devel/attachments/20200915/6fc70d78/attachment.htm>