[Scummvm-git-logs] scummvm-sites integrity -> 88aa0edf7658e742a5291fdf6d0e6e7225600629
sev-
noreply at scummvm.org
Fri Mar 21 16:53:21 UTC 2025
This automated email contains information about 4 new commits which have been
pushed to the 'scummvm-sites' repo located at https://github.com/scummvm/scummvm-sites .
Summary:
3d49d19205 Readme update, addition of sample_mysql_config.json, addition of requirements.txt and minor change in table order in sch
79d4625ba7 INTEGRITY: Remove extra 's' typo from file table in schema.py
50a0b8c561 INTEGRITY: Add newline at the end of sample_mysql_config.json
88aa0edf76 INTEGRITY: Correct spacing in README
Commit: 3d49d1920567e2ed96ddd68de8ee02db5adc51fd
https://github.com/scummvm/scummvm-sites/commit/3d49d1920567e2ed96ddd68de8ee02db5adc51fd
Author: ShivangNagta (shivangnag at gmail.com)
Date: 2025-03-22T00:53:17+08:00
Commit Message:
Readme update, addition of sample_mysql_config.json, addition of requirements.txt and minor change in table order in schema.py
Changed paths:
A requirements.txt
A sample_mysql_config.json
.gitignore
README.md
schema.py
diff --git a/.gitignore b/.gitignore
index 016f5c8..eea87ae 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,3 @@
*.dat
mysql_config.json
+__pycache__
diff --git a/README.md b/README.md
index 901fc1d..ba223ec 100644
--- a/README.md
+++ b/README.md
@@ -1,14 +1,116 @@
-# ScummVM File Integrity Check (GSoC 2024)
+# ScummVM File Integrity Check (GSoC 2025)
-This repository contains the server-side code for the upcoming file integrity check for game datafiles. This repository is part of the Google Summer of Code 2024 program.
+This repository contains the server-side code for the upcoming file integrity check for game datafiles. This repository is part of the Google Summer of Code 2025 program.
-This website needs a `mysql_config.json` in the root to run, in the form:
+## Prerequisites
+### Local
- {
- "servername": "<your servername>",
- "username": "<your username>",
- "password": "<your password>",
- "dbname": "<your db name>"
- }
+- Python 3.x
+- MySQL
+
+### Deployment
+- Apache2
+
+## Step-by-step Setup
+
+### 1. Clone the Repository
+```bash
+git clone https://github.com/scummvm/scummvm-sites.git
+```
+
+### 2. Navigate to the Project Directory
+```bash
+cd scummvm-sites
+```
+
+### 3. Fetch All Branches
+```bash
+git fetch --all
+```
+
+### 4. Checkout the integrity Branch
+```bash
+git checkout integrity
+```
+
+### 5. Install Required Python Packages
+You can also create a virtual environment (optional)
+```bash
+pip install -r requirements.txt
+```
+
+### 6. Install and Configure MySQL (if not already installed)
+Ensure MySQL is running and properly configured.
+
+### 7. Create the MySQL Configuration File
+Create a `mysql_config.json` file with the following structure:
+```json
+{
+ "host": "servername",
+ "user": "username",
+ "password": "your_password",
+ "database": "db_name"
+}
+```
+A `sample_mysql_config.json` is also present in the same directory.
+
+### 8. Run Schema to Create Tables
+```bash
+python schema.py
+```
+
+## General Usecases
+
+### 1. Manual Generation of dat files (scan.dat) from existing games collection
+This utility helps in generating dat files from the existing game collections with the developers and then upload it to the database.
+#### Dat File Generation :
+This will generate the `.dat` file with complete checksums but no metadata.
+
+```bash
+python compute_hash.py --directory <path_to_directory> --depth 0 --size 0
+```
+- `--directory` : Path of directory with game files
+- `--depth` : Depth from root to game directories
+- `--size` : Use first n bytes of file to calculate checksum
+
+#### Database upload (for developers ) :
+Uploading the `.dat` file to the database.
+```bash
+python dat_parser.py --upload <scanned_dat_file/scan>.dat --user <username> --skiplog
+```
+- `--upload` : Upload DAT file(s) to the database
+- `--match` : Populate matching games in the database
+- `--user` : Username for database
+- `-r` : Recurse through directories
+- `--skiplog` : Skip logging dups
+
+### 2. Uploading dat files (scummvm.dat) generated from detection entries to the DB (Initial Seeding)
+Upload the `.dat` file to the database using dat_parser script -
+```bash
+python dat_parser.py --upload <detection_dat_file/scummvm>.dat --user <username> --skiplog
+```
+`scummvm.dat` can be generated using -
+```bash
+./scummvm --dump-all-detection-entries
+```
+
+### 3. Uploading already existing dat files (set.dat) from old collections to the DB
+Upload the `.dat` file to the database using dat_parser script -
+```bash
+python dat_parser.py --upload <old_dat_file/set>.dat --user <username> --skiplog
+```
+
+
+### 4. Validate Game Files from Client Side (integrity.json)
+Make a POST request to the following endpoint:
+#### Local :
+```bash
+http://localhost:5000/validate
+```
+with the body in JSON format as shown in `sample_json_request.json` present in the main directory.
+There also exists a check_integrity button in the scummvm application itself which is under development.
+
+## Deployment
+
+The apache2 .conf file is located under `apache2-config/`.
-The apache2 .conf file is located under `apache2-conf/`.
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..41cebfb
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,10 @@
+blinker==1.9.0
+click==8.1.8
+Flask==3.1.0
+itsdangerous==2.2.0
+Jinja2==3.1.5
+MarkupSafe==3.0.2
+PyMySQL==1.1.1
+setuptools==75.8.0
+Werkzeug==3.1.3
+wheel==0.45.1
diff --git a/sample_mysql_config.json b/sample_mysql_config.json
new file mode 100644
index 0000000..13eee9b
--- /dev/null
+++ b/sample_mysql_config.json
@@ -0,0 +1,6 @@
+{
+ "servername": "localhost",
+ "username": "root",
+ "password": "12345678",
+ "dbname": "scummvm_db"
+}
\ No newline at end of file
diff --git a/schema.py b/schema.py
index 8e75085..3e78964 100644
--- a/schema.py
+++ b/schema.py
@@ -61,8 +61,22 @@ tables = {
FOREIGN KEY (engine) REFERENCES engine(id)
)
""",
+ "fileset": """
+ CREATE TABLE IF NOT EXISTS fileset (
+ id INT AUTO_INCREMENT PRIMARY KEY,
+ game INT,
+ status VARCHAR(20),
+ src VARCHAR(20),
+ `key` VARCHAR(64),
+ `megakey` VARCHAR(64),
+ `delete` BOOLEAN DEFAULT FALSE NOT NULL,
+ `timestamp` TIMESTAMP NOT NULL,
+ detection_size INT,
+ FOREIGN KEY (game) REFERENCES game(id)
+ )
+ """,
"file": """
- CREATE TABLE IF NOT EXISTS file (
+ CREATE TABLE IF NOT EXISTS file (s
id INT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(200) NOT NULL,
size BIGINT NOT NULL,
@@ -93,20 +107,6 @@ tables = {
FOREIGN KEY (fileset) REFERENCES fileset(id)
)
""",
- "fileset": """
- CREATE TABLE IF NOT EXISTS fileset (
- id INT AUTO_INCREMENT PRIMARY KEY,
- game INT,
- status VARCHAR(20),
- src VARCHAR(20),
- `key` VARCHAR(64),
- `megakey` VARCHAR(64),
- `delete` BOOLEAN DEFAULT FALSE NOT NULL,
- `timestamp` TIMESTAMP NOT NULL,
- detection_size INT,
- FOREIGN KEY (game) REFERENCES game(id)
- )
- """,
"log": """
CREATE TABLE IF NOT EXISTS log (
id INT AUTO_INCREMENT PRIMARY KEY,
Commit: 79d4625ba7f68dea332e9a1565f6012c313f63ec
https://github.com/scummvm/scummvm-sites/commit/79d4625ba7f68dea332e9a1565f6012c313f63ec
Author: Shivang Nagta (141389245+ShivangNagta at users.noreply.github.com)
Date: 2025-03-22T00:53:17+08:00
Commit Message:
INTEGRITY: Remove extra 's' typo from file table in schema.py
Changed paths:
schema.py
diff --git a/schema.py b/schema.py
index 3e78964..4a50e86 100644
--- a/schema.py
+++ b/schema.py
@@ -76,7 +76,7 @@ tables = {
)
""",
"file": """
- CREATE TABLE IF NOT EXISTS file (s
+ CREATE TABLE IF NOT EXISTS file (
id INT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(200) NOT NULL,
size BIGINT NOT NULL,
@@ -230,4 +230,4 @@ def insert_random_data():
# insert_random_data()
conn.commit()
-conn.close()
\ No newline at end of file
+conn.close()
Commit: 50a0b8c561ecc174de12eb8403a6a145df039c79
https://github.com/scummvm/scummvm-sites/commit/50a0b8c561ecc174de12eb8403a6a145df039c79
Author: Shivang Nagta (141389245+ShivangNagta at users.noreply.github.com)
Date: 2025-03-22T00:53:17+08:00
Commit Message:
INTEGRITY: Add newline at the end of sample_mysql_config.json
Changed paths:
sample_mysql_config.json
diff --git a/sample_mysql_config.json b/sample_mysql_config.json
index 13eee9b..cb337ae 100644
--- a/sample_mysql_config.json
+++ b/sample_mysql_config.json
@@ -3,4 +3,4 @@
"username": "root",
"password": "12345678",
"dbname": "scummvm_db"
-}
\ No newline at end of file
+}
Commit: 88aa0edf7658e742a5291fdf6d0e6e7225600629
https://github.com/scummvm/scummvm-sites/commit/88aa0edf7658e742a5291fdf6d0e6e7225600629
Author: Shivang Nagta (141389245+ShivangNagta at users.noreply.github.com)
Date: 2025-03-22T00:53:17+08:00
Commit Message:
INTEGRITY: Correct spacing in README
Changed paths:
README.md
diff --git a/README.md b/README.md
index ba223ec..5c9538e 100644
--- a/README.md
+++ b/README.md
@@ -73,7 +73,7 @@ python compute_hash.py --directory <path_to_directory> --depth 0 --size 0
- `--depth` : Depth from root to game directories
- `--size` : Use first n bytes of file to calculate checksum
-#### Database upload (for developers ) :
+#### Database upload (for developers) :
Uploading the `.dat` file to the database.
```bash
python dat_parser.py --upload <scanned_dat_file/scan>.dat --user <username> --skiplog
More information about the Scummvm-git-logs
mailing list