Merge pull request #1022 from ae-utbm/poors-man-docker

Poor mans docker compose

.env.example

@@ -8,4 +8,10 @@ SECRET_KEY=(4sjxvhz@m5$0a$j0_pqicnc$s!vbve)z+&++m%g%bjhlz4+g2
 DATABASE_URL=sqlite:///db.sqlite3
 #DATABASE_URL=postgres://user:password@127.0.0.1:5432/sith
 
-CACHE_URL=redis://127.0.0.1:6379/0
+REDIS_PORT=7963
+CACHE_URL=redis://127.0.0.1:${REDIS_PORT}/0
+
+# Used to select which other services to run alongside
+# manage.py, pytest and runserver
+PROCFILE_STATIC=Procfile.static
+PROCFILE_SERVICE=Procfile.service

.github/actions/setup_project/action.yml

@@ -9,6 +9,11 @@ runs:
         packages: gettext
         version: 1.0  # increment to reset cache
 
+    - name: Install Redis
+      uses: shogo82148/actions-setup-redis@v1
+      with:
+        redis-version: "7.x"
+
     - name: Install uv
       uses: astral-sh/setup-uv@v5
       with:

.github/workflows/ci.yml

@@ -10,6 +10,7 @@ on:
 env:
   SECRET_KEY: notTheRealOne
   DATABASE_URL: sqlite:///db.sqlite3
+  CACHE_URL: redis://127.0.0.1:6379/0
 
 jobs:
   pre-commit:

.gitignore

@@ -18,7 +18,14 @@ sith/search_indexes/
 .coverage
 coverage_report/
 node_modules/
+.env
+*.pid
 
 # compiled documentation
 site/
-.env
+
+### Redis ###
+
+# Ignore redis binary dump (dump.rdb) files
+
+*.rdb

Procfile.service

@@ -0,0 +1 @@
+redis: redis-server --port $REDIS_PORT

Procfile.static

@@ -0,0 +1 @@
+bundler: npm run serve

docs/tutorial/install-advanced.md

@@ -77,6 +77,58 @@ uv sync --group prod
     C'est parce que ces dépendances compilent certains modules
     à l'installation.
 
+## Désactiver Honcho
+
+Honcho est utilisé en développement pour simplifier la gestion
+des services externes (redis, vite et autres futures).
+
+En mode production, il est nécessaire de le désactiver puisque normalement
+tous ces services sont déjà configurés.
+
+Pour désactiver Honcho il suffit de ne sélectionner aucun `PROCFILE_` dans la config.
+
+```dotenv
+PROCFILE_STATIC=
+PROCFILE_SERVICE=
+```
+
+!!! note
+
+    Si `PROCFILE_STATIC` est désactivé, la recompilation automatique
+    des fichiers statiques ne se fait plus.
+    Si vous en avez besoin et que vous travaillez sans `PROCFILE_STATIC`,
+    vous devez ouvrir une autre fenêtre de votre terminal
+    et lancer la commande `npm run serve`
+
+## Configurer Redis en service externe
+
+Redis est installé comme dépendance mais pas lancé par défaut.
+
+En mode développement, le sith se charge de le démarrer mais
+pas en production !
+
+Il faut donc lancer le service comme ceci:
+
+```bash
+sudo systemctl start redis
+sudo systemctl enable redis  # si vous voulez que redis démarre automatiquement au boot
+```
+
+Puis modifiez votre `.env` pour y configurer le bon port redis.
+Le port du fichier d'exemple est un port non standard pour éviter
+les conflits avec les instances de redis déjà en fonctionnement.
+
+```dotenv
+REDIS_PORT=6379
+CACHE_URL=redis://127.0.0.1:${REDIS_PORT}/0
+```
+
+Si on souhaite configurer redis pour communiquer via un socket :
+
+```dovenv
+CACHE_URL=redis:///path/to/redis-server.sock
+```
+
 ## Configurer PostgreSQL
 
 PostgreSQL est utilisé comme base de données.

docs/tutorial/install.md

@@ -100,14 +100,6 @@ cd /mnt/<la_lettre_du_disque>/vos/fichiers/comme/dhab
     Python ne fait pas parti des dépendances puisqu'il est automatiquement
     installé par uv.
 
-Parmi les dépendances installées se trouve redis (que nous utilisons comme cache).
-Redis est un service qui doit être activé pour être utilisé.
-Pour cela, effectuez les commandes :
-
-```bash
-sudo systemctl start redis
-sudo systemctl enable redis  # si vous voulez que redis démarre automatiquement au boot
-```
 
 ## Finaliser l'installation
 
@@ -179,6 +171,11 @@ uv run ./manage.py runserver
     [http://localhost:8000](http://localhost:8000) 
     ou bien [http://127.0.0.1:8000/](http://127.0.0.1:8000/).
 
+!!!note
+
+    Le serveur de développement se charge de lancer redis
+    et les autres services nécessaires au bon fonctionnement du site.
+
 !!!tip
 
     Vous trouverez également, à l'adresse

docs/tutorial/structure.md

@@ -66,20 +66,24 @@ sith/
 │   └── ...
 ├── staticfiles/ (23)
 │   └── ...
+├── processes/ (24)
+│   └── ...
 │
-├── .coveragerc (24)
-├── .envrc (25)
+├── .coveragerc (25)
+├── .envrc (26)
 ├── .gitattributes
 ├── .gitignore
 ├── .mailmap
-├── .env (26)
-├── .env.example (27)
-├── manage.py (28)
-├── mkdocs.yml (29)
+├── .env (27)
+├── .env.example (28)
+├── manage.py (29)
+├── mkdocs.yml (30)
 ├── uv.lock
-├── pyproject.toml (30)
-├── .venv/ (31)
-├── .python-version (32)
+├── pyproject.toml (31)
+├── .venv/ (32)
+├── .python-version (33)
+├── Procfile.static (34)
+├── Procfile.service (35)
 └── README.md
 ```
 </div>
@@ -121,22 +125,27 @@ sith/
 23. Gestion des statics du site. Override le système de statics de Django.
     Ajoute l'intégration du scss et du bundler js
     de manière transparente pour l'utilisateur. 
-24. Fichier de configuration de coverage. 
-25. Fichier de configuration de direnv. 
-26. Contient les variables d'environnement, qui sont susceptibles
+24. Module de gestion des services externes.
+    Offre une API simple pour utiliser les fichiers `Procfile.*`.
+25. Fichier de configuration de coverage. 
+26. Fichier de configuration de direnv. 
+27. Contient les variables d'environnement, qui sont susceptibles
     de varier d'une machine à l'autre.
-27. Contient des valeurs par défaut pour le `.env`
+28. Contient des valeurs par défaut pour le `.env`
     pouvant convenir à un environnment de développement local
-28. Fichier généré automatiquement par Django. C'est lui
+29. Fichier généré automatiquement par Django. C'est lui
     qui permet d'appeler des commandes de gestion du projet
     avec la syntaxe `python ./manage.py <nom de la commande>`
-29. Le fichier de configuration de la documentation,
+30. Le fichier de configuration de la documentation,
     avec ses plugins et sa table des matières. 
-30. Le fichier où sont déclarés les dépendances et la configuration
+31. Le fichier où sont déclarés les dépendances et la configuration
     de certaines d'entre elles.
-31. Dossier d'environnement virtuel généré par uv
-32. Fichier qui contrôle quelle version de python utiliser pour le projet
-    
+32. Dossier d'environnement virtuel généré par uv
+33. Fichier qui contrôle quelle version de python utiliser pour le projet
+34. Fichier qui contrôle les commandes à lancer pour gérer la compilation
+    automatique des static et autres services nécessaires à la command runserver.
+35. Fichier qui contrôle les services tiers nécessaires au fonctionnement
+    du Sith tel que redis.
 
 ## L'application principale
 
@@ -220,4 +229,4 @@ comme suit :
 
 L'organisation peut éventuellement être un peu différente
 pour certaines applications, mais le principe
-général est le même.
+général est le même.

manage.py

@@ -13,13 +13,25 @@
 # OR WITHIN THE LOCAL FILE "LICENSE"
 #
 #
-
+import atexit
+import logging
 import os
 import sys
 
+from django.utils.autoreload import DJANGO_AUTORELOAD_ENV
+
+from sith.composer import start_composer, stop_composer
+from sith.settings import PROCFILE_SERVICE
+
 if __name__ == "__main__":
+    logging.basicConfig(encoding="utf-8", level=logging.INFO)
+
     os.environ.setdefault("DJANGO_SETTINGS_MODULE", "sith.settings")
 
     from django.core.management import execute_from_command_line
 
+    if os.environ.get(DJANGO_AUTORELOAD_ENV) is None and PROCFILE_SERVICE is not None:
+        start_composer(PROCFILE_SERVICE)
+        _ = atexit.register(stop_composer, procfile=PROCFILE_SERVICE)
+
     execute_from_command_line(sys.argv)

package.json

@@ -6,7 +6,7 @@
   "scripts": {
     "compile": "vite build --mode production",
     "compile-dev": "vite build --mode development",
-    "serve": "vite build --mode development --watch",
+    "serve": "vite build --mode development --watch --minify false",
     "analyse-dev": "vite-bundle-visualizer --mode development",
     "analyse-prod": "vite-bundle-visualizer --mode production",
     "check": "biome check --write"

pyproject.toml

@@ -47,6 +47,8 @@ dependencies = [
     "redis[hiredis]<6.0.0,>=5.2.0",
     "environs[django]<15.0.0,>=14.1.0",
     "requests>=2.32.3",
+    "honcho>=2.0.0",
+    "psutil>=7.0.0",
 ]
 
 [project.urls]
@@ -124,6 +126,13 @@ ignore = [
 [tool.ruff.lint.pydocstyle]
 convention = "google"
 
+[build-system] # A build system is needed to register a pytest plugin
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project.entry-points.pytest11]
+sith = "sith.pytest"
+
 [tool.pytest.ini_options]
 DJANGO_SETTINGS_MODULE = "sith.settings"
 python_files = ["tests.py", "test_*.py", "*_tests.py"]

sith/composer.py

@@ -0,0 +1,80 @@
+import logging
+import signal
+import subprocess
+import sys
+from pathlib import Path
+
+import psutil
+
+from sith import settings
+
+
+def get_pid_file(procfile: Path) -> Path:
+    """Get the PID file associated with a procfile"""
+    return settings.BASE_DIR / procfile.with_suffix(f"{procfile.suffix}.pid")
+
+
+def get_pid(procfile: Path) -> int | None:
+    """Read the PID file to get the currently running composer if it exists"""
+    file = get_pid_file(procfile)
+    if not file.exists():
+        return None
+    with open(file, "r", encoding="utf8") as f:
+        return int(f.read())
+
+
+def write_pid(procfile: Path, pid: int):
+    """Write currently running composer pid in PID file"""
+    file = get_pid_file(procfile)
+    if not file.exists():
+        file.parent.mkdir(parents=True, exist_ok=True)
+    with open(file, "w", encoding="utf8") as f:
+        _ = f.write(str(pid))
+
+
+def delete_pid(procfile: Path):
+    """Delete PID file for cleanup"""
+    get_pid_file(procfile).unlink(missing_ok=True)
+
+
+def is_composer_running(procfile: Path) -> bool:
+    """Check if the process in the PID file is running"""
+    pid = get_pid(procfile)
+    if pid is None:
+        return False
+    try:
+        return psutil.Process(pid).is_running()
+    except psutil.NoSuchProcess:
+        return False
+
+
+def start_composer(procfile: Path):
+    """Starts the composer and stores the PID as an environment variable
+    This allows for running smoothly with the django reloader
+    """
+    if is_composer_running(procfile):
+        logging.info(
+            f"Composer for {procfile} is already running with pid {get_pid(procfile)}"
+        )
+        logging.info(
+            f"If this is a mistake, please delete {get_pid_file(procfile)} and restart the process"
+        )
+        return
+    process = subprocess.Popen(
+        [sys.executable, "-m", "honcho", "-f", str(procfile), "start"],
+    )
+    write_pid(procfile, process.pid)
+
+
+def stop_composer(procfile: Path):
+    """Stops the composer if it was started before"""
+    if is_composer_running(procfile):
+        process = psutil.Process(get_pid(procfile))
+        if process.parent() != psutil.Process():
+            logging.info(
+                f"Currently running composer for {procfile} is controlled by another process"
+            )
+            return
+        process.send_signal(signal.SIGTERM)
+        _ = process.wait()
+        delete_pid(procfile)

sith/pytest.py

@@ -0,0 +1,20 @@
+import atexit
+
+import pytest
+
+from .composer import start_composer, stop_composer
+from .settings import PROCFILE_SERVICE
+
+# it's the first hook loaded by pytest and can only
+# be defined in a proper pytest plugin
+# To use the composer before pytest-django loads
+# we need to define this hook and thus create a proper
+# pytest plugin. We can't just use conftest.py
+
+
+@pytest.hookimpl(tryfirst=True)
+def pytest_load_initial_conftests(early_config, parser, args):
+    """Hook that loads the composer before the pytest-django plugin"""
+    if PROCFILE_SERVICE is not None:
+        start_composer(PROCFILE_SERVICE)
+        _ = atexit.register(stop_composer, procfile=PROCFILE_SERVICE)

sith/settings.py

@@ -50,8 +50,23 @@ from .honeypot import custom_honeypot_error
 env = Env()
 env.read_env()
 
+
+@env.parser_for("optional_file")
+def optional_file_parser(value: str) -> Path | None:
+    if not value:
+        return None
+    path = Path(value)
+    if not path.is_file():
+        return None
+    return path
+
+
 BASE_DIR = Path(__file__).parent.parent.resolve()
 
+# Composer settings
+PROCFILE_STATIC = env.optional_file("PROCFILE_STATIC", None)
+PROCFILE_SERVICE = env.optional_file("PROCFILE_SERVICE", None)
+
 # Quick-start development settings - unsuitable for production
 # See https://docs.djangoproject.com/en/1.8/howto/deployment/checklist/
 
@@ -219,8 +234,7 @@ DATABASES = {
     "default": env.dj_db_url("DATABASE_URL", conn_max_age=None, conn_health_checks=True)
 }
 
-if "CACHE_URL" in os.environ:
-    CACHES = {"default": env.dj_cache_url("CACHE_URL")}
+CACHES = {"default": env.dj_cache_url("CACHE_URL")}
 
 SESSION_ENGINE = "django.contrib.sessions.backends.cached_db"
 

staticfiles/management/commands/runserver.py

@@ -1,3 +1,4 @@
+import atexit
 import os
 
 from django.conf import settings
@@ -6,19 +7,19 @@ from django.contrib.staticfiles.management.commands.runserver import (
 )
 from django.utils.autoreload import DJANGO_AUTORELOAD_ENV
 
-from staticfiles.processors import JSBundler, OpenApi
+from sith.composer import start_composer, stop_composer
+from staticfiles.processors import OpenApi
 
 
 class Command(Runserver):
     """Light wrapper around default runserver that integrates javascirpt auto bundling."""
 
     def run(self, **options):
-        # OpenApi generation needs to be before the bundler
         OpenApi.compile()
-        # Only run the bundling server when debug is enabled
-        # Also protects from re-launching the server if django reloads it
-        if os.environ.get(DJANGO_AUTORELOAD_ENV) is None and settings.DEBUG:
-            with JSBundler.runserver():
-                super().run(**options)
-                return
+        if (
+            os.environ.get(DJANGO_AUTORELOAD_ENV) is None
+            and settings.PROCFILE_STATIC is not None
+        ):
+            start_composer(settings.PROCFILE_STATIC)
+            _ = atexit.register(stop_composer, procfile=settings.PROCFILE_STATIC)
         super().run(**options)

staticfiles/processors.py

@@ -99,12 +99,6 @@ class JSBundler:
         if process.returncode:
             raise RuntimeError(f"Bundler failed with returncode {process.returncode}")
 
-    @staticmethod
-    def runserver() -> subprocess.Popen:
-        """Bundle js files automatically in background when called in debug mode."""
-        logging.getLogger("django").info("Running javascript bundling server")
-        return subprocess.Popen(["npm", "run", "serve"])
-
     @staticmethod
     def get_manifest() -> JSBundlerManifest:
         return JSBundlerManifest(BUNDLED_ROOT / ".vite" / "manifest.json")

uv.lock

@@ -155,7 +155,7 @@ name = "click"
 version = "8.1.8"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
-    { name = "colorama", marker = "platform_system == 'Windows'" },
+    { name = "colorama", marker = "sys_platform == 'win32'" },
 ]
 sdist = { url = "https://files.pythonhosted.org/packages/b9/2e/0090cbf739cee7d23781ad4b89a9894a41538e4fcf4c31dcdd705b78eb8b/click-8.1.8.tar.gz", hash = "sha256:ed53c9d8990d83c2a27deae68e4ee337473f6330c040a31d4225c9574d16096a", size = 226593 }
 wheels = [
@@ -595,6 +595,18 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/cc/04/eaa88433249ddfc282018d3da4198d0b0018e48768e137bfad304aacb1ec/hiredis-3.1.0-cp313-cp313-win_amd64.whl", hash = "sha256:9020fd7e58f489fda6a928c31355add0e665fd6b87b21954e675cf9943eafa32", size = 22004 },
 ]
 
+[[package]]
+name = "honcho"
+version = "2.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "colorama", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/65/c8/d860888358bf5c8a6e7d78d1b508b59b0e255afd5655f243b8f65166dafd/honcho-2.0.0.tar.gz", hash = "sha256:af3815c03c634bf67d50f114253ea9fef72ecff26e4fd06b29234789ac5b8b2e", size = 45618 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/48/1c/25631fc359955569e63f5446dbb7022c320edf9846cbe892ee5113433a7e/honcho-2.0.0-py3-none-any.whl", hash = "sha256:56dcd04fc72d362a4befb9303b1a1a812cba5da283526fbc6509be122918ddf3", size = 22093 },
+]
+
 [[package]]
 name = "ical"
 version = "8.3.1"
@@ -807,7 +819,7 @@ version = "1.6.1"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
     { name = "click" },
-    { name = "colorama", marker = "platform_system == 'Windows'" },
+    { name = "colorama", marker = "sys_platform == 'win32'" },
     { name = "ghp-import" },
     { name = "jinja2" },
     { name = "markdown" },
@@ -1092,6 +1104,21 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/a9/6a/fd08d94654f7e67c52ca30523a178b3f8ccc4237fce4be90d39c938a831a/prompt_toolkit-3.0.48-py3-none-any.whl", hash = "sha256:f49a827f90062e411f1ce1f854f2aedb3c23353244f8108b89283587397ac10e", size = 386595 },
 ]
 
+[[package]]
+name = "psutil"
+version = "7.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/2a/80/336820c1ad9286a4ded7e845b2eccfcb27851ab8ac6abece774a6ff4d3de/psutil-7.0.0.tar.gz", hash = "sha256:7be9c3eba38beccb6495ea33afd982a44074b78f28c434a1f51cc07fd315c456", size = 497003 }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/ed/e6/2d26234410f8b8abdbf891c9da62bee396583f713fb9f3325a4760875d22/psutil-7.0.0-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:101d71dc322e3cffd7cea0650b09b3d08b8e7c4109dd6809fe452dfd00e58b25", size = 238051 },
+    { url = "https://files.pythonhosted.org/packages/04/8b/30f930733afe425e3cbfc0e1468a30a18942350c1a8816acfade80c005c4/psutil-7.0.0-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:39db632f6bb862eeccf56660871433e111b6ea58f2caea825571951d4b6aa3da", size = 239535 },
+    { url = "https://files.pythonhosted.org/packages/2a/ed/d362e84620dd22876b55389248e522338ed1bf134a5edd3b8231d7207f6d/psutil-7.0.0-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1fcee592b4c6f146991ca55919ea3d1f8926497a713ed7faaf8225e174581e91", size = 275004 },
+    { url = "https://files.pythonhosted.org/packages/bf/b9/b0eb3f3cbcb734d930fdf839431606844a825b23eaf9a6ab371edac8162c/psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4b1388a4f6875d7e2aff5c4ca1cc16c545ed41dd8bb596cefea80111db353a34", size = 277986 },
+    { url = "https://files.pythonhosted.org/packages/eb/a2/709e0fe2f093556c17fbafda93ac032257242cabcc7ff3369e2cb76a97aa/psutil-7.0.0-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a5f098451abc2828f7dc6b58d44b532b22f2088f4999a937557b603ce72b1993", size = 279544 },
+    { url = "https://files.pythonhosted.org/packages/50/e6/eecf58810b9d12e6427369784efe814a1eec0f492084ce8eb8f4d89d6d61/psutil-7.0.0-cp37-abi3-win32.whl", hash = "sha256:ba3fcef7523064a6c9da440fc4d6bd07da93ac726b5733c29027d7dc95b39d99", size = 241053 },
+    { url = "https://files.pythonhosted.org/packages/50/1b/6921afe68c74868b4c9fa424dad3be35b095e16687989ebbb50ce4fceb7c/psutil-7.0.0-cp37-abi3-win_amd64.whl", hash = "sha256:4cf3d4eb1aa9b348dec30105c55cd9b7d4629285735a102beb4441e38db90553", size = 244885 },
+]
+
 [[package]]
 name = "psycopg"
 version = "3.2.3"
@@ -1486,7 +1513,7 @@ wheels = [
 [[package]]
 name = "sith"
 version = "3"
-source = { virtual = "." }
+source = { editable = "." }
 dependencies = [
     { name = "cryptography" },
     { name = "dict2xml" },
@@ -1501,12 +1528,14 @@ dependencies = [
     { name = "django-phonenumber-field" },
     { name = "django-simple-captcha" },
     { name = "environs", extra = ["django"] },
+    { name = "honcho" },
     { name = "ical" },
     { name = "jinja2" },
     { name = "libsass" },
     { name = "mistune" },
     { name = "phonenumbers" },
     { name = "pillow" },
+    { name = "psutil" },
     { name = "pydantic-extra-types" },
     { name = "python-dateutil" },
     { name = "redis", extra = ["hiredis"] },
@@ -1561,12 +1590,14 @@ requires-dist = [
     { name = "django-phonenumber-field", specifier = ">=8.0.0,<9.0.0" },
     { name = "django-simple-captcha", specifier = ">=0.6.0,<1.0.0" },
     { name = "environs", extras = ["django"], specifier = ">=14.1.0,<15.0.0" },
+    { name = "honcho", specifier = ">=2.0.0" },
     { name = "ical", specifier = ">=8.3.0,<9.0.0" },
     { name = "jinja2", specifier = ">=3.1.4,<4.0.0" },
     { name = "libsass", specifier = ">=0.23.0,<1.0.0" },
     { name = "mistune", specifier = ">=3.0.2,<4.0.0" },
     { name = "phonenumbers", specifier = ">=8.13.52,<9.0.0" },
     { name = "pillow", specifier = ">=11.0.0,<12.0.0" },
+    { name = "psutil", specifier = ">=7.0.0" },
     { name = "pydantic-extra-types", specifier = ">=2.10.1,<3.0.0" },
     { name = "python-dateutil", specifier = ">=2.9.0.post0,<3.0.0.0" },
     { name = "redis", extras = ["hiredis"], specifier = ">=5.2.0,<6.0.0" },