uploading documentationHEADmaster

1 files changed, 83 insertions, 0 deletions

diff --git a/genesishosting/pmgenesisiorealignment.md b/genesishosting/pmgenesisiorealignment.md
new file mode 100644
index 0000000..8789bca
--- /dev/null
+++ b/genesishosting/pmgenesisiorealignment.md
@@ -0,0 +1,83 @@
+# Postmortem: Genesis I/O Realignment
+
+**Date:** May 8, 2025  
+**Author:** Doc  
+**Systems Involved:** minioraid5, shredder, chatwithus.live, zcluster.technodrome1/2, thevault  
+**Scope:** Local-first mirroring, permission normalization, MinIO transition
+
+---
+
+## 🎯 Objective
+
+To realign the Genesis file flow architecture by:
+
+- Making local block storage the **primary source** of truth for AzuraCast and Genesis buckets
+- Transitioning FTP uploads to target local storage instead of MinIO directly
+- Establishing **two-way mirroring** between local paths and MinIO buckets
+- Correcting inherited permission issues across `/mnt/raid5` using `find + chmod`
+- Preserving MinIO buckets as **backup mirrors**, not primary data stores
+
+---
+
+## 🔧 Work Performed
+
+### ✅ Infrastructure changes:
+- Deployed block storage volume to Linode Mastodon instance
+- Mirrored MinIO buckets (`genesisassets`, `genesislibrary`, `azuracast`) to local paths
+- Configured cron-based `mc mirror` jobs:
+  - Local ➜ MinIO: every 5 minutes with `--overwrite --remove`
+  - MinIO ➜ Local: nightly pull, no `--remove`
+
+### ✅ FTP Pipeline Adjustments:
+- Users now upload to `/mnt/spl/ftp/uploads` (local)
+- Permissions set so only admins access full `/mnt/spl/ftp`
+- FTP directory structure created for SPL automation
+
+### ✅ System Tuning:
+- Set `vm.swappiness=10` on all nodes
+- Apache disabled where not in use
+- Daily health checks via `pull_health_everywhere.sh`
+- Krang Telegram alerts deployed for cleanup and system state
+
+---
+
+## 🧠 Observations
+
+- **High load** on `minioraid5` during `mc mirror` and `chmod` overlap
+  - Load ~6.5 due to concurrent I/O pressure
+  - `chmod` stuck in `D` state (I/O wait) while `mc` dominated disk queues
+  - Resolved after `mc` completion — `chmod` resumed and completed
+
+- **MinIO buckets were temporarily inaccessible** due to permissions accidentally inherited by FTP group
+  - Resolved by recursively resetting permissions on `/mnt/raid5`
+
+- **Krang telemetry** verified:
+  - Mastodon swap usage rising under asset load
+  - All nodes had Apache disabled or dormant
+  - Health alerts triggered on high swap or load
+
+---
+
+## ✅ Outcome
+
+- Full Genesis and AzuraCast data now reside locally with resilient S3 mirrors
+- Mastodon running on block storage, no longer dependent on MinIO latency
+- FTP integration with SPL directory trees complete
+- Cleanup script successfully deployed across all nodes via Krang
+- Daily health reports operational with alerts for high swap/load
+
+---
+
+## 🔁 Recommendations
+
+- Consider adding snapshot-based ZFS backups for `/mnt/raid5`
+- Build `verify_mirror.sh` to detect drift between MinIO and local storage
+- Auto-trigger `chmod` only after `mc mirror` finishes
+- Monitor long-running background jobs with Krang watchdogs
+
+---
+
+**Signed,**  
+Doc  
+Genesis Hosting Technologies
+