1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
|
# Genesis Radio Internal Architecture Map
---
## 🏢 Core Infrastructure
| System | Purpose | Location |
|:---|:---|:---|
| Krang | Main admin server / script runner / monitoring node | On-premises / VM |
| SPL Server (Windows) | StationPlaylist Studio automation and playout system | On-premises / VM |
| Shredder | MinIO Object Storage / Cache server | On-premises / VM |
| PostgreSQL Cluster (db1/db2) | Mastodon database backend / Other app storage | Clustered VMs |
| Mastodon Server | Frontend social interface for alerts, community | Hosted at `chatwithus.live` |
---
## 🧠 Automation Components
| Component | Description | Hosted Where |
|:---|:---|:---|
| `mount_guardian.ps1` | Automatically ensures Rclone mounts (Q:\ and R:\) are up | SPL Server (Windows) |
| `rotate_mount_logs.ps1` | Weekly log rotation for mount logs | SPL Server (Windows) |
| `healthcheck.py` | Multi-node health and service monitor | Krang |
| Mastodon DM Alerts | Immediate alerting if something breaks (Mounts, Services) | Krang via API |
| Genesis Mission Control Landing Page | Web dashboard with Commandments + Live Healthcheck | Hosted on Krang's Nginx |
---
## 🎙️ Storage and Streaming
| Mount | Purpose | Backed by |
|:---|:---|:---|
| Q:\ (Assets) | Station IDs, sweepers, intro/outros, promos | GenesisAssets Bucket (Rclone) |
| R:\ (Library) | Full music library content | GenesisLibrary Bucket (Rclone) |
✅ Primary Cache: `L:\` (SSD)
✅ Secondary Cache: `X:\` (Spinning HDD)
---
## 📡 Communications
| Alert Type | How Sent |
|:---|:---|
| Mount Failures | Direct Mastodon DM |
| Healthcheck Failures (Disk, Service, SMART, RAID) | Direct Mastodon DM |
| Git Push Auto-Retry Failures (optional future upgrade) | Potential Mastodon DM |
---
## 📋 GitOps Flow
| Step | Script | Behavior |
|:---|:---|:---|
| Save changes | giteapush.sh | Auto stage, commit (timestamped), push to Gitea |
| Retry failed push | giteapush.sh auto-retry block | Up to 3x tries with 5-second gaps |
| Repo status summary | giteapush.sh final step | Clean `git status -sb` displayed |
✅ Follows GROWL commit style:
Good, Readable, Obvious, Well-Scoped, Logical.
---
## 📜 Policies and Procedures
| Document | Purpose |
|:---|:---|
| `OPS.md` | Healthcheck Runbook and Service Recovery Instructions |
| `GROWL.md` | Git Commit Message Style Guide |
| `Mission Control Landing Page` | Browser homepage with live dashboard + ops philosophy |
---
## 🛡️ Key Principles
- Calm is Contagious.
- Go Slow to Go Fast.
- Snappy Snaps Save Lives.
- Scripts are Smarter Than Sleepy Admins.
- If You Didn't Write It Down, It Didn't Happen.
---
# 🎙️ Genesis Radio Ops
Built with pride. Built to last. 🛡️🚀
|
