diff options
author | doc <doc@filenotfound.org> | 2025-06-30 20:06:28 +0000 |
---|---|---|
committer | doc <doc@filenotfound.org> | 2025-06-30 20:06:28 +0000 |
commit | 717fcb9c81d2bc3cc7a84a3ebea6572d7ff0f5cf (patch) | |
tree | 7cbd6a8d5046409a82b22d34b01aac93b3e24818 /blog | |
parent | 8368ff389ec596dee6212ebeb85e01c638364fb3 (diff) |
Diffstat (limited to 'blog')
-rw-r--r-- | blog/deathtominio.md | 39 | ||||
-rw-r--r-- | blog/docker.md | 118 | ||||
-rw-r--r-- | blog/minio.md | 77 | ||||
-rw-r--r-- | blog/toolkit.md | 69 |
4 files changed, 303 insertions, 0 deletions
diff --git a/blog/deathtominio.md b/blog/deathtominio.md new file mode 100644 index 0000000..cf423be --- /dev/null +++ b/blog/deathtominio.md @@ -0,0 +1,39 @@ +# Death to Object Storage: A Love Letter to Flat Files + +Once upon a time, I believed in MinIO. + +I really did. The idea was beautiful: S3-compatible object storage, self-hosted, redundant, robust — all those wonderful buzzwords they slap on the side of a Docker image and call “enterprise.” I bought into it. I built around it. I dreamed in buckets. + +And then, reality set in. + +What reality, you ask? + +- Media uploads timing out. +- Phantom 403s from ghosts of CORS configs past. +- Uploader works on Tuesday, breaks on Wednesday. +- “Why are all the thumbnails gone?” +- “Why does the backup contain *literally nothing*?” + +MinIO became that coworker who talks a big game but never shows up to help move the server rack. Sure, he says he's “highly available” — but when you need him? Boom. 503. + +So I did what any burned-out, overcaffeinated sysadmin would do. I tore it all down. + +Flat files. ZFS. Snapshots. The old gods. + +Now my media lives on Shredder. It’s fast. It’s simple. It scrubs itself weekly and never lies to me. Want to know if something's backed up? I check with my own eyes — not by playing 20 questions with a broken object path and a timestamp from the Nixon administration. + +I don’t have to `mc alias` anything. +I don’t need to care about ACLs. +I don’t need to learn how to spell “presigned URLs” ever again. + +It. Just. Works. + +So, farewell MinIO. You tried. You failed. You’re off my network. + +Long live `chmod -R`, long live ZFS, and long live sysadmins who know when to throw the whole stack in the trash and start over. + +--- + +📌 PS: If you’re still on object storage for your Mastodon instance… I’m sorry. I really am. + + diff --git a/blog/docker.md b/blog/docker.md new file mode 100644 index 0000000..f4b4e5f --- /dev/null +++ b/blog/docker.md @@ -0,0 +1,118 @@ +Fuck Docker +It works, but it gaslights you about everything. + +Docker is amazing when it works. And when it doesn’t? +It’s a smug little daemon that eats your RAM, forgets your volumes, lies about its health, and restarts things for reasons it refuses to explain. + +Scene 1: Everything Is Fine™ + +You run: +docker ps + +It tells you: +azuracast Up 30 seconds +db Up 31 seconds +nginx Up 30 seconds + +Everything is up. +Except the site is down. +The UI is dead. +curl gives you nothing. +The logs? Empty. + +Docker: “Everything’s running fine 👍” + +Scene 2: Logs Are a Lie + +docker logs azuracast + +Returns: + + Just enough output to give you hope + + Then nothing + + Then silence + +You tail it. +You restart it. +You exec into it. +It’s just a tomb with a PID. + +Scene 3: It Forgets Everything + +You reboot the host. + +Suddenly: + + Your containers forget their volumes + + Your docker-compose.override.yml is ignored + + Your networks vanish + + And the bridge interface is now possessed + +Scene 4: Volumes Are Haunted + +docker volume rm azuracast_station_data + +Error: volume is in use + +By what? +You stopped all containers. You nuked the services. +It’s still in use — by ghosts. + +Eventually you just: + +rm -rf /var/lib/docker + +Because therapy is cheaper than debugging this. + +Scene 5: docker-compose Is a Trick + +docker-compose down +docker-compose up -d + +Now: + + Some things are gone + + Some things are doubled + + Your stations/ folder is missing + + And your database container is holding a grudge + +You try to roll back. +There is no roll back. Only sadness. + +Scene 6: It’s Not Even Docker Anymore + +Modern Docker is: + + Docker + + Which is actually Moby + + Which uses containerd + + Which is managed by nerdctl + + Which builds with buildkit + + Which logs via journald + + Which stores data in an OCI-conforming mess of layers + +None of it can be managed with just docker. + +Final Thought + +Docker is powerful. +Docker is everywhere. +Docker changed the world. + +But once you run real infrastructure on it? + +Fuck Docker. diff --git a/blog/minio.md b/blog/minio.md new file mode 100644 index 0000000..b08df44 --- /dev/null +++ b/blog/minio.md @@ -0,0 +1,77 @@ +# MinIO: It Works, But It Hates Me + +*By someone who survived a 150,000-file sync and lived to tell the tale.* + +--- + +MinIO is fast. It's lightweight. It's compatible with Amazon S3. It’s everything you want in a self-hosted object storage system. + +Until you try to **use it like a filesystem**. + +Then it becomes the most temperamental, moody, selectively mute piece of software you've ever met. + +--- + +## What I Was Trying to Do + +All I wanted was to migrate ~40GB of Mastodon media from local disk into a MinIO bucket. Nothing fancy. Just a clean `rclone sync` and a pat on the back. + +--- + +## What Actually Happened + +- **Load average spiked to 33** +- `find` froze +- `rclone size` hung +- `zfs snapshot` stalled so long I thought the server died +- The MinIO **UI lied to my face** about how much data was present (5GB when `rclone` said 22GB) +- Directory paths that looked like files. Files that were secretly directories. I saw `.meta` and `.part.1` in my dreams. + +--- + +## The Root Problem + +MinIO is **not** a filesystem. + +It's a flat key-value object store that's just *pretending* to be a folder tree. And when you throw 150,000+ nested objects at it — especially from a tool like `rclone` — all the lies unravel. + +It keeps going, but only if: +- You feed it one file at a time +- You don’t ask it questions (`rclone ls`, `rclone size`, `find`, etc.) +- You don’t use the UI expecting it to reflect reality + +--- + +## The Fixes That Kept Me Sane + +- Switched from `rclone ls` to `rclone size` with `--json` (when it worked) +- Cleaned up thousands of broken `.meta`/`.part.*` directories using a targeted script +- Paused `rclone` mid-sync with `kill -STOP` to get snapshots to complete +- Used `du -sh` instead of `find` to track usage +- Lowered `rclone` concurrency with `--transfers=4 --checkers=4` +- Drank water. A lot of it. + +--- + +## The Moral of the Story + +If you're going to use MinIO for massive sync jobs, treat it like: + +- A **delicate black box** with fast internals but fragile mood +- Something that **prefers to be written to, not inspected** +- An S3 clone with boundary issues + +--- + +## Final Thought + +MinIO *does* work. It's powerful. It’s fast. But it also absolutely hates being watched while it works. + +And you won't realize how much until you're 100,000 files deep, snapshot frozen, and `rclone` is telling you you're doing great — while the UI smirks and says you're at 5 gigs. + +MinIO: It works. +But it hates you. + +--- + +**Filed under:** `disaster recovery`, `object storage`, `sync trauma`, `zfs`, `rclone`, `why me` diff --git a/blog/toolkit.md b/blog/toolkit.md new file mode 100644 index 0000000..6b86cb1 --- /dev/null +++ b/blog/toolkit.md @@ -0,0 +1,69 @@ +# Building Your Own CLI Toolkit: Introducing `genesisctl` + +After weeks of refining backup scripts, documenting resilience routines, and shoveling thousands of lines of shell logic into shape, we now have something more powerful than the sum of its parts: a unified, self-documenting, command-line interface for managing infrastructure scripts — `genesisctl`. + +## What is `genesisctl`? + +`genesisctl` is a simple but powerful Bash tool designed to manage and interact with the full suite of sysadmin scripts under the Genesis infrastructure umbrella. It pulls together documentation, logging, and execution into one cohesive interface. + +## Features + +* 🔍 `describe <tool.sh>` — Pretty-prints documentation and metadata from Markdown files auto-generated by our toolchain. +* 📋 `list` — Displays all installed tools based on your setup logs. +* 🚀 `run <tool.sh>` — Executes a script from your `bin/` folder like a command-line native. + +## Why This Matters + +When you're running dozens of bash scripts across multiple machines — backups, verifications, restores, syncs, DR drills — things get messy. With `genesisctl`, every script: + +* Has structured metadata +* Lives in a clean hierarchy +* Comes with a Markdown doc +* Can be queried or executed with a single, consistent command + +## Example Usage + +```bash +# Describe a script and its purpose +$ ./genesisctl describe backup.sh + +# List everything you've got installed +$ ./genesisctl list + +# Run a ZFS bootstrap script +$ ./genesisctl run zfs_bootstrap.sh +``` + +## Behind the Scenes + +Every time we run our scaffold script (`setup_genesis_tools.sh`), it: + +* Reorganizes the toolchain into folders (`bin/`, `docs/`, `archive/`) +* Generates Markdown from script headers (with frontmatter) +* Logs every install to a central file + +Then `genesisctl` reads that metadata in real-time — no guesswork, no rot. + +## What's Next + +This framework is rock solid for CLI use. Next steps may include: + +* `genesisctl doctor` — to validate all tools have docs and correct structure +* `genesisctl docgen` — regenerate Markdown docs on demand +* Static site export of docs with category tags + +But for now? It's stable, extensible, and battle-tested. + +## Final Thoughts + +If you’ve ever tried to manage 50+ bash scripts without structure, `genesisctl` is the toolkit you wish you had. With it, documentation isn't an afterthought — it's baked in. + +Stay tuned. This thing’s just getting started. + +--- + +📁 Repo: Coming soon to Gitea. + +📬 Ping @doc if you want help wiring this into your own ops stack. + +💀 FailZero approved. |