Stefano Marinelli@stefano@mastodon.bsd.cafe

I managed an e-commerce server for about ten years. It grew from a small local shop into a major national business, even handling international orders. They expanded to the point where they reduced their local brick-and-mortar store hours because the bulk of their revenue was coming from online sales.

Then one seller came along and convinced them that switching to Shopify would be the key to growing even further. Apparently, their €130/month bare-metal redundant setup - which boasted a calculated uptime of 99.995% over 10 years - just wasn't cutting it anymore.

They’ve been on Shopify for about six months now, and every now and then, I still get the alerts. I left the monitoring active via Uptime Kuma and ran the numbers. Over the last six months, their uptime dropped below 98%.
In other words, in just six months, they’ve been down for almost as many hours as they were during the entire previous decade.

I contacted the client - not because I want to take over the hosting again, but just to understand what on earth happened (we're on excellent terms). Their response was: "We don't know, but if it happened on Shopify, it means it was bound to happen anyway."

As long as we keep swallowing the lie that "the cloud" and "tech giants" are always the right solution for us, we completely deserve the cloud and the tech giants.

#IT #SysAdmin #OwnYourData

David Chisnall (*Now with 50% more sarcasm!*)@david_chisnall@infosec.exchange

Replying to @crankylinuxuser@infosec.exchange

@crankylinuxuser @stefano @wronglang

With small systems, you need complete duplicates for reliability because one failure can take out the entire system. But this isn’t true at larger scales.

If you run a personal mail server and want to handle failure of a single machine, you need two computers, which doubles the cost. If you are hosting enough that it’s worth having two computers and load balancing between them, if one can handle the load with slightly degraded performance, you can get away without redundancy, but if you want to preserve performance in the event of single-node failure, you need one spare computer, which adds only 50% to the cost.

At cloud scales, this should be much cheaper. Cloud storage is very reliable because they build rack-scale storage systems that expose virtual block devices to the rest of the system and do a load of error-correction coding across disks for redundancy. They assume some percentage of disks (based on models informed by prior failures) will fail and design the systems around this. For cool storage, MSR had a system (which, I think, was deployed around the time I left MS) that took advantage of really bad disks that were guaranteed for 50 rewrite cycles for cold data. Over 50% of the total used disk bandwidth was spent rewriting redundancy because those disks failed a lot, but that didn’t matter, the rack kept working.

And yet, in spite of that, everything else seems t9 have terrible uptime. When I was at MS, my personal email had higher uptime than my work email. My personal email was self hosted on a single machine. Work email was on M365 and was supposedly run on a cluster of mail servers spread across independent failure domains.