Amazon might think that they’re locking people in with the egress fees. But they’re also locking people out. As soon as you switch one part to a competitor, the high egress forces you to switch over everything.

It’s going to be complicated to switch, but it’s made easier by the fact that I didn’t fall into the trap of building my platform on Amazon-specific services.

You saved a lot of money but you'll spend a lot of time in maintenance and future headaches.

For backups we use both Velero and application-level backup for critical workloads (i.e. Postgres WAL backups for PITR). We also ensure all state is on at least two nodes for HA.

We also find bare metal to be a lot more performant in general. Compared to AWS we typically see service response times halve. It is not that virtualisation inherently has that much overhead, rather it is everything else. Eg, bare metal offers:

- Reduced disk latency (NVMe vs network block storage)

- Reduced network latency (we run dedicated fibre, so inter-az is about 1/10th the latency)

- Less cache contention, etc [1]

Anyway, if you want to chat about this sometime just ping me an email: adam@ company domain.

[0] https://lithus.eu

[1] I wrote more on this 6 months ago: https://news.ycombinator.com/item?id=45615867

I see the DigitalOcean vs Hetzner comparison as a tradeoff that we make in different domains all day long, similar to opening your DoorDash or UberEats instead of making your own dinner(and the cost ratio is similar too).

I work in all 3 major clouds, on-prem, the works. I still head to the DigitalOcean console for bits and pieces type work or proof of concept testing. Sometimes you just want to click a button and the server or bucket or whatever is ready and here's the access info and it has sane defaults and if I need backups or whatnot it's just a checkbox. Your time is worth money too.

However, the dealbreaker for me was that Hetzner IPs have a bad reputation. At work, I learned that one of the managed AWS firewall rules blocks many (maybe all) of their IPs. I can’t even open a website hosted on a Hetzner IP from my work laptop because it’s blocked by some IT policy (maybe this is not an issue for you if you are using CloudFlare or similar).

I've read online that the DDoS protection is very bad as well.

So in the end, I picked DO App Platform in one of the EU regions. Having the option to use a managed DB was a big plus as well.

Because with a single-server setup like this, I'd imagine that hardware (e.g. SSD) failure brings down your app, and in the case of SSD failure, you then have hours or days downtime while you set everything up again.

- reduce dns ttl (if not doing an ip swap)

- rsync website files

- rsync /etc/letsencrypt/ ssl certificates

- copy over database (if writes don't happen often and database is small enough, this can be done without replica, just go read_only during migration)

- test new server by putting new ip in local /etc/hosts

- turn off cron on old server

- convert old server nginx to reverse proxy to new server

- change dns (or ip swap between old and new server)

- turn on cron on new server

Nice to see it used _twice_ :D

  If you’re migrating a large MySQL database and you’re not 
  using mydumper/myloader, you’re doing it the hard way.

BTW, I've been a client of Hetzner (Cloud, Object Storage, and Storage Box) for a few years now, very happy with them!

Not really, a MITM could do anything here. It's not very likely to happen here, but I think this comment shows a misunderstanding of what certificates and verification does.

Sounds like from the requirement to live migrate you can't really afford planned downtime, so why are you risking unplanned downtime?

This reasoning does not add up. They could simply say they needed to move somewhere cheaper, like Hetzner. Inflation is still high but getting lower. Weakened Turkish Lira part is not correct because dollar is artificially suppressed for a very long time.

The issue is though, that you loose the managed part of the whole Cloud promise. For ephemeral services this not a big deal, but for persistent stuff like databases where you would like to have your data safe this is kind of an issue because it shifts additional effort (and therefore cost) into your operations team.

For smaller setups (attention shameless self-promotion incoming) I am currently working on https://pellepelster.github.io/solidblocks/cloud/index.html which allows to deploy managed services to the Hetzner Cloud from a Docker-Compose like definition. E.g. a PostgreSQL database with automatic backup and disaster recovery.

They do offer VPS in the US and the value is great. I was seriously looking at moving our academic lab over from AWS but server availability was bad enough to scare me off. They didn't have the instances we needed reliably. Really hoping that calms down.

What was the config on the receiving side to support this? Did you whitelist the old server IP to trust the forwarding headers? Otherwise you’d get the old server IP in your app logs. Not a huge deal for an hour but if something went wrong it can get confusing.

This isn't something others should use as an example.

https://slitherworld.com

My foray into multiplayer games.

None of it is mission critical - but it’s certainly something I’d use in production with a few more instances.

Networking over Tailscale works flawlessly with my Proxmox nodes at home.

you can basically go on hetzner and spin up a vps with linux that is exposed to the open internet with open ports and user security and within a few hours its been hacked, there is no like warning pop up that says "if you do this your server will be pwnd"

i especialy wonder with all the ai provisioned vps and postgres dbs what will happen here

Namely, all remote access (including serving http) must managed by a major player big enough to be part of private disclosure (e.g. Project Glasswing).

That doesn't mean we have to use AWS et al for everything, but some sort of zero trust solution actively maintained by one of them seems like the right path. For example, I've started running on Hetzner with Cloudflare Tunnels.

Anyone else doing something similar?

For OP though who is a Turkey-based company and want European region servers anyway, it might make sense.

If some people can chime in with their positive experiences I might switch.

Yeah - no, it's not. They made the MitM attack possible with this change. The exposure was limited to those 5 minutes, but it should have been a known risk.

Also not certain how they could check the apps on the new server with the read-only database, while it was a replica?

Still, nice to hear it succeeded, the reasons sound very familiar.

Asking the obvious question: why not your own server in a colo?

Over the years I've heard plenty of horror stories from Hetzner customers.

  > 30 MySQL databases (248 GB of data)
  > 34 Nginx virtual hosts across multiple domains
  > GitLab EE (42 GB backup)
  > Neo4J Graph DB (30 GB graph database)
  > Supervisor managing dozens of background workers
  > Gearman job queue
  > Several live mobile apps serving hundreds of thousands of users

I'm not against vertical scaling and stuff, but 30 db instances in one server is just crazy.

And i say it every time they came up: Their cloud UX is brilliant and simple! Compared to the big ones out there.

Moving away from the US also felt great.

Happened to me.

I now advise people to avoid clown-led services like Hetzner and stick to more reputable, if not as cheap, options.

Full of scanners, script kiddies and maybe worse.

When I’ve seen this work well, it’s either built into the product as an established feature, or it’s a devops procedure that has a runbook and is done weekly.

Doing it with low level commands and without a lot of experience is pretty likely to have issues. And that’s what happened here.

DigitalOcean just absolutely is just not an enterprise solution. Don't trust it with your data.

Oh, and did I mention I had been paying the upcharge for backups the entire time?

As such, I doubt the noted price reduction is reproducible. Combine this with Hetzner's sudden deletions of user accounts and services without warning, and it's a bad proposition. Search r/hetzner and r/vps for hetzner for these words: banned, deleted, terminated; there are many reports. What should stun you even more about it is that Hetzner could ostensibly be closely spying on user data and workloads, even offline workloads, without which they won't even know who to ban.

The only thing that Hetzner might potentially be good for is to add to an expendable distributed compute pool, one that you can afford to lose, but then you might as well also use other bottom-of-the-barrel untrustworthy providers for it too, e.g. OVH.

Cloud is ludicrously marked up.

Plus, this is not what DHH was doing, he was not saving few bucks, but unlocking potential for his company to thrive.

So they did same mistake all over again. Debian or Ubuntu would just be upgrade-in-place and migrate