Surely the notion of who owns an AS should be cryptographically held so that an update has to be signed. Updates should be infrequent so the cost is felt on the control plane, not on the data plane.

I'm sure there's a BGPSec or whatever like all the other ${oldTech}Sec but I don't know if there is a realistic solution here or if it's IPv6 style tech.

0: I looked it up before posting and it's 3000 leakers with 12 million leaks per quarter https://blog.qrator.net/en/q3-2022-ddos-attacks-and-bgp-inci...

I'll bet JGC can write his own ticket by now, but unretiring would be really bad optics. He's on the board though and still keeping a watchful eye. But a couple more of these and CFs reputation will be in the gutter.

In this case, the timeline states "IMPACT STOP" was at 20:50 UTC and the first post to their status page was 12 minutes later at 21:02 UTC:

"Cloudflare experienced a Network Route leak, impacting performance for some networks beginning 20:25 UTC. We are working to mitigate impact."

Is there any way to test these changes against a simulation of real world routes? Including to ensure that traffic that shouldn’t hit Cloudflare servers, continues to resolve routes that don’t hit Cloudflare?

I have to imagine there’s academic research on how to simulate a fork of global BGP state, no? Surely there’s a tensor representation of the BGP graph that can be simulated on GPU clusters?

If there’s a meta-rule I think of when these incidents occur, it’s that configuration rules need change management, and change management is only as good as the level of automated testing. Just because code hasn’t changed doesn’t mean you shouldn’t test the baseline system behavior. And here, that means testing that the Internet works.

Basically, my understanding (simplified) is:

- they originally had a Miami router advertise Bogota prefixes (=subnets) to Cloudflare's peers. Essentially, Miami was handling Bogota's subnets. This is not an issue.

- because you don't normally advertise arbitrary prefixes via BGP, policies were used. These policies are essentially if/then statements, carrying out certain actions (advertise or not, add some tags or remove them,...) if some conditions are matched. This is completely normal.

- Juniper router configuration for this kind of policy is (simplifying):

set <BGP POLICY NAME> from <CONDITION1>

set <BGP POLICY NAME> from <CONDITION2>

set <BGP POLICY NAME> then <ACTION1>

set <BGP POLICY NAME> then <ACTION2>

...

- prior to the incident, CF changed its network so that Miami didn't have to handle Bogota subnets (maybe Bogota does it on its own, maybe there's another router somewhere else)

- the change aimed at removing the configurations on Miami which were advertising Bogota subnets

- the change implementation essentially removed all lines from all policies containing "from IP in the list of Bogota prefixes". This is somewhat reasonable, because you could have the same policy handling both Bogota and, say, Quito prefixes, so you just want to remove the Bogota part.

HOWEVER, there was at least one policy like this:

(Before)

set <BGP POLICY NAME> from is_internal(prefix) == True

set <BGP POLICY NAME> from prefix in bogota_prefix_list

set <BGP POLICY NAME> then advertise

(After)

set <BGP POLICY NAME> from is_internal(prefix) == True

set <BGP POLICY NAME> then advertise

Which basically means: if you have an internal prefix advertise it

- an "internal prefix" is any prefix that was not received by another BGP entity (autonomous system)

- BGP routers in Cloudflare exchange routes to one another. This is again pretty normal.

- As a result of this change, all routes received by Miami through some other Cloudflare router were readvertised by Miami

- the result is CF telling the Internet (more accurately, its peers) "hey, you know that subnet? Go ask my Miami router!"

- obviously, this increases bandwidth utilization and latency for traffic crossing the Miami router.

Why even bother to write an article about it then haha