This deeply misunderstands the philosophy of Protobuf. proto3 doesn't even support required fields. https://protobuf.dev/best-practices/dos-donts/

> Never add a required field, instead add `// required` to document the API contract. Required fields are considered harmful by so many they were removed from proto3 completely.

Protobuf clients need to be written defensively, just like JSON API clients.

Sure it will blow up in your face when a field goes missing or value changes type.

People who advocate paying the higher cost ahead of time to perfectly type the entire data structure AND propose a process to do perform version updates to sync client/server are going to lose most of the time.

The zero cost of starting with JSON is too compelling even if it has a higher total cost due to production bugs later on.

When judging which alternative will succeed, lower perceived human cost beats lower machine cost every time.

This is why JSON is never going away, until it gets replaced with something with even lower human communication cost.

Unless your servers and clients push at different time, thus are compiled with different versions of your specs, then many safety bets are off.

There are ways to be mostly safe (never reuse IDs, use unknown-field-friendly copying methods, etc.), but distributed systems are distributed systems, and protobuf isn't a silver bullet that can solve all problems on author's list.

On the upside, it seems like protobuf3 fixed a lot of stuff I used to hate about protobuf2. Issues like:

> if the field is not a message, it has two states:

> - ...

> - the field is set to the default (zero) value. It will not be serialized to the wire. In fact, you cannot determine whether the default (zero) value was set or parsed from the wire or not provided at all

are now gone if you stick to using protobuf3 + `message` keyword. That's really cool.

Searched the article, no mention of gzip, and how most of the time all that text data (html, js and css too!) you're sending over the wire will be automatically compressed to...... an efficient binary format!

So really, the author should compare protobufs to gzipped JSON

Protobuf has advantages, but is missing support for a tons of use cases where JSON thrives due to the strict schema requirement.

A much stronger argument could be made for CBOR as a replacement for JSON for most use cases. CBOR has the same schema flexibility as JSON but has a more concise encoding.

https://en.wikipedia.org/wiki/ASN.1#Example_encoded_in_DER

Protobuf is ok but if you actually look at how the serializers work, it's just too complex for what it achieves.

Technically, it sounds really good but the actual act of managing it is hell. That or I need a lot of practice to use them, at that point shouldn't I just use JSON and get on with my life.

It appears possible in some cases but it's not universally the case. Which means that similar binary transport formats that do support zero-copy, like Cap'n Proto, offer most or all of the perks described in this post, with the addition of ensuring that serialization and deserialization are not a bottleneck when passing data between processes.

So my question is, why didn't Google just provide that as a library? The setup wasn't hard but wasn't trivial either, and had several "wrong" ways to set up the proto side. They also bait most people with gRPC, which is its own separate annoying thing that requires HTTP/2, which even Google's own cloud products don't support well (e.g App Engine).

P.S. Text proto is also the best static config language. More readable than JSON, less error-prone than YAML, more structure than both.

> → About 23 bytes.

This appears to be a nice ai-generated result. There are already at least 23 bytes of data there (number 42 (1 byte) + string of 5 chars + string of 17 chars + 1 boolean), so that plus field overhead will be more.

1. Size, biggest problem with JSON can happen when things gets too big. So here other formats might be better. Yet, as a reminder JSON has the binary version named BSON.

2. Zero batteries. JSON is readable by humans but also almost self explanatory format. Most languages has built in or quick drop in for json. Still, it’s easy to implement a limited JSON parser from scratch when in need (eg. Pure on func in C on a tiny device).

Working with Protobuf and MsgPack in the past, You have much more tooling involved especially if data passes between parts written in different languages.

3. Validation, JSON is simple. But there are solutions such as JSON Schema.

Plus - tooling.

JSON might not be simpler or strict but it gets the job done.

If JSON's size or performance is causing you to go out of business, you surely have bigger problems than JSON.

138 million downloads from npm in the last week. Yes, you can validate your JSON

The app 20req/sec

The app after optimizations: 20req/sec (It waits for db query anyway)

Due to very native nature of JSON in browsers and node backend it usually also the fastest data format

If for example you have C++ on backend and C++ on frontend - you'll definitely have some performance boost

But for browsers usage the goal is not so obvious

As an aside, like all things Google, their C++ library is massive (14mb dll) and painful to build (takes nearly 10 minutes on my laptop).

Distributing the contract with lock-step deploys between services is a bit much. JSON parsing time and size are not usually key factors in my projects. Protobuf doesn’t pose strict requirements for data validation, so I have to do that anyway. Losing data readability in transit is a huge problem.

Protobuf seems like it would be amazing in situations where data is illegible and tightly coupled such as embedded CAN or radio.

When you'll go through your own journey, and inevitably end back with json, do write another blog post :) ... we've all been there.

If you use good tooling, you can have a mutation change a variable type in the database and that type change is automatically reflected in the middleware/backend and the typescript UI code. Not only that libraries like HotChocolate for asp.net come with built-in functions for filtering, pagination, streaming etc.

We might disagree on what "efficient" means. OP is focusing on computer efficiency, where as you'll see, I tend to optimize for human efficiency (and, let's be clear, JSON is efficient _enough_ for 99% of computer cases).

I think the "human readable" part is often an overlooked pro by hardcore protobuf fans. One of my fundamental philosophies of engineering historically has been "clarity over cleverness." Perhaps the corollary to this is "...and simplicity over complexity." And I think protobuf, generally speaking, falls in the cleverness part, and certainly into the complexity part (with regards to dependencies).

JSON, on the other hand, is ubiquitous, human readable (clear), and simple (little-to-no dependencies).

I've found in my career that there's tremendous value in not needing to execute code to see what a payload contains. I've seen a lot of engineers (including myself, once upon a time!) take shortcuts like using bitwise values and protobufs and things like that to make things faster or to be clever or whatever. And then I've seen those same engineers, or perhaps their successors, find great difficulty in navigating years-old protobufs, when a JSON payload is immediately clear and understandable to any human, technical or not, upon a glance.

I write MUDs for fun, and one of the things that older MUD codebases do is that they use bit flags to compress a lot of information into a tiny integer. To know what conditions a player has (hunger, thirst, cursed, etc), you do some bit manipulation and you wind up with something like 31 that represents the player being thirsty (1), hungry (2), cursed (4), with haste (8), and with shield (16). Which is great, if you're optimizing for integer compression, but it's really bad when you want a human to look at it. You have to do a bunch of math to sort of de-compress that integer into something meaningful for humans.

Similarly with protobuf, I find that it usually optimizes for the wrong thing. To be clear, one of my other fundamental philosophies about engineering is that performance is king and that you should try to make things fast, but there are certainly diminishing returns, especially in codebases where humans interact frequently with the data. Protobufs make things fast at a cost, and that cost is typically clarity and human readability. Versioning also creates more friction. I've seen teams spend an inordinate amount of effort trying to ensure that both the producer and consumer are using the same versions.

This is not to say that protobufs are useless. It's great for enforcing API contracts at the code level, and it provides those speed improvements OP mentions. There are certain high-throughput use-cases where this complexity and relative opaqueness is not only an acceptable trade off, but the right one to make. But I've found that it's not particularly common, and people reaching for protobufs are often optimizing for the wrong things. Again, clarity over cleverness and simplicity over complexity.

I know one of the arguments is "it's better for situations where you control both sides," but if you're in any kind of team with more than a couple of engineers, this stops being true. Even if your internal API is controlled by "us," that "us" can sometimes span 100+ engineers, and you might as well consider it a public API.

I'm not a protobuf hater, I just think that the vast majority of engineers would go through their careers without ever touching protobufs, never miss it, never need it, and never find themselves where eking out that extra performance is truly worth the hassle.

It's the native format Erlang nodes use to serialize data to communicate with each other, but it's just a simple Type-Length-Value binary format, so anything can implement it.

It's small enough that you can create a reasonably complete and fast implementation of it for your language in an afternoon. It's self-describing, so if you can read ETF you can read any ETF message, as there are no out of band schemas. I love it.

[0] - https://www.erlang.org/doc/apps/erts/erl_ext_dist.html

But we have built protobuf into a web server that handles 2 requests per second. Why? We wanted to learn about it on the job.

I think that's 99% of Protobuf usage.

>If you develop or use an API, there’s a 99% chance it exchanges data encoded in JSON.

Just wondering if the inherent defiencies of JSON can somewhat be improved by CUE lang since the former is very much pervasive and the latter understand JSON [1],[2].

[1] Configure Unify Execute (CUE): Validate, define, and use dynamic and text‑based data:

https://cuelang.org/

[2] Cue – A language for defining, generating, and validating data:

https://news.ycombinator.com/item?id=20847943

If that is just your team, use whatever tech gets you there quick.

However, if you need to provide some guarantees to a second or third party with your API, embrace standards like JSON, even better, use content negotiation.

I too had this overly restrictive view of "APIs" for too long. One I started to think about it as the interface between any teo software components it really changed the way I did programming. In other words, a system itself is composed and that composition is done via APIs. There's no point treating "the API" as something special.

The design overhead involved in determining the correct URL and HTTP method adds a layer of subjectivity to the design and bike shedding arguments.

I’m not a huge fan of Protobuf/GRPC either, if there’s a better alternative I believe RPC is the right approach for exposing APIs.

I can't imagine using protobuf when you're in the first 5 years of a product.

Now, there is a serde_protobuf (I haven't used it) that I assume allows you to enforce nullability constraints but one of the article's points is that you can use the generated code directly and:

> No manual validation. No JSON parsing. No risk of type errors.

But this is not true—nullability errors are type errors. Manual validation is still required (except you should parse, not validate) to make sure that all of the fields your app expects are there in the response. And "manual" validation (again, parse don't validate) is not necessary with any good JSON parsing library, the library handles it.

Had the joy of implementing calling SOAP service with client generated from wsdl in .net core 2/3 times. Tooling was shit poorly undocumented amd with crappy errors at that time. Run only with very specific versions in very specific way. And not much you can do about it, rolling your own SOAP client would be too expensive for our team.

REST with JSON meanwhile is easy and we do it all the time, don't need any client, just give us request/response spec and any docs.

Getting everyone to agree on a standard was/is/will be the tougher part.

There is a really interesting discussion underneath of this as to the limitations of JSON along with potential alternatives, but I can't help but distrust this writing due to how much it sounds like an LLM.

A sizable portion of the integrations I've built have had to be built by hand, because there are inevitable stupid quirks and/or failures I've had to work around. For these usecases, using JSON is preferable, because it is easy for me to see what I have actually been sent, not what the partially up to date spec says I should've been sent.

This is consistent with the idea that communication over the internet should consist of (encrypted and compressed) plain text. It's because human beings are going to have to deal with human reality at the end of the day.

https://en.wikipedia.org/wiki/DCE/RPC

DCE/RPC worked in 1993, and still does today.

Protocol buffers is just another IDL.