Rewriting Rust - https://news.ycombinator.com/item?id=41654871 - Sept 2024 (385 comments)

A tricky thing that comes up with Rust comparisons is that often, Rust has a feature that's weird or hard to use, but it's because that's the only solution that makes sense within the constraints Rust has placed upon itself. Clicking some links gets me to https://git.yzena.com/Yzena/Yc/src/branch/master/docs/yao/de...

> Yao's biggest goals are: Correctness, Convenience, Performance, in that order.

Having clear goals is a great thing when making a language, because they give you guidance on what is appropriate to include and what is not.

Rust has certainly demonstrated having similar goals, but in a slightly different order: Correctness, Performance, and then Convenience. So it wouldn't shock me if Yao could do some things better than Rust, in accordance with its goals. But that also means that sometimes, Rust would be useful where Yao cannot be. Everything is tradeoffs.

Incidentally, I actually think that figuring out what your values are, and what your needs are, is a great way to pick a programming language. Decide what matters to you as an engineer, and then find a language that shares similar values. I gave a conference talk a few years back on this idea, and how I viewed Rust's values at the time https://www.infoq.com/presentations/rust-tradeoffs/

This was based off of bcantrill's Platform as a Reflection of Values, which was very influential on me. https://www.youtube.com/watch?v=Xhx970_JKX4

If you've ever heard about Oxide's focus on values, this is some of the older background on that.

- Is there really a major use case for function traits? Is more elaboration of the type system actually useful?

- The original "Rewriting Rust" article discussed replacing Rust's macro system. That's not mentioned here. If you had to write your macros in Rust, to be run in compile time, would that be a win? I doubt it. Each macro would start with a call to a parser, and then you generate code as strings. This would pay off only for complex cases such as a regular expression compiler. Rust already has compile-time build programs, where you can run Rust during the build process. (I use that to take in an API definition from another source and generate Rust.)

- "Runtime capabilities" seems to be an attempt to build a container system into the language. It's not clear that helps much. That's really the OS's job. Protecting programs against themselves is nice, but not usually the problem. I'd like to see something in Android that gives an app file access with less access than "all files". Like "Can read what was installed for this app, can write working subdirectory for this app, and nothing else."

* Language creators start with high hopes and big promises. "We've learned from the past! This will solve all the problems of previous X,Y,Z languages! It will be so much easier to use!"

* The language begins to be designed, implemented, and developed. Problems arise. Tradeoffs must be made. Invariably, the new language develops its own set of limitations, shortcomings, and weaknesses. These are inevitable with any new programming language.

* Ultimately the language either finds a userbase who are willing to overlook or work around the limitations and pitfalls, or it fades away.

* If it remains, eventually another new language arises saying "We've learned from the past! This will solve all the problems of previous X,Y,Z languages! It will be so much easier to use!" And the cycle repeats.

Now, is this a bad thing? I would argue no, not necessarily. Every time a new language is created, our vision of what is possible is expanded. We learn new lessons about what works and what doesn't work in programming languages.

So in short I'm looking forward to see how the state of the art is advanced by Yao. If it gets to a decent working state, I might even try it out if I have time. However, in light of the above statements forgive me for being skeptical of the following claim from Yao's website [1]:

> As powerful as C, As flexible as Lisp, As easy as Python, As provable as Ada/SPARK, More reliable than Rust, With the time semantics of HAL/S.

[1]: https://git.yzena.com/Yzena/Yc/src/branch/master/docs/yao/ma...

Hah, I wrote this section and was deeply aware of these problems at the time. Let’s just say it didn’t fit in the introduction. In fact, a lot of the understanding of this issue has been expressed later, blog post by blog post, largely without coordination. It’s almost eerie to see again and again people encountering the same issue, and spending huge effort to address it (this one takes the prize though - making your own language).

Let me try a to narrativize it: we have a fundamental problem with unrestricted concurrent control flow (very similar to how we used to have unrestricted sequential control flow when we had goto). This problem exists in most programming languages, but we mostly pretend it’s not there (and live with the subtle bugs instead). However, with Rust’s unique ownership- and safety model, the lack of structured concurrency was an unavoidable problem, at collision course with Rust itself. In the Rust we have today (without structured concurrency), a bunch of expected and reasonable behaviors cannot be achieved (such as borrowing from a parent task).

Koka is memory safe without using traditional GC, has effects, and is pretty cool over all.

[0] https://koka-lang.github.io/koka/doc/index.html

What happens when you need to add a new trait that all existing functions must be changed to have? That's why async is generally an attribute.

The answer is that we'd have to say that all functions have some default trait set that can be extended automatically as the language evolves, except those anti-traits they have. Then we could say that a function is async meaning "not sync", or even we could write !sync instead of async.

I rather like the idea that you have to write `!sync` instead of `async`.

"In fact, it might be better said that Yao is meant to be:

As powerful as C, As flexible as Lisp, As easy as Python, As provable as Ada/SPARK, More reliable than Rust, With the time semantics of HAL/S."

Even if I don't really believe in this incredible statement, I'm really curious to see what can be the result.

Serious mode off. When will we get Yao in the Linux kernel?

There are several presentations on the design and mechanisms. Eg

- https://www.youtube.com/watch?v=ws-Z8xKbP4w (Val: A Safe Language to Interoperate with C++ - Dimitri Racordon - CppCon 2022)

- https://www.youtube.com/watch?v=oFupPFniD9s ([IWACO23] Borrow checking Hylo)

According to the website I'm not allowed to ask about this though ;)

Wasn't it called Mojo? It's been a while since I saw traction of that new language over here.

Anyhow, it is interesting how many new languages are appearing every day. The bar is indeed constantly moving upwards.

It seems like Yao's whole project is solving the concurrency problem.

I'm like, why not drop it entirely (if performance isn't one of the language's top priorities anyway)? What does keeping it gain that's so important? (Context: I don't understand concurrency and multithreading.)

>Instead, Yao will be compiled to an LLVM-like IR (which already exists), and that’s how it will be distributed. Users will have to do the final compile step on their local machines.

Is this referring to the compiler itself, or programs written in Yao?

> But there is an even harder problem: the C10M Problem. Is it possible to use RSC to handle 10 million connections, of which 1 million need some processing every second?

(RSC being restricted structured concurrency.)

IMO the C10M problem requires: a) user-land network stacks or unikernel servers, b) NPROC (or fewer) threads or processes, c) evented, continuation-passing style code in each of those threads or processes.

I've written one such program (though without (a)), a HTTP read-only file server that does (b) and (c) and which supports "tailing" files by doing `GET`s with `Range: bytes=0-` (or some other starting offset, but no ending offset), and it scales to tens of thousands of concurrent clients trivially. To get to millions of clients I would need to use H/3 and QUIC, and also do (a). (That server also supports ETags and conditional requests. Tailing `GET`s end when the file is unlinked or renamed away. Essentially it is a poor person's Kafka.) This program stores state in a per-client data structure and in the form of a pointer to a continuation closure that is `{data structure, handler closure}` that is registered with the event system, and each handler does only non-blocking operations (if you can think of filesystem operations as non-blocking anyways, though one could also use aio).

I bet this works well with RSC if all the state is kept in statically allocated objects. In my program all state is kept in dynamically allocated heap objects, but I could have statically allocated them instead, perhaps at program startup, or else at compile time.

The point here is to absolutely minimize the representation of state. In a thread-per-client implementation the state is smeared on each thread's stack and also any necessary heap objects. In a CPS implementation the state is fully explicit, with none of it in any stack frames -- the programmer is in charge of producing the most compact state representation, and it's easy because the state has to be fully explicit.

I view evented CPS and thread-per-client as the ends of a concurrency spectrum where green threads and co-routines are somewhere in the middle. If you want C10M you will probably end up wanting evented CPS. Green threads / fibers can get close to CPS if you work to minimize state anyways.

If there were better ways to achieve these things, there would be much less inertia in the language space.

So mark "synchronous" (i.e., non-yielding) and "asynchronous" (i.e., non-blocking). Then we can have the synchronous colour, the asynchronous colour, functions that are both (e.g. pure CPU-bound tasks, like arithmetic), and functions that are neither (e.g. buggy code).