> Path names are UTF-8-encoded, unrooted, slash-separated sequences of path elements, like “x/y/z”
This is only for the "io/fs" package and its generic filesystem abstractions. The "os" package, which always operates on the real filesystem, doesn't actually specify how paths are encoded, nor does its associated helper package "path/filepath".
In practice, non-UTF-8 already wasn't an issue on Unix-like systems, where file paths are natively just byte sequences. You do need to be aware of this possibility to avoid mangling the paths yourself, though. The real problem was Windows, where paths are actually WTF-16, i.e. UTF-16 with unpaired surrogates. Go has addressed this issue by accepting WTF-8 paths since Go 1.21: https://github.com/golang/go/issues/32334#issuecomment-15500...
> There is no realistic risk of a SHA collision attack.
Indeed. To illustrate why:
1. It is not possible to "retroactively" find a SHA-1 collision for an already known hash. If somebody has produced a SHA-1 hash non-maliciously at any point in the past, it is safe from collisions. This is due to second-preimage resistance, which hasn't been broken for SHA-1 and doesn't seem likely to be broken any time soon.
2. The only way to obtain a SHA-1 collision is to do so knowingly when producing the original hash. You generate a pair of inputs at the same time that both hash to the same value. Certainly, this is an imaginable scenario; e.g. a trusted committer could push one half of the pair wittingly or a reviewer could be fooled into accepting one half of the pair unwittingly, both scenarios creating a timebomb where the malicious actor swaps the commit to the second half of the pair (which presumably carries a malicious payload) later. However, there are two blockers to this approach: Git (not just GitHub) will not accept a commit with a duplicate hash, always sticking with the original one, and GitHub specifically has implemented signature detection for the known SHA-1 collision-generating methods and will reject both halves of such a pair.
In short, there's just no practical way to exploit this weakness of SHA-1 with Git.
Many modern GC languages and runtimes have benefited greatly from improved escape analysis, which identifies memory allocations that have short, identifiable lifetimes, and removes those from the GC-managed heap entirely.
Of course, many GCs have indeed gotten better in their own right, but not using the GC at all remains the most potent optimization.
Floating-point non-determinism issues usually come from one of these sources:
- Inconsistent use of floating-point precision, which can arise surprisingly easily if e.g. you had optional hardware acceleration support, but didn't truncate the intermediate results after every operation to match the smaller precision of the software version and other platforms.
- Treating floating-point arithmetic as associative, whether at the compiler level, the network level, or even the code level. Common violations here include: changing compiler optimization flags between versions (or similar effects from compiler bugs/changes), not annotating network messages or individual events with strictly consistent sequence numbers, and assuming it's safe to "batch" or "accumulate" deltas before applying them.
- Relying on hardware-accelerated trigonometric etc. functions, which are implemented slightly differently from FPU to FPU, since these, unlike basic arithmetic, are not strictly specified by IEEE. There are many reasons for them to vary, too, like: which argument ranges should be closest to correct, should a lookup table be used to save time or avoided to save space, after how many approximation iterations should the computation stop, etc.
There's another gotcha. Consider positive, normal x and y where ulp(y) != ulp(x). Bitwise comparison, regardless of tolerance, will consider x to be far from y, even though they might be adjacent numbers, e.g. if y = x+ulp(x) but y is a power of 2.
This case actually works because for finite numbers of a given sign, the integer bit representations are monotonic with the value due to the placement of the exponent and mantissa fields and the implicit mantissa bit. For instance, 1.0 in IEEE float is 0x3F800000, and the next immediate representable value below it 1.0-e is 0x3F7FFFFF.
Signed zero and the sign-magnitude representation is more of an issue, but can be resolved by XORing the sign bit into the mantissa and exponent fields, flipping the negative range. This places -0 adjacent to 0 which is typically enough, and can be fixed up for minimal additional cost (another subtract).
I interpreted OP's "bit-cast to integer, strip few least significant bits and then compare for equality" message as suggesting this kind of comparison (Go):
with the sensitivity controlled by ignoreBits, higher values being less sensitive.
Supposing y is 1.0 and x is the predecessor of 1.0, the smallest value of ignoreBits for which equiv would return true is 24.
But a worst case example is found at the very next power of 2, 2.0 (bitwise 0x40000000), whose predecessor is quite different (bitwise 0x3FFFFFFF). In this case, you'd have to set ignoreBits to 31, and thus equivalence here is no better than checking that the two numbers have the same sign.
Yeah, that's effectively quantization, which will not work for general tolerance checks where you'd convert float similarity to int similarity.
There are cases where the quantization method is useful, hashing/binning floats being an example. Standard similarity checks don't work there because of lack of transitivity. But that's fundamentally a different operation than is-similar.
Sure, but that operator can propagate a carry all the way to the most significant bit, so a check for bitwise equality after "strip[ping] few least significant bits" will yield false in some cases. The pathologically worst case for single precision, for example, is illustrated by the value 2.0 (bitwise 0x40000000) and its predecessor, which differ in all bits except the sign.
Fil-C has two major downsides: it slows programs down and it doesn't interoperate with non-Fil-C code, not even libc. That second problem complicates using it on systems other than Linux (even BSDs and macOS) and integrating it with other safe languages.
I would never say it's impossible, and you've done some amazing work, but I do wonder if the second problem is feasibly surmountable. Setting aside cross-language interop, BYOlibc is not really tolerated on most systems. Linux is fairly unique here with its strongly compatible syscall ABI.
You're right that it's challenging. I don't think it's infeasible.
Here's why:
1. For the first year of Fil-C development, I was doing it on a Mac, and it worked fine. I had lots of stuff running. No GUI in that version, though.
2. You could give Fil-C an FFI to Yolo-C. It would look sort of like the FFIs that Java, Python, or Ruby do. So, it would be a bit annoying to bridge to native APIs, but not infeasible. I've chosen not to give Fil-C such an FFI (except a very limited FFI to assembly for constant time crypto) because I wanted to force myself to port the underlying libraries to Fil-C.
3. Apple could do a Fil-C build of their userland, and MS could do a Fil-C build of their userland. Not saying they will do it. But the feasibility of this is "just" a matter of certain humans making choices, not anything technical.
I think there's two main avenues for hardware acceleration: pointer provenance and garbage collection. The first dovetails with things like CHERI [1] but the second doesn't seem to be getting much hardware attention lately. It has been decades since Lisp Machines were made, and I'm not aware of too many other architectures with hardware-level GC support. There are more efficient ways to use the existing hardware for GC though, as e.g. Go has experimented with recently [2].
There are algorithms to align allocations and use metadata in unused pointer bits to encode object start addresses. That would allow Fil-C's shadow memory to be reduced to a tag bit per 8-byte word (like 32-bit CHERI), at the expense of more bit shuffling. But that shuffling could certainly be a candidate for hardware acceleration.
There is a startup working on "Object Memory Addressing" (OMA) with tracing GC in hardware [1], and its model seems to map quite well to Fil-C's.
I have also seen a discussion on RISC-V's "sig-j" mailing list about possible hardware support for ZGC's pointer colours in upper pointer bits, so that it wouldn't have to occupy virtual memory bits — and space — for those.
However, I think that tagged pointers with reference counting GC could be a better choice for hardware acceleration than tracing GC.
The biggest performance bottleneck with RC in software are the many atomic counter updates, and I think those could instead be done transparently in parallel by a dedicated hardware unit.
Cycles would still have to be reclaimed by tracing but modern RC algorithms typically need to trace only small subsets of the object graph.
The second bullet is a headline goal of Project Valhalla, however it is unlikely to be delivered in quite the way that a C# (or Go or Rust etc.) developer might expect. The ideal version would allow any object with purely value semantics [1] to be eligible for heap flattening [2] and/or scalarization [3], but in experimental builds that are currently available, the objects must be from a class marked with the "value" qualifier; importantly, this is considered an optimization and not a guarantee. More details: https://openjdk.org/projects/valhalla/value-objects
The third bullet (IIUC) is addressed with the Foreign Function & Memory API, though I'll admit what I've played around with so far is not nearly as slick as P/Invoke. See e.g. https://openjdk.org/jeps/454
[1] value semantics means: the object is never on either side of an == or != comparison; the equals and hashCode methods are never called, or are overridden and their implementation doesn't rely on object identity; no methods are marked synchronized and the object is never the target of a synchronized block; the wait, notify, and notifyAll methods are never called; the finalize method is not overridden and no cleaner is registered for the object; no phantom or weak references are taken of the object; and probably some other things I can't think of
[2] heap flattening means that an object's representation when stored in another object's field or in an array is reduced to just the object's own fields, removing the overhead from storing references to its class and monitor lock
[3] scalarization means that an object's fields would be stored directly on the stack and passed directly through registers
The third bullet is also presumably referring to C#'s ancient wider support for unsafe { } blocks for low level pointer math as well as the modern tools Span<T> and Memory<T> which are GC-safe low level memory management/access/pointer math tools in modern .NET. Span<T>/Memory<T> is a bit like a modest partial implementation of Rust's borrowing mechanics without changing a lot of how .NET's stack and heap work or compromising as much on .NET's bounds checking guarantees through an interesting dance of C# compiler smarts and .NET JIT smarts.
The FFM API actually does cover a lot of the same ground, albeit with far worse ergonomics IMO. To wit,
- There is no unsafe block, instead certain operations are "restricted", which currently causes them to emit warnings that can be suppressed on a per-module basis; it seems the warnings will turn into exceptions in the future
- There is no "fixed" statement and frankly nothing like it all, native code is just not allowed to access managed memory period; instead, you set up an arena to be shared between managed and native code
- MemorySegment is kinda like Memory<T>/Span<T> but harder to actually use because Java's type-erased generics are useless here
- Setting up a MemoryLayout to describe a struct is just not as nice as slapping layout attributes on an actual struct
- Working with VarHandle is just way more verbose than working with pointers
> - There is no unsafe block, instead certain operations are "restricted", which currently causes them to emit warnings that can be suppressed on a per-module basis; it seems the warnings will turn into exceptions in the future
Which sounds funny because C# effectively has gone the other direction. .NET's Code Access Security (CAS) used to heavily penalize unsafe blocks (and unchecked blocks, another relative that C# has that I don't think has a direct Java equivalent), limiting how libraries could use such blocks without extra mandatory code signing and permissions, throwing all sorts of weird runtime exceptions in CAS environments with slightly wrong permissions. CAS is mostly gone today so most C# developers only ever really experience compiler warnings and warnings-as-errors when trying to use unsafe (and/or unchecked) blocks. More libraries can use it for low level things than used to. (But also fewer libraries need to now than used to, thanks to Memory<T>/Span<T>.)
> There is no "fixed" statement and frankly nothing like it all, native code is just not allowed to access managed memory period; instead, you set up an arena to be shared between managed and native code
Yeah, this seems to be an area that .NET has a lot of strengths in. Not just the fixed keyword, but also a direct API for GC pinning/unpinning/locking and many sorts of "Unsafe Marshalling" tools to provide direct access to pointers into managed memory for native code. (Named "Unsafe" in this case because they warrant careful consideration before using them, not because they rely on unsafe blocks of code.)
> MemorySegment is kinda like Memory<T>/Span<T> but harder to actually use because Java's type-erased generics are useless here
It's the ease of use that really makes Memory<T>/Span<T> shine. It's a lot more generally useful throughout the .NET ecosystem (beyond just "foreign function interfaces") to the point where a large swathe of the BCL (Base Class Library; standard library) uses Span<T> in one fashion or another for easy performance improvements (especially with the C# compiler quietly preferring Span<T>/ReadOnlySpan<T> overloads of functions over almost any other data type, when available). Span<T> has been a "quiet performance revolution" under the hood of a lot of core libraries in .NET, especially just about anything involving string searching, parsing, or manipulation. Almost none of those gains have anything to do with calling into native code and many of those performance gains have also been achieved by eliminating native code (and the overhead of transitions to/from it) by moving performance-optimized algorithms that were easier to do unsafely in native code into "safe" C#.
It's really cool what has been going on with Span<T>. It's really wild some of the micro-benchmarks of before/after Span<T> migrations.
Related to the overall topic, it's said Span<T> is one of the reasons Unity wants to push faster to modern .NET, but Unity still has a ways to go to where it uses enough of the .NET coreclr memory model to take real advantage of it.
Yeah, coming to C# from Rust (in a project using both), I’ve been extremely impressed by the capabilities of Span<T> and friends.
I’m finding that a lot of code that would traditionally need to be implemented in C++ or Rust can now be implemented in C# at no or very little performance cost.
I’m still using Rust for certain areas where the C# type system is too limited, or where the borrow checker is a godsend, but the cooperation between these languages is really smooth.
For those wondering, Unity overloaded the == operator in two specific situations, such that obj == null will return true even though obj is not actually null. More details on this archived blogpost: https://web.archive.org/web/20140519040926/https://blogs.uni...
reply