> It is because of the hundreds of years of astonishing and unexpected success of the sciences that have convincingly shown that apparent metaphysical gaps are never such.
This has to be one of the most dumbfounding pseudo-philosophical sentences I've ever read. Metaphysics by definition is unfalsifiable and unscientific; it exists on a parallel plane from empiricism and is derived only through intuition, reason, and for the religious revelation. If this guy's claim for material consciousness simply rests on an intuitive argument from induction, it suffices as a counter argument to say "If I am mistaken, I am".
There are a lot of problems with the article, but this isn't one of them. The history of science has been one of dispatching one irreducible "essence" after another. Essence meaning some essential property of a phenomena that defines it and distinguishes it from all other phenomena. Science is in the business of reducing these once seemingly irreducible essences to more basic structure and dynamics. The last hold out is consciousness. It's reasonable to think it will also fall eventually.
It absolutely is a problem with the article. Science deals with physical phenomena; metaphysics quite literally means beyond physics. It's ridiculous to say that consciousness is the last hold out, as if there aren't a million other unanswered questions about meaning, essenence, and experience.
Here is a parallel argument for you. The history of science has been one discovery after another which leaves us with new, increasingly complex unanswered questions about phenomena. It is reasonable to think that if/when we reduce consciousness through science we will find that there are more increasingly complex unanswered metaphysical gaps.
Reducing a complex phenomena to more basic structure and dynamics just is to eliminate any open metaphysical questions about that phenomena. That's why the big philosophical debates center around monism vs dualism rather than n-pluralism. Science has dispatched all other essences from mainstream consideration.
Really? You are telling me that the discovery/development of general relativity or quantum mechanics has not thrown new increasingly complex doubts on the accuracy of previous physical models due to these new "essences" implying contradictions with classical "essences". What could possibly make you so confident that new datapoints, theories, and discoveries as it relates to consciousness will be completely flawless?
You're not using the term essence in the way philosophers mean it. An essence is a categorical descriptive/reasoning context. In mathematical terms, an essence is a lot like a descriptive/measurement basis. A naive scientist sees a world full of distinct reasoning contexts, length is categorically distinct from speed, which is categorically distinct from water, which is categorically distinct from life. The progress of science has been to progressively reduce reasoning contexts/measurement bases to other contexts/bases thus leading to a more unified theory of nature.
Quantum mechanics does increase the physics reasoning contexts owing to the incompatibility between classical and quantum mechanics. But this is not an in principle divergence in the way that philosophers understand essences. We can describe and reason about quantum mechanics and classical mechanics using the same language and the same descriptive tools, namely mathematics. When it comes to phenomenal consciousness and physical behavior, we cannot reason about them using the same descriptive language. Hence they count as distinct categorical essences until we discover the bridging principles that reduce consciousness to physical behavior.
Bambu absolutely could create a system where their printers both communicate with the cloud and local devices, they just don't want to do the difficult software engineering necessary because it is difficult. This is not theoretical either; I work on production devices with hybrid cloud and local functionality. Engineering around a zero-trust threat model (as in you assume the user can and will tamper with the device) is completely doable.
For instance, using a push-only RPC model where only the cloud can initiate a request is one zero-trust strategy that can be used for ensuring a predictable network load on cloud infrastructure, which seems to be their main concern.
This is fair and I should have been more clear that I meant “possible under their current self-imposed constraints;” of course it’s all software so anything is possible (for the record, I also agree that this is a much harder problem than people are giving it credit for).
Unless I am mistaken, it seems like there is a glaring flaw in this scheme, which is that without fsync you cannot guarantee the previous WAL blocks have been persisted before the current one, so a power loss event could leave a hole in the log and cause erroneous recovery. I believe that SSDs reorder writes internally so even having atomic batched O_DIRECT is not a strong enough guarantee for durability. I'll admit that I could be misunderstanding something about the system that alleviates this concern.
Assuming O_DIRECT actually blocks until the SSD has acked (this isn't actually what O_DIRECT's contract says, but what they rely on), you have to wait until each page write acks whenever you need a persistence barrier.
My guess is the preallocation + zeroing is what got them most of the win, and the O_DIRECT is actually hurting, not helping throughput. This has been the case 100% of the time I've benchmarked such things.
If you're doing this sort of stuff for real under Linux, check out sync_file_range. It's the only non-broken and performant sync API for ext4 (note that it's broken by design for many other file systems, and the API is terribly difficult to use correctly).
If you really care, it's probably just easier to use SPDK or something. Linux has historically been pretty hostile towards DBMS implementations.
That's a lot of valuable information and thanks for the input. Yes the original blog post is mainly focusing on reducing the metadata overhead due to fsync(), and I got a lot of good feedback from here and a lot of discussion is beyond our original scenario settings. We would like to incorporate all these enhancement suggestions without re-introducing fsync(), and make it work for more general environments.
Many storage devices guarantee that all successful DMA (e.g. O_DIRECT) writes are persisted even in the event of a power loss. This does not work on storage devices that do not offer this guarantee obviously. It also does not work if the filesystem does not support direct I/O or requires metadata updates.
This is not a new trick. It has been used in many storage engine designs to effect durability without an fsync.
if there is a hole in the log then the end of the log is before the hole. you do have to have checksums on log chunks, and better a kind of rolling hash, but you're really just talking about he number of entires that we would have liked to commit but didn't
Yeah this is a good point, and maybe a hole wasn't the right way to explain myself. The point is that the way a WAL is supposed to work is that the main data store always lags behind the WAL, so that if a partial operation (always idempotent) occurs on shutdown it is replayed on start up and fixed. In the case I describe, because of a lack of fsync it's possible for the WAL to lag the main data store, so partial operations will not be fixed on start up.
that's a much more interesting problem. fundamentally we're in a bad position by having two different formats, one optimized for writing and one for reading, that admit inconsistency between them. Postgres mitigates this slightly by having page level updates to the read indices also be present in the log (physiological), but that's always seemed like a huge waste to me.
if we give ourselves two definitions of persisted - logically(wal or write) and physically (index or read), it seems like we can maintain the invariant that P < L. (1) by keeping an in memory view of P-L that we have to consult on every read to assert eh delta and (2) an expensive but asynchronous flush path for updating P driven from reads verifying L has landed, then have we patched all the holes(?).
edit: of course one of the root problems here is the drive lying, so how can we understand that some log block has actually commit so that we can update P
thanks for feedback. actually it was pointed out in blog that we do not use append only log to avoid fsync due to size change. what we use is preallocate fixed size log file and we do write journal data and space reclaim by 4KB unit, also with direct-io.
For the uninitiated who didn't watch Terry's streams, HolyC is both an AOT and JIT language, but the JIT was in some ways much more rudimentary and in some ways much more powerful than a typical JIT compiler. Like this CJIT project, it basically could dynamically link and compile source code, spit the assembly into memory and proceed to immediately execute it. In fact, the system shell was literally JIT compiled HolyC. Which also meant you can do fun things like call kernel functions directly from the command line.
If all you care about is a windowed average you can implement the fps counter (really you keep track of seconds per frame) using an exponential moving average which has constant time and space complexity regardless of window size. The calculation you do once a frame is:
Which will give smoothing comparable to an n sample moving average. This is the same formula used for n-day exponential moving averages for stocks.
As the article points out, this is a sample based window which is not as good as a time based window, but it's also dead simple to implement.
Edit:
Just spit balling because I haven't thought about this problem in a while and asked AI to give a foruma for EMA with variable duration events, so take it with a grain of salt. Maybe for a time based window you could use a dynamic alpha (forgetting factor) with the following formula:
For dynamic alpha, start by pretending the new incoming, windowed data doesn't exist. Look at your old data. How quickly do you want it to disappear? It'll have some formula like e^-(bt) for some constant b governing what proportion from 0 to 1 of that data still remains.
So...T seconds have elapsed since your last frame, and you have a new data point you'd like to incorporate into your EMA (and, for this problem, that data point is T itself). You keep e^-(bT) of the old data, 1 minus that of the new data, and you're done. Alpha is indeed 1-e^-(b * cur_spf) for some constant b, just like the AI said.
Which b do you choose though? I usually prefer to think of it in terms of half lives. Let H be your desired half life, and set 1-e^-(bH) equal to 1/2. You get b=log(2)/H. That's similar to the AI answer, but it rescales window_secs into a parameter you can actually start to reason about. The AI answer gives you a 1/e life instead, which is a less comfortable constant to mentally process.
That answer also naturally generalizes to other "time" axes or denominators. Suppose you want to EMA using discrete events rather than wall-clock time. Replace T in your dynamic alpha with the discrete count of events you haven't updated the EMA with yet (e.g., if something happens every time tick you would usually take T=1, but you can increase that based on a few skipped frames or whatever if you'd like).
As a fun fact, you can tailor this sort of stateless solution to have almost any decay property you'd like. Start with your favorite ODE satisfying a few properties, and this equation falls out as the step a discrete solver would take to approximate the ODE.
That's basically a one-pole low pass filter. You can use this for smoothing any kind of data that arrives at a steady rate.
You can also dynamically calculate the filter coefficient based on the current delta time, which makes sure that the smoothing behavior is independent from the current framerate.
Thanks, I added a formula for a dynamic filter coefficient to my original comment. It makes sense intuitively to me, but I'm also not certain if the exact formula I provided is correct.
You really can't compare crates to a taxi service and I think the article lays out the reasoning nicely. The people running crates are a small team of mostly volunteers offering a free service. If someone offered a free taxi service as a small organization of volunteers then they could easily be forgiven for not having the same standards that a regulated for-profit business would.
Genuinely curious: is Tailscale actually providing any values to this use case beyond what you get from a raw Wiregaurd exit node with port forwarding instead of Tailscale's NAT traversal? I've never used Tailscale, but I have a Wiregaurd setup on my home server for the same purpose as described in the article, and I've never had any issues with it.
Edit:
Noticed some sibling comments asking effectively the same thing as me. I've been meaning to write a blog post covering the basic networking knowledge needed to DIY with just Wiregaurd. My impression is that many people don't realize just how easy it is or don't have the requisite background information.
If you're just doing hub-and-spoke anyway, yeah, you can do it yourself. I did for years. But holy smokes, is it a PITA to manually copy keys around to devices; especially when they might not even be yours. I have my Tailscale account hooked up to my self-hosted identity server and now it's just a matter of logging in on whatever device I want to be on the network.
Plus, I have the option of spinning up a random EC2 box whenever I want and instantly joining it to the network with basically no fuss.
I feel like articles like this do Tailscale a disservice to a certain degree. Most people know Tailscale helps with managing the mesh of connected devices. And as many people have said here you can do this manually with Wireguard, Netbird, Nebula, ZeroTier and many others. Why Tailscale is so helpful is the ACL system. I have about 40 devices connected to my Tailnet and depending on tags devices can or can't access direct communication and also certain exit node networks. Traditional VPNs generally suck because you dump out of a host and have flat access to everything. Tailscale allows you to segment access without disrupting general Internet access with minimal friction and ACLs allow segmentation to happen at the user / device level. Most people aren't using Tailscale ACLs, in fact I rarely hear it discussed. Also the article fails to mention Tailscale Peer Relays [0] which decreases the dependency on DERP relays significantly and are controlled by, you guessed it, ACLs.
The article does list what Tailscale adds on top of WireGuard:
> WireGuard by itself is mostly the data plane. Tailscale adds the control plane on top: identity/SSO, peer discovery, NAT traversal coordination, ACL distribution, route distribution (including exit node default routes), MagicDNS, and fast device revocation.
I think you missed the point. There's nothing in the article going into any of why this would help differentiate Tailscale from plain-old-Wireguard. Simply saying this and moving on is not that.
I have a phone and laptop; those are my only two "mobile" devices that I might ever use to access my home network remotely. I set them up once, it took a few minutes, and I won't have to do it again unless I replace one of them.
I can completely understand using Tailscale for enterprise networks, but it seems very overengineered for my personal VPN needs.
I have a family of four. Plus a couple relatives who like having access to some of my self-hosted stuff. So, that's 6 people, each with at least one phone and one laptop, but probably an iPad too, or an extra work laptop, or something else random. Plus my youngest is addicted to buying old laptops on eBay and switching to them.
You made me curious, so I looked it up: I have 17 machines. Yeah... I'm not going back to plain WireGuard. :D
i had this issue, with an even more wild set of restrictions, so i used Caddy to "output its own access log" and i had a cron job on any server at home that would hit that caddy server with a pre-defined key, so like `http://caddyserver.example.com/q?iamwebserver2j` for one server and "q?iamVOIP" for another.
And now i have bi-directional IP exposure. it's cute because you can't tell if you just drive by, it doesn't look like it does anything. you have to refresh to see your IP, which is a little obfuscation.
if you care about security, not sure what to tell you. use port knocking.
Please note: this doesn't require installing anything on any remote, just a cron job to curl a specific URL (arbitrary URL). I used it to find the IP to ssh on remote radio servers (like allstar, d-star) for maintenance, for example.
> Genuinely curious: is Tailscale actually providing any values to this use case beyond what you get from a raw Wiregaurd exit node with port forwarding instead of Tailscale's NAT traversal?
Yes, but I guess it depends on how much of an adoption barrier/pain you want to deal with. Tailscale's control plane is dead simple and they ship apps on basically every platform so its easy to onboard mobile devices in addition to anything else. I'm a literal former network engineer with over two decades of experience, and I tried Tailscale randomly one of the first few times it popped on HN and stuck with it precisely because of how easy it was and how trivial it was to verify the security of my tunnels. Doing this manually is definitely possible on devices you control, but it's not a fun time, and Tailscale is dead simple.
It has plenty of useful control plane features out of the box. Nothing much you _couldn’t_ do yourself but you don’t have to. Or with Headscale as the self-hosted open-source version
I've worked on large React and Solid codebases and don't agree at all. You can make a mess of either one if you don't follow good practices. Also dynamic dependency management is not just a nice to have, it's actually critical to why Solid's reactive system is more performant. Take a simple example of a useMemo/createMemo which contains conditional logic based on a reactive variable, in one branch a calculation is done that references a lot of reactive state while the other branch doesn't. In React, the callback will constantly be re-executed when the state changes even if it is not being actively used in the calculation, while this is not the case in Solid because dependencies are tracked at runtime.
> Take a simple example of a useMemo/createMemo which contains conditional logic based on a reactive variable, in one branch a calculation is done that references a lot of reactive state while the other branch doesn't.
Then you create two useMemos, and recalculate based on the branch. Or you can cache the calculation in a useRef variable and just not re-do it inside the memo callback. There's also React Compiler that attempts to do the static analysis for automatic memoification.
> it's actually critical to why Solid's reactive system is more performant.
I've yet to see a large Solid/Vue project that is more performant than React.
> In React, the callback will constantly be re-executed when the state changes even if it is not being actively used in the calculation, while this is not the case in Solid because dependencies are tracked at runtime.
And in Solid the dependency tracker can't track _all_ the dependencies, so you end up using "ref.value" all the time. Often leading to the code that that has a parallel type system just for reactivity. While in React, you just use regular objects.
> I struggle to see the point. The paper in question doesn't claim to be practically faster...
I struggle to see the point of your comment. The blog post in question does not say that the paper in question claims to be faster in practice. It simply is examining if the new algorithm has any application in network routing; what is wrong with that?
I publish a Firefox plugin and needed help a few years ago. Not to get too far down that rabbit hole, but they suddenly blocked my plugin because they couldn't build my source code, even though the issue with their build environment was pretty obvious. Anyways, I had to use their Matrix support channel and they recommended Element. I was immensely frustrated with how buggy the experience was, and it turned me off from ever trying it again.
This has to be one of the most dumbfounding pseudo-philosophical sentences I've ever read. Metaphysics by definition is unfalsifiable and unscientific; it exists on a parallel plane from empiricism and is derived only through intuition, reason, and for the religious revelation. If this guy's claim for material consciousness simply rests on an intuitive argument from induction, it suffices as a counter argument to say "If I am mistaken, I am".
reply