Literate programming fails because it I'm a programmer and so it is telling things I already know. I understand how to read code.
Literate programming makes sense if you are writing raw machine code, or assembly where you cannot always name something important. DoSomething() better do what the names says it does so you don't need a comment to say anything more about it. Look at all the examples - why are they saying anything about "#include <stdio.h>" - if you don't know what that line does then you have no business reading C code - go learn C and then come back. Sure you could read the literate programming comments without knowing C - but I cannot think of any reason anyone would ever do that.
Comments that tell me what the code cannot are very helpful. Telling me that this complex sequence of weird code is just the standard AES algorithm is helpful (I haven't seen AES code, but I've seen other encryption algorithms that are weird bit manipulations and I assume AES is the same), but even then if I need to touch that code I better refer to the AES documentation so don't write how AES works. (If it is a custom algorithm for your company maybe document how it works, but the code is not the place for that). The important part is the code can tell me why is happening but not why it is that way so comments should tell me why you are doing something.
> Literate programming fails because it I'm a programmer and so it is telling things I already know.
Code is just a medium. You’re supposed to know it. What you don’t know is the domain knowledge the code is solving, and that’s what’s written in the literate program, along with diagrams, math, etc to aid your understanding.
Look again at the examples - very often it isn't that at all. My example - where Knuth himself wrote about "#include <stdio.h>" proves my point. If you don't know what stdio.h is about you are not competent to be reading this code in the first place.
> What you don’t know is the domain knowledge the code is solving
THIS IS UTTERLY FALSE! The company hired me to work on this code and trained me. Or I got interested in the problem and learned the domain knowledge already. If you don't have this basic knowledge you either won't read the code in the first place, or you can only make the most minimal basic contributions for the next few months while learning it. Unless this problem is very trivial there are lots of other source files and lots of other knowledge needed that is outside the scope of the current file.
Write comments to remind the experts (which is you in 6 months!) of the tricky details that they dare not forget. Write comments to explain to people who are experts in a different area of your program who this works - but you can assume a lot of shared domain knowledge because they work on the same project.
So you can read a section of code and immediately know what it’s doing and why it’s written that way? And the decisions and context that led to it? And you have no further questions?
furthmore you are familiar with all techniques your coworkers might employ?
Wow!
> Write comments to remind the experts
What if I want images in comments or a math formula? What if want to refer to another section of code?
> So you can read a section of code and immediately know what it’s doing and why it’s written that way?
No, but the documentation you have been advocating (at least so far here) wouldn't help either.
> What if I want images in comments or a math formula?
I don't have a good answer. I've seen some interesting ascii art. It looked cool, but it was out of date from what the code did and so needed to be deleted as I'm not enough of an artist to fix it.
> What if want to refer to another section of code? Use your IDE to go to the definition of the code in question, (hopefuly you can do this).
I understand (or think I understand) your problem. I often have problems with code that I don't understand. I don't think literate programming helps though as nobody actually is writing documentation of the type that would answer the questions I really have.
> nobody actually is writing documentation of the type that would answer the questions I really have.
That’s a choice though, right? The tools enable you to document in ways you haven’t before.
The most valuable use case I have seen is code that’s not too long and is read a lot of times. That’s when getting every decision, diagram, etc in one place helps a lot of people.
I have one particular case at work where every line of code has had several meetings about it to ensure the parameters are correct, etc.
> I constantly run into functions called doSomething that also do something else.
You have to have some trust in your fellow coders that doSomething doesn't do something else and this is often false. Still in most cases it doesn't matter on a first read as doSomething is close enough to only doing that something. LP won't help here anyway. Documentation can be wrote just as much as the function name.
You say LP is not about teaching the language, but the examples from Knuth himself are often documenting the language and not the codebase. This is a common problem with documentation.
As people repeatedly say in trying to rebut my OP, it's important not to cherry-pick examples. I try to be careful to find the best examples when criticizing. In that spirit, here's a couple of examples from https://cs.stanford.edu/~knuth/programs.html
So you reckon you could learn how compilers work by reading nothing but the GCC source code? While I'm sure it's possible, I think it would take you 10x as long as it would if you had read a few books about compilers first.
I'm saying LP is the wrong way to write a book about how compilers work. LP might be useful for a toy example compiler as part of the how compilers work book, but that is it. Reading a LP version of GCC is the wrong way to learn how compilers work - once you see one (or maybe 10) optimization you know how they work and can just read source code for the rest - I don't know how many different optimizations gcc has, but I'm sure it is at least thousands. Similar for the parser, there are a lot of edge cases in gcc (C is a terrible language to parse - your compiler example should use something with a simple grammar so LR or other standard parser is used not the mess that gcc must have because C).
You're describing staged learning -- and the example at the bottom of my OP is trying to demonstrate exactly that. My https://akkartik.name/post/wart-layers describes the mechanism in more detail for starting from a simple example and gradually adding concerns.
It's true that the very first example a student sees shouldn't be some eldritch horror in all its complexity.
It's also true that in the real world today, people study eldritch horrors on their own after learning the basics in kiddie pools.
But we can do better than the way we've always done it. For key pieces of software that have eaten the world, it seems worthwhile to gradually chip away steps on the cliff side to help future learners more easily understand the real-world complexity.
Literate programming makes sense if you are writing raw machine code, or assembly where you cannot always name something important. DoSomething() better do what the names says it does so you don't need a comment to say anything more about it. Look at all the examples - why are they saying anything about "#include <stdio.h>" - if you don't know what that line does then you have no business reading C code - go learn C and then come back. Sure you could read the literate programming comments without knowing C - but I cannot think of any reason anyone would ever do that.
Comments that tell me what the code cannot are very helpful. Telling me that this complex sequence of weird code is just the standard AES algorithm is helpful (I haven't seen AES code, but I've seen other encryption algorithms that are weird bit manipulations and I assume AES is the same), but even then if I need to touch that code I better refer to the AES documentation so don't write how AES works. (If it is a custom algorithm for your company maybe document how it works, but the code is not the place for that). The important part is the code can tell me why is happening but not why it is that way so comments should tell me why you are doing something.