I wonder if going the other way, maxing out semantic density per token, would improve LLM ability (perhaps even cost).
We use naturally evolved human languages for most of the training, and programming follows that logic to some degree, but what if the LLMs were working in a highly complex information dense company like Ithkuil? If it stumbles on BF, what happens with the other extreme?
Or was this result really about the sparse training data?
I wonder the same. I think a language like pascal is more semantically rich than C-like languages. Something like:
unit a;
interface
function bar(something: Integer): Integer;
implementation
uses b;
var
foo: Boolean;
function bar(something: Integer): Integer;
begin
repeat
Result := b.code(something);
until Result <> 0;
end;
end.
Probably holds more semantically significant tokens than the C-counterpart.
But with LLM's the problem's gotta be training data. But if there was as much training data in Pascal as there is in C it would be pretty cool to see benchmarks, I have a hunch Pascal would do better.
(Sorry for the bad pascal I haven't programmed in ages)
This requires a preset architecture standard that explains specifically what boxes and arrows are, and how boxes/arrows interact. Lots of software smears a logical box out over several folders in the code, sometimes even with entirely different names. I don't just mean people write non-cohesive code, I mean frameworks tend to prefer organisation by layer ("the views go in the view folder!") instead of organisation by module ("These things work and change together with a defined boundary").
You can have what you're asking for if you agree to a predefined architecture and everyone agrees to write code that way.
Honestly we have the same problem outside of engineering. C4 is really just a way to focus conversations along one level of abstraction.
How many times have you talked to someone who bounces up and down from high detail high complexity to zoomed-out simplicity in the same thought, and not understood what they wanted to tell you? We have the same issue when devs think "I will just yolo some boxes and arrows on this board, it makes sense in my head so it is a good diagram!"
Having structured architecture conventions becomes more important when you have many architecture conversations across multiple people and would like to gain insights from the diagrams too.
But you don't need and official standard that everyone learns. Just use words. If your box is a class, write "class Foo". If your arrow means "contains 1 or more" just write that next to the arrow.
Yes, that’s exactly the convention that C4 specifies: add legends on every diagram and make sure things are labeled in a way that people can understand.
You don't organize code the way you mentally model it in many projects, and nearly all languages lack a way to solve this. Annotating code is prone to the same issue as keeping a diagram up to date, and the same issue as keeping comments or documentation up to date.
The only thing I’ve seen that goes in this direction is Knuth’s literate programming. I’ve tried it. In its current form it’s still clumsy, lacks tool support and IMO doesn’t fully solve the problem of how to deal with documenting a changing piece of software yet. Knuth got his requirements correct on the first try; the rest of us aren’t so lucky.
There is a medium-to-weak argument that enforcing these minimum standards even in clearly benign places raises standards everywhere, which means it is much less likely to show up in the really bad places.
I'm just making debate however, I don't think most people are playing 3d chess when they ask for these changes, they just want the line item on their report to clear up.
When I meet someone with two PhDs in Europe I started to evaluate if they're contributing massive brain-blasts to civilization or if they just wanted to delay the next chapter of adulthood and use free education as a camouflage.
A Ph.d is mostly cheap labour for the relevant research institution. I wouldn't count that as a cost to society unless the work done is not useful, in which case it is not the Ph.d graduate but the research institution that funded the project that is to blame. Get a Ph.d position is quite competitive and the ones which are less "useful" from an immediate economic perspective are paid much worse.
I have to say I feel like there's a lot of comments in this thread that are just totally detached from reality. I moved from Toronto to Berlin in 2017 and a lot of comments in here about living in fear of Muslims is just ... I don't know what part of Germany you are living in, and I visited a lot of it. I went to Karnival in Köln, and small beer festivals in the Bavarian countryside, I talk to locals in every city I go to.
I do however distinctly remember that when I lived in Toronto my social media feed was absolutely polluted with hype about how refugees are taking over Europe and the migrants are everywhere and there are countless no-go zones and endless rapes and the end of Europe was imminent.
Then I moved here, and literally not even once have I seen or felt any of that. I have however seen the far-right media endlessly look for bad events and a way to pin it on migrants. I have seen AfD marketing to tell people they will somehow "fix things" by replacing burkas with burgundy wine.
I've also seen that the neo-Nazis try to overthrow the government (Reuß) and carry out domestic terrorist acts (Hanau for example), the constant planning for "Tage X" and stealing weapons from the army, and the arm of the government charged with constitutional protection constantly needing to stop Nazi activity or investigate problems with the AfD or the military or other important authorities.
There are multiple areas in Berlin alone that the police officialy declares as No-Go Areas, among them:
- Alexanderplatz
- Görlitzer Park and Wrangelkiez
- Warschauer Brücke
- Hermannstraße and Bahnhof Neukölln
- Hermannplatz and Donaukiez
- Rigaer Straße
- Kottbusser Tor
Görlitzer Park is officially lost to African drug dealers, they just group raped a woman in front of her boyfriend.
In other cities it's similar. I go to Hamburg and Dresden regularly, can't even befin to describe how much downhill St. Pauli and Alaunpark are going. No sane person goes there after dark anymore.
Do you live in Berlin? I do, right near Bahnhof Neukölln. I live or have lived in many of the areas you list, and have spent time in all of them. Neither myself, nor my wife, nor my friends have any fear about these areas, even if there are sometimes people we would avoid (like any big city). None of these areas are no-go zones for the police, the police are all over the place.
Alexanderplatz being a no-go zone? Are you joking? It’s full of tourist traps and has some pickpockets potentially, but that’s because they are popular areas full of people.
The gang rape in Gorlitzer is horrible, but is by no means a regular occurrence.
Your entire description reads like a farcical joke. I don’t expect to convince you of anything, but for anyone else reading this, it’s a laughable description.
These DSL tools can be absolutely fantastic and feel powerful in the hands of one person, but the issues come when multiple people get involved.
I train teams to use C4 diagrams, one of the most common issues they tell me they had in the past is the ivory-tower: someone somewhere created all the diagrams alone and dumps them on everyone hoping it will make the world better. The problem is that mode lacks the collaboration and mutual context-building to get everyone on the same page. Not everything needs to be a team effort, but a lot of your diagram work should shift towards a day-to-day tactical discussion (the deeper the C4 level the faster moving things will be). Shifting to a culture of shared context and the discipline of speaking the same language lets everyone have high clarity and move quickly.
The problem with DSLs is they are often nudging people to work alone. Text editors like that are often not multi-player. You can get around it with a screen-share or even pair programming a diagram, but often the tools nudge behaviours of people into a mode where they just work alone and dump stuff out of the ivory tower.
When I train teams who are coming in new to something like C4 I will use Miro specifically because they don't need any special DSL knowledge, and also especially because it is multi-player (everyone gets to draw and move stuff around). I find often people get a bit shy about touching the diagrams but in the training it is really important to get the whole team into the practice of seeing "oh yeah, this is a diagram I can touch too".
For teams who've been through the basic training and gotten used to C4 diagrams to do specific jobs in their tech org, I move them into https://icepanel.io/ because the problems of that team have changed a lot. The initial problem was "I need to know how to structure a story and model my architecture at the same time". Once they got good at explaining their architecture they end up needing to model their architecture (a diagram is something different than a model) at a bigger scale (all those connections that make your diagrams too messy, the boxes that are important in context A but not context B, etc). I like IcePanel because I can slice out a "domain" of my model and show just that view of the world. For teams that have been trained to empower everyone to draw (instead of a single Benevolent Diagrammer For Life), having a multi-player system to manage the model and pick how to present a multi-dimensional subset of that better than just having a static DSL file or a Miro board (note: Miro is "fine" but it can quickly reach its limits). Basically, You get to keep the complexity of your model but only have a focused discussion on the relevant parts.
Beyond that, there's a whole world of techniques on how to actually read an architecture diagram to spot problems, but that's just too much stuff to post in a comment here.
The tl;dr is: Getting your teams to manage their architecture is multiple skills you need to build into the people on your team: collaborating, diagramming, modeling, analysis, and story-telling. I suggest starting in any tool where everyone can participate, and I don't think that's a DSL-based tool because of the UX. Those DSLs "nudge" your culture towards one where one person in the ivory tower drops an inaccurate and overly complicated one-size-fits-all diagram every 9 months and nobody knows with to do with it. It doesn't always happen, but it does increase the chances.
Full disclosure: I'm the trainer mentioned in the article. Happy to answer questions here if anyone wants to debate or pick my brain.
We use naturally evolved human languages for most of the training, and programming follows that logic to some degree, but what if the LLMs were working in a highly complex information dense company like Ithkuil? If it stumbles on BF, what happens with the other extreme?
Or was this result really about the sparse training data?