Zompist Bboard Again

malloc wrote: ↑Sun Sep 06, 2026 8:26 pm

linguistcat wrote: ↑Sat Sep 05, 2026 2:28 pmOr create images, navigate a maze, identify baked goods or cancers, create conlangs, do basic math correctly, etc. And even if we have individual AI for each of these (or calculators in the last case), none of them do the others, or do other things well. LLMs are bad at math, AI for identification cannot write.

Sure but AI in the aggregate can do all of those things, which is enough to challenge humans in the workplace and administration. Meanwhile the technology just keeps advancing with no signs of slowing down.

We had to close one AI thread because it was nothing but your AI worship/fear over and over. Stop it.

malloc wrote: ↑Sun Sep 06, 2026 8:26 pm

linguistcat wrote: ↑Sat Sep 05, 2026 2:28 pmOr create images, navigate a maze, identify baked goods or cancers, create conlangs, do basic math correctly, etc. And even if we have individual AI for each of these (or calculators in the last case), none of them do the others, or do other things well. LLMs are bad at math, AI for identification cannot write.

Sure but AI in the aggregate can do all of those things, which is enough to challenge humans in the workplace and administration. Meanwhile the technology just keeps advancing with no signs of slowing down.

Integrating them might not be so easy as you think. After all, we've waited a lifetime for quantum mechanics and general relativity to be integrated, and that still hasn't been achieved.

jal wrote: ↑Mon Sep 07, 2026 6:05 am [LLMs] have mastered the human language, and are able to respond to it like a human would (sort of), beating the Turing test (or at least initally, nowadays it's pretty easy to spot an LLM text or reply).

Having read an explanation of Searle's 'Chinese room' argument, I am left wondering how well one should be able to distinguish AI-generated story telling from unreviewed formulaic writing.

zompist wrote: ↑Sun Sep 06, 2026 8:53 pmWe had to close one AI thread because it was nothing but your AI worship/fear over and over. Stop it.

I would assume that now this thread has been split from the original one, we can (at least for now) continue the discussion for a while?


JAL

malloc wrote: ↑Sun Sep 06, 2026 8:26 pmSure but AI in the aggregate can do all of those things, which is enough to challenge humans in the workplace and administration. Meanwhile the technology just keeps advancing with no signs of slowing down.

I don't see AI as functionally different from other workplace automation. It's just that a) a new approach (LLMs) has been adopted and b) it's pushed by billionaires taking huge losses and investments, seemingly without an end goal.

As for "keeps advancing" and "not slowing down" that's just rubbish. It's not advancing nearly as fast as predicted, and it is definitely slowing down. New LLM models aren't the leaps that have been promised, and the limits of LLMs are clearly showing.


JAL

not to mention that, given the rate of infrastructure growth and the fact that the industry is still in the capture-and-hype phase of doctorow's cycle, it's remarkable that we're seeing any slowdown at all.

Torco wrote: ↑Tue Sep 08, 2026 3:23 pm not to mention that, given the rate of infrastructure growth and the fact that the industry is still in the capture-and-hype phase of doctorow's cycle, it's remarkable that we're seeing any slowdown at all.

A lot of people here in the US don't want new data centers, and various locales have introduced legislation banning or restricting them.

Moose-tache wrote: ↑Tue Sep 08, 2026 4:57 pm"But the abacus has allowed Mathematics to do things no unaided human has done before! Surely we will all be replaced by abacuses in the next few years!"

I will concede that LLMs have some use case somewhere as a tool. They are incremental improvements of existing technology. And calling them MArkov chains was meant to be slightly hyperbolic. But my point still stands. What they call "AI" is hyped up by people who have Capital-related reasons to hype it up, not because of any real capacity that they have.

(From the conworld random thread, but more fitting here) Sure, but the abacus isn't proving and refuting theorems like the latest AI models. How does matching the capabilities of academic mathematicians amount to hype?

malloc wrote: ↑Tue Sep 08, 2026 7:17 pm

Moose-tache wrote: ↑Tue Sep 08, 2026 4:57 pm"But the abacus has allowed Mathematics to do things no unaided human has done before! Surely we will all be replaced by abacuses in the next few years!"

I will concede that LLMs have some use case somewhere as a tool. They are incremental improvements of existing technology. And calling them MArkov chains was meant to be slightly hyperbolic. But my point still stands. What they call "AI" is hyped up by people who have Capital-related reasons to hype it up, not because of any real capacity that they have.

(From the conworld random thread, but more fitting here) Sure, but the abacus isn't proving and refuting theorems like the latest AI models. How does matching the capabilities of academic mathematicians amount to hype?

The thing is that many of these AI-generated proofs are completely inelegant and need significant cleanup by human mathematicians before they are of any acceptable quality.

Oh and also ─ AI-generated proofs very often contain subtle errors that may not be immediately apparent.

Also responding to seomthing in the conworlding forum:

WeepingElf wrote:I don't know much about LLM technology, but from what little I understand, they are more sophisticated than Markov chains - but they are still much closer to Markov chains than to human minds, I'd guess. As you say, it's all about token prediction, not about true intelligence. Doesn't Latin intelligere mean 'to understand'? LLMs don't understand anything; they are only quite good at predicting which token follows next.

Markov generators are a very good place to start in understanding LLMs, because the idea is similar. They are trained on a set of texts; they amalgate the texts down to a set of statistics; they have the same goal (given some text, predict the next word); they are surprisingly good at generating humanlike text. And they are easy enough for a human to understand how they work.

The limitation of a Markov generator are of course that all the input that determines the next word is minimal, just a few sequential words. It has no way of looking back further or keeping a context in mind.

Without explaining LLMs in depth, you could picture extending the Markov generator in those ways:

1) the next-word prediction is affected not just by the previous few words, but by bits and pieces of the entire conversation.
2) it puts more weight on the user conversation (the "prompt") than on its training data.

I'm tempted to simulate this by extending my Markov program to accept a one-word prompt. I'll report back if I can get that working.

Travis B. wrote: ↑Tue Sep 08, 2026 7:23 pmThe thing is that many of these AI-generated proofs are completely inelegant and need significant cleanup by human mathematicians before they are of any acceptable quality.

Inelegant perhaps but still correct and that is all that matters for qualifying as intelligence. If proving a mathematical conjecture that has stumped mathematicians for decades doesn't qualify as intelligence, then what does?

Travis B. wrote: ↑Tue Sep 08, 2026 7:41 pm Oh and also ─ AI-generated proofs very often contain subtle errors that may not be immediately apparent.

So do human proofs. Not for nothing did we get exam questions where we had to spot the error in an alleged proof.

malloc wrote: ↑Tue Sep 08, 2026 9:58 pm Inelegant perhaps but still correct and that is all that matters for qualifying as intelligence. If proving a mathematical conjecture that has stumped mathematicians for decades doesn't qualify as intelligence, then what does?

From what I've heard, what happened was plodding marshalling of multitudinous existing results, rather than any fresh insight.

Travis B. wrote: ↑Tue Sep 08, 2026 7:23 pm The thing is that many of these AI-generated proofs are completely inelegant and need significant cleanup by human mathematicians before they are of any acceptable quality.

I've heard the same about AI-generated code.

I am sure that everyone here has either heard already or will hear shortly that OpenAI has a claimed solution to the Navier–Stokes problem, one of the most significant unsolved problems in mathematics. I felt the need to write down some of my thoughts on this. In no particular order:

• I still see some people claiming that ‘LLMs are just a marketing scam’ and suchlike. Over the past few months this view has been becoming increasingly untenable, and this is the last nail in the coffin. LLMs can make really significant advantages, and it’s high time we all understood that so that we can act accordingly.
• The announcement was complicated by a statement from Tristan Buckmaster (a mathematician at NYU), who alongside Levent Alpöge (a mathematician employed by Anthropic) had developed the approach which eventually led to OpenAI’s solution. He alleges that OpenAI achieved their solution by, essentially, stealing their approaches (most likely from their prompts to OpenAI’s models), then pumping massive amounts of money and resources to scoop them before they could achieve the full Navier–Stokes result. They then allegedly contacted Buckmaster, offering him first-authorship on the condition that Alpöge be excluded. If true, this would be a significant instance of corporate malfeasance as well as a breach of mathematical ethics. Nonetheless, none of this has any direct relationship to the LLMs themselves, which were being used extensively by both parties.
• Also in the statement, Buckmaster explicitly states: ‘The ideas making this line of attack possible are due to Córdoba and Martínez-Zoroa. Let me make plain what I have said to colleagues in private: in view of this body of work, I believe Luis Martínez-Zoroa deserves a Fields Medal.’ This is completely true and very important. The impact of significant advances in human mathematics to this result cannot be understated. Despite the number of problems LLMs have already solved, as far as I’m aware they have not yet made a conceptual, theoretical advance to human mathematics of this level.
• Relatedly: all the LLM-powered solutions so far have been in the form of counterexamples, i.e. specific demonstrations that a certain conjecture is false. I’m not yet aware of a significant proof generated by LLMs in the form of universal quantification, i.e. that such a statement is true or false in all possible cases.
• The power of the LLM in this case was arguably due to breadth and perseverance more than raw intelligence. OpenAI explicitly stated that ‘the group that produced the Navier–Stokes resolution involved on the order of 10,000 concurrent agents’. If, theoretically, you got 10,000 human mathematicians together to all work on this single approach, I think it’s quite possible they would have landed on a solution too — though obviously due to human limitations you could never do that in practice.
• I’ve seen some mathematicians (notably Terence Tao) making the correct and insightful observation that the point of mathematics is not merely to come up with proofs of results: it’s to explain why those results are true, in ways humans can understand, and, through that process and the process of developing those proofs, to create new mathematical theories which let us reveal even more patterns and attack even more problems. It is this ability to create new theories which is destroyed by LLM mathematics: when an LLM solves a problem, we know the fact that it is true, but not why it is true. Fortunately in this case, as mentioned above, the theories leading to this result were already developed by Martínez-Zoroa; but in the future this could prove significantly corrosive to our understanding of mathematics.
• For completeness’ sake, I should note that the proof has not yet been fully checked by human mathematicians: rather it is encoded as a computer-verified proof object formalised in the Lean language. This is almost certainly correct. However, there remains a slim chance that this could be due to a correctness bug in Lean — LLMs have proven rather good at taking advantage of such bugs in the past. So it still needs some human checking. That said, I find it most likely that it is indeed correct.
• The ‘nice’ thing about mathematics is that it’s purely theoretical. The natural sciences are significantly more difficult due to the need to do experiments. So far there is no way to replace humans in the lab.

So, in sum, this development leaves me quite depressed, but not entirely devoid of hope. The essential challenge is keeping alive the practice of human mathematics, in the form of the development of theories which are meaningful to us, even in a (as yet hypothetical) world where LLMs could give us solutions to arbitrary problems ‘on demand’. I don’t know how to do that but I don’t think it’s impossible, either. This will become most urgent if LLMs are able to do more than ‘merely’ finding counterexamples to things; thankfully this is not yet the case.

bradrn wrote: ↑Wed Sep 09, 2026 8:11 am

• I still see some people claiming that ‘LLMs are just a marketing scam’ and suchlike. Over the past few months this view has been becoming increasingly untenable, and this is the last nail in the coffin. LLMs can make really significant advantages, and it’s high time we all understood that so that we can act accordingly.

What one must remember, though, is that LLM's commonly don't produce the same quality as humans and don't 'think' like humans. E.g. sure, they can program, but they have a strong tendency to produce highly inelegant code that is difficult to review and which requires significant refactoring before it can match what humans can produce, and which often contains subtle bugs that may get by reviewers and unit tests (and if the LLM is told to conform to a set of unit tests, they are liable to precisely pass the unit tests but produce code that fails where there are subtle gaps in the unit tests).

bradrn wrote: ↑Wed Sep 09, 2026 8:11 am I still see some people claiming that ‘LLMs are just a marketing scam’ and suchlike. Over the past few months this view has been becoming increasingly untenable, and this is the last nail in the coffin. LLMs can make really significant advantages, and it’s high time we all understood that so that we can act accordingly.

There's a lot of potential for actually useful tools there! But the 'marketing scam' element is there too, alright; as you've outlined there's significant breach of ethics in service to marketing there.

It's a pity that we should have that kind of technological advance right at the time when the tech industry is at its most toxic. Though both are correlated in an ugly feedback loop: you get fast results if you're willing to do away with ethics and sanity, and the results validate the malignancy.

The power of the LLM in this case was arguably due to breadth and perseverance more than raw intelligence. OpenAI explicitly stated that ‘the group that produced the Navier–Stokes resolution involved on the order of 10,000 concurrent agents’. If, theoretically, you got 10,000 human mathematicians together to all work on this single approach, I think it’s quite possible they would have landed on a solution too — though obviously due to human limitations you could never do that in practice.

I didn't know that -- that makes it an incredible case of using brute force; brute force squared really, as LLMs are themselves a brute force approach! I'll note that solving mathematical problems by brute force is nothing new.

I’ve seen some mathematicians (notably Terence Tao) making the correct and insightful observation that the point of mathematics is not merely to come up with proofs of results: it’s to explain why those results are true, in ways humans can understand, and, through that process and the process of developing those proofs, to create new mathematical theories which let us reveal even more patterns and attack even more problems. It is this ability to create new theories which is destroyed by LLM mathematics: when an LLM solves a problem, we know the fact that it is true, but not why it is true. Fortunately in this case, as mentioned above, the theories leading to this result were already developed by Martínez-Zoroa; but in the future this could prove significantly corrosive to our understanding of mathematics.

Yes, I've heard other mathematicians make comments to the same effect. I understand that mathematics these days seems to revolve a lot around trophy hunts -- which I don't think really expands human knowledge.

Travis B. wrote: ↑Tue Sep 08, 2026 4:37 pm

Torco wrote: ↑Tue Sep 08, 2026 3:23 pm not to mention that, given the rate of infrastructure growth and the fact that the industry is still in the capture-and-hype phase of doctorow's cycle, it's remarkable that we're seeing any slowdown at all.

A lot of people here in the US don't want new data centers, and various locales have introduced legislation banning or restricting them.

quite, and rightly so. jussayin the fact that the chatbots have more and more chips powering them as the years go by and that despite that growth we're seeing a slowdown, this suggests even more strongly that the tech is maturing into the status of, say, a gasoline car: you can add bells and whistles and screens and sensors and so on to a car, but for the most part it's mature, stable technology in that a car is a car is a car

Travis B. wrote: ↑Wed Sep 09, 2026 8:44 am

bradrn wrote: ↑Wed Sep 09, 2026 8:11 am

• I still see some people claiming that ‘LLMs are just a marketing scam’ and suchlike. Over the past few months this view has been becoming increasingly untenable, and this is the last nail in the coffin. LLMs can make really significant advantages, and it’s high time we all understood that so that we can act accordingly.

What one must remember, though, is that LLM's commonly don't produce the same quality as humans and don't 'think' like humans. E.g. sure, they can program, but they have a strong tendency to produce highly inelegant code that is difficult to review and which requires significant refactoring before it can match what humans can produce, and which often contains subtle bugs that may get by reviewers and unit tests (and if the LLM is told to conform to a set of unit tests, they are liable to precisely pass the unit tests but produce code that fails where there are subtle gaps in the unit tests).

Indeed. (Even in Buckmaster’s statement he apologises for providing what ‘can only be described as AI slop’, since they were forced to publish early.) That said, in mathematics the use of very strongly typed proof languages like Lean ensures the worst failures are mostly avoided.

Ares Land wrote: ↑Wed Sep 09, 2026 9:34 am It's a pity that we should have that kind of technological advance right at the time when the tech industry is at its most toxic. Though both are correlated in an ugly feedback loop: you get fast results if you're willing to do away with ethics and sanity, and the results validate the malignancy.

Pretty much my feeling, yes. I do feel that LLMs are an intrinsically dangerous technology, but the sense of badness I get from them is very different to the way in which the big tech companies are bad.

(Like, OpenAI’s marketing here is aggressive and unethical, but that doesn’t mean the claimed capabilities of their LLMs are purely marketing; it just seems like ‘aggressive and unethical’ is OpenAI’s default mode of operating, so to speak.)

Yes, I've heard other mathematicians make comments to the same effect. I understand that mathematics these days seems to revolve a lot around trophy hunts -- which I don't think really expands human knowledge.

That reminds me of Perelman‘s refusal of the Fields Medal and Millenium Prize. Beyond that, not being a mathematician, I can’t really comment further on what the state of the field is like today. (I don’t really get the impression that it revolves around ‘trophy hunts’ any more than it ever did, but maybe it does; I wouldn’t know.)

Torco wrote: ↑Wed Sep 09, 2026 9:47 am jussayin the fact that the chatbots have more and more chips powering them as the years go by and that despite that growth we're seeing a slowdown

I really do not see any slowdown occurring. (Quite the reverse, in fact — the capability growth seems to be exponential.)