Rendered at 12:00:51 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
martincsweiss 23 hours ago [-]
Co-author here - first paper from my postdoc accepted to NeurIPS main conference.
We built a way to sample diverse ideas in machine learning - ideas that make scientific sense, but that no existing research community is positioned to propose.
In short, the way it works is like this:
1. Take 16,068 LLM papers (NeurIPS, ICLR, ICML, ACL, EMNLP, 2017 to 2025) are broken into 273 "idea atoms". Every paper, and every candidate idea, becomes a set of atoms.
2. Train a Coherence Model: a 42.8M-parameter transformer trained on papers as sequences of atoms. Basically a language model whose tokens are ideas.
3. Train a Cognitive-Availability Model: a dual encoder trained on the publication histories of 39,791 researchers, which predicts where each person could plausibly go next.
4. Use those models to score sampled tuples using reciprocal rank fusion (you want ideas that go together well, but that are not cognitively available to anyone).
We tested it in a bunch of ways. One interesting one was that we took all the papers at the end of 2024, then checked which high-scoring tuples showed up in 2025 papers. It worked well - one examples is an ICLR 2025 oral, goes from rank 1,489 with coherence alone to the top 25 of 1.14M samples.
A funny one: if you ask Claude to come up with an ML research idea, it is often about interpretability. If you then run an autoresearch pipeline with some A100s on the idea, the result tends to be interpretability 8/10 times. Our Alien Sampling method stayed at 0 interpretability ideas even after letting claude cook on them.
We also paid 50 people with graduate degrees in CS/AI to judge the research reports and found that Alien’s reports had the best average quality rank and produced 3 of the top 5 reports.
Oh and then we scaled it way the hell up to a couple million papers and 10,000,000's of concept elements and generated 25,000 new research proposals that you can search over here: https://lacuna.tiptreesystems.com/explore?view=hypotheses
I was really impressed by the scale of the lacuna tool you created, but then I dug into the niche topics (steganography, covert communication) that constitute my PhD research area and my expectations were tempered.
There’s a lot of noise in the research literature. Lots of papers get published that really don’t contribute much of anything to the field, but the existence of a bunch of papers often sends a signal that some concept is important and needs to be studied.
I find that often a big cluster of papers around a particular topic, none of which seems to move the needle much, is actually an indication that we need a new approach and writing more and more incremental papers just digs deeper into a local minimum.
edit: clarification: the content itself is fine, in fact I'm curious!
Kuinox 40 minutes ago [-]
This is the same thing as "but AI can only repeat what it already saw" but the goalpost moved again.
arethuza 2 hours ago [-]
I think if an LLM had read any discussions on how best to remove burrs from pet coats then it might eventually get to Velcro - I believe the inventor of Velcro got the idea from burrs?
peri-cl 26 minutes ago [-]
> "I believe the inventor of Velcro got the idea from burrs?"
I think there's a provocative human-intelligence analog to be found there: "humans will never discover anything that doesn't already exist somewhere in nature. We can only recombine concepts we've seen—never invent truly new ones".
It's said that mathematics is "unreasonably effective" at describing the natural world. But what about the other way around? What if mathematics is unreasonably ineffective at escaping the convex-hull of concepts and patterns found in nature?
breakyerself 1 hours ago [-]
Ok do you want to share any interesting ideas the method produced?
dmos62 4 hours ago [-]
Kudos. The proposals are actually interesting to browse through. Good paper name by the way.
dansitu 3 hours ago [-]
Congratulations, this is a really cool approach and result. I lead a research team and I'm very interested in tooling for improved collaboration, especially between human and agent researchers.
Do you have any evidence that this approach scales to cross-domain research?
JPLeRouzic 5 hours ago [-]
Thank you, that an excellent idea and an excellent execution!
But I wonder why when someone types 'medicine' there is only 6 proposals while when someone type 'physics' there is 447 proposals?
unsupp0rted 2 hours ago [-]
I love this. My personal completely unsupported theory is we’re cognitively capable of conceiving of less than 1% of what’s true/accurate about the universe, and cognitively capable of understanding 1% of that 1%
dachris 2 hours ago [-]
I offer my support for that theory. There's so many facets of reality, many of which no one will ever explore or understand.
- art style
- a doctor might see indications of a health issue that Lisa has
- what it says about the society of that time
- geology of the landscape
- contemporary production of the color pigments
- what the picture does with you when you look at it
- and so on
Good Sci-Fi novels bring along some of these - aliens that have access to different concepts of reality, e.g. the dark forest hypothesis. The hard part is that you actually have to come up with that understanding for writing it, instead of just hand-waving it.
negura 2 hours ago [-]
> we’re cognitively capable of conceiving of less than 1% of
If you can't conceive it, then you can't take a percentage of "it", since you've not conceived that "it".
tauntz 1 hours ago [-]
A kind-of similar concept made the rounds yesterday but for games, instead of science.
Damn, this was the killer app for semantic spaces I'd been waiting for since Word2vec.
jan_m_savage 3 hours ago [-]
this reminds of the short novel by Abbott: Flatland where the inhabitants live in a 2D world and are unaware of Space: the 3D environment.
krige 2 hours ago [-]
I'm on the other hand reminded of a certain Lem sub-story from A Perfect Vacuum, which was about geniuses whose discoveries were too divorced from the mainstream to be useful / worth following upon.
curt15 1 hours ago [-]
Humans could also benefit such a model of idea atoms to consciously explore the "alien space" of a field.
Makes one wonder if there were technologies that did something similar to humanity in the real world…
I wonder if all the investment in smartphones and LLM’s may have taken away from focus on other things…
mechagodzilla 5 minutes ago [-]
I think the way computers developed is kind of an interesting example of this in the real world. Babbage basically realized the idea of a modern computer, but tried to build a too-complicated 'supercomputer' (40 decimal digit precision!) because he didn't know how incredibly useful a much simpler, more limited computer could be. We had to wait another 75 years and spend an incredible amount of money to develop relay and vacuum-tube supercomputers, before realizing that the idea could be scaled down, and something like a PDP-8 / PLC / microcontroller was incredibly useful for industrial control and, with sufficient cleverness, even do a lot of scientific computing.
I think if you went back in time you could pay some power loom engineers to build you a pretty decent 4-8 bit microcontroller operating at 1-5 Hz (comparable to power looms at the time) and computer-controlled textiles would have paid for the computer revolution 75-100 years ahead of time.
Gravityloss 2 hours ago [-]
A bit of a tangent, regarding innovation. It's interesting to think about counterfactuals in history. There are questions like what if solar panels became widespread 100 years earlier? Could we have avoided global warming and all the wars over coal and oil? (Not possible because lack of purity technology in materials and understanding of matter.) But are there other things where some easy discovery was waiting just around the corner but it wasn't discovered for some reason? Or something was discovered in one society but not in another for a long time, from similar starting conditions. Washing hands certainly is something that one could go back with, probably thousands of years, to bring immediate benefits. Some cultures have done it but some have absolutely refused it for a long time.
And then applying that kind of thinking to today. Looking back to 2026 from 2126, what would a historian facepalm over? Or what is already being done in one field or social scene but not in another, that would be greatly beneficial? The classic recent example of success is applying airline pilots' checklists to surgery. Airline pilots probably are a little bit like aliens, from surgeon point of view. How many people had already tried similar things, but failed with the institutions and people's inability to receive innovation? Maybe with AI tooling we can facilitate better innovation diffusion. It's not without its dangers of course.
negura 2 hours ago [-]
I've filtered the generative proposals for biology [0] because that's a field I'm a bit familiar with. I think it's a bunch of nonsense, tho the confident language really gaslights me.
We built a way to sample diverse ideas in machine learning - ideas that make scientific sense, but that no existing research community is positioned to propose.
In short, the way it works is like this:
1. Take 16,068 LLM papers (NeurIPS, ICLR, ICML, ACL, EMNLP, 2017 to 2025) are broken into 273 "idea atoms". Every paper, and every candidate idea, becomes a set of atoms.
2. Train a Coherence Model: a 42.8M-parameter transformer trained on papers as sequences of atoms. Basically a language model whose tokens are ideas.
3. Train a Cognitive-Availability Model: a dual encoder trained on the publication histories of 39,791 researchers, which predicts where each person could plausibly go next.
4. Use those models to score sampled tuples using reciprocal rank fusion (you want ideas that go together well, but that are not cognitively available to anyone).
We tested it in a bunch of ways. One interesting one was that we took all the papers at the end of 2024, then checked which high-scoring tuples showed up in 2025 papers. It worked well - one examples is an ICLR 2025 oral, goes from rank 1,489 with coherence alone to the top 25 of 1.14M samples.
A funny one: if you ask Claude to come up with an ML research idea, it is often about interpretability. If you then run an autoresearch pipeline with some A100s on the idea, the result tends to be interpretability 8/10 times. Our Alien Sampling method stayed at 0 interpretability ideas even after letting claude cook on them.
We also paid 50 people with graduate degrees in CS/AI to judge the research reports and found that Alien’s reports had the best average quality rank and produced 3 of the top 5 reports.
Oh and then we scaled it way the hell up to a couple million papers and 10,000,000's of concept elements and generated 25,000 new research proposals that you can search over here: https://lacuna.tiptreesystems.com/explore?view=hypotheses
Code (MIT): https://github.com/alejandrohdez00/alien-space-of-science
There’s a lot of noise in the research literature. Lots of papers get published that really don’t contribute much of anything to the field, but the existence of a bunch of papers often sends a signal that some concept is important and needs to be studied.
I find that often a big cluster of papers around a particular topic, none of which seems to move the needle much, is actually an indication that we need a new approach and writing more and more incremental papers just digs deeper into a local minimum.
https://blog.karliner.net/posts/chatgpt-and-the-velcro-test/
edit: clarification: the content itself is fine, in fact I'm curious!
I think there's a provocative human-intelligence analog to be found there: "humans will never discover anything that doesn't already exist somewhere in nature. We can only recombine concepts we've seen—never invent truly new ones".
It's said that mathematics is "unreasonably effective" at describing the natural world. But what about the other way around? What if mathematics is unreasonably ineffective at escaping the convex-hull of concepts and patterns found in nature?
Do you have any evidence that this approach scales to cross-domain research?
But I wonder why when someone types 'medicine' there is only 6 proposals while when someone type 'physics' there is 447 proposals?
If you take a look at the https://en.wikipedia.org/wiki/Mona_Lisa , it can be seen from so many aspects
- art style - a doctor might see indications of a health issue that Lisa has - what it says about the society of that time - geology of the landscape - contemporary production of the color pigments - what the picture does with you when you look at it - and so on
Good Sci-Fi novels bring along some of these - aliens that have access to different concepts of reality, e.g. the dark forest hypothesis. The hard part is that you actually have to come up with that understanding for writing it, instead of just hand-waving it.
If you can't conceive it, then you can't take a percentage of "it", since you've not conceived that "it".
https://x.com/re_skob/status/2113043120128483349
The Road not Taken by Harry Turtledove
https://pastebin.com/aJQfubrK
Makes one wonder if there were technologies that did something similar to humanity in the real world…
I wonder if all the investment in smartphones and LLM’s may have taken away from focus on other things…
I think if you went back in time you could pay some power loom engineers to build you a pretty decent 4-8 bit microcontroller operating at 1-5 Hz (comparable to power looms at the time) and computer-controlled textiles would have paid for the computer revolution 75-100 years ahead of time.
And then applying that kind of thinking to today. Looking back to 2026 from 2126, what would a historian facepalm over? Or what is already being done in one field or social scene but not in another, that would be greatly beneficial? The classic recent example of success is applying airline pilots' checklists to surgery. Airline pilots probably are a little bit like aliens, from surgeon point of view. How many people had already tried similar things, but failed with the institutions and people's inability to receive innovation? Maybe with AI tooling we can facilitate better innovation diffusion. It's not without its dangers of course.
[0] https://lacuna.tiptreesystems.com/explore?view=hypotheses&q=...