This reminded me a great deal of a paper I read like 10 years ago, which I just looked up because I wanted to see if it was cited here, "Distilling Free-Form Natural Laws from Experimental Data" [0]. Lo and behold it is not only cited, but is written by the same authors. So the novelty here, as discussed in the paper [1], is that they are doing something similar but from video instead of from sensor streams. Which is quite interesting, as it opens up the available information to systems that are hard to 'sense' apart from pointing a camera at them, like the lava lamp example.
I did see a presentation a few years ago on a similar topic by Erwin Couwans, author of Bullet physics engine, who was discussing doing neural inference of physics. He basically wanted to replace Bullet with a neural network, which I thought was kind of funny at the time, but cool if it worked. (Physical laws at this level of mechanics are mostly quite well understood, but the devil is in the details and actually solving time-discretized non-smooth dynamics, as well as performing good contact detection is less than obvious the more you get into it, not to mention friction, so I could see why a "learned" solution could be attractive, but at the time I found it funny that he was proposing to learn the physics that were already simulated by Bullet.) Looking back, it appears that those ideas culminated in a paper [2] two years ago, which I'll have to read now -- from the abstract, the differentiability of the simulation has benefits not only for learning the physics (mostly friction models it appears), but for learning controllers for plants within the simulated physics. It seems to be not cited in the linked article but maybe it is a slightly different, but related, topic. In any case, fascinating stuff.
Out of interest, how would such a system derive material properties from a still 3d scene?
We know the leaves of a house plant will bow in the wind, because you have observed it in the wild, as a child several times. Even given non-still image training material, the nn would have to learn all of the physical properties of all the things from scratch, like a human.
To train a physics model like this to derive the properties of material from the world is going to be bound to hilarious failures.
A mountain with smoke hanging over it, is obviously a house plant.
I would think that what you're suggesting could be done by doing a bunch of Fourier transforms. If there is no relative motion of the camera you could assume that objects with high frequencies would be less massive than objects with low frequencies that could give you a useful bound on their mass/density. (Assuming a fairly uniform energy density in the scene)
For my own curiosity, I wonder if some of the variables they're missing are based more on the context of the video, rather than the content. Like, colors, for instance. Maybe if you ran everything through in black and white you could get it closer to an integer lower bound?
I'd imagine you could do a frame by frame to identify points of interest and then "predict" the track they should follow into the next frame, training the model on each point of interest as you go.
In this case, you wouldn't be aiming to identify the material properties specifically, but given a model that's tightly fitted enough, there should be constants that stand in for the missing material properties when it's all said and done.
Just an initial thought, I'd love to hear why I'm wrong from those in the field.
That still gives you only indicators on the physical material properties under condition x. Means, you know that with slight wind, a palm tree waves. The net still does not know, how much force it takes to break it and how the material deforms and keeps a different shape beyond certain forces.
Why will a lamp post bend permanently in a hurricane, but a palmtree wont? You can derive those properties as a child, by bending a thin metal pipe and watching mums houseplant recover from a application of force. As an adult, you then project that information about thing "classifications" upon the world.
Lets take another extreme, a floaties palmtree. How does on derive from the observed visual differences, that this palmtree is stiff and bouncy.
Or circumstantial decorations as pysic change indicator. If there is snow on the palmtree, the tree might break in strong winds.
To develop this physics reasoning is quite a larger part of childhood and i think the idea, that one can train this into a Neural Net beyond simple "all things are solid and bound to gravity" with references alone is quite a challenging endavour.
That’s interesting thanks for sharing. I always was bearish on neural networks when advertised as “AI”, but now I think that perhaps when used as a tool rather than a panacea they could be very useful. Humans learn to operate heavy machinery in industrial settings through experience and use via our own neural networks, and we have incomplete understanding of physics, so why not use a neural network to learn the physically best way to drive an industrial machine and use the physics as backup for safety? Seems to work well based on all these stories so far, so maybe learning neural networks are the next differential equations for engineering.
The trick is to remember that a neural network is a function approximator. A good deal of AI research is in the business of casting "intelligence" as a "function" so that you can pose a problem and figure out how to feed it data, ie., well-defined input and output. That's why different function approximators can be used for AI, such as decision trees, etc., not just neural networks. What to model, and how to model it, are orthogonal problems, both interesting in their own right. It just happens that NN are particularly good at handling high-dimensional inputs, which are necessary for perception tasks, as well as for handling large vocabulary language modeling, so they are doing rather well lately.
On the other hand, there are lots of places where "functions" are useful, that have nothing to do with "intelligence", but where actually writing the function with full fidelity can be difficult or intractable. These are also opportunities where learnable function approximation can provide some great benefit, provided you can figure out how to pose the problem such that data is available for the learning part.
A good example is in physically-based rendering, you have lots of complicated aspects of light transport for which we can model the physics quite well, but when you get into complicated reflections and scattering, etc., this can all be modeled by a complicated function called the BRDF [0]. Hand-writing a good BRDF is possible, and quite typical of high-end renderers in fact, but it's no surprise that there's been research in replacing it with a "neural BRDF" [e.g. 1].
That's just to give an example of a place where a single, very targeted and small (but complicated) part of a larger framework can stand to benefit from data-driven modeling, and a neural network can be one good way to do that. Another example is similar usage in computation fluid dynamics [2], where we can hand write a pretty good model, but to capture what is missing, it can be useful to have an approximator. The problem there is that having approximated the function well, it doesn't necessarily lead to better human understanding of the phenomenon. Discovering the true, sparse latent variables in a way that is interpretable, instead of a black box, is a useful step towards that. (Which I guess the current article is aiming at, but I haven't read it in full yet.) But sometimes all you want is results, as in the case of synthesizing a good controller. For example in CFD, if you can use the blackbox model to generate a good stabilizer for an ocean platform, you don't really care about the physics, as long as it's accurate enough to be trustworthy. So the utility of these methods, like most things, is relative to what your goals are.
Piggy backing here... but this dimension-reduction or system-identification stuff always reminds me of Takens' theorem [0]. He even did some neural network stuff back in the "dark ages" of the topic [1].
Thanks for those links! Back in school, I loved wondering about whether things like the inverse square law for gravity could be (have been) discovered from raw experimental data via ML - cool to see that it's actually been done already.
This is cool and I personally believe this type of work may lead to breakthroughs in messy data rich fields like biology where we can arrive at a higher levels of abstraction maybe not exactly to "laws" like physics but highly correlative rules around phenomena. I think this is more on the side of knowledge creation and is human friendly as opposed to being more of a black box prediction like deep neural networks. Though I think both things are complimentary since human curiosity isn't satisfied by prediction alone.
If anyone else is interested in this line of work I recommend checking out Kathleen Champion, Steve Brunton, and J. Nathan Kutz's work on Discovering governing equations from data by sparse identification of nonlinear dynamical systems(https://www.pnas.org/doi/full/10.1073/pnas.1517384113).
Thank you for mentioning J. Nathan Kutz! Reading through this article, I saw similarities to Dynamic Mode Decomposition (I am not literate enough on the topic to elaborate). His Coursera courses and book were a fascinating dive into orthogonal basis functions, lower-rank approximations like PCA... I'm not sharp enough anymore (over a decade since grad school) to fully grok it, but damn his work is so cool!
I'm not seeing the significance of that. Surely there's plenty of alternate descriptions of physics/systems that can be equally predictive and certainly there's lots of equivalences to shift from one framework to another.
It's true that there are many mathematically equivalent ways to describe physical systems. But the important point is that some are more useful than others. For example, Lagrangian mechanics and Hamiltonian mechanics are equivalent to Newtonian mechanics, but they can give much better intuition for certain problems. Feynman diagrams are equivalent to grinding out the QFT algebra by hand à la Schwinger, but they give a completely different intuition for the underlying Physics.
More importantly, though, they could use this NN on systems that have not yet successfully been modeled, perhaps complex dynamical systems, to discover good parameters and conserved quantities.
> For example, Lagrangian mechanics and Hamiltonian mechanics are equivalent to Newtonian mechanics, but they can give much better intuition for certain problems. Feynman diagrams are equivalent to grinding out the QFT algebra by hand à la Schwinger, but they give a completely different intuition for the underlying Physics.
I just read about Langrangian and Hamiltonian mechanics. I didn't encounter those at all in my EE physics, and they are fascinating. Great examples! Are you a physics professor, or is this stuff undergrad physics majors learn?
Used to be third-year in the major under Classical Mechanics.
There's a good series of videos on YT, with the title Variational Calculus and the Euler-Lagrange equation on channel Structural Dynamics. I have only seen the first few. This first video should give you the full playlist:
> More importantly, though, they could use this NN on systems that have not yet successfully been modeled, perhaps complex dynamical systems, to discover good parameters and conserved quantities.
That would only make sense to try if the model could do this for systems we already understand. By the sound of the article, it can't even do that. Despite many efforts the researchers couldn't even understand the second pair of parameters. That doesn't correspond to my understanding of "good parameters".
Good summary. There's no end to the kind of delusion some AI proponents will cling too. In a few days they'll say this is another proof we live in a simulation!
They seem to be uncorrelated with real-world physics. Whether they are "new" is anybody's guess. They're variables identified in a virtual environment, rather than the real world so there's very little chance they correspond to something in the real world, let alone physics, much less "new physics".
Well, it does seem like a "made up", not directly measurable variable. We just have it for convenience of understanding the world. Or am I missing something?
Nothing is "directly measurable". Everything in science is a story that we make up to explain our personal experience, including "measurement" and "observation".
My current thinking (I’m a total dilettante….) is that energy is a state in closed system. The system of course designed by the observer. For example, a cart half way up a hill. Does it have positive or negative potential energy? Depends on how you define your system.
Having been programming a while, when confronted with "Well, it got the answer 4.7, and we figured out 2, and we can't figure what the other 2.7 are", I wouldn't leap to "Wow, the AI has found a whole new physics". I'd go with "bug", until I can very concretely describe otherwise.
State variables of dynamic systems can be arbitrary. The fact that two that were selected by an automatic process happened to correspond to the two angles is probably more interesting than the other two being uncorrelated with the momenta.
I guess it's because it's a serendipitous discovery by roboticists rather than an experiment explicitly set up to discover "alternative physics," but wouldn't it be much more efficient to feed the NN actual physical measurements rather than a video if discovering physical models was the goal?
It's amazing that the AI found two variables that corresponded to actual physical phenomena just from video footage but it seems highly unlikely that it found an alternative description of physics given how far off the answer was. The NN could be modeling any kind of weird "fantasy" for how the visual signal it was being fed could be produced that might not even correspond to 3D space in a meaningful way.
There should be four state variables for a double pendulum system - angle 1, angle 2, momentum 1, momentum 2. But they don't have to be neatly divided up like that. Angle 1 could be represented by some proportion of all four quantities. The individual variables can have no physical significance in of themselves whatsoever as long as there is a mechanism for turning them into the final answer.
It's interesting that the angular explanatory variables are also those that are easily measured in a 2d still image. Whereas the angular velocity requires differential information and is not be immediately represented in each individual image.
So these guys trained "an AI" to identify hidden variables by looking at videos
of physical phenomena, without prior knowledge of physics (except what was
implicitly and obscurely encoded in "the AI's" neural net architecture, and its
training set and hyperparameters, by the researchers).
The "AI" came up with a slightly off number and the researchers struggle to
understand why. They struggle because "the AI" is a black box that lacks
explainability and cannot produce an explanatory model, only a predictive
model [1]. The simplest conclusion is that their "AI" has not learned any useful
models, and has only learned to accurately predict their test set, to which
the authors had acces throughout the training of "the AI" [2]. The simplest
explanation is that their model isn't very good at doing what they wanted it to
do, but they opt to explain it as a scientific mystery, instead. Why?
It seems the only reason to try and see a scientific mystery where a simple
failure would suffice as an explanation is because the model is "an AI". There
seems to be some kind of expectation that "an AI" must have some deeper
understanding of physics than humans, even when we don't understand what it's
doing, even whe it's not doing very well.
In short, this seems to be based on very wrong assumptions and to be coming up
with very wrong conclusions, but then again it makes for a great headline and so
here we are, discussing it on HN.
____________________
[1] To clarify: a predictive model is one that can predict novel events. An
explainable model is one that can explain how it made a prediction. An
explanatory model is one that explains how the world works and why certain
predictions are true, or not. Predictive and explainable models are useful, but
most scientists aim to build explanatory models, because an explanatory model is
necessarily also explainable and ultimately better at predictions, while a
predictive model is not necessarily explainable or explanatory and an
explainable model is not necessarily predictive or explanatory. Deep neural nets
can only build predictive models, which are sometimes explainable, but they
can't build explanatory models. That's because they can only identify
correlations in data, but not explain those correlations with generalised
theories, based on previous knowledge.
To put it plainly, neural nets can't come up with scientific theories, but they
can estimate probabilities of things happening. But scientists can and want to
come up with scientific theories, not just predictions.
[2] When the experimenter has access to the test data and can tune the learner's
model until it scores highly on the test data, that leads to a model that
overfits to the test data. Such a model is useless for prediction over unseen
data (i.e. data not available to the researchers during training).
>To clarify: a predictive model is one that can predict novel events. An explainable model is one that can explain how it made a prediction.
Sorry but that distinction doesn't mean anything falsifiable. Did you mean an explanatory model is one that's easier to understand for a human? If so, that's certainly useful, but it may well turn out that the simplest working model of physics is incomprehensible.
No, I distinguish between "explainable" and "explanatory". To give an example, the theory of epicycles is a predictive model, that is also explainable, but it is not explanatory. Kepler's laws of planetary motion are an explanatory model that is also explainable and predictive.
The theory of epicycles is predictive because it predicts the motions of the planets, as they are observed in the night sky. It is explainable because anyone can perform the necessary calculations and understand how the predicted motions are, well, predicted. The theory of epicycles is not an explanatory model because it does not explain why the planets should move in circular orbits with epicyclical sub-orbits. Kepler's laws are explanatory because they explain planetary motion as a result of Newton's law of universal gravitation, are predictive because they can be used to predict the motion of the planets and are explainable because anyone can plug in the numbers and see how the results are calculated.
So an "explanatory" model is a theory that explains why things happen they way they happen. A predictive model only predicts that some things will happen. An explainable model explains why it made a prediction, but it does not explain why this prediction should hold or with what frequency.
Another way to see this is that an explanatory model explains past observations and predicts future observations, while a predictive model only predicts future observations and has no explanatory power, cannot explain why past phenomena were observed.
An "explainable" model is just a model that people can understand. It's not more complicated than that.
>The theory of epicycles is not an explanatory model because it does not explain why the planets should move in circular orbits with epicyclical sub-orbits. Kepler's laws are explanatory because they explain planetary motion as a result of Newton's law of universal gravitation, are predictive because they can be used to predict the motion of the planets and are explainable because anyone can plug in the numbers and see how the results are calculated.
That's not a qualitative difference, it's a quantitative difference. Here you somewhat implicitly claim that models of lower Kolmogorov complexity for same, or better, accuracy are better, but how is that 'explanatory'? The term has no meaning. Ironically Kepler's laws are a better equivalent of epicycles, because they reduce to an imperfect approximation of Newtonian mechanics. Predicting actual orbits in a Newtonian Solar system of perfectly spherical objects with fully known matter distribution can only be done with an iterative solution.
The actual, physical reality continuously diverges from all current models, and all we can try to achieve is to reduce that error. It can't ever be eliminated unless the universe is in fact a simulation, and whatever is controlling it decides to copy humans into the parent world, then gives them external access to the full underlying state.
Oh yes, that's absolutely necessary. That's how science works, right? Every new bit of knowledge builds upon older knowledge. It's theories all the way down, until we hit arbitrary axioms on which all our knowledge is based, though we hope those are somehow based on solid observations. And that's how we understand the world.
But just to be clear, I understand there's a colloquial meaning of "theory" as in what people mean when they say "that's just a theory". What I mean by "theory" is an epistemic object with either explanatory, or predictive power, or both. A theory can include multiple laws and hypotheses etc. And a theory "is just a theory" only until we can refute it, or find a better theory that does a better job at explaining things.
>> If so, that's certainly useful, but it may well turn out that the simplest working model of physics is incomprehensible.
I just re-read your comment and I notcied that.
I think by "the simplest working model of physics" you mean quantum mechanics? I hope I clarified how I mean "explainable" vs. "explanatory" but yeah, I think that's absolutely spot on. Quantum mechanics is predictive, but not explanatory. I do think it's "explainable" though in the sense I sort-of define it, because it's a bunch of formulae that anyone can plug in the numbers to, and see how they come up with results. I don't reckon there's many theories in the sciences that are not explainable. The lack of explainability only becomes an issue with black box models like neural nets.
Perhaps I should have used the word "comprehensible" or "interpretable" instead of "explainable" since it causes confusion by being too close to "explanatory". But those words have their own problems ("comprehensible"... by whom?)
As an aside, I'm on the camp that hopes that quantum mechanics is somehow fundamentally wrong and we'll figure it out in the next big paradigm shift. It bothers me deeply that we seem to have hit a wall where we can predict, but don't have a clue and can't understand why. I think every other big leap in scientific ... understanding (hint) has come from the ability to make explanatory theories, that tell us how things work and why.
The generic response to your type of reasoning is that you might need to rethink what you think "why" and "how" and "have a clue" are defined :)
The human mind really wants to understand things in terms of the paltry slow coarse-grained 3D world we inhabit, despite 100 years of QM and QFT-based experiments showing that the world simply doesn't work that way. Can't other realms have other "why"'s? Tens of thousands of professional solid state physicists and QM researchers would probably not refer to their field as "not having a clue" for example, any more than anybody working in a field dominated by Newtonian mechanics would have a clue at least.
In fact, if you make a simulated world in a computer game, it's trivial to come up with algorithms that do alternative physics in the game that can create interesting behaviour, despite being far from explanatory for a being inside the sim.. I'm rather amazed the reality we inhabit is so easy to comprehend as it is even despite QM/QFT being pretty far from Newtonian.
Well I'm far from an expert on physics, to be fair, so apologies for misrepresenting how phycisists see their field.
I think what you're saying is that it may be impossible to fully understand the world in the same way that we have tried to, in all of science, until now. I agree, in fact I suspect we probably can't, because it's obvious to me our intelligence is limited (as a species) and we're going to hit our limit sooner or later, if we haven't indeed hit it already.
But that's not for me a reason to stop trying to understand as much as we can, neither is it a good reason to replace understanding with ... something else. Because the day when we finally hit our limit of understanding is the day our civilisation stops dead in its tracks.
I for one am not prepared to go gently into that good night, any time soon.
Yeah I agree with you here, it might very well be that the next breakthrough level has to come from even more non-intuitive (from a human standpoint) rules. After all Newtonian mechanics is intuitive just because humans evolved in a macroscopic world ruled by it.
We can resort to math (like we have done in the last 100 years with QM) because while we rely on math being stringent we don't rely on it having to be intuitive on the level of our own experiences. On the other hand, even in a mathematical treatment of QM/QFT a lot of "classical" intuition and terminology is still used even though it, in my opinion, is hurting.
For example the insistence to apply properties like position and momentum to particles in the same physical framework even though QM/QFT has consistently shown for 100 years that conceptually you should pick one of them, the other is a dual, and failure to get rid of that baggage leads you into "weird" things like Heisenberg's uncertainty relation which is only mysterious when you insist on mapping properties to the classical world one by one. That is certainly one aspect where humans just can't seem to shake off the classical notions..
> To put it plainly, neural nets can't come up with scientific theories, but they can estimate probabilities of things happening. But scientists can and want to come up with scientific theories, not just predictions.
Current science is famous for "shut up and calculate" principle.
As I understand it, that just applies to quantum mechanics. For example, in computer science, we certainly try to prove the properties of algorithms, such as their computational complexity or their soundness and completeness etc.
This is not meant to be personally offensive. I normally wouldn't be this blunt, but I was one of the downvoters and you asked for feedback.
Your comment comes across as arrogant and ignorant. I have not formally studied quantum mechanics, but I do know theoretical physics is all about identifying and testing theories to better explain phenomenon. And the tests are statistically rigorous.
This is a thread specifically regarding an AI getting a wrong answer and practitioners suddenly deciding it's discovered new physics. Yet you claim CS is rigorous and quantum mechanics is not.
There is a tendency of someone well learned in one subject (eg CS) deciding that makes them an expert on any subject they casually observe. This is very off-putting.
Thanks for the explanation and I appreciate the candour. I am really surprised by what you say. I didn't expect that my comment would come across as arrogant and ignorant. I'm not an expert on physics, and I started my comment by saying "as far as I understand it". Reading it again I can't see that the language I used is particularly emotionally, or otherwise, charged.
I also absolutely did not try to claim that "CS is rigorous and quantum mechanics is not", as you took my comment to mean. The conversation here is not about rigour, or at least I didn't realise it was about rigour. Is the "shut up and calculate" dictum about rigour? In the context of the conversation and without looking it up I took it to mean that we should not try to understand and only try to calculate. I didn't put any more meaning on it.
I think you (I don't know about the other users who downvoted my comment) misunderstood both the meaning and the intent of my comment. May I suggest that clarifications should be sought, in the future?
Not that it will kill me to have my comments downvoted once in a while, it's just that this time I really didn't understand what I got so wrong.
EDIT:
>> There is a tendency of someone well learned in one subject (eg CS) deciding that makes them an expert on any subject they casually observe. This is very off-putting.
I'm with you on that, but I think if you look through my comments you'll see that I generally only discuss things I know. In particular with physics, I read but not comment on such conversations because I'm really not an expert.
I already upvoted your first comment and just upvoted this one.
I wasn't telling you what your comment meant to you personally -- only you know that. I was telling you how it came across and likely the reason it was downvoted significantly. I think a lot of people here don't have a lot of time to give feedback when they see an off-putting comment. And usually if someone has a counter-argument they'll say something instead of downvote. It's a good rule of thumb to consider how a tweet comes across that you may not have meant if it is just downvoted and not replied to.
I've thought about this myself before so this was an interesting read! I've thought in particular about quantum theories and how they are so hard to reconcile with gravity, and the hierarchy problem. Maybe we do simply struggle because this reality is so distant from the one we experience that we have trouble understanding it. Or maybe it is because we formalized it empirically but we got off on the wrong foot unlike Einstein's part that seems to come together better. Max Planck built his framework, not because he understood why quantum physics is the way it is, but because he observed how it worked. Since it works, we build upon that, what else can we do... So it looks like it's right to build on but we're having the damndest time to move forward. This can be very hard to later backtrack on!
> A particularly interesting question was whether the set of variable was unique for every system, or whether a different set was produced each time the program was restarted.
indeed! i'm assuming it finds the same variables over multiple runs with different rng seeds. if that is true, it seems the next interesting question is if multiple kinematic problems can be used for the training of the same net. one key feature of physics is that it was invented to describe general principles that apply to multiple problems, so it doesn't surprise me that it might find some more optimal representation that doesn't necessarily generalize to other physical problems.
:::Tangent::: So if you had a new neural network, observing the physical world, and put a simple chaotic physics system within it's field of view, it would learn to encode such systems, and perhaps begin to understand the laws of physics?
So that's why we put dangly things above cribs for babies to focus on.
Apart from some very slow atom and particle simulations...
Is not most physics based on very extreme oversimplifications?
They are still very useful because errors average out, but for example: we don't know why fluid simulations work, in fact aerodynamics is an entire empirical field.
This was seen in F1 this year. All cars bouncing on straights, it was even given the name porpoising by the pilots. The teams never expected it, because the simulations never discovered anything about it.
Also: all the work of AlphaFold predicting protein structures.
Physics are awesome at predicting planetary motion, but stuff at smaller scales require a lot of empirical engineering to figure out.
The porpoising is apparently because the more expensive higher speed wind tunnel analysis is banned.
Engineering needs engineering,. Physics does not. The interesting thing about aero in physics is the emergence of extremely complicated phenomena, not the equations as per se.
Why not? It will surely have the same bounding box as our own physics, but the laws may be represented in completely different fashion. They would mostly map to each other, though.
Or maybe not. Maybe the "AI physics" has predictions for phenomena which our physics never bothered to figure out, whereas it would ignore large chunks of our physics area as irrelevant.
This is more like the empirical "laws" that various engineering disciplines use. I.e. some number crunching that has no theoretical foundations whatsoever, but seems to be working anyway.
What is physics, if not a transition between states? Whether the merchanism that predicts those transitions is machine- or human-interpretable is orthogonal.
An attempt to describe fundamental nature of the word, best as we can get at it. Unless you're an antirealist, or you think the universe is just information.
I don't see why that's relevant, a video camera is just another instrument that records data, not essentially different from the detectors at the LHC, albeit completely un-optimized - which is necessary for this experiment to work.
If all the AI is seeing of the world is a digital image, then they are likely to mistake the digital image for the world.
As far as I know, the article claims that the AI has discovered new physical variables, yet the researchers are unsure as to what they are. For all we know, these variables are the distance of objects from the edge of the image frame.
I'm not sure why you don't think it's Physics. It's about formulating laws that describe the behaviour of physical systems - that's the essence of what Physics is. I have a PhD in high energy theory and this really seems like Physics to me.
Sure, if the pseudoscientific description of factor analysis applied to images is correct, then it's phyiscs.
As-is, it's pseudoscience. What happens when you do a factor analysis on images? You get some measure of the axes of geometrical variance across those images.
Are those axes "related" to any physical variables, sure -- but almost never directly. To suppose the system itself had these properties is to suppose, for example, constellations actually exist and cause your personality traits.
Everything we want to know is what phyical properties of the system give rise to the observed consistent correlations in geometrical properties. *THAT* is physics.
Showing these geometrical properties exist and are consistent is just what we're trying to explain.
You cannot go from images to the domain of physics -- there are an infinite number of theories consistent with these images domains. And this is pseudoscience.
It's really easy to test whether or not it works - see how well the model predicts on out-of-training sample data. That wouldn't work with astrology.
There's no such thing as "physical properties of the system" other than measurable quantities that can be used to make predictions, which is what this does. There's no reason to be sure that temperature, for example, is a "real" physical property of a system rather than just one of many variables that would help us model it and understand it.
Do you think it's pseudoscientific because there's no theory-ladenness in the predictions?
It's pseudoscience because there's nothing in the geometrical properties of those images called "gravity", etc. One can generate those pixel patterns from an infinite number of theories with an infinite variety of causally efficacious parameters.
From the article, it doesn't work. They found on known physics it gives 4.7 dimensions, of which only two are explicable -- 4 is correct; the others have no known physical interpretation. No surprises: those two are just the geometric properties of the system (angles) which are actually properties of the image. The others are pure bullshit.
Since, of course, the real physical parameters of the system we take to have generated those images are not present in them. The images are distal effects of these things
Only in cases where the geometric properties of the target system are causally relevant to its actual causal properties will this work -- ie., only when "angles matter"
Thinking you can infer laws of nature from images is pseudoscience, and these guys need to think more carefully about why we experiment in the first place
eg., Consider that if mass is a relevant causal property, there'd be no way of inferring it from images: two objects can be visually identical whilst having radically different masses... making images *OBVIOUSLY* not a measure of mass...
this project almost defines the modern kind of schizophrenic pseudoscience born of this wave of AI
It also incidentally underscores the amazing predictive powers of Noam Chomsky, when he thinks he's describing something that common sense indicates is dumb, and then a few years later someone actually goes out and does it, and does it in earnest, and unironically, and tries to promote it as an actual advance:
So for example, take an extreme case, suppose that somebody says he wants to eliminate the physics department and do it the right way. The “right” way is to take endless numbers of videotapes of what’s happening outside the video, and feed them into the biggest and fastest computer, gigabytes of data, and do complex statistical analysis — you know, Bayesian this and that [Editor’s note: A modern approach to analysis of data which makes heavy use of probability theory.] — and you’ll get some kind of prediction about what’s gonna happen outside the window next. In fact, you get a much better prediction than the physics department will ever give. Well, if success is defined as getting a fair approximation to a mass of chaotic unanalyzed data, then it’s way better to do it this way than to do it the way the physicists do, you know, no thought experiments about frictionless planes and so on and so forth. But you won’t get the kind of understanding that the sciences have always been aimed at — what you’ll get at is an approximation to what’s happening.
But he's saying it is bad to do it that way instead of doing traditional physics because you get no understanding, which is true, but in this study they're not using it as a "physics engine" to pilot aircraft or whatever, they're using it as a trick to generate novel hypotheses, which could then be theorised and investigated properly, not as a replacement to theory.
Unless you have videos of experiments designed to observe measurement devices we have created, on systems we have designed, it's all useless.
The only useful thing in figuring out how nature works is creating truely novel experimental circumstances and measuring them with novel devices created for that purpose.
You cannot do science as a statitics of images; that's pseudoscience. And chomsky is here only half-right; it's actually much worse than he's sayign.
I think I understand your criticism. There is no inherent ground truth in the image. The mass example is great since a 2D plane can't capture the quantity mass it's literally impossible the dimensions don't work. At best a 2D plane could show you correlations of mass (mass vs something plotted out). Hence this is just modern AI aka pattern-matching on steroids.
I think a counter argument would be that if there is SOME signal in the photos AND there's enough training data that does have the correct ground truth signal that the scientists are matching up then you can have SOME level of accuracy. If the training set can reasonably cover the space of possibility that we're interested in then we can get reasonable interpretations.
However in this case the insane number of physical phenomena will always be larger than any training set so this approach should NEVER generalize there will always be way too much noise which is what the scientists have figured out here. So I agree with you that it's extremely limited but I don't think I'd call it pseudoscience there might be very limited domains where for example the only data we have available are images and so such a tool may be appropriate.
I definitely share your frustration though since any half way decent scientist should have just done a thought experiment instead and figured that this wouldn't work well. This smells like BS academic marketing where they always inflate their own impact and significance.
Who cares if there's no gravity? Gravity wasn't sent down to us from heaven on a stone tablet, it's just a concept that lets us make predictions. At school I was taught it was a force and at university I was taught it was a pseudoforce resulting from fixing a non-inertial frame. Both approaches give correct answers, even though they're conceptually very different. There's no objective way to say which is right; they're just different approaches to modelling.
And maybe the 4.7 is actually more correct? The 4-parameter model is an approximation that neglects friction and air resistance. Moreover the double pendulum is a chaotic system and chaotic systems sometimes have dynamics described by laws with non-integer exponents such as Lyapunov dimension. I'm just spitballing, but the point is that it's not a priori ridiculous.
It's definitely possible to estimate mass from images. How do you think we know the masses of asteroids and planets? No-one put them on a scale, we just record their motion and work out which value fits best.
> Who cares if there's no gravity? Gravity wasn't sent down to us from heaven on a stone tablet, it's just a concept that lets us make predictions.
A concept that says gravity is the result of bending spacetime, with the speed of light being constant. It's not just a model, it's saying the universe is 4D spacetime, which explains why GR is so predictive.
It is just a model though! Everything in science is just a model. We better hope it's just a model, because it's incompatible with quantum field theory, which is another very accurate model. The only consistent model that bridges the two, superstring theory, says that spacetime and gravity could fundamentally be many things, from closed strings travelling between D3-branes to the holographic projection of a conformal theory - and you still get the same predictions.
I show someone a photo of a bowling ball and a styrofoam ball of the same shape and size. If someone thinks they can infer from a simple visual scan of the scene (analyzing the factors you see) are they delusional?
Perhaps they could leverage their lifelong training set which correlates scenes that look like they have bowling balls with scenarios that have a high mass movable sphere.
Perhaps we could have a good laugh together by painting a bowling ball to look like styrofoam and painting styrofoam to look like a bowling ball- then we could watch the silly ai/human apply an incorrect mental model and fail to grasp the causal reality! Ohohoho
None of this works without astrology, since it was the guiding theory behind Brahe and Kepler's measurements. The out-of-sample training data that Newton used for confirmation was comet orbits. Would ML really have created an elegant, closed-form theory about the elliptical shape of orbits and the power-law dependence of the period? Without these insights, there would be no inverse-square law in the first place, and perhaps we would only have an effective theory.
The purpose of Brahe's measurements, and the reason he hired Kepler, was to gather data for astrological predictions. The principles of astrology led them to look for simple, basic principles in a way that a computer would not, unless directly programmed to do so. The astronomical measurements alone were not enough.
>> It's about formulating laws that describe the behaviour of physical systems - that's the essence of what Physics is.
I didn't see any attempt to formulate laws. The researchers trained a neural net model to predict the next event in a sequence. That is not a natural law, it's a maximum probability estimator.
To clarify, a natural law would be a formula with variables that one can plug in numbers to, in order to predict the behaviour of a system. For example, Newton's law of gravitation is a natural law, Kepler's laws of planetary motion are natural laws, the laws of thermondynamics are natural laws. But a neural net model trained to predict the next frame in a video? How is that a "law"?
I don't see any fundamental difference. A deep neural network is a universal function approximator. It uses different language from what we're used to (weights and activations instead of analytic functions and calculus) but that's not a big deal. The point is that it uses only a handful of latent variables to describe the state of the system at a given time, and these can be used to predict the system's behaviour, which is fundamentally the same thing that a scientist would try to do.
So, to be clear on what you are saying, if I understand corectly you are saying that training a neural net to approximate a function is formulating a law, like for example a natural law? Is that right?
As a for instance, if I train a neural net to predict the motions of the planets, the trained model is a law of planetary motion, like Kepler's laws of planetary motion? Is that correct?
I would say it's essentially equivalent, especially if you choose a neural network architecture with a very low-dimensional layer in the middle with only a handful of variables.
Then the first half of the network (before the low-dimensional layer) will learn how to "encode" the state of the system in the video in as few variables as possible, such as the orientations and angular momenta of the double pendulum. This is equivalent to what humans do when we look at a messy physical system like the Solar System and model it with a few quantitative parameters.
The bottleneck layer will represent the handful of state variables, and then finally the other half of the network will learn the mathematical function that predicts the system's evolution. This is equivalent to what humans do when we work out physical laws and equations of motion.
OK, thanks for clarifying. I feel that your description of neural nets' inner
workings is a bit idealised and I'm not convinced that we have seen any evidence
that they are as powerful in representing real-world phenomena as you suggest.
But that's a big discussion so let's leave this aside for a moment.
I can agree that a neural net can learn a model that can predict the behaviour
of a system, to some extent, within some margin of error.
That's not enough for me to see neural net models as (scientific) "laws". For
the sake of having a common definition of what a scientific law is, I'm going
with what wikipedia describes as a scientific law: a statement that describes or
predicts some set of natural phenomena, according to some observations
(paraphrasing from: https://en.wikipedia.org/wiki/Scientific_law). Sorry for not
introducing this definition earlier on. If you disagree with it, then that's my
bad for not estabilishing common terminology beforhand.
In that sense, neural net models are not scientific laws because, while they can
predict (but not describe) they are not "statements". Rather they are systems.
They have behaviour and their behaviour may match that of some target system,
like the weather say. But like a simulation of the economy, or an armillary
sphere are not, themselves "laws", even though they are possibly based on
"laws", a neural net's model can't be said to be a "law", even if it's based on
observations and even if it has an internal structure that makes its behaviour
consistent with some (known or unknown) law.
There is also the matter of usability: neural net models are, as we know, "black
boxes" that can't be inspected or queried, except by asking them to analyse some
data. While useful, that's not a "law", because it does not help us understand
the systems they model. If this sounds like a semantic quibble, it isn't. To me
anyway it doesn't make sense to base scientific knowledge on a bunch of
inscrutable black boxes. Scientific laws and scientific theories are not black
boxes.
As an aside, neural nets fall short of what Donald Michie (father of AI in the
UK) called "ultra-strong machine learning" [1]. That's the property fo a machine
learning system that improves not only its own performance, but that of its
user, also. Current techniques aren't even close to that.
____________________
[1] Machine Learning: the next five years, Donald Michie, 1988
I see why you would say that: these neural networks probably have thousands or millions of weights while the equations of motion can probably be written on an index card.
But I would argue that this parsimony is illusory. There's a lot of implicit knowledge needed for the interpretation of physical laws. The laws are written using specialized mathematical notation such as special functions, partial differential equations, in a certain conceptual framework such as Lagrangian mechanics. You need to understand the concept of abstracting and quantifying a dynamic system (most people wouldn't imagine you can do this) and then you have to learn all the tips and tricks of how to reformulate and solve systems.
For example, I could write a mathematical representation of quantum electrodynamics (the theory of how electrons and photons interact) on a single index card. However, I would need to dig into my two shelves of QFT textbooks to actually make any quantitative experimental predictions, on top of my degree, PhD and post doc experience, which I need to even be able to read the textbooks (and I would still mess up the minus signs).
I think it's important to remember that these neural networks are doing all of that - not just finding the physics, but also all the abstraction, calculation and interpretation that is usually taken for granted but actually very non-trivial.
I sort of agree with both myself and yourself. This point of mine technically must be qualified when interpreted like this but I actually mean something slightly more subtle than just parameterization.
The tools of physics have a lot of implicit assumptions that guide the end result in ways that I would describe as parsimonious in terms of how much the output state space must be reduced. They are much more free, which is why they can be amazing for some very hard shit, but proving they're behaving exactly in "physical" way is very hard.
"Time is defined so that motion looks simple" is my favourite quote from MTQ for this reason. It's intuitive and yet also very physically "rigorous" in a way that people don't necessarily realize is a thing in physics beyond just using mathematics.
Maybe we can just train the AI to do the maths for us, dunno, but I think currently this tabula Rasa approach will inform the physics-y-ness. I still call it physics personally, but I don't really think it's interesting from a purely physical perspective.
There have been some works deriving conservation's laws and so on from empirical motion, which I think is very impressive at scale, but I don't know what that does for physics as opposed to the applications of said physics.
I did see a presentation a few years ago on a similar topic by Erwin Couwans, author of Bullet physics engine, who was discussing doing neural inference of physics. He basically wanted to replace Bullet with a neural network, which I thought was kind of funny at the time, but cool if it worked. (Physical laws at this level of mechanics are mostly quite well understood, but the devil is in the details and actually solving time-discretized non-smooth dynamics, as well as performing good contact detection is less than obvious the more you get into it, not to mention friction, so I could see why a "learned" solution could be attractive, but at the time I found it funny that he was proposing to learn the physics that were already simulated by Bullet.) Looking back, it appears that those ideas culminated in a paper [2] two years ago, which I'll have to read now -- from the abstract, the differentiability of the simulation has benefits not only for learning the physics (mostly friction models it appears), but for learning controllers for plants within the simulated physics. It seems to be not cited in the linked article but maybe it is a slightly different, but related, topic. In any case, fascinating stuff.
[0]: https://www.science.org/doi/10.1126/science.1165893
[1]: https://arxiv.org/abs/2112.10755
[2]: https://arxiv.org/abs/2011.04217