Cosmology is arguably the most ambitious of all the sciences. It is built on astronomy, to be sure, but also a great deal of particle physics and quantum theory in energy regions where we have little or no data.
Learning anything about the universe as a whole requires enormous investments in astronomical observations at all wavelengths and statistical analysis of the collected data. On the back end, these results are compared to computer simulations that start with a few simple assumptions and then follow the physics from the Big Bang to the present, and, finally translate the large-scale picture to a prediction of what we should be seeing as we look out from our present vantage.
Every fact that we have about the universe as a whole was obtained at enormous cost in time, dollars, and effort. Because the facts we have about the universe are few, in good faith we can only entertain theories that are very simple. For example, if we have 7 measured facts about the universe and a theory with 6 free parameters, the theory is on shaky ground. If we need 7 free parameters to explain 7 observations, then we are open to the charge that any other theory might serve just as well.
The heyday of physical cosmology began with the discovery of the cosmic microwave background in 1965 and continued through 1997. Back in the 1970s and 80s, when I was studying astrophysics, there was an enormous sense of excitement and pride because of one agreement between theory and observation.
28% of matter in stars (and interstellar clouds) is hydrogen, and the rest is helium. If we extrapolate the present temperature of the universe (3 degrees above absolute zero as inferred from the cosmic microwave background) back to the big bang, then we could calculate what nuclear reactions would have occurred in the first three minutes that the universe existed. The answer we got was 28% helium, 72% hydrogen. (This story is related in Steven Weinberg's very readable (but dated) book, The First Three Minutes.)
What happened in 1997 is that two independent measurements of the expansion of the universe both showed that the expansion is speeding up, and this required a negative gravity to explain it. A negative gravity substance was postulated and given the name "dark energy", and another kind of matter, "dark matter" was needed as well, partially to make up for the dark energy. We know absolutely nothing about either DM or DE except that they utterly resist detection by any means outside these observations of the structure of galaxies and clusters of galaxies.
Since 1997, cosmology has been in an uncomfortable zone where science really can't do business: there are more assumptions that go into the model than there are facts that the model can explain. Clearly, we need more observations, more facts, more opportunities to test and constrain existing theories.
This has been a golden age for automated observation of the heavens and sophisticated data analysis. As the new observations come in, The problem has been getting worse. Sober, conservative leaders in the field speak of a crisis in cosmology. Two such articles have appeared in recent months. The one that I just linked (by my Harvard classmate of 50 years ago, Joseph Silk) notes that theorists have indulged in a kind of cheating to make their models appear consistent with the data. They have chosen parameters for the expansion and the density of the universe that are halfway between values measured by two kinds of methodologies. If you compare Observation A to the model, it is just on the edge of being plausible. If you compare Observation B to the model, it is just on the edge of being plausible in the other direction. But, as Dr Silk points out, if you compare Observation A directly to Observation B, you realize that the two are too far apart to be compatible, and that our research and analysis methods must be called into question.
Dr Becky Smethurst emphasizes that one implication of the new perspective is that the universe is closed and finite and will not expand forever.
The other recent article notes that all of the measurements that pointed to speed-up in the expansion (and the need for dark energy) came from one direction in the sky. If you look in the opposite direction, the epansion is slowing down. Maybe it's not that the whold universe is changing its expansion at all, but only that our little neighborhood has shifted direction. But we're out of the frying pan, into the fire, because the same evidence suggests that the universe may not be completely uniform and symmetrical, as theories have always assumed. The trouble with asymmetrical models is that they call into question the very simple equations that are our hope for staying within 6 or 7 free parameters. There's a worse problem, actually: The equations of gravity (Einstein's General Relativity) are so insanely complicated that they cannot be solved even with the largest supercomputers we have except in the case where the equations are enormously simplified either by (1) a very high degree of symmetry that vastly reduces the complexity, or (2) weak fields, called the "Newtonian limit". If this situation persists, we will have
Sabine Hossenfelder does a good job of explaining the context in this video.
