This book identifies the organizing concepts of physical and biological phenomena by an analysis of the foundations of mathematics and physics. Our aim is to propose a dialog between different conceptual universes and thus to provide a unification of phenomena. The role of "order" and symmetries in the foundations of mathematics is linked to the main invariants and principles, among them the geodesic principle (a consequence of symmetries), which govern and confer unity to various physical theories. Moreover, an attempt is made to understand causal structures, a central element of physical intelligibility, in terms of both symmetries and symmetry breakings. A distinction between the principles of (conceptual) construction and of proofs, both in physics and in mathematics, guides most of the work.The importance of mathematical tools is also highlighted to clarify differences in the models for physics and biology that are proposed by continuous and discrete mathematics, such as computational simulations.Since biology is particularly complex and not as well understood at a theoretical level, we propose a "unification by concepts" which in any case should precede mathematization. This constitutes an outline for unification also based on highlighting conceptual differences, complex points of passage and technical irreducibilities of one field to another. Indeed, we suppose here a very common monist point of view, namely the view that living objects are "big bags of molecules". The main question though is to understand which "theory" can help better understand these bags of molecules. They are, indeed, rather "singular", from the physical point of view. Technically, we express this singularity through the concept of "extended criticality", which provides a logical extension of the critical transitions that are known in physics. The presentation is mostly kept at an informal and conceptual level.