How can scientists tell the difference between fresh and saltwater fossils (fish or reptiles)?
For some fossils, scientists can measure the oxygen isotopes in the fossilized bones, because the oxygen in the bones comes largely from the water the animal lived in. Sea water contains a higher proportion of heavier oxygen isotopes than does freshwater. This is because water with the lighter isotope of oxygen evaporates more easily, and freshwater forms largely from evaporated sea water. Thus, fossil bone containing more heavy oxygen can be an indication that the animal lived in saltwater, and bone containing lighter oxygen can indicate freshwater. For example, the 52-million-year-old fossils of the early whale Pakicetus were found to contain oxygen isotopes suggesting that they lived in freshwater.
For other fossils, the geological history of the rocks they are found in can be used to estimate the salinity of the fossil's environment. For example, the Fossil Butte National Monument in Wyoming has a collection of fossils known to have formed in an approximately 50-million-year-old freshwater lake. Thus, the fish and crocodile fossils found in this formation are thought to be freshwater organisms. Reptile fossils from South Dakota, which are understood to have formed in a saltwater sea that covered South Dakota at the time, are thus thought to be marine animals.
Many other clues to whether a fossil is from freshwater or saltwater can be found in the fossil's depositional environment (e.g., whether the fossil is embedded in sand or silt, what other kinds of animals and plants are fossilized nearby). For any particular fossil, all of the available evidence from all of these different sources of information can be compared to determine whether the animal lived in freshwater or saltwater.
For more information
Story from New Scientist magazine, June 1, 1996, about Pakicetus fossil oxygen isotope study that concluded Pakicetus lived in freshwater:
From Northern State University, Marine Reptiles of South Dakota:
Fossil Butte National Monument in Wyoming: