Dr Paul Smith and a team from the School of Earth Sciences at Birmingham University, found the remains in sandstone rock 500 million years old near Colorado Springs in the United States.
The shark scales are 25 million years older than the earliest shark remains known previously, which were discovered in Siberia. The thelodont fossils extend the previous record by 10 million years.
Both remains date from the Ordovician period, which began 505 million years ago and lasted 70 million years. The oldest sharks had been dated back to the Silurian period, which started 430 million years ago.
The new evidence indicates that fish evolved earlier than had previously been thought.
Writing in the science journal Nature, Dr Smith said the finds suggested that "the major period of diversification within the lower vertebrates was already well under way during the Ordovician, not the Silurian as has been suggested classically".
Another member of the team, Dr Ivan Sampson, said the finds would force a complete re-think of the way fish evolved.
He said: "The traditional textbook idea is that fish first appeared at some stage in the Ordovician and underwent a major evolutionary episode 50 million years later.
"What we have found ... is that fish went through a big stage of evolution soon after they first appeared, much earlier than had previously been thought," he said.
He said other clues found in South America and Australia seemed to support the theory.
It is still uncertain whether any fish possessed teeth and jaws 455 million years ago, when scientists place the new finds. Dr Sampson said it was possible that the earliest sharks did not have jaws and, like the thelodonts, were similar to the modern eel-like lamprey.
However, a number of tooth-like objects, about a millimetre long, were found with the scales, but have not yet been identified.
The team is planning to return to the site this summer in the hope of making further finds. "What we would really like are more complete fossil remains, possibly the whole body of one of these creatures," Dr Sampson said.Reuse content