And did some caveman just drop their meat into the fire or something?

Bread cooked in stone ovens tended to develop a very thick crust on the bottom. This was generally cut off before it was served to the wealthy. The remainder was then given to the servants to eat. Toast was then simply the cast offs from the more well offs bread.