In total there are 5 coloured gems (Red, Purple, Blue, Green, Yellow) and they are needed to reveal optional Gem paths in some levels that are otherwise inaccessible. The video below will show you where to find each of the Gems.
Red Gem
Level 12: Deep Trouble - Use the '!' exclamation point box at the end of the level to start a secret course. Pick up the Gem during the secret course.
Purple Gem
Level 13: High Time - Found on death route (must get there without losing a life).
Blue Gem
Level 20: Tomb Wader - Found on the death route (must get there without losing a life).
Green Gem
Level 23: Flaming Passion - Found on death route (must get there without losing a life).
Yellow Gem
Level 27: Hang’em High - Found on the main path at the end of Level 27: Hang’ Em High. To access the level you must earn 10 Relics by doing speedruns. Once you have 10 relics step on the orange dot in the middle of the central warp room (where you select levels) and it will lead you to a secret 6th warp room in the basement. From here you can start level 27.