Backlinks linking to the video on the web, number of views, likes, comments, NLP analysis of the title and the description of the video and yea references made to each video in other video descriptions is also good idea.
It's an interesting idea. But my gut feel is that only a minority of content actually is referenced elsewhere. It would be an interesting idea to test though!
You would use ranking algorithms meaning you should crawl, scrape and analyse YouTube videos which would make YouTube probably unhappy.