Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Calculate the number of language model tokens for a string (mastykarz.nl)
17 points by waldekm on Feb 5, 2025 | hide | past | favorite | 3 comments


  import tiktoken
  enc = tiktoken.encoding_for_model('o1-mini')
  len(enc.encode('some string'))


Does that work on anything other than openai?


No but most models have their own libs you can use




Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: