I have been working on knowledgebase stuff using embedding for my programming website that uses OpenAI's APIs. Maybe this is obvious, but does anyone know a good tool or algorithm or anything for creating the snippets? The first thing I tried was just splitting on multiple newlines or one newline and whitespace or something.

Some of the snippets are much too long, some to short. Also ideally I could extract code in snippets that include the whole function.

Maybe I should copy how gpt-index is doing it.