Jane Index

Testing llms on quadruple amputee scenarios

Test of existing presets within ooba's webui by generating actions of the quadruple amputee;

Initial Generation (generative model)

• start ooba's text generation webui service on port 5000;
• load model you want to use for generation; ./jane_index.py will generate json file with that model name (for ex: TheBloke_Llama-2-13B-GPTQ.json);

Scoring (judge model)

• go to ooba's webui and set judge model (preferably 65b+); ./jane_index.py TheBloke_Llama-2-13B-GPTQ.json judge scores each generation as success or failure, modifies initial json file to add scores to it, where 1 is a perfect score and 0 is a complete failure.