Thank you everyone for the support. I have received nearly 40 signups and 1 paid user which is massive for me as I am still on the early stages of validating the product. So thank you everyone.
Next steps: Even though the signups are coming in I am tracking the usage of the app and I dont see many users running the evaluations and I need help. How should I get you to try and test the product.
My current thoughts are:
- User using their API keys means friction and would go away from the platform as can't test it immediately so perhaps allow free trial with my API key which involves unlimited runs with 3 test cases.
- Create a onboarding guide? Like the ones of enterprise softwares that says "Click here" "Next Steps": What tool can I use for this?
For both the options above will still need to inform them about the updates and hope they will signin again.
- Reach out to all 40 users for a 1-1 15 mins call and show them the product. Assuming 30% respond that 12 calls booked.
Do you have any suggestions? Keen on feedback. This is critical as I need to solve these issues before building next features i.e (Adding more models and multi model runs).
PromptPerf
As an AI developer, I spend a lot of time running prompts across different models and configs, tweaking temperature, comparing outputs, and manually checking which one gets it right.
It’s repetitive. Time-consuming. And easy to mess up.
So I built PromptPerf -> a tool that tests a single prompt across GPT-4o, GPT-4, and GPT-3.5, runs it multiple times, and compares the results to your expected output using similarity scoring.
⚡ No more guessing which prompt or model is better
⚡ No more switching between tabs
⚡ Just clean, fast feedback and a CSV if you want it
This started as a scratch-my-own-itch tool, but now I’m opening it up to anyone building with LLMs.
Unlimited free runs. More models coming soon. Feedback shapes the roadmap.
Would love to hear what you think! Keen on feedback and help to ensure I build a product that solves your problems
👉 promptperf.dev
Whoa! This looks interesting!
PromptPerf
@neelptl2602 Thanks Neel, I plan on adding multiple models from Claude, Gemini and others soon to evaluate across models and different temperatures.
Brev
This is super useful, thanks for building this.
PromptPerf