A draft structure for tracking costly AI evaluation work
The open source framework draft for user-centric separates AI service metrics by where they run. The main split is versus cloud API processing.
covers fast work with no network call, such as regex checks or basic math. Cloud API processing covers slower or work, such as calls.
A shared AppLayer holds a score and a map of metric scores so the front end can find values quickly. The IChatMetric structure proposes automatic namespace registration, so developers do not have to create IDs by hand.
Key points
- Metrics are divided into and cloud API processing.
- is meant for quick checks with no network cost.
- Cloud API processing marks expensive or slower checks like .
- AppLayer stores scores and metric values for quick front-end lookup.
- IChatMetric aims to register namespaces automatically instead of making developers manage IDs.