Can we use a System One model in the browser?

What happened

The article describes the testing of Laya, an open-source decision model, in a browser environment. It evaluates the model's performance in terms of download time, speed, and accuracy. The model was tested using different quantization levels (8-bit and 4-bit) and hardware configurations (CPU and GPU). The results show that while the model can run in the browser, there are trade-offs in speed, memory, and accuracy depending on the setup.

Why it matters

For AI developers and web app creators, this article provides insights into deploying models directly in the browser, which can reduce dependency on servers and improve privacy. However, it also highlights limitations such as memory usage and performance degradation in background tabs, which are important considerations for real-world applications.

• Running models in the browser is feasible for quick, user-triggered decisions but has limitations in background execution.
• Quantization (4-bit vs. 8-bit) affects speed and accuracy, with GPU support being crucial for performance.
• Memory usage is significant, and background tabs in Chrome can severely impact performance.
• Fine-tuning the model improves accuracy, and the model should not be trusted without further validation.
• Browser compatibility and hardware limitations must be considered when deploying models in the browser.

The bigger picture

The ability to run models in the browser can enhance privacy and reduce latency for certain applications, such as ticket routing or lead scoring. However, the limitations in background execution and accuracy with 4-bit quantization suggest that server-side processing may still be necessary for critical tasks.

The article concludes that while it is possible to run a System One model like Laya in the browser, the practicality depends on the use case. It is suitable for applications that require quick, user-initiated decisions but not for background tasks or high-accuracy guardrail systems.

Technical details

• Laya is a 421M-parameter model with a decision head, built on ModernBERT-large.
• The model was ported to ONNX Runtime Web with 8-bit and 4-bit quantization.
• The 8-bit build uses int8 block-128 MatMulNBits weights, while the 4-bit uses int4 block-32 weights.
• The model runs on Apple Silicon Mac with Chromium-based browsers, and performance is measured using session.run() and memory usage.

FAQ