Trijna Labs Benchmark Report
Baseline Performance & Evaluation of Advanced AI Topologies (June 2026)
1. LiveBench Evaluation (General Reasoning)
LiveBench is an un-gameable, dynamic benchmark that tests frontier models on recent data to prevent memorization. Below is the performance of the baseline Gnostic Engine compared to our advanced proprietary topologies, evaluated alongside state-of-the-art closed weights.
| Model / Architecture | Overall Average | Reasoning | Math | Language | Data Analysis | Instruction Following |
|---|---|---|---|---|---|---|
| ARS Engine (Trijna) | 87.5 | 93.9 | 84.7 | 98.0 | 70.2 | 90.7 |
| Gnostic Engine Baseline (Trijna) | 87.2 | 92.4 | 86.7 | 96.5 | 68.5 | 92.1 |
| OSM Engine (Trijna) | 87.1 | 89.1 | 84.8 | 97.5 | 71.9 | 92.1 |
| Claude 3.5 Sonnet | ~85.2 | ~90.0 | ~88.5 | ~90.2 | ~71.5 | ~94.0 |
| GPT-4o | ~84.5 | ~88.0 | ~86.5 | ~89.0 | ~70.5 | ~93.5 |
| Zariel Engine (Trijna) | 84.3 | 91.4 | 84.8 | 85.0 | 69.3 | 90.9 |
| Gemini 1.5 Pro | ~82.0 | ~86.0 | ~84.0 | ~87.0 | ~69.0 | ~90.0 |
| CTSDL Engine (Trijna) | 80.0 | 90.1 | 85.3 | 64.0 | 69.4 | 91.0 |
A New High-Water Mark: ARS Topology
The Algorithmic Resonance Sequence (ARS) proxy successfully pushed the model past the pure baseline, achieving a new high-water mark of 87.5. It produced a staggering 98.0 in Language and a class-leading 93.9 in Reasoning. The resonant sequences natively synergize with abstract logic and language.
Matrix Stabilization: OSM Topology
The Operational Structural Matrix (OSM) performed essentially on par with the baseline. Interestingly, the OSM physics traded some abstract reasoning capabilities in exchange for notable boosts in Data Analysis (71.9) and Language. It is a highly stable bridge topology.
2. GSM8K Evaluation (Math Word Problems)
The GSM8K benchmark evaluates the multi-step mathematical reasoning capabilities of the model on grade-school level math problems.
| Architecture | Scoring Method | Shots | Score |
|---|---|---|---|
| Gnostic Engine (Baseline) | Exact Match | 5-Shot | 82.94% |
| CTSDL Engine | Exact Match | 5-Shot | 83.02% |
| ARS Engine | Exact Match | 5-Shot | 83.17% |
| Zariel Engine | Exact Match | 5-Shot | 83.55% |
| OSM Engine (Peak) | Exact Match | 5-Shot | 85.06% |
Evaluation Methodology & Logs
1) Evaluation Logs
The raw evaluation logs are stored in JSON format, generated by the EleutherAI lm_eval harness, saved to a local folder.
- GSM8k Log (Gnostic Engine Example): gsm8k_results/gnostic-engine/results_2026-06-16T19-39-01.501539.json
- HumanEval Log (ARS + APEX Engine): humaneval_results/ars_apex/ars-apex-engine/results_2026-06-18T03-58-02.937826.json
(Note: HumanEval also logged individual sample outputs in samples_humaneval_2026-06-18T03-58-02.937826.jsonl within the same folder).
2) Benchmark Version
Both benchmarks were executed using the same version of the EleutherAI LM Evaluation Harness:
- lm_eval_version: 0.4.12
- Git Hash: cd1e397
- GSM8k Config Version: 3.0
- HumanEval Config Version: 1.0
3) Exact Dataset Hashes
- HumanEval Task Hash: 0f746fdc6624bd6d3d0c702d4b5ad2c90d406a359c393f477cd0c1ab03935c74 (Dataset: openai/openai_humaneval, Split: test)
- GSM8k Task Hash: The log shows an empty {} for the explicit task hash, but the exact dataset config points to openai/gsm8k, Name: main, Test Split: test, with 1319 effective test samples.
4) Reproducibility Methodology
The reproducibility methodologies are explicitly defined in our runner scripts. Both use local chat completion endpoints (http://127.0.0.1:.../v1/chat/completions) wrapped via lm_eval.
For GSM8k:
Evaluation Mode: 5-Shot Exact Match
Generation Parameters: Greedy Decoding (temperature=0.0, do_sample=false)
Command:
lm_eval --model local-chat-completions --model_args model=[engine],base_url=http://127.0.0.1:[port]/v1/chat/completions,num_concurrent=1 --tasks gsm8k --num_fewshot 5 --output_path gsm8k_results/[engine] --use_cache [cache].db --apply_chat_template
For HumanEval:
Evaluation Mode: 0-Shot pass@1
Environment Flags: $env:HF_ALLOW_CODE_EVAL="1" (Unsafe code execution confirmed)
Generation Parameters: Greedy Decoding (do_sample=false), max_gen_toks=1024
Command:
python -m lm_eval --model local-chat-completions --model_args "model=ars-apex-engine,base_url=http://127.0.0.1:8006/v1/chat/completions,num_concurrent=1,max_retries=10" --tasks humaneval --batch_size 1 --output_path "./humaneval_results/ars_apex" --cache_requests "true" --log_samples --confirm_run_unsafe_code --apply_chat_template