Llama.cpp is a port of Facebook's LLaMA model in C/C++ developed by Georgi Gerganov. Llama.cpp allows the inference of LLaMA and other supported models in C/C++. For CPU inference Llama.cpp supports AVX2/AVX-512, ARM NEON, and other modern ISAs along with features like OpenBLAS usage.
To run this test with the Phoronix Test Suite, the basic command is: phoronix-test-suite benchmark llama-cpp.
OpenBenchmarking.org metrics for this test profile configuration based on 206 public results since 10 January 2024 with the latest data as of 4 May 2024.
Below is an overview of the generalized performance for components where there is sufficient statistically significant data based upon user-uploaded results. It is important to keep in mind particularly in the Linux/open-source space there can be vastly different OS configurations, with this overview intended to offer just general guidance as to the performance expectations.
Based on OpenBenchmarking.org data, the selected test / test configuration (Llama.cpp b1808 - Model: llama-2-13b.Q4_0.gguf) has an average run-time of 5 minutes. By default this test profile is set to run at least 3 times but may increase if the standard deviation exceeds pre-defined defaults or other calculations deem additional runs necessary for greater statistical accuracy of the result.
Based on public OpenBenchmarking.org results, the selected test / test configuration has an average standard deviation of 0.3%.
Yes, based on the automated analysis of the collected public benchmark data, this test / test settings does generally scale well with increasing CPU core counts. Data based on publicly available results for this test / test settings, separated by vendor, result divided by the reference CPU clock speed, grouped by matching physical CPU core count, and normalized against the smallest core count tested from each vendor for each CPU having a sufficient number of test samples and statistically significant data.
Notable instruction set extensions supported by this test, based on an automatic analysis by the Phoronix Test Suite / OpenBenchmarking.org analytics engine.
This test profile binary relies on the shared libraries libopenblas.so.0, libm.so.6, libc.so.6, libgfortran.so.5, libquadmath.so.0.
This benchmark has been successfully tested on the below mentioned architectures. The CPU architectures listed is where successful OpenBenchmarking.org result uploads occurred, namely for helping to determine if a given test is compatible with various alternative CPU architectures.
1 System - 236 Benchmark Results |
AMD Ryzen 9 7900 12-Core - ASRockRack 1U4LW-B650/2L2T B650D4U-2L2T/BCM - AMD Device 14d8 Ubuntu 24.04 - 6.8.0-31-generic - GNOME Shell 46.0 |
1 System - 7 Benchmark Results |
AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.8.7-zen1-1-zen - Xfce 4.18 |
1 System - 3 Benchmark Results |
AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.8.2-zen2-1-zen - Xfce 4.18 |
1 System - 341 Benchmark Results |
AMD Ryzen 9 7950X 16-Core - ASUS ProArt X670E-CREATOR WIFI - AMD Device 14d8 Pop 22.04 - 6.6.10-76060610-generic - GNOME Shell 42.5 |
2 Systems - 8 Benchmark Results |
AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.9-zen1-1-zen - X Server 1.21.1.11 |
1 System - 1 Benchmark Result |
AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.9-zen1-1-zen - X Server 1.21.1.11 |
3 Systems - 79 Benchmark Results |
Intel Core i7-1185G7 - Dell XPS 13 9310 0DXP1F - Intel Tiger Lake-LP Ubuntu 23.10 - 6.7.0-060700rc5-generic - GNOME Shell 45.1 |
2 Systems - 52 Benchmark Results |
2 x INTEL XEON PLATINUM 8592+ - Quanta Cloud QuantaGrid D54Q-2U S6Q-MB-MPS - Intel Device 1bce Ubuntu 23.10 - 6.6.0-060600-generic - GCC 13.2.0 |
2 Systems - 28 Benchmark Results |
AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18 |
2 Systems - 23 Benchmark Results |
AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18 |
2 Systems - 23 Benchmark Results |
AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18 |
2 Systems - 18 Benchmark Results |
AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18 |
2 Systems - 13 Benchmark Results |
AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18 |
5 Systems - 587 Benchmark Results |
Intel Core i5-14500 - ASUS PRIME Z790-P WIFI - Intel Raptor Lake-S PCH Ubuntu 23.10 - 6.7.3-060703-generic - GNOME Shell 45.2 |
1 System - 12 Benchmark Results |
AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18 |
6 Systems - 162 Benchmark Results |
AMD Ryzen 7 8700G - ASRock B650 Pro RS - AMD Device 14e8 Ubuntu 23.10 - 6.7.0-060700-generic - GNOME Shell 45.0 |
18 Systems - 154 Benchmark Results |
Intel Core i9-14900K - ASUS PRIME Z790-P WIFI - Intel Device 7a27 Ubuntu 23.10 - 6.7.0-060700-generic - GNOME Shell 45.0 |
3 Systems - 83 Benchmark Results |
AMD EPYC 8534P 64-Core - AMD Cinnabar - AMD Device 14a4 Ubuntu 23.10 - 6.5.0-5-generic - GNOME Shell |
5 Systems - 587 Benchmark Results |
Intel Core i5-14600K - ASUS PRIME Z790-P WIFI - Intel Raptor Lake-S PCH Ubuntu 23.10 - 6.7.3-060703-generic - GNOME Shell 45.2 |
5 Systems - 149 Benchmark Results |
AMD EPYC 7601 32-Core - TYAN B8026T70AE24HR - AMD 17h Ubuntu 23.10 - 6.6.9-060609-generic - GNOME Shell 45.0 |
4 Systems - 100 Benchmark Results |
AMD Ryzen Threadripper PRO 5965WX 24-Cores - ASUS Pro WS WRX80E-SAGE SE WIFI - AMD Starship Ubuntu 23.10 - 6.5.0-13-generic - GNOME Shell 45.0 |
3 Systems - 69 Benchmark Results |
Intel Xeon Silver 4216 - TYAN S7100AG2NR - Intel Sky Lake-E DMI3 Registers Debian 12 - 6.1.0-11-amd64 - X Server |
2 Systems - 262 Benchmark Results |
AMD EPYC 7F32 8-Core - ASRockRack EPYCD8 - AMD Starship Debian 12 - 6.1.0-11-amd64 - X Server |
16 Systems - 168 Benchmark Results |
AMD Ryzen 7 7700 8-Core - ASRock B650 Pro RS - AMD Device 14d8 Ubuntu 23.10 - 6.7.0-060700-generic - GNOME Shell 45.0 |
2 Systems - 52 Benchmark Results |
2 x INTEL XEON PLATINUM 8592+ - Quanta Cloud QuantaGrid D54Q-2U S6Q-MB-MPS - Intel Device 1bce Ubuntu 23.10 - 6.6.0-060600-generic - GCC 13.2.0 |
2 Systems - 31 Benchmark Results |
Intel Core i9-10980XE - ASRock X299 Steel Legend - Intel Sky Lake-E DMI3 Registers Ubuntu 22.04 - 6.2.0-39-generic - GNOME Shell 42.2 |
2 Systems - 28 Benchmark Results |
AMD EPYC 7R13 48-Core - Supermicro H12SSL-I v1.02 - AMD Starship EndeavourOS rolling - 6.7.4-zen1-1-zen - Xfce 4.18 |
3 Systems - 27 Benchmark Results |
AMD EPYC 7601 32-Core - TYAN B8026T70AE24HR - AMD 17h Ubuntu 23.10 - 6.6.9-060609-generic - GNOME Shell 45.0 |
7 Systems - 167 Benchmark Results |
AMD EPYC 8534PN 32-Core - AMD Cinnabar - AMD Device 14a4 Ubuntu 23.10 - 6.6.9-060609-generic - GNOME Shell 45.0 |
Featured Processor Comparison |
AMD Ryzen 5 8500G - ASRock B650 Pro RS - AMD Device 14e8 Ubuntu 23.10 - 6.7.3-060703-generic - GNOME Shell 45.2 |