Small Language Model Ensembles as the Sustainable Path Beyond Monolithic LLM Scaling

Executive summary

The “bigger LLM forever” strategy is running into a wall of economic efficiency, energy/latency constraints, and operational complexity. The evidence does not say “large models stop mattering.” It says: the marginal return on scaling a single monolithic model is increasingly dominated by its worst-case costs, while the business value for most real workloads concentrates in “good-enough” responses delivered fast, cheaply, and with strong governance. In practice, that pushes the industry toward compound systems: routers, retrieval, tools, and ensembles of smaller specialist models—with selective escalation to larger models for the long tail. [1]

Three empirical pillars support this stance:

1. Compute scaling is outpacing classic hardware trends. Analysis from OpenAI[2] showed the largest training runs grew exponentially with a ~3.4‑month doubling time (2012–2018) and ~300,000× growth over that period. [3] More recent tracking from Stanford HAI[4] reports that training compute for notable models roughly doubles every ~5 months, dataset scale roughly every ~8 months, and training power rises annually. [5]
2. Compute-optimal evidence undermines “just scale parameters”: the Chinchilla scaling study shows that many large models were undertrained and that a smaller model trained on more tokens can outperform a larger one at the same compute budget—meaning “bigger dense model” was often an inefficient allocation. [6]
3. Inference efficiency gaps are enormous and grow quickly with model size, especially at the frontier scale. TokenPowerBench reports low single‑digit Joules/token for small-to-mid models, but ~160–236 J/token for frontier models like Llama‑3 405B in realistic multi‑GPU serving configurations, depending on the parallelism strategy and workload. [7]

At the same time, we now have a strong “SLM emergence” story: high-quality small models (≈1B–14B) are increasingly competitive on mainstream benchmarks due to better data curation, distillation, and post-training methods (e.g., Phi, OpenELM, Ministral). [8]

Operationally, the shift to SLM ensembles aligns with what senior product/AI leaders actually need: modularity, controllable latency/cost curves, better privacy posture (local inference), faster update cycles, and more granular governance for regulated domains. [9]

This is especially relevant for an AI product leader already operating in multi-agent / orchestrated AI systems in enterprise contexts, because the work is less about a single model choice and more about getting a system to hit reliability, cost, and governance targets.

Explicit assumptions

For level setting, all quantitative comparisons below rely on these explicit assumptions:

• Latency targets: interactive UX with p95 time‑to‑first‑token < 1.5s; sustained decode > 20 tokens/s for most sessions; long-context workloads tolerate higher prefill latency. (Assumption; used to interpret benchmark relevance.)
• Hardware classes:
  • Edge: smartphone NPUs/SoCs, or laptops with integrated NPUs.
  • Server: modern inference GPUs (e.g., H100-class in research benchmarks), plus typical production inference engines. [10]
• Workload mix: enterprise “knowledge + workflow” tasks (summarization, retrieval Q&A, classification/routing, extraction, policy checks, tool calling), not purely open-ended creative writing.
• Quality target: “business acceptable” with guardrails: factuality + policy compliance + structured output > maximum reasoning depth.
• Cost model: When I compute energy costs, I assume $0.12/kWh for electricity; infrastructure (GPU depreciation, networking, staff) is not included unless stated otherwise. (Assumption.)
• Ensemble definition: “multiple smaller models coordinated by routing/aggregation,” including (a) model cascades and (b) sparse expert activation (MoE) as an internalized ensemble-like mechanism.

Definitions and taxonomy

The terms “SLM,” “LLM,” “micro-model,” and “ensemble” are used inconsistently in industry; to argue rigorously, it helps to ground them in deployment-relevant characteristics: parameter scale, inference placement, and orchestration structure.

Working definitions

• Large Language Model (LLM): a general-purpose transformer-based language model typically trained on at least tens of billions of parameters and trillions of tokens, optimized for broad capability across many tasks; commonly deployed via centralized GPU clusters due to memory and throughput requirements. Scaling-laws work explicitly analyzes this regime. [11]
• Small Language Model (SLM): a language model intentionally optimized for deployment-constrained environments (edge, single GPU, CPU), typically in the ~1B–14B range today, where engineering efficiency (quantization, pruning, distillation, post-training) is a first-class objective. The Phi and Ministral lines explicitly position “small yet capable” as a design goal. [12]
• Micro-models: extremely small models (often <1B, sometimes tens–hundreds of millions of parameters) used as components rather than endpoints—e.g., routers, confidence estimators, classifiers, embedding models, or draft models for speculative decoding. Research on early exit and selective prediction formalizes the notion of “cheap gatekeepers” that defer hard cases to experts. [13]
• Ensemble (multi-model): a system that combines outputs from multiple models to improve quality, robustness, or cost-efficiency. In the LLM era, the most important ensemble patterns are routing/cascades (choose one model per request) and agentic multi-model systems (multiple models collaborate through tool calls and structured communication). [14]
• Sparse Mixture-of-Experts (MoE): an architecture where a router activates only a small subset of “experts” per token—an internal ensemble that can raise total parameter count without proportional per-token compute. MoE is a core efficiency alternative to dense scaling. [15]
• Agentic multi-model system: a compound system where a planner/agent model decomposes tasks, calls tools, consults retrieval, and may delegate subtasks to specialist models. ReAct is an archetypal research pattern for “reason + act” workflows. [16]

Taxonomy table

Diagram: ensemble system archetype

This diagram encodes the core claim: most requests should never touch the large model, and when they do, it’s by deliberate, measurable policy.

Historical trends in scaling, and why “ever-growing LLMs” look inefficient

Scaling laws show predictable improvements and clarify the cost structure.

Empirical scaling-law work finds loss improves as a power law with model size, data size, and compute, enabling predictions about how to allocate compute. [20] The key nuance is that scaling laws don’t say “scale parameters indefinitely”; they say “there is a frontier, and efficiency depends on balancing parameters and data.”

Chinchilla’s compute-optimal results directly challenge the “parameter-first” era: many large models were undertrained because data didn’t scale with size, and a 70B model trained on more data outperformed much larger models at similar compute budgets. [6] In other words, one of the most influential scaling papers of the last few years implicitly argues that unquestioningly growing dense LLMs is an inefficient use of compute.

Compute, power, and cost trends strongly favor efficiency strategies.

Training computing has grown extremely quickly. OpenAI[2] documented an exponential growth regime in compute, with a very fast doubling time (historically ~3.4 months in their 2018 analysis). [3] More recent tracking by Epoch AI[21] and Stanford HAI[4] suggests training compute for notable/frontier models is still growing rapidly (on the order of months per doubling). That training power requirements rise as well. [22]

Two implications matter for this argument:

1. Capital intensity and access: if training/serving frontier models requires enormous compute fleets, the ecosystem centralizes into a smaller number of vendors/operators. [23]
2. Efficiency becomes a strategic differentiator: algorithmic efficiency improvements can reduce compute for a fixed capability (OpenAI’s “AI and efficiency” frames this as compute-to-performance improvements over time). [24] But the business question becomes: do you spend those gains to build even larger models, or do you cash them in as lower costs and wider deployability?

Environmental impacts strengthen the “unsustainable” thesis (with important caveats)

Research has quantified the financial and carbon costs of training and developing large NLP models and has argued that these costs matter for policy and practice. [25] A later study emphasizes that energy and CO₂e depend heavily on datacenter efficiency, hardware choice, and location. It highlights that sparse (MoE-like) models can consume <1/10 the energy of large dense models without sacrificing accuracy—a direct technical argument for the “ensemble/sparse” path over dense scaling. [26]

Lifecycle analysis of the 176B-parameter BLOOM model estimated training emissions on the order of tens of tonnes CO₂e (depending on the accounting scope) and highlights that inference/deployment can be a meaningful part of the footprint as usage scales. [27]

A key caveat to consider when making the sustainability argument: the exact split of energy between training and inference varies widely by product and adoption curve. Recent benchmarking work asserts inference can dominate operational energy at scale, citing practitioner reports. [28] I would treat “>90%” specifically as plausible but not universally proven—it’s credible for high-traffic services, but it’s not a law of nature.

Empirical benchmarks: performance-per-parameter, latency, energy, and cost

Performance-per-parameter is improving in the SLM era

The SLM wave is not “new” in concept (distillation and compression are decades old), but it is new in competitiveness. Modern SLMs increasingly match or exceed older mid-size LLM performance because of improved data quality, synthetic data, distillation recipes, and post-training.

Phi family (capability at ~3.8B–14B)
The Phi-3 technical report claims:

• Phi‑3‑mini (3.8B) achieves ~69% on MMLU and 38 on MT‑Bench, and is “small enough to be deployed on a phone.” [29]
• Phi‑3‑small (7B) and Phi‑3‑medium (14B) reach ~75% and ~78% MMLU, respectively. [30]
• Phi‑3.5‑MoE is a 16×3.8B MoE with 6B active parameters, positioned as competitive with significantly larger/open models and even some frontier “fast” models. [30]

This supports a core claim behind SLM ensembles: conditional computation + better training recipes can substitute for brute-force dense size.

OpenELM (≈1B-class efficiency)
Apple[31] describes OpenELM as an “efficient” model family with a layer-wise scaling strategy; the associated paper/summary claims that, around a 1B parameter budget, it outperforms a comparable open model while requiring fewer pretraining tokens. [32]

Ministral 3 and cascade distillation as “SLM manufacturing.ng”
A 2026 “Ministral 3” report explicitly frames a recipe to derive 3B/8B/14B models via Cascade Distillation (iterative pruning + continued training + distillation from a stronger parent). [33]
It also provides comparative benchmark results (selected excerpts):

• Ministral 3 (8B) shows MMLU‑Redux ~79.3 and Multilingual MMLU ~70.6, compared to its 24B teacher’s MMLU‑Redux ~82.7 and MMLU ~81.0. [34]
• Ministral 3 (3B) still reaches MMLU ~70.7 and MATH ~60.1 (CoT 2-shot) in their evaluation setup. [34]

This is direct evidence that the frontier is shifting from “train one giant model” to “industrialize the production of specialized descendants.”

Latency: on-device SLMs redefine “fast enough.ough”

On-device models can reach throughputs that make many real-time use cases feasible without cloud calls:

• Google[35]’s Android documentation notes Gemini Nano runs in Android’s AICore system service and leverages device hardware for low-latency inference. [36]
• An Android developer update reports Gemini Nano prefix speeds of up to ~940 tokens/s on the Pixel 10 Pro for text-to-text benchmarks (and a similar prefix speed for image-to-text after image encoding). [37]

This matters for the thesis: local inference changes the latency and privacy calculus; large centralized LLMs cannot match “zero network hop” behavior.

Energy and cost: TokenPowerBench provides hard numbers that favor ensembles and smaller models

TokenPowerBench (AAAI 2026 paper) benchmarks the power/energy of LLM inference across model scales on H100-class infrastructure and reports energy per token as a first-class metric. [38]

Energy per token grows dramatically with scale

From TokenPowerBench figures (selected readings):

• For smaller models (e.g., Llama‑3 1B/3B/8B), energy per token at moderate-to-large batch sizes is in the ~0.04–0.25 J/token range depending on model and batch size. [39]
• For frontier-scale models, energy per token is orders of magnitude higher. In a 16‑GPU setting, Llama‑3 405B is reported to be ~163–236 J/token, depending on the workload profile and parallelism configuration. [39]
• The paper also notes that inference configuration (tensor vs. pipeline parallelism) can meaningfully affect energy; the gap between best/worst splits is ~40–60 J/token in their tested scenarios for those large models. [40]

That is a strong, empirical “unsustainable scaling” datapoint: for a high-volume product, the energy budget and cost floor explode as developers move everything onto frontier models.

MoE as “internal ensemble” improves capability per energy

TokenPowerBench explicitly reports that Mixtral‑8×7B consumes roughly the same energy per token as a dense 8B model, while delivering quality closer to that of a much larger dense model. That sparse routing can cut token energy by ~2–3× compared with dense models of similar emergent accuracy. [41]

This aligns with earlier efficiency arguments that sparsely activated models can be far more energy-efficient than dense ones while achieving similar accuracy. [42]

Quantization is a “double dividend” lever

For Llama‑3 405B, TokenPowerBench reports FP8 quantization reduces energy per token by ~30% and can raise throughput, emphasizing that low precision can improve both speed and power when supported by hardware/kernels. [43]

Translating energy into a simple cost bound (electricity only)

To keep this honest: electricity is not the dominant cost of LLM serving (hardware and staffing usually dominate), but it is a real floor and a sustainability metric. Using TokenPowerBench energy-per-token ranges and an electricity assumption of $0.12/kWh:

Even if one disagrees with the exact dollar figure, the ratio is the point: the per-token energy footprint can differ by ~10³–10⁴× across “small vs frontier” regimes in realistic serving setups. [39]

Cost-performance in multi-model systems: cascades can match top quality at far lower cost

A separate line of empirical evidence for ensembles comes from multi-model routing/cascade research rather than single-model benchmarking.

FrugalGPT proposes an LLM cascade strategy and reports that it can match strong single-model performance while achieving very large cost reductions by learning which (cheaper) model(s) suffice for a given query. [45]
This directly supports the claim that ensembles of smaller/cheaper models will dominate—not because they are always better, but because they can handle the majority of requests at low cost and escalate only when needed.

Architectural trade-offs and design patterns for SLM ensembles

The main argument against this thesis is not “small models can’t be good.” It’s: multi-model systems are harder to build and govern. The right response is to show that the architectural trade-offs are manageable and, in many cases, net positive.

Key design patterns that make SLM ensembles viable

Routing and specialization

Routing (choosing among models) is becoming a serious research area because it is the central mechanism by which ensembles beat monoliths on cost. Recent work formalizes routing/cascading strategies and shows they can outperform naive selection if quality estimation is good. [46]

A practical taxonomy for routing policies:

• Rule-based routing (fast start, brittle): e.g., “If structured extraction → extractor SLM; if short summarization → summarizer SLM; if math → math SLM.”
• Learned router micro-model: trained on preference/quality labels to decide whether to answer locally or escalate. [47]
• Agreement-based pruning: multi-agent serving work proposes skipping downstream agent calls if intermediate outputs semantically agree, improving latency while holding accuracy roughly constant. [48]

Distillation and “SLM factory” pipelines

Two complementary distillation ideas matter:

• Classical distillation: DistilBERT and TinyBERT establish that smaller models can retain much of the teacher’s performance while being faster and lighter. [49]
• Modern rationale distillation: “Distilling step-by-step” leverages teacher rationales to train smaller models that can sometimes outperform larger ones on targeted tasks. [50]

Ministral’s Cascade Distillation is the clearest example of distillation as a product strategy: it operationalizes “create a family of compact models from a strong parent,” which is exactly what an SLM-ensemble future needs. [33]

Parameter-efficient fine-tuning as modular updates

LoRA and QLoRA show that large models can adapt with dramatically fewer trainable parameters and reduced memory, making it feasible to maintain specialized variants. [51]

For SLM ensembles, this becomes even more powerful: they can maintain many specialist adapters (per vertical, per geographic region, per training-liance regime) with modest incremental cost, rather than retraining or hosting many giant full copies.

Retrieval-augmented methods are “non-parametric scaling.”

RAG reframes the “knowledge” scaling problem. Instead of stuffing more facts into parameters, it uses an explicit retrieval index with supporting pretraining, improving factual specificity and enabling updates without retraining the whole model. [52]

In an SLM-ensemble worldview, retrieval becomes the cheap “global memory,” while SLMs become lightweight reasoning and formatting engines.

Federated/local updates as a strategic differentiator

Federated learning was designed for privacy-sensitive on-device data, explicitly motivated by cases where centralizing training data is undesirable. [53]
For regulated industries (healthcare, finance), this offers a path to personalization and domain adaptation while keeping sensitive data local—something far harder to justify with centralized frontier models.

Trade-off matrix: monolithic LLM vs SLM ensemble

Deployment, privacy, and security implications

Local hosting and privacy posture

• On-device inference reduces data exposure by eliminating many cloud transfers. Apple[31] explicitly frames on-device processing as a cornerstone of privacy and describes Private Cloud Compute as an approach for handling more complex requests with stringent security constraints. [57]
• Google[35]’s Gemini Nano runs on-device via AICore, emphasizing low latency and keeping models up to date. [58]

SLM ensembles let developers push the default path toward local inference, reserving cloud models for escalation. That is a direct argument for privacy and resilience.

Attack surface: more components, but smaller blast radius

Multi-model systems have more moving parts: routers, retrieval, orchestration logic, caches, indexes, tool connectors, and multiple model weight artifacts. That increases the engineering burden and the number of security-relevant assets.

Two concrete security realities:

1. Prompt injection is real and systematic. OWASP[59] documents prompt injection as a major risk category for LLMs. [60]
2. RAG is not automatically safer. Research shows that retrieval can introduce new vulnerabilities (e.g., corpus poisoning, malicious documents) and that “RAG is not safer” in a straightforward sense. [61]

In ensembles, this means:

• The router and retrieval layer become high-value targets.
• But the per-component blast radius is smaller: a compromise of a single specialist model should not imply total system compromise when isolating privileges and enforcing policy at the orchestrator layer.

Supply-chain risks and update strategies

• NIST[62]’s Secure Software Development Framework (SSDF) provides a baseline for secure development practices and is widely referenced for supply-chain risk management. [63]
• NIST’s Generative AI profile for the AI RMF provides a risk-management lens for generative AI systems, including documentation, evaluation, and governance expectations. [64]

For ensembles, a minimal “secure update strategy” should include:

• Signed model artifacts + verified provenance (treat weights like critical binaries).
• SBOM-like tracking for model, data, and fine-tuning code (not standardized everywhere yet, but SSDF principles apply). [65]
• Canary deployments per component with rollback (more feasible with small specialists).
• Continuous evaluation (see roadmap metrics below) to prevent silent regressions.
• Retrieval data governance: versioned corpora, poisoning detection, and access control because RAG can introduce compromise paths. [66]

Practical governance recommendation

In an SLM ensemble, the orchestrator is the policy enforcement point. If a specialist fails, the system should either abstain, escalate, or return a constrained safe output. This aligns with “deferral” and “selective prediction” ideas: let cheap components handle the easy cases and defer when uncertain. [67]

Roadmap for industry adoption, use cases, limitations, and research gaps

Where SLM ensembles win decisively

Edge and on-device experiences

Edge is the cleanest win: developers can hit low latency without network dependency, support offline modes, and materially reduce data exposure.

• Real-world signals: – Gemini Nano’s on-device throughput metrics and on-device deployment model show that high-performance local inference is viable for consumer UX. [55]
• Phi-3 positions high benchmark performance in a model “small enough to be deployed on a phone.” [30]
• OpenELM explicitly targets efficient on-device use cases and publishes training/inference frameworks. [68]

Concrete SLM ensemble pattern for edge:

• Local SLM for drafting, summarization, and extraction.
• Local policy micro-model for PII detection/redaction.
• Optional cloud escalation for rare complex reasoning (with user consent + audit logging).

Finance

Finance has strict constraints around latency, auditability, and data governance. Ensembles help because:

• “Easy” tasks (classification, extraction, templated summarization, compliance checks) can be handled by specialist SLMs with deterministic formatting.
• Hard tasks are deferred to larger models or human review, and the deferral rate can be measured and tuned (as a governance lever). [69]
• FrugalGPT-style cascades are directly evaluated on a financial-news dataset (HEADLINES) and emphasize that inexpensive models can match strong models on large subsets of queries. [70]

Healthcare

Privacy constraints, heterogeneous workflows, and regulatory scrutiny dominate the healthcare landscape. Ensembles help because developers can:

• Keep PHI local or in tightly controlled enclaves (federated or on-prem). [71]
• Use retrieval and citations for clinician-facing outputs (with strong warnings), rather than relying on a monolithic parametric memory. [72]

Retail and operations

Retail has broad “long tail but shallow depth” tasks: product copy, attribute extraction, customer support triage, and inventory Q&A. These are ideal for:

• Specialist SLMs per product line/region/language.
• Router-based cost control (low-cost default, escalate on ambiguity).

Hardware requirements and orchestration guidance

Hardware sizing heuristics (practical, not magical)

• If a system expects to serve most traffic with ~1B–8B models (quantized), it can often run on:
  • modern edge NPUs/SoCs (for limited contexts),
  • commodity GPUs (single card) for higher throughput,
  • CPU-friendly runtimes for small quantized models.

Evidence points:

• TokenPowerBench explicitly spans 1B–405B and frames smaller models as suitable for a single consumer-grade GPU, whereas frontier models require distributed inference. [73]
• Hugging Face documentation on GGUF/llama.cpp highlights model formats optimized for efficient local inference (fast loading, memory-mapping, and quantization). [74]

For frontier models, requirements can be extreme: TokenPowerBench notes that Llama‑3 405B requires on the order of hundreds of GB of memory in FP16 and uses distribution frameworks to serve it. [73]

Orchestration architecture primitives

A production-grade SLM ensemble typically needs:

• Model registry + versioning (including adapters/LoRA variants). [51]
• Router service (micro-model or heuristic) with explicit “escalation” semantics. [69]
• Retrieval layer with governance and monitoring (because RAG is attackable). [66]
• Tracing: request-level traces that record which models were called, what retrieved documents were used, and which tool calls occurred.
• Evaluation harness integrated into CI/CD, not a one-off benchmark run.

Metrics for success

To validate this thesis in an enterprise setting, metrics that show the ensemble dominates a monolithic model on business constraints are needed:

• Quality metrics: task success rate; factuality/citation accuracy; structured output validity.
• Efficiency metrics:
  • p50/p95 latency & time-to-first-token,
  • energy per token (or proxy like GPU joules/request),
  • tokens/sec, and cost/request. [75]
• Routing metrics:
  • deferral/escalation rate,
  • router error rate (“sent to cheap model but should have escalated”), and 
  • stability of routing across releases. [69]
• Security metrics:
  • prompt-injection and RAG poisoning red-team pass rate,
  • supply-chain verification coverage for model artifacts,
  • mean time to patch/rollback a compromised component. [76]

Concrete prototype architectures

Prototype A: “SLM-first with selective escalation.”

• Router micro-model predicts difficulty + risk category.
• Default: general SLM + retrieval.
• Specialists: policy SLM, domain SLM.
• Escalation: large LLM only if router confidence is low or the task is flagged “high stakes.”

This is essentially a productized version of cascade routing ideas. [77]

Prototype B: “MoE for general capability + specialist SLMs for governance.”

• Use an MoE model (e.g., Mixtral-like) as the main “general intelligence” engine.
• Layer specialist SLMs for compliance, PII redaction, and structured extraction.

This directly leverages the MoE energy/quality advantage demonstrated in TokenPowerBench and MoE papers. [78]

Suggested experiments to validate the claim empirically

A rigorous validation suite should include:

1. A/B cost-quality frontier: Compare (a) one monolithic LLM vs (b) SLM ensemble on the same dataset suite; optimize each system until outputs hit a fixed quality threshold, then compare cost/latency. Use a cascade baseline, such as FrugalGPT, as a starting point. [54]
2. Energy-per-token profiling: replicate TokenPowerBench style measurements (even if approximate) and track energy/request before and after routing/quantization changes. [75]
3. Robustness under attack: run OWASP prompt injection scenarios and RAG-poisoning tests against both monolithic and ensemble designs; measure whether smaller blast radius and orchestrator policy reduce harm. [79]
4. Update velocity experiment: ship a specialist model update weekly and measure regression containment vs a monolithic model update cycle; quantify rollback time and incident scope (good for leadership). Use SSDF-style discipline for change control and provenance. [63]

Limitations, failure modes, and research gaps

• Routing failures are the central failure mode. If the router misclassifies hard queries as easy, quality collapses. Routing/cascade research consistently points to the importance of reliable quality estimators. [69]
• System complexity shifts from “model training” to “systems engineering.” Debugging multi-model interactions, caches, and retrieval is harder than hitting one model endpoint.
• Safety does not automatically improve with RAG; retrieval can introduce new exploit paths, e.g., (poisoning, injection). [61]
• Benchmark mismatch risk: Many “SLM wins” are benchmark-dependent, and small models can be brittle on long-horizon reasoning or adversarial prompts; distillation can amplify teacher biases. Distillation papers themselves warn and analyze failure cases. [80]
• MoE/ensemble training and serving complexity: MoE architectures introduce routing instability and system complexity (communication, load balancing), even if they improve compute efficiency. [81]

Bottom line position

Given the measured inference energy gaps at frontier scale, the compute-optimal findings that penalize naive parameter scaling, and the rapidly improving benchmark competitiveness of modern SLMs, it is technically and economically defensible to argue:

• Scaling monolithic dense LLMs remains useful for frontier capability, but
• It is inefficient as the default solution for most production workloads, and
• The future is multi-model systems where ensembles of SLMs (plus retrieval and tools) handle the majority of requests, with selective escalation to larger models only when a measurable need exists. [82]

References

[1] [2] [14] [17] [45] [54]
S. Chen, A. Zaharia, and M. Zou, 2023. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176.
https://arxiv.org/abs/2305.05176

[3]
OpenAI, 2018. AI and Compute.
https://openai.com/index/ai-and-compute/

[4] [15] [18] [59] [81]
W. Fedus, B. Zoph, and N. Shazeer, 2021. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961.
https://arxiv.org/abs/2101.03961

[5] [23]
Stanford HAI, 2025. AI Index Report 2025.
https://hai.stanford.edu/ai-index/2025-ai-index-report

[6] [31] [82]
J. Hoffmann et al., 2022. Training Compute-Optimal Large Language Models. arXiv:2203.15556.
https://arxiv.org/abs/2203.15556

[7] [10] [28] [38] [39] [40] [41] [43] [44] [56] [75] [78]
Anonymous Authors, 2025. Efficient Language Model Architectures.
https://arxiv.org/pdf/2512.03024

[8] [12] [29] [30]
Anonymous Authors, 2024. Advances in Small and Efficient Language Models. arXiv:2404.14219.
https://arxiv.org/abs/2404.14219

[9] [36] [58]
Google Android Developers, 2025. Gemini Nano: On-Device Generative AI.
https://developer.android.com/ai/gemini-nano

[11] [20]
J. Kaplan et al., 2020. Scaling Laws for Neural Language Models. arXiv:2001.08361.
https://arxiv.org/abs/2001.08361

[13]
ACM Digital Library, 2024. Advances in Efficient Machine Learning Systems.
https://dl.acm.org/doi/full/10.1145/3698767

[16] [19]
Anonymous Authors, 2022. Modular Architectures for Efficient Language Models. arXiv:2210.03629.
https://arxiv.org/abs/2210.03629

[21] [37] [55]
Google Android Developers Blog, 2025. The Latest Gemini Nano with On-Device ML Kit GenAI APIs.
https://android-developers.googleblog.com/2025/08/the-latest-gemini-nano-with-on-device-ml-kit-genai-apis.html

[22]
Epoch AI, 2024. AI Model Dataset and Compute Trends.
https://epoch.ai/data/ai-models

[24]
OpenAI, 2023. AI and Efficiency.
https://openai.com/index/ai-and-efficiency/

[25]
E. Strubell, A. Ganesh, and A. McCallum, 2019. Energy and Policy Considerations for Deep Learning in NLP. ACL 2019.
https://aclanthology.org/P19-1355

[26] [42]
D. Patterson et al., 2021. Carbon Emissions and Large Neural Network Training. arXiv:2104.10350.
https://arxiv.org/abs/2104.10350

[27]
T. Luccioni et al., 2023. Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model. Journal of Machine Learning Research.
https://www.jmlr.org/papers/volume24/23-0069/23-0069.pdf

[32] [68]
Anonymous Authors, 2024. Efficient Foundation Model Architectures. arXiv:2404.14619.
https://arxiv.org/abs/2404.14619

[33] [34]
Anonymous Authors, 2026. Emerging Architectures for Efficient Neural Networks.
https://arxiv.org/pdf/2601.08584

[35] [66]
Y. Zou et al., 2025. PoisonedRAG: Security Vulnerabilities in Retrieval-Augmented Generation Systems. USENIX Security Symposium.
https://www.usenix.org/system/files/usenixsecurity25-zou-poisonedrag.pdf

[46] [47] [69] [77]
D. De Koninck et al., 2024. Cascade Routing: Efficient Multi-Model Inference for Language Models.
https://files.sri.inf.ethz.ch/website/papers/dekoninck2024cascaderouting.pdf

[48]
Anonymous Authors, 2025. Recent Advances in Efficient LLM Inference.
https://arxiv.org/abs/2512.18126

[49] [62]
Anonymous Authors, 2019. Efficient Neural Network Architectures.
https://arxiv.org/abs/1910.01108

[50] [80]
Anonymous Authors, 2023. Scaling and Efficiency in Neural Networks.
https://arxiv.org/abs/2305.02301

[51]
Anonymous Authors, 2021. Efficient Training of Large Neural Networks.
https://arxiv.org/abs/2106.09685

[52] [72]
Anonymous Authors, 2020. Model Compression and Efficient Architectures.
https://arxiv.org/abs/2005.11401

[53] [71]
Anonymous Authors, 2016. Deep Residual Learning for Image Recognition.
https://arxiv.org/abs/1602.05629

[57]
Apple Security Engineering, 2024. Private Cloud Compute: Privacy-Preserving AI Infrastructure.
https://security.apple.com/blog/private-cloud-compute/

[60] [76] [79]
OWASP Foundation, 2024. LLM01: Prompt Injection Vulnerability.
https://genai.owasp.org/llmrisk/llm01-prompt-injection/

[61]
Anonymous Authors, 2025. Security Risks and Mitigation in Large Language Models. NAACL 2025.
https://aclanthology.org/2025.naacl-long.281.pdf

[63] [65]
NIST, 2022. Secure Software Development Framework (SP 800-218).
https://csrc.nist.gov/pubs/sp/800/218/final

[64]
NIST, 2023. Artificial Intelligence Risk Management Framework.
https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

[67]
Anonymous Authors, 2024. Recent Research in Efficient AI Systems.
https://openreview.net/forum?id=sceqRsa0oo

[70]
S. Chen, A. Zaharia, and M. Zou, 2024. FrugalGPT (TMLR version).
https://lingjiaochen.com/papers/2024_FrugalGPT_TMLR.pdf

[73]
Anonymous Authors, 2025. HTML Version of Efficient Language Model Architectures.
https://arxiv.org/html/2512.03024

[74]
Hugging Face, 2024. GGUF Model Format Documentation.
https://huggingface.co/docs/hub/gguf