# GreenServe: Energy-Aware Scheduling of Large Language Model Inference Workloads in Serverless Edge Computing Environments

**Author:** Independent Research Study
**Field:** Information Technology — Distributed Systems, Cloud/Edge Computing, Green AI
**Date:** September 2026
**Paper Type:** Original Research Article

---

## Abstract

The rapid adoption of Large Language Models (LLMs) has shifted the dominant cost of artificial intelligence from training to inference. Industry estimates indicate that inference now accounts for 80–90% of the total lifecycle energy consumption of a deployed model. At the same time, latency-sensitive applications (conversational assistants, real-time translation, code completion, and augmented reality) are pushing inference from centralized hyperscale data centers toward heterogeneous serverless edge nodes. This creates a three-way tension between **latency**, **energy consumption**, and **carbon intensity** that existing serverless schedulers—designed for short, stateless, CPU-bound functions—are not equipped to resolve.

This paper proposes **GreenServe**, an energy-aware scheduling framework for LLM inference in serverless edge environments. GreenServe introduces (i) a *token-level cost model* that predicts energy and latency for a request based on prompt length, expected output length, and model quantization level; (ii) a *carbon-aware placement algorithm* that routes requests across edge, regional, and cloud tiers using real-time grid carbon-intensity signals; and (iii) a *speculative cold-start mitigation strategy* that pre-warms quantized model variants based on short-horizon demand forecasting. We evaluate GreenServe using a discrete-event simulation calibrated with published measurements from NVIDIA Jetson Orin, NVIDIA A10G, and NVIDIA H100 accelerators and real-world carbon-intensity traces from four electricity grids. Across a 24-hour trace derived from the Azure Functions and LMSYS-Chat-1M datasets, GreenServe reduces total energy consumption by **31.4%** and operational carbon emissions by **42.7%** compared to a latency-only baseline, while keeping the 95th-percentile time-to-first-token (TTFT) within the 500 ms service-level objective for 97.8% of requests. We conclude that energy and carbon can be treated as first-class scheduling objectives for LLM inference without materially sacrificing user experience.

**Keywords:** Large Language Models, Serverless Computing, Edge Computing, Green AI, Carbon-Aware Scheduling, Inference Optimization, Sustainable Computing

---

## 1. Introduction

### 1.1 Background and Motivation

Since the release of transformer-based Large Language Models at scale, the information technology industry has undergone a structural transformation in how compute resources are consumed. Early sustainability research focused almost exclusively on the training phase of models, where a single training run of a frontier model can consume several gigawatt-hours of electricity. However, as models are deployed to hundreds of millions of users, the **aggregate energy consumed by inference** has surpassed training by a wide margin. A single LLM query is estimated to consume 10–30× the energy of a traditional web search, and global LLM inference is now projected to draw on the order of tens of terawatt-hours annually.

Simultaneously, three technological trends are converging:

1. **Serverless computing** has become the default deployment model for event-driven and bursty workloads because it abstracts infrastructure management and bills per invocation.
2. **Edge computing** is placing GPU-equipped nodes in metropolitan points of presence, telecom base stations, and enterprise premises to reduce round-trip latency.
3. **Model compression** techniques—4-bit and 8-bit quantization, speculative decoding, and distillation—have made it feasible to run 7B–13B parameter models on edge-class accelerators.

The intersection of these trends—**serverless LLM inference at the edge**—is now a practical deployment target. Yet the schedulers that govern where and when a function runs were designed under assumptions that no longer hold. Traditional serverless functions execute for milliseconds and consume a few hundred megabytes of memory; an LLM inference request may execute for several seconds, requires gigabytes of GPU memory, exhibits highly variable execution time depending on output length, and suffers cold-start penalties measured in tens of seconds when a model must be loaded from storage.

### 1.2 Problem Statement

Existing serverless platforms schedule primarily for **latency** and **resource utilization**. They are blind to two variables that dominate the sustainability profile of LLM inference:

- **Energy per token**, which varies by more than an order of magnitude across accelerator classes and quantization levels.
- **Carbon intensity of the electricity grid**, which varies by a factor of 5–10× between regions and by 2–3× within a single region over the course of a day.

Consequently, a request that could be served on a nearby, solar-powered edge node using a 4-bit quantized model is frequently routed to a distant cloud region powered by a coal-heavy grid simply because the cloud node has a warm container available. The result is unnecessary energy consumption and carbon emissions, with no benefit to the end user.

### 1.3 Research Questions

This study addresses the following research questions:

- **RQ1:** Can the energy and latency of an LLM inference request be predicted accurately enough at admission time to inform scheduling decisions?
- **RQ2:** How much energy and carbon reduction is achievable by incorporating carbon-intensity signals into multi-tier (edge–regional–cloud) placement decisions?
- **RQ3:** Can cold-start penalties—the primary obstacle to serverless LLM deployment—be mitigated through demand-forecast-driven pre-warming without eroding energy savings?
- **RQ4:** What is the trade-off frontier between energy, carbon, and tail latency, and where should operators position themselves on it?

### 1.4 Contributions

The contributions of this paper are:

1. A **token-level energy and latency cost model** for LLM inference that accounts for prompt length, predicted output length, quantization level, and accelerator class (Section 4.1).
2. **GreenServe**, a three-tier carbon-aware placement algorithm formulated as a constrained multi-objective optimization solved with a lightweight heuristic suitable for sub-millisecond scheduling decisions (Section 4.2).
3. A **speculative pre-warming strategy** that uses exponential-smoothing demand forecasting to keep quantized model variants resident on edge nodes (Section 4.3).
4. A **simulation-based evaluation** calibrated against published hardware measurements and real carbon-intensity traces (Sections 5 and 6).
5. An **open discussion** of limitations, threats to validity, and a roadmap for production deployment (Sections 7 and 8).

### 1.5 Paper Organization

Section 2 reviews related work. Section 3 describes the system model and assumptions. Section 4 presents the GreenServe design. Section 5 details the experimental methodology. Section 6 reports results. Section 7 discusses implications and limitations. Section 8 concludes and outlines future work.

---

## 2. Literature Review

### 2.1 Energy Consumption of Machine Learning

Strubell et al. (2019) first quantified the carbon footprint of training NLP models, sparking the "Green AI" movement. Patterson et al. (2021) refined this analysis, showing that choice of hardware, data-center location, and model architecture can change emissions by up to 100–1000×. Luccioni et al. (2023) extended lifecycle analysis to inference, estimating that the BLOOM-176B model emitted approximately 19 kg CO₂ per day during deployment. More recent work by Samsi et al. (2023) benchmarked LLaMA-family models across GPU generations and demonstrated that energy per generated token drops sharply with quantization and batching. These works establish the measurement foundation upon which GreenServe's cost model is built, but none of them propose a scheduling mechanism.

### 2.2 Serverless Computing and Cold Starts

Serverless platforms such as AWS Lambda, Azure Functions, and Google Cloud Functions have been extensively studied. Shahrad et al. (2020) analyzed the Azure Functions production trace and showed that invocation patterns are highly bursty and that a small fraction of functions account for most invocations. They proposed a hybrid histogram policy for keep-alive that reduces cold starts. Wang et al. (2018) reverse-engineered the resource-allocation behavior of major providers. Jonas et al. (2019) articulated the vision and limitations of serverless for stateful and data-intensive workloads—limitations that apply acutely to LLM inference, where model weights constitute multi-gigabyte state.

### 2.3 Serving Large Language Models

Orca (Yu et al., 2022) introduced iteration-level scheduling for transformer inference, enabling continuous batching. vLLM (Kwon et al., 2023) introduced PagedAttention, which manages the key-value cache as virtual memory and dramatically increases throughput. Splitwise (Patel et al., 2024) proposed separating the prefill and decode phases onto different hardware pools. These systems optimize throughput and latency within a single cluster; they do not consider geographic placement or grid carbon intensity.

### 2.4 Carbon-Aware Computing

Radovanović et al. (2022) described Google's carbon-intelligent computing platform, which shifts flexible batch workloads in time to align with low-carbon periods. Wiesner et al. (2021) introduced "temporal workload shifting" and quantified potential savings across grids. Souza et al. (2023) proposed Ecovisor, a virtualized energy system for carbon-aware applications. Hanafy et al. (2023) developed CarbonScaler for elastic batch jobs. These approaches assume workloads are *deferrable* by hours—an assumption that does not hold for interactive LLM inference, where latency budgets are measured in hundreds of milliseconds. GreenServe fills this gap by exploiting **spatial** rather than temporal flexibility.

### 2.5 Edge AI and Model Quantization

Frantar et al. (2023) introduced GPTQ, a post-training quantization method that compresses LLMs to 4 bits with minimal accuracy loss. Lin et al. (2024) proposed AWQ (Activation-aware Weight Quantization). Dettmers et al. (2022) demonstrated LLM.int8() for 8-bit inference. These techniques make edge deployment feasible but introduce a new scheduling dimension: the *choice of quantization level* becomes a lever for trading quality against energy and latency.

### 2.6 Research Gap

To our knowledge, no prior work jointly considers (a) the multi-tier edge–cloud topology, (b) real-time grid carbon intensity, (c) quantization-level selection, and (d) LLM-specific cold-start behavior in a unified scheduling framework for latency-constrained inference. GreenServe addresses this gap.

---

## 3. System Model

### 3.1 Infrastructure Topology

We consider a three-tier infrastructure:

| Tier | Example Hardware | Typical Nodes | Network RTT to User | Grid Region |
|------|------------------|---------------|---------------------|-------------|
| **Edge** | NVIDIA Jetson AGX Orin (64 GB) | 50–500 per metro | 2–10 ms | Local |
| **Regional** | NVIDIA A10G / L4 (24 GB) | 5–20 per region | 15–40 ms | Regional |
| **Cloud** | NVIDIA H100 (80 GB) | Elastic | 40–120 ms | Hyperscale DC |

Each node $$n$$ is characterized by its accelerator class $$c(n)$$, memory capacity $$M_n$$, current set of resident (warm) model variants $$W_n$$, and the carbon intensity $$\kappa_n(t)$$ of its electricity grid at time $$t$$ (in gCO₂e/kWh).

### 3.2 Workload Model

Each inference request $$r$$ arrives with a prompt of $$p_r$$ tokens and a *predicted* output length $$\hat{o}_r$$ tokens. The request has a latency service-level objective (SLO) defined on **time-to-first-token (TTFT)**, denoted $$L^{\max}_r$$, and optionally on **inter-token latency (ITL)**. Requests specify a minimum acceptable quality tier $$q_r \in \{\text{fp16}, \text{int8}, \text{int4}\}$$, where lower precision is permitted only if the application tolerates it.

### 3.3 Model Variants

A base model $$m$$ (e.g., a 8B-parameter instruction-tuned model) is available in three variants $$v \in V_m = \{v_{16}, v_8, v_4\}$$ corresponding to fp16, int8, and int4 quantization. Each variant has a memory footprint $$\mu_v$$ and per-accelerator throughput and power characteristics.

### 3.4 Assumptions

1. Carbon-intensity forecasts are available at 5-minute granularity with ±10% error (consistent with public APIs such as Electricity Maps and WattTime).
2. Model weights are pre-distributed to all tiers via a content-distribution network; cold start cost is dominated by loading weights from local NVMe into GPU memory, not by network download.
3. Requests are independent (no multi-turn KV-cache reuse across requests), a conservative assumption that slightly disadvantages edge nodes.
4. The scheduler is centralized per metro area with a global view of node state, updated every 100 ms.

---

## 4. GreenServe Design

### 4.1 Token-Level Energy and Latency Cost Model

#### 4.1.1 Output-Length Prediction

Output length is unknown at admission time but strongly influences both energy and latency. We train a lightweight gradient-boosted regressor on features extracted from the prompt: token count, presence of instruction keywords ("summarize," "list," "explain in detail"), system-prompt category, and historical mean output length for the requesting application. On the LMSYS-Chat-1M dataset, the model achieves a mean absolute percentage error (MAPE) of 27.3%, and—more importantly for scheduling—correctly classifies requests into short (<64 tokens), medium (64–256), and long (>256) buckets with 81.6% accuracy.

#### 4.1.2 Latency Model

Inference latency decomposes into prefill and decode phases:

$$T_r(n, v) = T_{\text{cold}}(n, v) \cdot \mathbb{1}[v \notin W_n] + \frac{p_r}{\theta^{\text{pre}}_{c(n), v}} + \hat{o}_r \cdot \tau^{\text{dec}}_{c(n), v} + 2 \cdot \text{RTT}_{n}$$

where $$\theta^{\text{pre}}$$ is prefill throughput in tokens/s, $$\tau^{\text{dec}}$$ is per-token decode latency, and $$T_{\text{cold}}$$ is the model load time if the variant is not resident. TTFT is the sum of the cold-start term, the prefill term, and one RTT.

#### 4.1.3 Energy Model

Energy is modeled as the integral of power over execution time, with static and dynamic components:

$$E_r(n, v) = P^{\text{static}}_{c(n)} \cdot T^{\text{exec}}_r + e^{\text{pre}}_{c(n), v} \cdot p_r + e^{\text{dec}}_{c(n), v} \cdot \hat{o}_r + E^{\text{cold}}_{c(n), v} \cdot \mathbb{1}[v \notin W_n]$$

where $$e^{\text{pre}}$$ and $$e^{\text{dec}}$$ are per-token dynamic energy coefficients (J/token) calibrated from hardware measurements. Table 1 summarizes the calibration values used in this study.

**Table 1. Calibrated cost model parameters for an 8B-parameter model (values derived from published benchmarks and vendor specifications; ±15% uncertainty).**

| Accelerator | Variant | Decode ITL (ms/token) | Decode Energy (J/token) | Static Power (W) | Cold Start (s) |
|-------------|---------|----------------------|-------------------------|------------------|----------------|
| Jetson AGX Orin | int4 | 48 | 1.9 | 15 | 11.2 |
| Jetson AGX Orin | int8 | 82 | 3.4 | 15 | 18.7 |
| A10G | int4 | 14 | 1.1 | 40 | 4.1 |
| A10G | int8 | 21 | 1.7 | 40 | 6.8 |
| A10G | fp16 | 36 | 3.0 | 40 | 12.3 |
| H100 | int8 | 6 | 0.9 | 110 | 3.2 |
| H100 | fp16 | 9 | 1.4 | 110 | 5.9 |

Note that while the H100 has the lowest energy *per token* at high utilization, its high static power and location in a hyperscale data center (with associated PUE overhead of ~1.12 and, frequently, higher carbon intensity) can make it the worse choice for short requests in aggregate.

#### 4.1.4 Carbon Model

Operational carbon for a request is:

$$C_r(n, v) = E_r(n, v) \cdot \text{PUE}_n \cdot \kappa_n(t)$$

where PUE (Power Usage Effectiveness) captures cooling and distribution overhead (1.05 for edge micro-DCs with free-air cooling, 1.12 for hyperscale, 1.25 for legacy regional facilities).

### 4.2 Carbon-Aware Multi-Tier Placement

#### 4.2.1 Problem Formulation

For each arriving request $$r$$, the scheduler selects a node $$n$$ and variant $$v$$ to minimize a weighted objective subject to SLO and feasibility constraints:

$$\min_{n, v} \; \alpha \cdot \frac{C_r(n,v)}{C^{\text{ref}}} + \beta \cdot \frac{E_r(n,v)}{E^{\text{ref}}} + \gamma \cdot \frac{T_r(n,v)}{L^{\max}_r}$$

subject to:

$$\text{TTFT}_r(n, v) \leq L^{\max}_r \quad \text{(latency SLO)}$$

$$v \succeq q_r \quad \text{(quality floor)}$$

$$\mu_v + \text{load}_n \leq M_n \quad \text{(memory feasibility)}$$

The weights $$(\alpha, \beta, \gamma)$$ are operator-tunable; $$C^{\text{ref}}$$ and $$E^{\text{ref}}$$ are normalization constants equal to the cost of serving on the cloud tier with fp16.

#### 4.2.2 Heuristic Solution

The candidate space is small (tens of nodes × three variants), so the scheduler evaluates all feasible candidates directly. To avoid herd behavior and queue build-up, we apply **power-of-two-choices** sampling among nodes with equal accelerator class and use a *queue-aware latency estimate* that adds the expected waiting time based on the node's current queue depth and average decode time. The complete decision executes in under 200 microseconds on a single CPU core in our implementation.

#### 4.2.3 Degradation Ladder

If no candidate satisfies the SLO with the requested quality floor, GreenServe walks a degradation ladder: (1) relax to the next lower quantization level if the application permits soft degradation; (2) route to the cloud tier ignoring carbon; (3) admit with an SLO-violation flag for observability. This guarantees that carbon optimization never causes request rejection.

### 4.3 Speculative Pre-Warming

Cold starts of 4–19 seconds are unacceptable for interactive workloads. GreenServe maintains, for each edge node, a **residency plan** updated every 60 seconds:

1. **Forecast demand** per (metro, model, quality-tier) using Holt-Winters exponential smoothing with a 24-hour seasonal period, trained on the trailing 7 days.
2. **Compute expected utility** of keeping variant $$v$$ warm on node $$n$$ as the forecast number of requests it would serve in the next horizon multiplied by the cold-start energy and latency it would avoid, minus the static energy of holding memory.
3. **Solve a knapsack** per node over memory capacity to select the residency set $$W_n$$.
4. **Prefer int4 variants** at the edge because they occupy ~4× less memory than fp16, allowing multiple models to remain resident.

Crucially, pre-warming is *carbon-gated*: when the forecast carbon intensity for the edge region exceeds that of the regional tier by more than a threshold (default 30%), the planner reduces the edge residency set and lets the regional tier absorb demand.

---

## 5. Experimental Methodology

### 5.1 Simulation Framework

We implemented a discrete-event simulator in Python (SimPy) modeling request arrival, scheduling, queueing, execution, and node state. The simulator replays a 24-hour workload trace and applies the cost model of Section 4.1. Each configuration was run with 10 random seeds; we report means and 95% confidence intervals.

### 5.2 Workload Trace

We constructed a synthetic-but-realistic trace by combining:

- **Arrival patterns** from the Azure Functions 2019 trace (Shahrad et al., 2020), scaled to a peak of 1,200 requests/second for a metropolitan area.
- **Prompt and response lengths** sampled from LMSYS-Chat-1M (Zheng et al., 2023), giving a median prompt of 58 tokens and median response of 142 tokens with a heavy tail.
- **Quality-tier distribution**: 60% of requests tolerate int4, 30% require int8 or better, 10% require fp16 (modeling code-generation and medical applications).
- **SLO**: 500 ms TTFT for 90% of requests; 1,000 ms for the remainder.

### 5.3 Infrastructure Configuration

- 200 edge nodes (Jetson AGX Orin) distributed across one metro area.
- 12 regional nodes (A10G) in a facility 300 km away.
- Elastic cloud capacity (H100) in a hyperscale region 1,100 km away, with a 5-node minimum floor.

### 5.4 Carbon-Intensity Traces

We used publicly reported hourly carbon-intensity data for four grid profiles representing distinct characteristics:

| Grid Profile | Mean (gCO₂e/kWh) | Daily Range | Character |
|--------------|------------------|-------------|-----------|
| G1 — Hydro/nuclear-dominant | 45 | 30–70 | Low, flat |
| G2 — High solar penetration | 210 | 80–410 | Strong diurnal swing |
| G3 — Wind-heavy | 240 | 90–520 | Volatile |
| G4 — Coal/gas-dominant | 580 | 480–680 | High, flat |

In the primary experiment, the edge tier is on G2 (solar), the regional tier on G3 (wind), and the cloud tier on G4 (coal/gas), reflecting the common reality that hyperscale regions are sited for land and power cost rather than grid cleanliness. Section 6.5 reports sensitivity to alternative assignments.

### 5.5 Baselines

1. **Latency-Only (LO):** Route to the node with lowest predicted TTFT; always use requested quality tier. Representative of today's serverless platforms.
2. **Cloud-Only (CO):** Route everything to the H100 cloud tier. Representative of centralized deployment.
3. **Edge-First (EF):** Route to the edge whenever a warm variant exists, falling back to regional then cloud. Carbon-blind.
4. **GreenServe-NoPrewarm (GS-NP):** GreenServe placement without speculative pre-warming (ablation).
5. **GreenServe (GS):** Full system with weights $$(\alpha, \beta, \gamma) = (0.5, 0.2, 0.3)$$.

### 5.6 Metrics

- **Total energy** (kWh) and **operational carbon** (kgCO₂e) over 24 hours.
- **p50, p95, p99 TTFT** and **SLO attainment** (fraction of requests meeting TTFT SLO).
- **Cold-start rate** (fraction of requests incurring model load).
- **Quality degradation rate** (fraction of requests served below requested tier).

---

## 6. Results

### 6.1 Energy and Carbon (RQ2)

**Table 2. Primary results over the 24-hour trace (mean ± 95% CI over 10 seeds).**

| Policy | Energy (kWh) | Δ Energy | Carbon (kgCO₂e) | Δ Carbon | p95 TTFT (ms) | SLO Attain. | Cold-Start Rate |
|--------|-------------|----------|-----------------|----------|---------------|-------------|-----------------|
| Cloud-Only (CO) | 1,842 ± 21 | +18.9% | 1,196 ± 14 | +52.3% | 312 | 99.1% | 0.4% |
| Latency-Only (LO) | 1,549 ± 18 | — | 785 ± 11 | — | 288 | 99.3% | 1.1% |
| Edge-First (EF) | 1,118 ± 24 | −27.8% | 531 ± 15 | −32.4% | 604 | 88.2% | 9.7% |
| GS-NoPrewarm | 1,203 ± 22 | −22.3% | 502 ± 13 | −36.1% | 517 | 93.5% | 6.3% |
| **GreenServe (GS)** | **1,063 ± 19** | **−31.4%** | **450 ± 12** | **−42.7%** | **421** | **97.8%** | **1.8%** |

GreenServe reduces energy by 31.4% and carbon by 42.7% relative to the Latency-Only baseline. Carbon savings exceed energy savings because GreenServe deliberately shifts load toward the solar-powered edge tier during daylight hours and toward the wind-powered regional tier at night, exploiting the differential in $$\kappa(t)$$ rather than merely reducing joules.

The Edge-First baseline achieves substantial energy savings but at an unacceptable cost: p95 TTFT balloons to 604 ms and SLO attainment collapses to 88.2%, driven by a 9.7% cold-start rate. This confirms that naive edge placement is not viable without cold-start mitigation.

### 6.2 Latency and SLO Attainment (RQ4)

Figure 1 (described) shows the TTFT cumulative distribution. GreenServe's p50 TTFT (148 ms) is actually *lower* than Latency-Only's (171 ms) because the majority of requests are served on nearby edge nodes with 2–10 ms RTT. The tail is longer—p99 of 890 ms vs. 512 ms—because a small fraction of long-prompt requests routed to edge nodes suffer slow prefill on the Jetson's limited compute. The 2.2% of requests missing SLO are concentrated in the long-prompt (>512 tokens), fp16-required category. We consider this an acceptable trade-off but note that operators with stricter tail requirements can raise $$\gamma$$.

### 6.3 Cold-Start Mitigation (RQ3)

Speculative pre-warming reduces the cold-start rate from 6.3% (GS-NP) to 1.8% (GS), and this reduction *also* reduces energy consumption by 11.6% relative to GS-NP, because each avoided cold start saves 15–25 kJ of model-loading energy. The pre-warming overhead—static power of holding models resident—amounts to 4.1% of total energy, comfortably below the savings.

The forecast model achieved a 24-hour-ahead MAPE of 14.2% on aggregate request rate per metro, sufficient for residency planning. Errors were concentrated at the sharp morning ramp (07:00–09:00 local), where a 15-minute lag in adaptation produced a transient cold-start spike to 5.8% before recovering.

### 6.4 Cost-Model Accuracy (RQ1)

Comparing predicted to simulated actual values (which incorporate the true rather than predicted output length):

| Quantity | MAPE | Bucket Accuracy |
|----------|------|-----------------|
| Output length | 27.3% | 81.6% (3 buckets) |
| Per-request energy | 22.8% | — |
| TTFT | 9.4% | — |

TTFT is predicted accurately because it depends only on prompt length (known) and cold-start state (known). Energy prediction inherits output-length error. Importantly, scheduling quality is robust to this error: an oracle variant of GreenServe with perfect output-length knowledge improved carbon savings by only 2.1 percentage points (44.8% vs. 42.7%), indicating that bucket-level accuracy is sufficient.

### 6.5 Sensitivity Analysis

**Grid assignment.** When all three tiers are placed on the same grid (G2), carbon savings fall to 27.9%—still substantial, now driven purely by energy reduction and diurnal shifting between edge residency and regional capacity. When the cloud tier is on the cleanest grid (G1) and the edge on the dirtiest (G4), GreenServe correctly *reverses* its preference, routing 71% of traffic to the cloud, and still achieves 8.3% carbon savings over Latency-Only by choosing lower quantization where permitted.

**Weight sensitivity.** Sweeping $$\alpha$$ from 0 to 0.8 (holding $$\gamma$$ at 0.3) traces a smooth Pareto frontier: carbon savings rise from 19% to 46% while SLO attainment falls from 99.0% to 95.1%. The chosen operating point (0.5) sits at the knee of this curve.

**Quality tolerance.** If only 30% of requests (rather than 60%) tolerate int4, energy savings fall to 23.7% and carbon savings to 35.2%, confirming that quantization tolerance is a major lever.

### 6.6 Scheduler Overhead

Placement decisions averaged 187 μs (p99: 412 μs) on a single core. Residency-planning knapsack solves took 8–14 ms per node per 60-second cycle. Overheads are negligible relative to inference times of 0.5–10 seconds.

---

## 7. Discussion

### 7.1 Implications for Practitioners

The results suggest three actionable guidelines for organizations deploying LLM inference:

1. **Treat quantization tolerance as an API-level contract.** The single largest lever in our study was the fraction of requests permitted to use int4. Applications should declare their quality floor explicitly rather than defaulting to full precision.
2. **Site edge inference for grid cleanliness, not just proximity.** Edge nodes co-located with rooftop solar or in regions with high renewable penetration deliver disproportionate carbon savings for daytime workloads.
3. **Invest in cold-start mitigation before scaling out the edge.** Edge-First without pre-warming was worse on user experience than staying in the cloud; the two must be deployed together.

### 7.2 Embodied Carbon

This study considers operational carbon only. Deploying 200 Jetson-class edge nodes carries embodied emissions from manufacturing (estimated at 100–200 kgCO₂e per unit). At the carbon-savings rate observed (335 kgCO₂e/day for the metro), embodied carbon of the edge fleet is amortized in approximately 60–120 days, after which savings are net positive. A full lifecycle analysis is left to future work.

### 7.3 Threats to Validity

- **Simulation vs. reality.** Our cost model is calibrated from published benchmarks, not from measurements on a deployed testbed. Real-world thermal throttling, memory fragmentation, and network jitter may reduce achievable savings.
- **Workload realism.** The trace combines two datasets from different domains; real LLM traffic may exhibit different burstiness or length distributions.
- **Carbon-intensity data.** Marginal versus average emissions factors remain a debated methodological choice. We used average intensity; using marginal factors could change the magnitude (but likely not the direction) of results.
- **Single-metro scope.** We modeled one metropolitan area. Multi-metro interactions, cross-region failover, and data-residency regulations add complexity.

### 7.4 Ethical and Societal Considerations

Carbon-aware routing could inadvertently shift computational load—and its local externalities such as heat, noise, and grid strain—onto communities hosting edge infrastructure. Operators should pair carbon optimization with transparency and community engagement. Additionally, serving lower-precision models to users who cannot express a preference raises fairness questions if quality degradation is unevenly distributed; the quality-floor mechanism in GreenServe is designed to give applications explicit control, but defaults matter.

---

## 8. Conclusion and Future Work

This paper presented GreenServe, a scheduling framework that treats energy and carbon as first-class objectives for LLM inference in serverless edge environments. By combining a token-level cost model, carbon-aware multi-tier placement, and speculative pre-warming of quantized model variants, GreenServe reduced energy by 31.4% and operational carbon by 42.7% in simulation while maintaining 97.8% SLO attainment. These results demonstrate that the sustainability of LLM inference is not solely a hardware or model-architecture problem—it is a **systems and scheduling problem** that can be addressed with existing infrastructure.

Future work will proceed along four directions:

1. **Testbed validation** on physical Jetson and A10G hardware with power meters to close the simulation-to-reality gap.
2. **Multi-turn KV-cache locality**, where routing consecutive turns of a conversation to the same node avoids recomputing the prefix and further reduces energy.
3. **Learned scheduling policies** using reinforcement learning to replace hand-tuned weights and adapt to non-stationary workloads.
4. **Integration with speculative decoding**, where a small draft model at the edge and a large verifier in the cloud form a natural two-tier pipeline whose energy profile deserves independent study.

As LLM inference becomes a utility-scale consumer of electricity, the responsibility to serve it efficiently falls on the systems community. We hope GreenServe provides both a practical blueprint and a call to treat every token as a unit of energy worth optimizing.

---

## References

1. Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. *Advances in Neural Information Processing Systems (NeurIPS)*.
2. Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. *International Conference on Learning Representations (ICLR)*.
3. Hanafy, W. A., Liang, Q., Bashir, N., Irwin, D., & Shenoy, P. (2023). CarbonScaler: Leveraging Cloud Workload Elasticity for Optimizing Carbon-Efficiency. *Proceedings of the ACM on Measurement and Analysis of Computing Systems*, 7(3).
4. Jonas, E., Schleier-Smith, J., Sreekanti, V., et al. (2019). Cloud Programming Simplified: A Berkeley View on Serverless Computing. *arXiv:1902.03383*.
5. Kwon, W., Li, Z., Zhuang, S., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. *Proceedings of the 29th Symposium on Operating Systems Principles (SOSP)*.
6. Lin, J., Tang, J., Tang, H., et al. (2024). AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. *Proceedings of Machine Learning and Systems (MLSys)*.
7. Luccioni, A. S., Viguier, S., & Ligozat, A.-L. (2023). Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model. *Journal of Machine Learning Research*, 24.
8. Patel, P., Choukse, E., Zhang, C., et al. (2024). Splitwise: Efficient Generative LLM Inference Using Phase Splitting. *International Symposium on Computer Architecture (ISCA)*.
9. Patterson, D., Gonzalez, J., Le, Q., et al. (2021). Carbon Emissions and Large Neural Network Training. *arXiv:2104.10350*.
10. Radovanović, A., Koningstein, R., Schneider, I., et al. (2022). Carbon-Aware Computing for Datacenters. *IEEE Transactions on Power Systems*, 38(2).
11. Samsi, S., Zhao, D., McDonald, J., et al. (2023). From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference. *IEEE High Performance Extreme Computing Conference (HPEC)*.
12. Shahrad, M., Fonseca, R., Goiri, Í., et al. (2020). Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider. *USENIX Annual Technical Conference (ATC)*.
13. Souza, A., Bashir, N., Murillo, J., et al. (2023). Ecovisor: A Virtual Energy System for Carbon-Efficient Applications. *ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)*.
14. Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and Policy Considerations for Deep Learning in NLP. *Annual Meeting of the Association for Computational Linguistics (ACL)*.
15. Wang, L., Li, M., Zhang, Y., Ristenpart, T., & Swift, M. (2018). Peeking Behind the Curtains of Serverless Platforms. *USENIX Annual Technical Conference (ATC)*.
16. Wiesner, P., Behnke, I., Scheinert, D., Gontarska, K., & Thamsen, L. (2021). Let's Wait Awhile: How Temporal Workload Shifting Can Reduce Carbon Emissions in the Cloud. *ACM/IFIP International Middleware Conference*.
17. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., & Chun, B.-G. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. *USENIX Symposium on Operating Systems Design and Implementation (OSDI)*.
18. Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset. *arXiv:2309.11998*.

---

## Appendix A. Notation Summary

| Symbol | Meaning |
|--------|---------|
| $$r$$ | An inference request |
| $$p_r, \hat{o}_r$$ | Prompt length and predicted output length (tokens) |
| $$L^{\max}_r$$ | TTFT latency SLO for request $$r$$ |
| $$q_r$$ | Minimum acceptable quantization tier |
| $$n, c(n)$$ | Node and its accelerator class |
| $$W_n$$ | Set of model variants warm (resident) on node $$n$$ |
| $$v \in \{v_{16}, v_8, v_4\}$$ | Model variant (fp16, int8, int4) |
| $$\kappa_n(t)$$ | Grid carbon intensity at node $$n$$ at time $$t$$ (gCO₂e/kWh) |
| $$E_r, C_r, T_r$$ | Energy, carbon, and latency of serving $$r$$ |
| $$\alpha, \beta, \gamma$$ | Objective weights for carbon, energy, latency |

## Appendix B. Reproducibility Checklist

- Simulator: SimPy 4.x, Python 3.11; 10 seeds per configuration.
- Cost-model parameters: Table 1; uncertainty ±15%.
- Workload: Azure Functions 2019 trace (arrivals) × LMSYS-Chat-1M (lengths), scaled to 1,200 req/s peak.
- Carbon traces: hourly average intensity for four representative grid profiles (Section 5.4).
- Forecasting: Holt-Winters additive seasonal, period = 24 h, trained on trailing 7 days.
- Placement weights: $$(\alpha, \beta, \gamma) = (0.5, 0.2, 0.3)$$ unless stated otherwise.