AI Compute & ServerlessNebius vs Modal vs AWS Lambda

Best answers, written after the sections were taught. Try each one before opening it.

1. Q1

   You have def add(a, b): return a + b and are told: “run this function in the cloud.” What does the cloud actually have to do between add(2, 3) and 5? No “serverless handles it” unless you can say what that means underneath.

   ▸Reveal best answerHide answerfundamentals

   Somewhere a physical machine with a CPU and memory must exist. Your code has to get onto it, a Python interpreter has to be available there, a process has to start that runs the interpreter, the CPU executes the addition, and the result has to travel back over the network to you.

   • find capacity → place code + runtime there → start a process → execute → return the result → free or reuse the capacity
   • VMs, containers, schedulers, HTTP APIs are how a provider does this safely for millions of users — not the essence of it
2. Q2

   A: you rent a machine for 24 hours, your program finishes in 10 seconds, the machine keeps running for 23h 59m 50s. B: you hand over the program, call it, and compute exists only for the duration of the execution. What is the fundamental difference?

   ▸Reveal best answerHide answerintuition

   Who owns the lifecycle of the compute, and what the unit of purchase is. In A you buy machine-time: the machine (and the bill) exists whether or not anything runs, and you decide what runs on it. In B you buy execution: the provider allocates compute for the work and takes it back afterwards. A is infrastructure; B is a service that happens to need infrastructure.

3. Q3

   resize_image(image) is deployed as an AWS Lambda function and a user uploads an image. Fill in the two ??? in Upload → ??? → Python function → ??? → Result. Who creates the machine that executes your code?

   ▸Reveal best answerHide answerlambda

   • Upload lands in storage (S3) → the storage service emits an event → the Lambda service receives it
   • Lambda finds or creates an execution environment with your code and a Python runtime, then invokes your handler with the event
   • your function runs → the result is written somewhere / returned → the environment is kept warm for a while or shut down

   The machine is created by AWS’s Lambda service on its own fleet. You never asked for one, and you cannot see it.

4. Q4

   train_model() takes 9 hours. Why might Lambda be the wrong abstraction — beyond “Lambda has a time limit”?

   ▸Reveal best answerHide answerintuition

   Lambda’s model is many short, independent, stateless invocations in environments the platform is free to spin up, reuse, or tear down. It assumes work can be interrupted and redistributed. A 9-hour training run is one long, stateful process that needs the same resources (likely GPUs, large memory, checkpoints) continuously.

   The 15-minute cap is a symptom of that design, not the reason.

5. Q5

   model = load_llama_70b() then generate(prompt). Why can’t this go in a normal Lambda function? Name the underlying resource problem.

   ▸Reveal best answerHide answerresources

   A 70B model in 16-bit needs roughly 140 GB of weights sitting in GPU memory plus an accelerator to run at usable speed. Lambda environments are CPU-only, capped at about 10 GB of memory, and short-lived. Even if the weights fit, loading tens of gigabytes on every cold start would dwarf the actual work.

   The workload needs a GPU and persistent hot state. Lambda offers neither.

6. Q6

   Given this Dockerfile, what problem does the container solve? Does a container itself provide a CPU or GPU?

   FROM python:3.12
   RUN pip install torch transformers
   COPY app.py .
   CMD ["python", "app.py"]

   ▸Reveal best answerHide answercontainers

   It packages your code with its userspace dependencies (Python version, torch, transformers, system libraries) so the same thing runs the same way anywhere a container runtime exists. It solves “works on my machine”.

   A container provides no hardware. It is a way of packaging and isolating processes; CPU and GPU come from the host it runs on.

7. Q7

   A GPU machine runs Linux, which runs Docker containers A, B, C. Who is actually providing the GPU — the VM, Docker, Linux, the cloud provider, something else?

   ▸Reveal best answerHide answerlayers

   The GPU is physical silicon in the machine, owned and supplied by the cloud provider. The hypervisor passes it (or a slice of it) into the VM; Linux’s driver exposes it as a device; Docker, with the NVIDIA toolkit, grants a container access to that device.

   Nobody inside the container provides a GPU. Each layer only passes access downward.

8. Q8

   What happens between calling run_model.remote() and your code executing on an H100?

   @app.function(gpu="H100")
   def run_model():
       ...

   ▸Reveal best answerHide answermodal

   • your client sends the function’s identity and serialized inputs to Modal’s control plane
   • Modal finds (or provisions) a container built from your image on a machine with a free H100 — or reuses a warm one
   • it starts / reuses your function’s process, runs it, streams the result back, and keeps the container warm briefly in case more calls arrive

   The decorator is a requirement declaration. Turning it into a machine is Modal’s job, not yours.

9. Q9

   You obtain an H100 VM from Nebius. Is “Nebius gives you an H100 → you SSH in → install CUDA → install PyTorch → run model” roughly right, or is Nebius doing something more abstract?

   ▸Reveal best answerHide answernebius

   For the VM product, roughly right: you get a machine with an H100 attached (often from an image with drivers and CUDA pre-installed), you SSH in, build your environment, and run. Nebius also offers higher-level building blocks — managed Kubernetes, GPU clusters, storage — but the mental unit is still infrastructure you operate, not functions you invoke.

10. Q10

    Traffic is 100 requests/hour but extremely bursty (see the chart above). Would you rather A keep GPUs running continuously, or B allocate compute when requests arrive and release it when demand disappears? And: what new problem does B create that A doesn’t?

    ▸Reveal best answerHide answerthe important one

    B is cheaper for bursty traffic — A pays for GPUs that idle most of the day. But B creates a problem A never had: latency and capacity at the spike. When demand appears, compute must be found, environments created, weights loaded — cold starts — and if everyone spikes at once the provider may not have enough free GPUs.

    A trades money for predictability; B trades predictability for money. Real systems hedge: keep a minimum warm, scale the rest.

11. Q11

    You own one H100. A customer needs it for 10 seconds, another for 20 seconds, then nobody for 5 minutes, then 50 customers at once. Describe the actual engineering problem without the words serverless, autoscaling, orchestration, Kubernetes, Modal, or Nebius.

    ▸Reveal best answerHide answerno jargon allowed

    Time-sharing a scarce, expensive, indivisible thing across unpredictable demand — with switching costs.

    • decide who gets it right now, and how quickly the next tenant can be switched in (clean up the last one, load the next one’s environment and weights)
    • decide what to do while it idles — hold it hot for the likely next caller, or let someone else use it
    • decide what happens when 50 arrive and there is one GPU: queue, refuse, or find more hardware elsewhere
    • and never let one customer see another’s data or work

Questions reconstructed from the review of my answers; the corrections are verbatim.

1. Q1

   In your own words: program, process, container, virtual machine, physical machine. What is each one, really?

   ▸Reveal best answerHide answer

   • Program — instructions sitting somewhere (a file, a bundle)
   • Process — those instructions currently executing, with their own memory
   • Container — an isolated userspace environment in which processes run, sharing the host’s kernel. Not “a machine with dependencies”
   • Virtual machine — a virtual computer with virtualized hardware, managed by a hypervisor
   • Physical machine — the actual hardware
2. Q2

   A physical machine has many CPUs and 8 GPUs. VM A is given 8 CPUs, 64 GB and 1 GPU. What is doing the giving, and what is true of the resources VM A did not get?

   ▸Reveal best answerHide answerintuition

   The hypervisor maintains the mapping between virtual resources and physical resources. The rest of the machine is allocatable to other VMs — but not automatically “free” in the economic sense: it may be reserved, fragmented into unusable slices, or simply idle and unsold.

   Where I slipped

   I said “the other resources are free”. Allocatable, not free.

3. Q3

   Does a container “have” a GPU?

   ▸Reveal best answerHide answerintuition

   No. A container can be given access to a GPU; the GPU stays in the machine. The chain is Physical H100 → Host / VM → Container → PyTorch → your model, and each layer only passes access along.

   Follow-up that leads to scheduling: if ten containers want the same H100, who decides which one gets it?

4. Q4

   You delete the container. What happens to the Python process inside it? To the VM it ran on?

   ▸Reveal best answerHide answer

   The process disappears — it only ever existed inside the container’s isolation boundary. The VM stays exactly as it was: Physical machine → VM, with the container layer gone.

5. Q5

   What does it minimally take to run a Python program?

   ▸Reveal best answerHide answerfundamental vs implementation

   Hardware that executes instructions, working memory, something that interprets Python (the runtime), and the code itself. That is all. On your laptop it is machine → macOS → Python → your process.

   A VM and a container are implementation and abstraction choices a provider makes to serve many people safely. They are not requirements for execution.

   Where I slipped

   I listed “VM” and “container” as requirements. They are how clouds do it, not what Python needs.

6. Q6

   Given the stack, what is Modal “hiding” when you write a decorated function and call it remotely?

   ▸Reveal best answerHide answerprediction

   The whole Container → VM → Physical machine band — and more precisely the decisions about where and when those things exist. “Hiding” turns out to mean: taking over provisioning and lifecycle. That is exactly the boundary Section 1.5 defines.

1. Q1

   Explain in your own words: “serverless doesn’t mean there are no servers.” What does serverless mean from the developer’s side?

   ▸Reveal best answerHide answer

   There are servers — lots of them. You just never manage or think about them as individual machines. Serverless means the provider manages the allocation and lifecycle of the underlying compute, while you describe the workload you want executed. What moved is the abstraction boundary.

2. Q2

   A: “Give me an H100 VM.” B: “Run this Python function using an H100.” What does the provider have to manage in B that you manage in A? List as much as you can.

   ▸Reveal best answerHide answer

   • finding capacity with a free H100, in some region
   • creating the execution environment: OS, drivers, CUDA compatibility, runtime, your dependencies
   • scheduling your function onto that GPU, and onto more GPUs as calls multiply
   • lifecycle: start, keep warm, tear down; recovering from failures
   • networking so the call reaches the function; isolation from other tenants; metering per execution

   In A everything from the OS upward is yours. The invariant is what matters — “whatever infrastructure is needed to execute your workload” — not one specific stack.

3. Q3

   def hello(): return "hello" runs through Lambda in 5 ms. Does AWS need a CPU? RAM? An operating system? A Python runtime? A server? Yes/no with a reason for each.

   ▸Reveal best answerHide answerfundamental vs implementation

   • CPU — yes. Instructions must execute somewhere
   • RAM — yes. The process has state, however small
   • OS / execution environment — yes. Something must start the process, give it memory, and isolate it
   • Python runtime — yes. Someone has to interpret the code
   • A server — yes. A physical machine exists; you just don’t see it

   Not fundamentally needed: a VM, a container, persistent storage. Those are how AWS chooses to provide the above.

   Where I slipped

   My yeses were right, but I justified them with “the program has to be loaded into a VM inside a container”. That bakes an implementation into the definition.

4. Q4

   Why is “serverless = pay only while the function runs” an incomplete definition? What is the deeper abstraction?

   ▸Reveal best answerHide answer

   Because the pricing is a consequence of the abstraction, not the abstraction itself. The deeper idea: you ask the provider to execute your workload, and the provider decides how to provision the infrastructure to do it. You never asked for a machine — so you are not renting one.

   Where I slipped

   I said “we would also need to pay for the VM used to run the function.” That is the exact thing FaaS hides. You pay for the meal, not for the restaurant’s kitchen.

5. Q5

   generate() (which loads a model) is called at 10:00 and again at 10:01. Should Modal destroy everything after the first call and rebuild at 10:01? What would a good system do?

   ▸Reveal best answerHide answerprediction

   No. Destroying means repeating the expensive initialization — environment, runtime, dependencies, weights — for a call that arrives a minute later. A good system keeps the environment idle for a while, reuses it at 10:01, and tears it down only after an idle timeout.

   The tradeoff it is managing: idle resource cost vs re-initialization cost.

6. Q6

   1,000 users call generate() simultaneously, each needing an H100. The provider has 200. What problem does it have to solve? Don’t say “autoscaling”.

   ▸Reveal best answerHide answerintuition

   Resource allocation under constraints. There are 1,000 things to run and 200 places to run them; the scheduler must continuously map workload → resource while respecting GPU type, memory, region, latency, fairness, cost and capacity — and decide what the other 800 do: wait, get rejected, or trigger new capacity.

   Prioritization and latency are inputs to that decision, not the whole problem.

   Where I slipped

   I described only “prioritize requests and serve in least waiting time”. That is half of it — the placement problem is the core.

1. Q1

   AWS receives an invocation and no environment exists. What sequence of events has to happen before your function can return “hello”?

   ▸Reveal best answerHide answer

   request → no environment → find capacity → create / start an environment → initialize the runtime → load your code → invoke → return the result.

   “Find a suitable VM” is an implementation detail; the abstraction to keep is request → execution environment → runtime → your code.

2. Q2

   Explain the difference between a cold start and a warm invocation without using the words “cold” or “warm”.

   ▸Reveal best answerHide answerintuition

   In one case the platform must prepare an environment — capacity, runtime, code, initialization — before the function can run, so all that work sits on the request’s critical path. In the other, a prepared idle environment already exists and the request goes straight to invocation. The difference is whether preparation is on the critical path.

3. Q3

   Model 20 GB, GPU H100, function execution 100 ms. Why is serverless execution problematic if the model is loaded from scratch on every invocation? Reason about relative costs.

   ▸Reveal best answerHide answerintuition

   Loading 20 GB — disk or network I/O plus the copy into GPU memory — takes seconds to tens of seconds; the actual work takes 100 ms. Initialization is 100–1000× the useful work, so latency is dominated by preparation and the GPU spends most of its time waiting rather than computing.

   The loaded model in GPU memory is the valuable state. The whole design question becomes: how do we keep it, and what does keeping it cost while idle?

   Where I slipped

   I said “we would need to keep the server state and that adds latency.” It is the reverse: losing the state is what adds latency.

4. Q4

   generate.remote() is called 100 times in 10 seconds. Reusing one environment (Strategy B) is obviously attractive — what problems can it create? Think users, state, isolation, concurrency, memory, failures.

   ▸Reveal best answerHide answer

   • state leakage — globals, caches, temp files from one request visible to the next
   • isolation — different users or tenants sharing a process
   • concurrency — one process serving several requests at once: thread safety, GPU memory contention
   • memory growth and leaks over a long-lived process; a crash corrupting an environment many requests depend on
   • stale code after a deploy; idle cost while it waits

   Reuse does not require requests to be similar or grouped — Alice, Bob and Charlie can all hit the same environment one after another.

   Where I slipped

   I said the platform “would need to group similar requests into one batch”. Reuse, concurrency and batching are three separate ideas; batching is an optional optimization.

5. Q5

   “The platform keeps an environment alive after your function finishes.” Does that mean your function is still running?

   ▸Reveal best answerHide answerintuition

   No. The environment exists — runtime, libraries, code, model weights all loaded — but nothing is executing. It sits idle until a request arrives, runs the function, returns, and goes idle again. Environment exists ≠ process is executing.

6. Q6

   One request becomes 1,000 simultaneous requests. Does Lambda (A) make one Python process handle all 1,000, (B) create / reuse multiple execution environments, (C) create one giant VM, or (D) something else?

   ▸Reveal best answerHide answerprediction

   B. The system scales out: more execution environments, possibly with some concurrency inside each, fed by a scheduler and queues. Not one process doing everything; not one giant machine.

   Scaling is the fundamental resource-allocation response. Batching is a separate optimization that may or may not be layered on.

   Where I slipped

   I said it “would batch them”. Scaling adds environments; batching merges requests. Different levers.