Sandbox bench: cloud sandboxes for agents, measured

Where the time goes

Step by step, from median timings. Colour is the provider's time; grey is ours.

TraceCold start · median of each stepE2B

Total time

1.44s

559ms provider · 882ms ours

Cost per task

$0.00049

create, replay, delete

Calls

3

median of 100 starts

Failures

0

of 100 attempts

Span

0s0.5s1s1.5s

Cold startcreate → ready
1.44s
create()provider API
449ms
Bench bookkeepingnot counted
104ms
Ownership checkours · excluded
101ms
Journal writeours · excluded
2.9ms
First commandecho ok
110ms
SDK setupSandbox SDK
778msSLOWEST

E2B's timeOurs: Sandbox SDKOurs, left out of every chart (shown here)Times are medians

TraceCold start · median of each stepE2B

Total time

1.92s

906ms provider · 1.01s ours

Cost per task

$0.0016

create, replay, delete

Calls

3

median of 100 starts

Failures

0

of 100 attempts

Span

0s0.5s1s1.5s2s

Cold startcreate → ready
1.92s
create()provider API
737ms
Bench bookkeepingnot counted
113ms
Ownership checkours · excluded
111ms
Journal writeours · excluded
2.9ms
First commandecho ok
169ms
SDK setupSandbox SDK
899msSLOWEST

E2B's timeOurs: Sandbox SDKOurs, left out of every chart (shown here)Times are medians

Start-up

Cold start is the wait before an agent can do anything: from create() until the first command runs, the provider's own time. Ready through Sandbox SDK adds the setup our SDK does after that, marked as ours.

Cold start

create() to first command, without our SDK's setup · Median seconds · Lower is better

Superservemedian 0.51 sp95 3.4 s95% CI 0.50 s–0.53 s100 attemptsClick for where the time goes

E2Bmedian 0.56 sp95 3.2 s95% CI 0.55 s–0.57 s100 attemptsClick for where the time goes

Freestylemedian 0.63 sp95 1.4 s95% CI 0.61 s–0.64 s100 attemptsClick for where the time goes

Tenkimedian 0.87 sp95 4.1 s95% CI 0.85 s–0.89 s100 attemptsClick for where the time goes

boxdmedian 1.4 sp95 4.2 s95% CI 1.4 s–1.4 s100 attemptsClick for where the time goes

Daytonamedian 1.6 sp95 5.8 s95% CI 1.4 s–1.8 s100 attemptsClick for where the time goes

Upstash Boxmedian 3.0 sp95 23 s95% CI 2.7 s–4.2 s100 attemptsClick for where the time goes

0.51

Superservemedian 0.51 sp95 3.4 s95% CI 0.50 s–0.53 s100 attemptsClick for where the time goes

0.56

E2Bmedian 0.56 sp95 3.2 s95% CI 0.55 s–0.57 s100 attemptsClick for where the time goes

0.63

Freestylemedian 0.63 sp95 1.4 s95% CI 0.61 s–0.64 s100 attemptsClick for where the time goes

0.87

Tenkimedian 0.87 sp95 4.1 s95% CI 0.85 s–0.89 s100 attemptsClick for where the time goes

1.4

boxdmedian 1.4 sp95 4.2 s95% CI 1.4 s–1.4 s100 attemptsClick for where the time goes

1.6

Daytonamedian 1.6 sp95 5.8 s95% CI 1.4 s–1.8 s100 attemptsClick for where the time goes

3.0

Upstash Boxmedian 3.0 sp95 23 s95% CI 2.7 s–4.2 s100 attemptsClick for where the time goes

Notes (2)

1E2B: ran E2B's default base template, 2 vCPU and 512 MB. E2B's pricing page says its default sandbox has 4 GB.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.

Ready through Sandbox SDK

create() until our SDK's handle is ready · Median seconds of each part · Lower is better

Provider cold start, the number on the Cold start tabOur SDK's extra setup, not the provider's

Superserveprovider cold start 0.51 sour SDK setup +0.81 sready through the SDK, median 1.3 s100 cold startsClick for where the time goes

E2Bprovider cold start 0.56 sour SDK setup +0.78 sready through the SDK, median 1.3 s100 cold startsClick for where the time goes

Freestyleprovider cold start 0.63 sour SDK setup +0.97 sready through the SDK, median 1.6 s100 cold startsClick for where the time goes

Tenkiprovider cold start 0.87 sour SDK setup +1.0 sready through the SDK, median 2.0 s100 cold startsClick for where the time goes

boxdprovider cold start 1.4 sour SDK setup +1.8 sready through the SDK, median 3.2 s100 cold startsClick for where the time goes

Daytonaprovider cold start 1.6 sour SDK setup +2.6 sready through the SDK, median 5.8 s100 cold startsClick for where the time goes

Upstash Boxprovider cold start 3.0 sour SDK setup +7.3 sready through the SDK, median 12 s100 cold startsClick for where the time goes

1.3

Superserveprovider cold start 0.51 sour SDK setup +0.81 sready through the SDK, median 1.3 s100 cold startsClick for where the time goes

1.3

E2Bprovider cold start 0.56 sour SDK setup +0.78 sready through the SDK, median 1.3 s100 cold startsClick for where the time goes

1.6

Freestyleprovider cold start 0.63 sour SDK setup +0.97 sready through the SDK, median 1.6 s100 cold startsClick for where the time goes

1.9

Tenkiprovider cold start 0.87 sour SDK setup +1.0 sready through the SDK, median 2.0 s100 cold startsClick for where the time goes

3.2

boxdprovider cold start 1.4 sour SDK setup +1.8 sready through the SDK, median 3.2 s100 cold startsClick for where the time goes

4.2

Daytonaprovider cold start 1.6 sour SDK setup +2.6 sready through the SDK, median 5.8 s100 cold startsClick for where the time goes

10

Upstash Boxprovider cold start 3.0 sour SDK setup +7.3 sready through the SDK, median 12 s100 cold startsClick for where the time goes

Notes (2)

The hatched part is work Sandbox SDK does after the sandbox can run commands, such as preparing the workspace. It is ours, so it is kept out of cold start. Each part is its own median, so the stack can differ a little from the median total shown on hover.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.

Cold start

create() to first command, without our SDK's setup · Median seconds · Lower is better

Freestylemedian 0.66 sp95 2.8 s95% CI 0.64 s–0.68 s100 attemptsClick for where the time goes

E2Bmedian 0.95 sp95 3.1 s95% CI 0.92 s–1.0 s100 attemptsClick for where the time goes

Tenkimedian 0.96 sp95 3.0 s95% CI 0.91 s–1.1 s78 attemptsClick for where the time goes

Daytonamedian 1.1 sp95 3.0 s95% CI 1.0 s–1.3 s100 attemptsClick for where the time goes

boxdmedian 1.4 sp95 3.9 s95% CI 1.4 s–1.4 s100 attemptsClick for where the time goes

Upstash Boxmedian 2.7 sp95 9.1 s95% CI 2.6 s–2.7 s100 attempts, 1 failedClick for where the time goes

0.66

Freestylemedian 0.66 sp95 2.8 s95% CI 0.64 s–0.68 s100 attemptsClick for where the time goes

0.95

E2Bmedian 0.95 sp95 3.1 s95% CI 0.92 s–1.0 s100 attemptsClick for where the time goes

0.96

Tenkimedian 0.96 sp95 3.0 s95% CI 0.91 s–1.1 s78 attemptsClick for where the time goes

1.1

Daytonamedian 1.1 sp95 3.0 s95% CI 1.0 s–1.3 s100 attemptsClick for where the time goes

1.4

boxdmedian 1.4 sp95 3.9 s95% CI 1.4 s–1.4 s100 attemptsClick for where the time goes

2.7

Upstash Boxmedian 2.7 sp95 9.1 s95% CI 2.6 s–2.7 s100 attempts, 1 failedClick for where the time goes

Notes (3)

1boxd: ran 4 vCPU, 16 GB, not 4 vCPU, 8 GB. boxd has no 8 GB size at 4 vCPU; this is its nearest.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.
Superserve: not in this track: our Superserve plan caps templates at 2 vCPU, so there was no 4 vCPU machine to test. A paid plan may lift this; we will add it if it does.

Ready through Sandbox SDK

create() until our SDK's handle is ready · Median seconds of each part · Lower is better

Provider cold start, the number on the Cold start tabOur SDK's extra setup, not the provider's

Freestyleprovider cold start 0.66 sour SDK setup +0.96 sready through the SDK, median 1.7 s100 cold startsClick for where the time goes

E2Bprovider cold start 0.95 sour SDK setup +0.90 sready through the SDK, median 2.1 s100 cold startsClick for where the time goes

Tenkiprovider cold start 0.96 sour SDK setup +1.0 sready through the SDK, median 2.0 s78 cold startsClick for where the time goes

Daytonaprovider cold start 1.1 sour SDK setup +1.8 sready through the SDK, median 3.4 s100 cold startsClick for where the time goes

boxdprovider cold start 1.4 sour SDK setup +1.8 sready through the SDK, median 3.2 s100 cold startsClick for where the time goes

Upstash Boxprovider cold start 2.7 sour SDK setup +5.2 sready through the SDK, median 8.9 s99 cold startsClick for where the time goes

1.6

Freestyleprovider cold start 0.66 sour SDK setup +0.96 sready through the SDK, median 1.7 s100 cold startsClick for where the time goes

1.9

E2Bprovider cold start 0.95 sour SDK setup +0.90 sready through the SDK, median 2.1 s100 cold startsClick for where the time goes

2.0

Tenkiprovider cold start 0.96 sour SDK setup +1.0 sready through the SDK, median 2.0 s78 cold startsClick for where the time goes

3.0

Daytonaprovider cold start 1.1 sour SDK setup +1.8 sready through the SDK, median 3.4 s100 cold startsClick for where the time goes

3.2

boxdprovider cold start 1.4 sour SDK setup +1.8 sready through the SDK, median 3.2 s100 cold startsClick for where the time goes

7.8

Upstash Boxprovider cold start 2.7 sour SDK setup +5.2 sready through the SDK, median 8.9 s99 cold startsClick for where the time goes

Notes (4)

1boxd: ran 4 vCPU, 16 GB, not 4 vCPU, 8 GB. boxd has no 8 GB size at 4 vCPU; this is its nearest.
The hatched part is work Sandbox SDK does after the sandbox can run commands, such as preparing the workspace. It is ours, so it is kept out of cold start. Each part is its own median, so the stack can differ a little from the median total shown on hover.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.
Superserve: not in this track: our Superserve plan caps templates at 2 vCPU, so there was no 4 vCPU machine to test. A paid plan may lift this; we will add it if it does.

Agent work

Real coding-agent tasks, so CPU, disk and network count as much as the API.

Agent tool replay

100 tool calls in a row: write, read, edit, list, grep, run a test · Median seconds · Lower is better

Daytonamedian 11 sp95 31 s95% CI 10 s–14 s30 attemptsClick for where the time goes

Superservemedian 12 sp95 15 s95% CI 12 s–13 s30 attemptsClick for where the time goes

E2Bmedian 15 sp95 17 s95% CI 15 s–15 s30 attemptsClick for where the time goes

Tenkimedian 17 sp95 21 s95% CI 17 s–18 s30 attemptsClick for where the time goes

Freestylemedian 20 sp95 23 s95% CI 19 s–20 s30 attemptsClick for where the time goes

Upstash Boxmedian 24 sp95 30 s95% CI 24 s–25 s30 attempts, 1 failedClick for where the time goes

boxdmedian 31 sp95 34 s95% CI 31 s–32 s30 attemptsClick for where the time goes

11

Daytonamedian 11 sp95 31 s95% CI 10 s–14 s30 attemptsClick for where the time goes

12

Superservemedian 12 sp95 15 s95% CI 12 s–13 s30 attemptsClick for where the time goes

15

E2Bmedian 15 sp95 17 s95% CI 15 s–15 s30 attemptsClick for where the time goes

17

Tenkimedian 17 sp95 21 s95% CI 17 s–18 s30 attemptsClick for where the time goes

20

Freestylemedian 20 sp95 23 s95% CI 19 s–20 s30 attemptsClick for where the time goes

24

Upstash Boxmedian 24 sp95 30 s95% CI 24 s–25 s30 attempts, 1 failedClick for where the time goes

31

boxdmedian 31 sp95 34 s95% CI 31 s–32 s30 attemptsClick for where the time goes

Notes (2)

1boxd: runs in Germany, so every call from our Florida server crosses the Atlantic.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.

Agent tool replay

100 tool calls in a row: write, read, edit, list, grep, run a test · Median seconds · Lower is better

Daytonamedian 11 sp95 29 s95% CI 9.9 s–12 s30 attemptsClick for where the time goes

E2Bmedian 15 sp95 16 s95% CI 15 s–15 s30 attemptsClick for where the time goes

Freestylemedian 20 sp95 28 s95% CI 20 s–22 s30 attemptsClick for where the time goes

Tenkimedian 21 sp95 34 s95% CI 20 s–25 s30 attemptsClick for where the time goes

Upstash Boxmedian 28 sp95 54 s95% CI 24 s–30 s30 attemptsClick for where the time goes

boxdmedian 33 sp95 49 s95% CI 32 s–41 s30 attemptsClick for where the time goes

11

Daytonamedian 11 sp95 29 s95% CI 9.9 s–12 s30 attemptsClick for where the time goes

15

E2Bmedian 15 sp95 16 s95% CI 15 s–15 s30 attemptsClick for where the time goes

20

Freestylemedian 20 sp95 28 s95% CI 20 s–22 s30 attemptsClick for where the time goes

21

Tenkimedian 21 sp95 34 s95% CI 20 s–25 s30 attemptsClick for where the time goes

28

Upstash Boxmedian 28 sp95 54 s95% CI 24 s–30 s30 attemptsClick for where the time goes

33

boxdmedian 33 sp95 49 s95% CI 32 s–41 s30 attemptsClick for where the time goes

Notes (3)

1boxd: ran 4 vCPU, 16 GB, not 4 vCPU, 8 GB. boxd has no 8 GB size at 4 vCPU; this is its nearest. Runs in Germany, so every call from our Florida server crosses the Atlantic.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.
Superserve: not in this track: our Superserve plan caps templates at 2 vCPU, so there was no 4 vCPU machine to test. A paid plan may lift this; we will add it if it does.

Everyday calls

What an agent does hundreds of times per task: run a command, move files, reattach to a sandbox.

Run a command

A no-op command, mostly network and API overhead · Median seconds · Lower is better

Daytonamedian 0.12 sp95 0.29 s95% CI 0.11 s–0.13 s100 attempts

E2Bmedian 0.12 sp95 0.28 s95% CI 0.12 s–0.12 s100 attempts

Freestylemedian 0.15 sp95 0.20 s95% CI 0.14 s–0.15 s100 attempts

Superservemedian 0.15 sp95 0.23 s95% CI 0.15 s–0.16 s100 attempts

Tenkimedian 0.17 sp95 0.19 s95% CI 0.16 s–0.17 s100 attempts

Upstash Boxmedian 0.22 sp95 0.32 s95% CI 0.22 s–0.22 s100 attempts

boxdmedian 0.29 sp95 0.29 s95% CI 0.29 s–0.29 s100 attempts

0.12

Daytonamedian 0.12 sp95 0.29 s95% CI 0.11 s–0.13 s100 attempts

0.12

E2Bmedian 0.12 sp95 0.28 s95% CI 0.12 s–0.12 s100 attempts

0.15

Freestylemedian 0.15 sp95 0.20 s95% CI 0.14 s–0.15 s100 attempts

0.15

Superservemedian 0.15 sp95 0.23 s95% CI 0.15 s–0.16 s100 attempts

0.17

Tenkimedian 0.17 sp95 0.19 s95% CI 0.16 s–0.17 s100 attempts

0.22

Upstash Boxmedian 0.22 sp95 0.32 s95% CI 0.22 s–0.22 s100 attempts

0.29

boxdmedian 0.29 sp95 0.29 s95% CI 0.29 s–0.29 s100 attempts

Notes (1)

Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.

Run a command

A no-op command, mostly network and API overhead · Median seconds · Lower is better

Daytonamedian 0.11 sp95 0.28 s95% CI 0.10 s–0.11 s100 attempts

E2Bmedian 0.12 sp95 0.14 s95% CI 0.12 s–0.12 s100 attempts

Freestylemedian 0.15 sp95 0.22 s95% CI 0.14 s–0.15 s100 attempts

Tenkimedian 0.17 sp95 0.20 s95% CI 0.16 s–0.17 s78 attempts

Upstash Boxmedian 0.23 sp95 0.50 s95% CI 0.22 s–0.23 s99 attempts

boxdmedian 0.28 sp95 0.29 s95% CI 0.28 s–0.28 s100 attempts

0.11

Daytonamedian 0.11 sp95 0.28 s95% CI 0.10 s–0.11 s100 attempts

0.12

E2Bmedian 0.12 sp95 0.14 s95% CI 0.12 s–0.12 s100 attempts

0.15

Freestylemedian 0.15 sp95 0.22 s95% CI 0.14 s–0.15 s100 attempts

0.17

Tenkimedian 0.17 sp95 0.20 s95% CI 0.16 s–0.17 s78 attempts

0.23

Upstash Boxmedian 0.23 sp95 0.50 s95% CI 0.22 s–0.23 s99 attempts

0.28

boxdmedian 0.28 sp95 0.29 s95% CI 0.28 s–0.28 s100 attempts

Notes (3)

1boxd: ran 4 vCPU, 16 GB, not 4 vCPU, 8 GB. boxd has no 8 GB size at 4 vCPU; this is its nearest.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.
Superserve: not in this track: our Superserve plan caps templates at 2 vCPU, so there was no 4 vCPU machine to test. A paid plan may lift this; we will add it if it does.

Pause & snapshots

Pausing stops paying for idle time while an agent waits on a human, with every process kept. Where stop and start restarts the sandbox instead, we mark pause as not supported. Snapshots start many sandboxes from one prepared state.

Resume

Resume a paused sandbox until a command runs · Median seconds · Lower is better

Freestylemedian 0.32 s100 attemptsClick for where the time goes

boxdmedian 0.41 s100 attemptsClick for where the time goes

E2Bmedian 0.45 s100 attemptsClick for where the time goes

Superservemedian 0.54 s100 attemptsClick for where the time goes

Upstash Boxmedian 0.59 s100 attemptsClick for where the time goes

Tenkimedian 2.3 s100 attemptsClick for where the time goes

0.32

Freestylemedian 0.32 s100 attemptsClick for where the time goes

0.41

boxdmedian 0.41 s100 attemptsClick for where the time goes

0.45

E2Bmedian 0.45 s100 attemptsClick for where the time goes

0.54

Superservemedian 0.54 s100 attemptsClick for where the time goes

0.59

Upstash Boxmedian 0.59 s100 attemptsClick for where the time goes

2.3

Tenkimedian 2.3 s100 attemptsClick for where the time goes

Notes (2)

Daytona: not supported. Daytona's container sandboxes have no pause: stop and start restarts the sandbox and ends every running process. We don't time a restart as a pause.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.

Resume

Resume a paused sandbox until a command runs · Median seconds · Lower is better

Freestylemedian 0.35 s100 attemptsClick for where the time goes

boxdmedian 0.40 s99 attemptsClick for where the time goes

E2Bmedian 0.51 s100 attemptsClick for where the time goes

Upstash Boxmedian 0.57 s99 attemptsClick for where the time goes

Tenkimedian 2.4 s77 attemptsClick for where the time goes

0.35

Freestylemedian 0.35 s100 attemptsClick for where the time goes

0.40

boxdmedian 0.40 s99 attemptsClick for where the time goes

0.51

E2Bmedian 0.51 s100 attemptsClick for where the time goes

0.57

Upstash Boxmedian 0.57 s99 attemptsClick for where the time goes

2.4

Tenkimedian 2.4 s77 attemptsClick for where the time goes

Notes (4)

1boxd: ran 4 vCPU, 16 GB, not 4 vCPU, 8 GB. boxd has no 8 GB size at 4 vCPU; this is its nearest.
Daytona: not supported. Daytona's container sandboxes have no pause: stop and start restarts the sandbox and ends every running process. We don't time a restart as a pause.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.
Superserve: not in this track: our Superserve plan caps templates at 2 vCPU, so there was no 4 vCPU machine to test. A paid plan may lift this; we will add it if it does.

Bursts

10 or 25 sandboxes started at once, as an agent fanning out would, timed until the last one runs a command.

10 at once, all ready

10 creates at once, until all 10 run a command · Median seconds · Lower is better

Superservemedian 0.72 sp95 0.72 s95% CI 0.64 s–0.72 s3 attempts

E2Bmedian 1.0 sp95 1.2 s95% CI 0.98 s–1.2 s3 attempts

Freestylemedian 1.1 sp95 1.1 s95% CI 0.85 s–1.1 s3 attempts

Tenkimedian 1.2 sp95 1.8 s95% CI 1.1 s–1.8 s3 attempts

boxdmedian 2.2 sp95 2.9 s95% CI 2.0 s–2.9 s3 attempts

Daytonamedian 2.8 sp95 4.4 s95% CI 2.2 s–4.4 s3 attempts

Upstash Boxmedian 3.8 sp95 13 s95% CI 2.8 s–13 s3 attempts

0.72

Superservemedian 0.72 sp95 0.72 s95% CI 0.64 s–0.72 s3 attempts

1.0

E2Bmedian 1.0 sp95 1.2 s95% CI 0.98 s–1.2 s3 attempts

1.1

Freestylemedian 1.1 sp95 1.1 s95% CI 0.85 s–1.1 s3 attempts

1.2

Tenkimedian 1.2 sp95 1.8 s95% CI 1.1 s–1.8 s3 attempts

2.2

boxdmedian 2.2 sp95 2.9 s95% CI 2.0 s–2.9 s3 attempts

2.8

Daytonamedian 2.8 sp95 4.4 s95% CI 2.2 s–4.4 s3 attempts

3.8

Upstash Boxmedian 3.8 sp95 13 s95% CI 2.8 s–13 s3 attempts

Notes (1)

Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.

10 at once, all ready

10 creates at once, until all 10 run a command · Median seconds · Lower is better

Freestylemedian 1.0 sp95 1.0 s95% CI 0.89 s–1.0 s3 attempts

Tenkimedian 1.2 sp95 1.3 s95% CI 1.2 s–1.3 s3 attempts

E2Bmedian 1.4 sp95 1.6 s95% CI 1.2 s–1.6 s3 attempts

Upstash Boxmedian 21 sp95 25 s95% CI 4.8 s–25 s3 attempts

1.0

Freestylemedian 1.0 sp95 1.0 s95% CI 0.89 s–1.0 s3 attempts

1.2

Tenkimedian 1.2 sp95 1.3 s95% CI 1.2 s–1.3 s3 attempts

1.4

E2Bmedian 1.4 sp95 1.6 s95% CI 1.2 s–1.6 s3 attempts

21

Upstash Boxmedian 21 sp95 25 s95% CI 4.8 s–25 s3 attempts

Notes (3)

No result for Daytona (not run: our Daytona account's 10 vCPU quota can't hold ten 4 vCPU sandboxes at once); boxd (ran 4 vCPU, 16 GB, not 4 vCPU, 8 GB. boxd has no 8 GB size at 4 vCPU; this is its nearest. One of its three bursts of 10 was cut short by our own ownership check (1 create failed on our side), so it is not counted; coming in the next update).
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.
Superserve: not in this track: our Superserve plan caps templates at 2 vCPU, so there was no 4 vCPU machine to test. A paid plan may lift this; we will add it if it does.

Machines

What each provider ran: what we asked for, what we measured inside, and the hardware underneath.

What each provider ran

Each provider's default sandbox, and the vCPU and RAM measured inside it

E2B
Asked for 2 vCPU, 512 MB, template base
Measured 2 vCPU, 473 MB
E2B's default base template is 2 vCPU and 512 MB. E2B's pricing page says its default sandbox has 4 GB, so E2B ran smaller here than that.
Daytona
Asked for 1 vCPU, 1 GB
Measured 1 vCPU, 1 GB
Upstash Box
Asked for 2 vCPU, 4 GB
Measured 2 vCPU, 4 GB
boxd
Asked for 2 vCPU, 8 GB
Measured 2 vCPU, 7.8 GB
Freestyle
Asked for 4 vCPU, 8 GB
Measured 4 vCPU, 7.8 GB
Tenki
Asked for provider default
Measured 2 vCPU, 3.8 GB
Tenki documents a 5 GiB default disk but not a default vCPU or RAM, so those are what we measured.
Superserve
Asked for 1 vCPU, 1 GB, template sandbox-bench-node22-py311-1x1-pinned
Measured 1 vCPU, 994 MB
Superserve ran a template we built with Node 22 and Python 3.11 preinstalled, at its default 1 vCPU, 1 GB RAM and 4 GB disk. Superserve's own templates have Node (node-22) or Python (python-3.11), not both, and our tests need both.

Provider	Asked for	Measured inside
E2B	2 vCPU, 512 MB, template baseE2B's default base template is 2 vCPU and 512 MB. E2B's pricing page says its default sandbox has 4 GB, so E2B ran smaller here than that.	2 vCPU, 473 MB
Daytona	1 vCPU, 1 GB	1 vCPU, 1 GB
Upstash Box	2 vCPU, 4 GB	2 vCPU, 4 GB
boxd	2 vCPU, 8 GB	2 vCPU, 7.8 GB
Freestyle	4 vCPU, 8 GB	4 vCPU, 7.8 GB
Tenki	provider defaultTenki documents a 5 GiB default disk but not a default vCPU or RAM, so those are what we measured.	2 vCPU, 3.8 GB
Superserve	1 vCPU, 1 GB, template sandbox-bench-node22-py311-1x1-pinnedSuperserve ran a template we built with Node 22 and Python 3.11 preinstalled, at its default 1 vCPU, 1 GB RAM and 4 GB disk. Superserve's own templates have Node (node-22) or Python (python-3.11), not both, and our tests need both.	1 vCPU, 994 MB

Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.

Hardware inside each sandbox

Chip, memory, cloud and region, and how each sandbox is isolated

Named means the CPU reports its own model. Inferred means we worked it out from what the sandbox shows, such as model number, cache and clock, or from the chip's published spec. Measured memory was read inside the sandbox. From a hardware probe on Oct 22, 2026, checked against the benchmark's fingerprints. Open a row for what each sandbox hides.

E2BChipIntel Xeon Platinum 8373CIce Lake, 2.6 GHz, 54 MB L3 · inferredMemoryDDR4inferredWhereGoogle Cloudus-west1, The Dalles, OregonRuns asFirecracker microVMnested inside a Google Cloud VM
Hidden or different from the real machine
The CPU name, microcode and clock leaves are hidden. The chip is identified from its model number, cache size and 2.6 GHz clock.
The default 2 vCPUs are one physical core with two hyperthreads, so two-thread work scales well below twice one thread.
No DMI or PCI devices, so the memory type can't be read inside; DDR4 follows from the chip.
Memory
DDR4-3200, the chip's memory type (inferred)
Disk
23 GB virtio disk, ext4
Kernel
Linux 6.1, Debian 12
Size
E2B's default base template is 2 vCPU and 512 MB. E2B's pricing page says its default sandbox has 4 GB, so E2B ran smaller here than that.
DaytonaChipAMD EPYC 9354P, 9354, 9254 or 9275FZen 4 (Genoa); the 9275F is Zen 5 (Turin). Hosts vary · named by the CPUMemoryDDR5measuredWhereBare-metal serversLocation not visible; control plane on AWS us-east-1Runs asDocker containershares the host's kernel; cgroups set its CPU and RAM
Hidden or different from the real machine
Nothing at the CPU level is hidden: the real CPU name, microcode, server model (Dell PowerEdge R6615 in the probe, a Supermicro AS-2115GT-HNTR in the benchmark) and disks are visible.
nproc and lscpu show every host thread (48 or 64) and /proc/meminfo shows the host's RAM, while the default sandbox may use 1 CPU and 1 GB. Tools that size thread pools or caches from those numbers oversubscribe.
Hosts differ in clock speed, so single-thread results vary from sandbox to sandbox. Neighbours share the host kernel.
Outbound internet is blocked on our account tier, so the sandbox's own network location could not be read.
Memory
Registered DDR5, read from the memory controller on a 9254 host (measured)
Disk
3 GB container disk (10 GB at 4 vCPU) on the host's NVMe RAID
Kernel
Host kernel 6.17 (Ubuntu 24.04), Debian 13 userland
Chips in this track
9354P ×59, 9254 ×43, 9354 ×27, 9275F ×1, from the benchmark's fingerprints
Upstash BoxChipAWS Graviton4Arm Neoverse V2, 2.8 GHz; Graviton4 from the m8g instance type · inferredMemoryDDR5inferredWhereAWS EC2us-east-1, Ashburn, VirginiaRuns asDocker containeron a shared m8g.16xlarge VM; cgroups set its CPU and RAM
Hidden or different from the real machine
The only Arm provider here: x86-only binaries don't run, and compile and run times differ for that reason alone.
nproc and /proc/meminfo show the host's 64 CPUs and 256 GB, while a box may use 2 CPUs and 4 GB (4 and 8 at the medium size).
No DMI tables or clock readings inside, so memory type and clock come from AWS's published spec.
Neighbours share the host kernel and the chip's L3 cache.
Memory
DDR5-5600, from AWS's Graviton4 spec (inferred)
Disk
5 GB overlay root and a 5 GB /workspace, on EBS
Kernel
Host kernel 7.0 (Ubuntu 24.04 for AWS), Debian 12 userland
boxdChipAMD EPYC 9554P, likelyZen 4 (Genoa), 3.1 GHz · inferredMemoryDDR5inferredWhereLatitude.sh bare metalFrankfurt, GermanyRuns asFirecracker microVMwith Docker installed inside the VM
Hidden or different from the real machine
The CPU name is trimmed to "AMD EPYC" and the microcode and topology are hidden. The chip is inferred from its 3.1 GHz clock and the Genoa servers Latitude.sh sells; a dual-socket 9554 can't be ruled out.
No DMI, so the memory type is inferred.
AVX-512 is exposed, which can help vectorised work against hosts that hide it.
The disk reports itself as rotational although it is virtual, so tools that tune for hard drives may behave differently.
Memory
DDR5-4800, the Genoa platform's memory (inferred)
Disk
100 GB virtio disk, ext4
Kernel
Linux 6.18 (custom build), Ubuntu 24.04
FreestyleChipAMD EPYC 7763, likelyZen 3 (Milan), 2.45 GHz · inferredMemoryDDR4inferredWhereFreestyle's own network (AS36320)San Francisco Bay AreaRuns asFirecracker microVM4 vCPU, 8 GB, no hyperthreads exposed
Hidden or different from the real machine
The CPU name, microcode and topology are hidden. The chip is inferred from its 2.45 GHz clock, cache and Zen 3 features.
Memory is added after boot: about 3 GB is free at first, 7.9 GB later.
No DMI, so the memory type is inferred.
Memory
DDR4-3200, the Milan platform's memory (inferred)
Disk
32 GB virtio disk, ext4
Kernel
Linux 6.1
TenkiChipAMD EPYC 7B13, as reportedZen 3 (Milan), 2.25 GHz. The 4 vCPU VM also showed EPYC 9575F (Zen 5, Turin) cores · conflictingMemoryNot knownNo memory data inside the VMWhereLuxor (AS19538)SeattleRuns asCloud Hypervisor microVMwith a container inside
Hidden or different from the real machine
On the 4 vCPU VM, two vCPUs report an EPYC 7B13 (Milan) and two an EPYC 9575F (Turin). That fits a snapshot taken on a Milan host and restored on a Turin one, so the real host may be newer than the name says.
CPU features are capped at the Milan set on every vCPU, so no AVX-512 even on Turin.
The clock reads 2.25 GHz, the 7B13's base, which may be carried over from the snapshot.
Memory
No memory data inside the VM
Disk
5 GB virtio disk, ext4
Kernel
Linux 6.18
Size
Tenki documents a 5 GiB default disk but not a default vCPU or RAM, so those are what we measured.
SuperserveChipIntel Xeon, Sapphire RapidsLikely Xeon Platinum 8481C; 2.2 GHz, 105 MB L3 · inferredMemoryDDR5inferredWhereGoogle Cloudus-east4, Northern Virginia, from its IP rangeRuns asFirecracker microVMlikely inside a Google Cloud VM
Hidden or different from the real machine
The CPU name and microcode are hidden. The chip family is clear from its model number; the exact SKU is not.
The old 4.14 guest kernel can't turn on the chip's AMX units, so AMX workloads can't use them.
No DMI, so the memory type is inferred.
Memory
DDR5, the Sapphire Rapids platform's memory (inferred)
Disk
4 GB virtio disk, ext4
Kernel
Linux 4.14, Debian 12
Size
Superserve ran a template we built with Node 22 and Python 3.11 preinstalled, at its default 1 vCPU, 1 GB RAM and 4 GB disk. Superserve's own templates have Node (node-22) or Python (python-3.11), not both, and our tests need both.

What each provider ran

Asked for 4 vCPU, 8 GB; measured inside each sandbox

E2BMatches
Asked for 4 vCPU, 8 GB, template r1h7gud12rf1f6iu29r0
Measured 4 vCPU, 7.8 GB
DaytonaMatches
Asked for 4 vCPU, 8 GB, template daytona-large
Measured 4 vCPU, 8 GB
Upstash BoxMatches
Asked for 4 vCPU, 8 GB ("medium")
Measured 4 vCPU, 8 GB
boxdDiffers
Asked for 4 vCPU, 16 GB
Measured 4 vCPU, 16 GB
Ran 4 vCPU, 16 GB, not 4 vCPU, 8 GB. boxd has no 8 GB size at 4 vCPU; this is its nearest.
FreestyleMatches
Asked for 4 vCPU, 8 GB
Measured 4 vCPU, 7.8 GB
TenkiMatches
Asked for 4 vCPU, 8 GB
Measured 4 vCPU, 7.8 GB

Provider	Asked for	Measured inside	4 vCPU, 8 GB?
E2B	4 vCPU, 8 GB, template r1h7gud12rf1f6iu29r0	4 vCPU, 7.8 GB	Matches
Daytona	4 vCPU, 8 GB, template daytona-large	4 vCPU, 8 GB	Matches
Upstash Box	4 vCPU, 8 GB ("medium")	4 vCPU, 8 GB	Matches
boxd	4 vCPU, 16 GBRan 4 vCPU, 16 GB, not 4 vCPU, 8 GB. boxd has no 8 GB size at 4 vCPU; this is its nearest.	4 vCPU, 16 GB	Differs
Freestyle	4 vCPU, 8 GB	4 vCPU, 7.8 GB	Matches
Tenki	4 vCPU, 8 GB	4 vCPU, 7.8 GB	Matches

Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.
Superserve: not in this track: our Superserve plan caps templates at 2 vCPU, so there was no 4 vCPU machine to test. A paid plan may lift this; we will add it if it does.

Hardware inside each sandbox

Chip, memory, cloud and region, and how each sandbox is isolated

Named means the CPU reports its own model. Inferred means we worked it out from what the sandbox shows, such as model number, cache and clock, or from the chip's published spec. Measured memory was read inside the sandbox. From a hardware probe on Oct 22, 2026, checked against the benchmark's fingerprints. Open a row for what each sandbox hides.

E2BChipIntel Xeon Platinum 8373CIce Lake, 2.6 GHz, 54 MB L3 · inferredMemoryDDR4inferredWhereGoogle Cloudus-west1, The Dalles, OregonRuns asFirecracker microVMnested inside a Google Cloud VM
Hidden or different from the real machine
The CPU name, microcode and clock leaves are hidden. The chip is identified from its model number, cache size and 2.6 GHz clock.
The default 2 vCPUs are one physical core with two hyperthreads, so two-thread work scales well below twice one thread.
No DMI or PCI devices, so the memory type can't be read inside; DDR4 follows from the chip.
Memory
DDR4-3200, the chip's memory type (inferred)
Disk
23 GB virtio disk, ext4
Kernel
Linux 6.1, Debian 12
At 4 vCPU
The 4 vCPU, 8 GB template ran on the same CPU in the same region: two cores with two threads each.
DaytonaChipAMD EPYC 9354P, 9354, 9254 or 9275FZen 4 (Genoa); the 9275F is Zen 5 (Turin). Hosts vary · named by the CPUMemoryDDR5measuredWhereBare-metal serversLocation not visible; control plane on AWS us-east-1Runs asDocker containershares the host's kernel; cgroups set its CPU and RAM
Hidden or different from the real machine
Nothing at the CPU level is hidden: the real CPU name, microcode, server model (Dell PowerEdge R6615 in the probe, a Supermicro AS-2115GT-HNTR in the benchmark) and disks are visible.
nproc and lscpu show every host thread (48 or 64) and /proc/meminfo shows the host's RAM, while the default sandbox may use 1 CPU and 1 GB. Tools that size thread pools or caches from those numbers oversubscribe.
Hosts differ in clock speed, so single-thread results vary from sandbox to sandbox. Neighbours share the host kernel.
Outbound internet is blocked on our account tier, so the sandbox's own network location could not be read.
Memory
Registered DDR5, read from the memory controller on a 9254 host (measured)
Disk
3 GB container disk (10 GB at 4 vCPU) on the host's NVMe RAID
Kernel
Host kernel 6.17 (Ubuntu 24.04), Debian 13 userland
Chips in this track
9254 ×35, 9354P ×34, 9354 ×24, 9275F ×7, from the benchmark's fingerprints
Upstash BoxChipAWS Graviton4Arm Neoverse V2, 2.8 GHz; Graviton4 from the m8g instance type · inferredMemoryDDR5inferredWhereAWS EC2us-east-1, Ashburn, VirginiaRuns asDocker containeron a shared m8g.16xlarge VM; cgroups set its CPU and RAM
Hidden or different from the real machine
The only Arm provider here: x86-only binaries don't run, and compile and run times differ for that reason alone.
nproc and /proc/meminfo show the host's 64 CPUs and 256 GB, while a box may use 2 CPUs and 4 GB (4 and 8 at the medium size).
No DMI tables or clock readings inside, so memory type and clock come from AWS's published spec.
Neighbours share the host kernel and the chip's L3 cache.
Memory
DDR5-5600, from AWS's Graviton4 spec (inferred)
Disk
5 GB overlay root and a 5 GB /workspace, on EBS
Kernel
Host kernel 7.0 (Ubuntu 24.04 for AWS), Debian 12 userland
At 4 vCPU
The medium size (4 vCPU, 8 GB) runs on the same m8g.16xlarge hosts.
boxdChipAMD EPYC 9554P, likelyZen 4 (Genoa), 3.1 GHz · inferredMemoryDDR5inferredWhereLatitude.sh bare metalFrankfurt, GermanyRuns asFirecracker microVMwith Docker installed inside the VM
Hidden or different from the real machine
The CPU name is trimmed to "AMD EPYC" and the microcode and topology are hidden. The chip is inferred from its 3.1 GHz clock and the Genoa servers Latitude.sh sells; a dual-socket 9554 can't be ruled out.
No DMI, so the memory type is inferred.
AVX-512 is exposed, which can help vectorised work against hosts that hide it.
The disk reports itself as rotational although it is virtual, so tools that tune for hard drives may behave differently.
Memory
DDR5-4800, the Genoa platform's memory (inferred)
Disk
100 GB virtio disk, ext4
Kernel
Linux 6.18 (custom build), Ubuntu 24.04
At 4 vCPU
The 4 vCPU, 16 GB size ran on the same CPU type.
FreestyleChipAMD EPYC 7763, likelyZen 3 (Milan), 2.45 GHz · inferredMemoryDDR4inferredWhereFreestyle's own network (AS36320)San Francisco Bay AreaRuns asFirecracker microVM4 vCPU, 8 GB, no hyperthreads exposed
Hidden or different from the real machine
The CPU name, microcode and topology are hidden. The chip is inferred from its 2.45 GHz clock, cache and Zen 3 features.
Memory is added after boot: about 3 GB is free at first, 7.9 GB later.
No DMI, so the memory type is inferred.
Memory
DDR4-3200, the Milan platform's memory (inferred)
Disk
32 GB virtio disk, ext4
Kernel
Linux 6.1
TenkiChipAMD EPYC 7B13, as reportedZen 3 (Milan), 2.25 GHz. The 4 vCPU VM also showed EPYC 9575F (Zen 5, Turin) cores · conflictingMemoryNot knownNo memory data inside the VMWhereLuxor (AS19538)SeattleRuns asCloud Hypervisor microVMwith a container inside
Hidden or different from the real machine
On the 4 vCPU VM, two vCPUs report an EPYC 7B13 (Milan) and two an EPYC 9575F (Turin). That fits a snapshot taken on a Milan host and restored on a Turin one, so the real host may be newer than the name says.
CPU features are capped at the Milan set on every vCPU, so no AVX-512 even on Turin.
The clock reads 2.25 GHz, the 7B13's base, which may be carried over from the snapshot.
Memory
No memory data inside the VM
Disk
5 GB virtio disk, ext4
Kernel
Linux 6.18
At 4 vCPU
The 4 vCPU, 8 GB VM is the one that showed both chips.

Reliability

How often each provider failed, and why. Only failures the provider caused are counted; Method lists the ones we set aside.

Failed attempts

Per 1,000 attempts across the tests on this page · Lower is better

E2B0 of 1,076 attempts failed

Daytona0 of 973 attempts failed

Freestyle0 of 1,068 attempts failed

Superserve0 of 1,076 attempts failed

Tenki1 of 1,075 attempts failed

boxd1 of 1,073 attempts failed

Upstash Box3 of 1,070 attempts failed

0.0

E2B0 of 1,076 attempts failed

0.0

Daytona0 of 973 attempts failed

0.0

Freestyle0 of 1,068 attempts failed

0.0

Superserve0 of 1,076 attempts failed

0.9

Tenki1 of 1,075 attempts failed

0.9

boxd1 of 1,073 attempts failed

2.8

Upstash Box3 of 1,070 attempts failed

Every failure and its cause · 5 runs across 3 providers

Tenki
- 1× Tenki request failed (not_found). (not_found, 5, operation=snapshot)Take snapshot
boxd
- 1× boxd request failed. (provider)Write + read 10 MB
Upstash Box
- 1× Upstash Box exec failed (provider). (provider)Agent tool replay
- 1× Upstash Box files.write failed (unavailable, HTTP 500). (unavailable, HTTP 500, operation=files.write)npm install
- 1× Guest exit 143: Cloning into '/workspace/bench-clone'..git clone

Provider	Runs	Cause	Tests
Tenki	1	Tenki request failed (not_found). (not_found, 5, operation=snapshot)	Take snapshot
boxd	1	boxd request failed. (provider)	Write + read 10 MB
Upstash Box	1	Upstash Box exec failed (provider). (provider)	Agent tool replay
Upstash Box	1	Upstash Box files.write failed (unavailable, HTTP 500). (unavailable, HTTP 500, operation=files.write)	npm install
Upstash Box	1	Guest exit 143: Cloning into '/workspace/bench-clone'..	git clone

Failed attempts

Per 1,000 attempts across the tests on this page · Lower is better

E2B0 of 1,076 attempts failed

Daytona0 of 910 attempts failed

Tenki0 of 897 attempts failed

Freestyle1 of 1,073 attempts failed

Upstash Box1 of 1,062 attempts failed

boxd2 of 1,064 attempts failed

0.0

E2B0 of 1,076 attempts failed

0.0

Daytona0 of 910 attempts failed

0.0

Tenki0 of 897 attempts failed

0.9

Freestyle1 of 1,073 attempts failed

0.9

Upstash Box1 of 1,062 attempts failed

1.9

boxd2 of 1,064 attempts failed

Every failure and its cause · 4 runs across 3 providers

Freestyle
- 1× Preview HTTP 502, no first bytePreview URL first byte
Upstash Box
- 1× Upstash Box exec failed (provider). (provider)Cold start
boxd
- 1× boxd request failed. (invalid_argument)Pause + resume
- 1× boxd request failed. (provider)Write + read 10 MB

Provider	Runs	Cause	Tests
Freestyle	1	Preview HTTP 502, no first byte	Preview URL first byte
Upstash Box	1	Upstash Box exec failed (provider). (provider)	Cold start
boxd	1	boxd request failed. (invalid_argument)	Pause + resume
boxd	1	boxd request failed. (provider)	Write + read 10 MB

Cost

What one tool replay task cost, read from each provider's billing where it reports one, and published prices for comparison.

Cost per 1,000 tasks

Create a sandbox, run the 100-call tool replay, delete it · US dollars per 1,000 tasks · Lower is better

Billed: the provider's own usage charge
Metered: the provider's recorded usage at its published rates
Estimate: list price, where the provider's billing gives no amount

Upstash BoxBilled $0.23 per 1,000 tasks$0.00023 for one taskBilled over 10 tasks, high confidence

DaytonaBilled $0.24 per 1,000 tasks$0.00024 for one taskBilled over 10 tasks, medium confidence

SuperserveEstimate $0.25 per 1,000 tasks$0.00025 for one taskEstimate over 10 tasks, low confidence

E2BMetered $0.49 per 1,000 tasks$0.00049 for one taskMetered over 10 tasks, medium confidence

TenkiMetered $0.90 per 1,000 tasks$0.00090 for one taskMetered over 10 tasks, medium confidence

boxdMetered $1.00 per 1,000 tasks$0.0010 for one taskMetered over 10 tasks, medium confidence

FreestyleEstimate $1.71 per 1,000 tasks$0.0017 for one taskEstimate over 10 tasks, low confidence

$0.23

Upstash BoxBilled $0.23 per 1,000 tasks$0.00023 for one taskBilled over 10 tasks, high confidence

$0.24

DaytonaBilled $0.24 per 1,000 tasks$0.00024 for one taskBilled over 10 tasks, medium confidence

$0.25

SuperserveEstimate $0.25 per 1,000 tasks$0.00025 for one taskEstimate over 10 tasks, low confidence

$0.49

E2BMetered $0.49 per 1,000 tasks$0.00049 for one taskMetered over 10 tasks, medium confidence

$0.90

TenkiMetered $0.90 per 1,000 tasks$0.00090 for one taskMetered over 10 tasks, medium confidence

$1.00

boxdMetered $1.00 per 1,000 tasks$0.0010 for one taskMetered over 10 tasks, medium confidence

$1.71

FreestyleEstimate $1.71 per 1,000 tasks$0.0017 for one taskEstimate over 10 tasks, low confidence

Notes (4)

A task is: create a sandbox, run the 100-call tool replay, delete it. Each was its own sandbox in a separate cost run, billed per sandbox.
Superserve, Freestyle: list-price estimates, because billing gives no amount for them.
Billed and metered amounts are usage charges before credits, tax and plan minimums, not invoices. Estimates use compute list price only; disk, transfer and subscriptions are left out.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.

Agent work vs cost

Median tool replay time against what 1,000 tool replay tasks cost · Both lower is better

Faster and cheaper than the medianBilledMeteredEstimate

Cost per 1,000 tasks (USD)

Median tool replay, seconds (log scale)

Notes (4)

A task is: create a sandbox, run the 100-call tool replay, delete it. Each was its own sandbox in a separate cost run, billed per sandbox. Time is the median tool replay from the main tests. Labels show the vCPUs it ran on.
Hollow points are list-price estimates, not billing: Superserve, Freestyle.
Boat and boxd run in Europe, so every call from our Florida server crosses the Atlantic.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.

Cost per 1,000 tasks

Create a sandbox, run the 100-call tool replay, delete it · US dollars per 1,000 tasks · Lower is better

Billed: the provider's own usage charge
Metered: the provider's recorded usage at its published rates
Estimate: list price, where the provider's billing gives no amount

Upstash BoxBilled $0.47 per 1,000 tasks$0.00047 for one taskBilled over 8 tasks, high confidence

DaytonaBilled $1.19 per 1,000 tasks$0.0012 for one taskBilled over 10 tasks, medium confidence

E2BMetered $1.55 per 1,000 tasks$0.0016 for one taskMetered over 10 tasks, medium confidence

FreestyleEstimate $1.78 per 1,000 tasks$0.0018 for one taskEstimate over 10 tasks, low confidence

TenkiMetered $1.89 per 1,000 tasks$0.0019 for one taskMetered over 10 tasks, medium confidence

boxdMetered $2.00 per 1,000 tasks$0.0020 for one taskMetered over 10 tasks, medium confidence

$0.47

Upstash BoxBilled $0.47 per 1,000 tasks$0.00047 for one taskBilled over 8 tasks, high confidence

$1.19

DaytonaBilled $1.19 per 1,000 tasks$0.0012 for one taskBilled over 10 tasks, medium confidence

$1.55

E2BMetered $1.55 per 1,000 tasks$0.0016 for one taskMetered over 10 tasks, medium confidence

$1.78

FreestyleEstimate $1.78 per 1,000 tasks$0.0018 for one taskEstimate over 10 tasks, low confidence

$1.89

TenkiMetered $1.89 per 1,000 tasks$0.0019 for one taskMetered over 10 tasks, medium confidence

$2.00

boxdMetered $2.00 per 1,000 tasks$0.0020 for one taskMetered over 10 tasks, medium confidence

Notes (6)

1boxd: ran 4 vCPU, 16 GB, not 4 vCPU, 8 GB. boxd has no 8 GB size at 4 vCPU; this is its nearest.
A task is: create a sandbox, run the 100-call tool replay, delete it. Each was its own sandbox in a separate cost run, billed per sandbox.
Freestyle: list-price estimates, because billing gives no amount for them.
Billed and metered amounts are usage charges before credits, tax and plan minimums, not invoices. Estimates use compute list price only; disk, transfer and subscriptions are left out.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.
Superserve: not in this track: our Superserve plan caps templates at 2 vCPU, so there was no 4 vCPU machine to test. A paid plan may lift this; we will add it if it does.

Agent work vs cost

Median tool replay time against what 1,000 tool replay tasks cost · Both lower is better

Faster and cheaper than the medianBilledMeteredEstimate

Cost per 1,000 tasks (USD)

Median tool replay, seconds (log scale)

Notes (6)

A task is: create a sandbox, run the 100-call tool replay, delete it. Each was its own sandbox in a separate cost run, billed per sandbox. Time is the median tool replay from the main tests. Labels show the vCPUs it ran on.
boxd: ran 4 vCPU, 16 GB, not 4 vCPU, 8 GB. boxd has no 8 GB size at 4 vCPU; this is its nearest.
Hollow points are list-price estimates, not billing: Freestyle.
Boat and boxd run in Europe, so every call from our Florida server crosses the Atlantic.
Boat: coming in the next update. Our Boat plan allows 200 creates a day, and this run used them up; a higher plan or quota may lift that.
Superserve: not in this track: our Superserve plan caps templates at 2 vCPU, so there was no 4 vCPU machine to test. A paid plan may lift this; we will add it if it does.

Method

How the numbers were made, and what could bias them.

This release: Default-machine timings come from one run on one harness commit. Snapshots, bursts, the same-machine track and cost per task come from runs on a later harness commit whose changes only touched cleanup. Tenki's same-machine run stopped early at 78 of 100 cold starts; each chart shows its attempt count. A burst size is published for a provider only when all three of its bursts finished, and a chart names the reason for any provider it leaves out. Failures caused by our own code are set aside, not counted. Boat will be added in the next update.
One SDK: Every provider ran through Sandbox SDK, our own TypeScript SDK. Version 0.2.0 is public on npm, and the 1.0 used here will be published soon. Each adapter calls the provider's official SDK or API and uses its native path for commands and files where one exists.
Cold start: Timed from create() until a bare true command succeeds in the new sandbox: the provider's own time. Our SDK's setup after that point is timed separately and shown as Ready through Sandbox SDK. It is never part of cold start or bursts.
Superserve: Superserve ran a template we built with Node 22 and Python 3.11 preinstalled. Superserve offers a Node template (node-22) and a Python template (python-3.11), but our tests need both in one sandbox; before this template, our setup installed Python on every create.
Two tracks: Default machines: each provider's default sandbox, so sizes differ. E2B's default base template is 2 vCPU and 512 MB, though its pricing page says 4 GB. Same machine: every provider asked for 4 vCPU and 8 GB, Boat's default. boxd has no such size and ran 4 vCPU, 16 GB, its nearest. Superserve is left out because its account caps templates at 2 vCPU. Machines that differ from the track are marked on every chart.
Billing: Cost per task comes from a separate cost run: 10 sandboxes per provider, each created, running the 100-call tool replay once and deleted, so one sandbox's charge is the cost of one task. The main benchmark sandboxes are not priced, because providers ran different mixes of tests in them. Billed is the provider's own per-sandbox charge (Upstash Box; Daytona, whose bill can still change for up to 48 hours). Metered is the provider's own recorded usage times its published rates (E2B and boxd usage meters; Tenki's server-recorded session start and end). Estimates use the provider's published per-second rates and billing rules on our measured create-to-delete time, because there is no per-sandbox billing data (Freestyle has no billing API; Superserve reports usage only per account). Credits, tax and plan minimums are left out. One task costs a fraction of a cent, so charts show the cost of 1,000 tasks; results.json and results.md keep the per-task amount too.
One location: Every run started from one server in Florida, US. Distance to each provider's region affects every call; the Hardware tab under Machines shows where each sandbox ran.
Plans and limits: We used each provider's self-serve account. Where a plan or account limit changed what we could test, the chart says so: Superserve's plan caps templates at 2 vCPU, so it is not in the 4 vCPU track; Boat's plan allows 200 creates a day; Daytona's account quota of 10 vCPU kept its 4 vCPU snapshot and burst tests from running. A paid plan or a quota increase may lift these, and we will add the results when it does.
Attempts: 100 runs per quick test, 30 per agent task and 30 per snapshot test. Bursts: 3 of each size, each timed until every sandbox in it runs a command.
Independent: No provider paid for these results or had any say in them. Some may have seen an early copy to check for errors. We may work with some of these companies; that did not change how tests ran or how results are reported.
Failures: Only failures the provider caused count against it. We set aside failures our own code caused and ones we can't pin on either side, such as our SDK timing out while waiting. Attempts blocked by our own plan limits were rerun. Set aside:
Default machines, boxd: 1 run, 21 tests. Our ownership check failed during one burst of 25 (16 creates, 5 deletes), so that burst is not counted.
Default machines, Freestyle: 8 runs, 8 tests. Our SDK timed out or wrapped the error moving 10 MB, so we can't tell whose side failed.
Default machines, Tenki: 1 run, 2 tests. Our cleanup check could not confirm the sandbox after Tenki's snapshot failed, so we can't tell whose side the delete failure was on.
Same machine, Upstash Box: 3 runs, 5 tests. Our ownership check could not confirm the sandbox before deleting it, so we can't tell whose side failed.
Same machine, Upstash Box: 3 runs, 4 tests. Our ownership check failed on 4 snapshot cleanup steps; none is counted.
Same machine, Upstash Box: 3 runs, 3 tests. Our ownership check failed on 3 burst deletes; none is counted.
Same machine, boxd: 12 runs, 12 tests. Our ownership check failed on 4 snapshot attempts, and our 30-second cleanup deadline ran out 8 times; none is counted.
Same machine, boxd: 4 runs, 14 tests. Our ownership check failed on 1 burst create and 13 burst deletes; none is counted.
Same machine, Freestyle: 3 runs, 3 tests. Our SDK timed out or wrapped the error moving 10 MB, so we can't tell whose side failed.
Same machine, Tenki: 1 run, 1 test. The run was stopped during this attempt, so it is not counted.
Ties: Medians within each other's 95% confidence intervals are effectively tied. Focus or hover a bar for its interval and attempt count.
Left out: Download speed, because the public test host rate-limited several providers.

Cloud sandboxes, measured

Cold start→

Agent work, 100 tool calls→

Cost per 1,000 tasks→

Start from snapshot→

Cold start→

Agent work, 100 tool calls→

Cost per 1,000 tasks→

Start from snapshot→

Where the time goes

Start-up

Cold start

Ready through Sandbox SDK

Every cold start

Cold start

Ready through Sandbox SDK

Every cold start

Agent work

Agent tool replay

npm install

pip install

git clone

Dev server ready

Preview URL first byte

Edit, rebuild, reload

Agent tool replay

npm install

pip install

git clone

Dev server ready

Preview URL first byte

Edit, rebuild, reload

Everyday calls

Run a command

Write + read 1 KB

Write + read 10 MB

Reconnect

Destroy

Run a command

Write + read 1 KB

Write + read 10 MB

Reconnect

Destroy

Pause & snapshots

Resume

Pause

Pause + resume

Take snapshot

Start from snapshot

Resume

Pause

Pause + resume

Take snapshot

Start from snapshot

Bursts

10 at once, all ready

25 at once, all ready

10 at once, all ready

25 at once, all ready

Machines

What each provider ran

Default machine size

vCPU

RAM

Disk

Hardware inside each sandbox

CPU, one core

CPU, all cores

Memory bandwidth

Disk write

Disk read

4K random write

4K random read

What each provider ran

Hardware inside each sandbox

CPU, one core

CPU, all cores

Memory bandwidth

Disk write

Disk read

Cold start

Agent work, 100 tool calls

Cost per 1,000 tasks

Start from snapshot

Cold start

Agent work, 100 tool calls

Cost per 1,000 tasks

Start from snapshot