All articles

The nightly run took numbers and never waited for a code: how automated tests quietly cut your concurrent order limit

A nightly CI run failed three tests out of twenty. In the morning, a manager opens the dashboard. The 7-day share of orders with a code has dropped, and the current concurrent order limit is lower than it was. The money for orders without a code went back to the balance on its own, so the problem doesn't show up in spending. The cause is usually one of three pipeline mistakes, and you fix all three in the test code.

How a run eats into the limit

How many orders an account can keep open at the same time depends on its reputation. The base ceiling is 100. The dashboard gets the current limit and current usage from the server, and it also shows the share of orders with a code over the last 7 days. That's the number to keep high so the limit doesn't get cut.

A run hits both metrics. First, usage. An open order holds a slot in the limit until it's closed or until its time runs out. That time is set per service in the catalog and is currently one hour. For the whole period, the money for the order sits in reserve. Second, reputation. Every order closed without a code lowers the share of orders with a code. The window is seven days, and a week of nightly runs fills it completely.

The number is ordered before the app is ready to send the code

Here's a common pattern. The number is ordered in a setup fixture or in beforeAll so it's ready when needed. The order's clock starts at getNumber, but the test is still waiting for the environment: containers are starting, migrations are running. If the environment doesn't come up or the test gets skipped, the SMS is never sent, and the order expires empty.

Call getNumber at the exact step after which the app immediately sends a code to that number. Check that the environment is ready before you order. For setup, getPrices and getBalance are all you need. They don't order anything and don't take up a slot in the limit.

The test failed, but the order stayed open

The test framework has its own timeout, and the order has its own. The test waited a couple of minutes for the code, failed, and the process exited. The order, meanwhile, stays alive until its time runs out. It takes up a slot, holds the reserve, and if the code never arrives, it closes empty.

Close every order explicitly in teardown, whatever the test outcome. Code received: setStatus 6, and the order is completed. No code: setStatus 8, and the order is canceled. Canceling doesn't make the order successful, but it frees the slot and the reserve right away. An order can't be canceled during its first minute. So if the test fails earlier than that, teardown should wait and retry the cancellation instead of swallowing the error.

Retries are a separate trap. The runner reruns the failed test from scratch, the test calls getNumber again, and one scenario ends up with two or three orders. At best, one of them gets a code. For tests that use a number, turn off whole-test retries or pass the already open order into the retry. To request another code on the same number, use setStatus 3.

More parallelism than you need

The runner config has 32 workers because that's how many cores the CI machine has. If each worker holds its own number, a single run takes 32 slots, and with a reduced limit that's almost the entire limit. Manual orders your colleagues place from the same account share that usage with the run.

The second effect is polling. Twenty tests, each calling getStatus once a second, add up to 1200 requests per minute. The API limit is 600 requests per minute per key, which is half that. Requests over the limit get rejected, and a test that reads a rejection as "no code" abandons the order. Poll every few seconds and budget your requests for the whole run.

Move tests that need a number into a separate group with lower parallelism. For CI, create a separate named key in the dashboard and put it in your secrets store right away, because the key is shown only once. The last-used date tells you whether the schedule actually ran, and revoking this key won't affect your main integration. The key doesn't come with its own order limit, though. The account has a single usage pool.

Before the next run

The same rules apply to manual orders. Take a number when the service is already ready to send the code, and close the order yourself instead of waiting for it to expire.

Pipeline checklist:

  • getNumber sits at the step after which the app sends the code;
  • teardown closes every order with setStatus 6 or 8 and retries the cancellation if the test failed in the first minute;
  • a retry doesn't order a new number, and a repeat code goes through setStatus 3;
  • there are no more workers holding numbers than tests that need a number;
  • total getStatus polling stays within 600 requests per minute per key.

Before adding workers, open the dashboard and check the current limit, current usage, and the 7-day share of orders with a code. If the share dropped after nightly runs, we recommend starting with where getNumber is called and with teardown, and raising parallelism only after that.

Your first number in a minute

Signing up takes a minute, topping up a couple more, and the code arrives in 20 seconds.

Buy a number
No subscription feeCancel with a refund in the first hour