How We Improved Moove's Performance By 59%

Distance was our slowest dependency

Most people who open the Moove app are on mobile networks in Nigeria and Indonesia, thousands of kilometres from the servers that answer them. This month we changed how their requests reach us, and our time to first byte (TTFB), the time between a phone sending a request and the first piece of the answer arriving, fell by 59%, from 0.74 seconds to 0.31 seconds. We did not move a single server to get there.

This post explains why distance cost us more than we expected, what we changed, the part that turned out to be harder than the change itself, and what we would tell another team about to do the same.

Why one long trip becomes three

Engineers call the delay between sending something and it arriving latency, and the time for a message to reach a server and for the reply to come back the round-trip time (RTT). From Lagos to our servers, the RTT is between 200 and 300 milliseconds. On its own, that is tolerable. The problem is that a request rarely travels alone.

Before a phone can ask for anything over a secure connection, it has to complete two handshakes, the short exchanges that set a connection up. The TCP handshake opens the connection, and the TLS handshake then agrees the encryption keys that keep it private. Each one costs a full round trip. Only then does the request itself go out, which is a third trip. At 250 milliseconds a trip, a user waits roughly three quarters of a second before our servers have done any work at all.

On mobile networks, that cost is paid far more often than you would hope. Phones move between cell towers, radios go to sleep to save battery, and apps are sent to the background. Each of those events can drop the connection, and the next request has to start from a cold connection, one with no handshakes done yet, and pay the full setup cost again. A warm connection, already open and secured, skips straight to the request.

We could not move Lagos closer to our servers. We could move the setup closer to Lagos.

Moving the handshake, not the servers

A content delivery network (CDN) is usually described as a cache: a set of locations around the world that keep copies of files so they can be served from nearby. We used one for a different reason. For our API, the interface the app talks to for everything it does, caching is switched off entirely. Every request still travels to our origin, the CDN's name for the servers behind it.

What changes is where the connection starts. A phone now opens its connection to the nearest edge location, also called a point of presence (PoP): one of the CDN's sites around the world, often tens of milliseconds away rather than hundreds. The edge performs TLS termination, meaning the secure connection from the phone ends there. It then reuses a pool of warm connections to our origin, kept open with keep-alive, a setting that leaves a connection open between requests instead of closing it. The two expensive setup trips happen over the short hop, and only the request itself crosses the ocean, on a connection that is already warm.

Three long trips become one long trip and two short ones. That is the whole idea.

We kept the rest deliberately boring:

• Clients noticed nothing. The app and the website kept the same addresses, so no app release was needed and nobody had to update anything.
• Failure behaviour stayed the same. Timeouts match what was there before, so anything that failed before still fails in the same way. We wanted no new kinds of error.
• The edge closes idle connections first. Every connection has an idle timeout, how long it can sit unused before it is closed. If our servers closed a quiet connection while the edge still thought it was open, the next request would fail. Setting the edge's timeout shorter means it always gives up first and opens a fresh connection instead.
• Newer protocols for phones. Phones can connect over HTTP/3, the latest version of the web's request protocol. It runs on QUIC, which folds both handshakes into a single round trip and copes better with the packet loss common on mobile networks.
• No region left out. Cheaper tiers of an edge network often skip locations in exactly the parts of the world where our users live, so we chose full coverage.

What we measured

We measured from Sydney, which is roughly as far from our servers as Lagos is, comparing the old path with the new one. For each, we opened five fresh cold connections, the case that matters most on mobile networks, and took the median, also written p50: the middle result, which half the requests beat and half do not. The same project also switched on compression for our website's scripts, which had been sent at full size. Compression shrinks a file before it is sent and the browser expands it again on arrival; we used Brotli, a compression format built for the web.

Time to first byte is where the 59% comes from. A connection that used to take half a second to set up is now ready in under a tenth of one, and a first visit to our website now downloads two thirds less, on connections where many people pay for every megabyte.

These figures come from one city, not a global benchmark. The arithmetic carries over to Lagos and Jakarta, though, and that is where our users are.

The hard part was knowing who is calling

Putting something in front of your servers is easy. Keeping your protections working behind it is where most of the effort went.

Like most payment platforms, Moove uses rate limiting, a cap on how many requests any one caller can make in a given time, and those limits are keyed on the caller's IP address, the number that identifies a device on the internet. Behind an edge network, every request arrives from an edge location's address instead of the user's. If nothing else had changed, everyone in Lagos would have shared one budget, and one busy afternoon would have started turning away people who had done nothing wrong.

The fix rests on two ideas.

Trust exactly the layers you have. A CDN is a kind of reverse proxy, a server that sits in front of yours and passes requests on. Each proxy adds the address it received a request from to a standard header called X-Forwarded-For, so the real caller sits a fixed number of steps, or hops, back in that list. Getting that number wrong is a bug in both directions. Count too few layers and every user looks like their edge location. Count too many and you start trusting an entry the caller wrote themselves. So we made the number a property of the infrastructure rather than of the code. It changes at the same moment the network does, and our service refuses to start if the value makes no sense, rather than guessing.

Let the edge vouch for the caller, and believe only the edge. The edge now writes the real caller's address onto every request and overwrites anything the caller sent in its place. Our protections only believe that value when they can verify the request really came through our edge. Every limit that used to key on the network address now keys on the real caller for edge traffic. We added these rules alongside the old ones instead of changing them, so each change in the rollout was a pure addition that anyone could review in one read.

A bug we caught before it shipped

Moove Profiles can live on a merchant's own domain. Requests for those domains were passed through our website with the merchant's domain attached, so the site knew which profile to show.

The new edge in front of the website strips that kind of information from incoming requests, which is exactly right, because a visitor should never be able to set it. It also meant that once the website moved behind the edge, every merchant domain would have quietly shown the Moove home page instead of the merchant's profile.

We caught it in review, moved the merchant-domain path to its own route first, and made our infrastructure refuse to apply the website change anywhere that move had not happened. A mistake that once lived only in a reviewer's head is now something the build stops.

The general lesson: when you add a layer that cleans up inputs, look for everything that depended on the inputs you are about to clean.

How we rolled it out

Payments infrastructure does not get "deploy and see", so we changed staging first and production after, and a few choices kept the cutover uneventful. We lowered the TTL (time to live) on our DNS records, the internet's directory that turns a name like moove.xyz into a server address. The TTL is how long the rest of the internet may cache that answer, and we cut it from an hour to five minutes, a full hour before anything moved, so the old answer had expired everywhere and a rollback would take effect within minutes. We kept address records in a form that could be switched in place, because removing and re-creating them would have left a gap, and a gap is an outage. Each risky step was its own small change, with the steps to undo it written down in order.

What we would tell another team

If your users are far from your servers, an edge network is worth having even when it caches nothing. Before you cut over:

• List everything that reads the caller's address, including rate limits, allowlists, fraud signals, audit logs and security rules. Each of them sees the edge's address unless you change it.
• Make the number of trusted layers part of your infrastructure, and refuse to start when it is wrong.
• Let the edge record the real caller, and trust that record only on traffic you can verify came from your edge.
• Make sure your servers accept traffic only through the edge, and watch before you enforce.
• If you cache pages, build the cache key, the set of request details a cache uses to decide whether two requests can share one stored response, from everything the response actually varies on. Servers list those details in a Vary header, so read it rather than relying on memory.
• Measure before and after, and look at handshake time and time to first byte separately, so you can tell connection setup apart from the time your servers spend working.

Looking ahead

Most of our pages are personalised, so the website's edge cache mostly sits idle for now. The connection improvement applies to every page regardless, and individual pages can now opt into caching when it is safe to share them. Caching API responses would need its own design and review, and it is the next thing we want to look at.

Money should move at the speed of the network, not the speed of light across an ocean.

If you build on top of Moove, read our introduction to Web3 payment APIs and SDKs and the Moove Developer Program.

Written by the moove.xyz engineering team.

About moove.xyz

moove.xyz is a global Web3 fintech platform built for the permissionless and effortless movement of value. We empower businesses and consumers anywhere to send, receive, stake, and swap any cryptocurrencies across any blockchains, all in one single platform.

We are one of the first Web3 fintech companies globally to innovate and build a full-stack crypto payments and decentralised finance infrastructure, enabling an integrated and comprehensive coverage across multi-chain wallet access, personalised wallet handles, cross-chain token swaps, embedded cross-chain transactions and a decentralised social financial network. Our key products include Moove Profile, Moove Send, Moove Receive, Moove Stake, Moove Swap, Moove Rewards, Moove Discover and more.

Our mission is simple — to create and distribute permissionless and effortless financial technology for the next 1 billion Web3 users. We fundamentally believe that the future of the movement of money and value shall be costless, borderless, permissionless, effortless, and built for everyone — and we're building the ultimate Web3 fintech platform to make that future real.

Your money. Your move.