ExpressVPN Trust Center
Four key strategies
In general, our approach to security is four-fold: Make a system very difficult to compromise; minimize the damage if it is compromised; minimize the time it can stay compromised; validate all of the above, internally and externally.
To illustrate these four goals, let’s think about security at a physical location, like a castle. How might you defend a castle from thieves?
1. Make it very difficult to compromise
High walls. Steel gates. A moat filled with bloodthirsty crocodiles. These security features of a castle make it very difficult for thieves to penetrate.
2. Minimize the damage if it is compromised
If thieves do manage to enter the castle, they shouldn’t be able to steal much. The treasure room should be kept as bare as possible.
3. Minimize the time it can stay compromised
Have guards patrol the castle at regular intervals. If anyone tries to steal anything, it won’t be long before they get ejected.
4. Validate all of the above, internally and externally
How secure is the castle? You can’t know for sure unless you practice attacking it. Better yet, invite qualified outsiders to do the same and report the results.
Security at ExpressVPN: General applications
So how do we apply these strategies in practice? Here are some security measures common to multiple systems at ExpressVPN:
1. Making things difficult to hack
Several of our systems automatically destroy and rebuild their infrastructure, pulling in the latest patches, starting with the OS, daily. This automatically ensures we’re running the latest patches where those patches come from package managers.
We have teams responsible for subscribing to CVE announcements for all relevant software used by a given system, triaging the announcements, and applying patches quickly where necessary. This ensures we run the latest patches, even where those patches need to be applied manually, such as for custom-built software.
We set each system’s firewall to reject all traffic by default and only allow intended traffic types.
SSH access is restricted to modern protocols and only key-based authentication, using an IP whitelist, and bastion boxes. SSH keys for many systems are protected with YubiKeys and require a physical touch to log in.
We treat data as a toxic asset.
We maintain strong IT policies, well-explained and enforced.
- Even when hiring for non-technical positions, we look for candidates with an appreciation for, and interest in, security.
We use YubiKeys for GitHub commits, website logins, and SSH.
We require per-person password-protected SSH keys.
Where applicable we use dedicated hardware to access production systems, separate from daily-use workstations.
We prefer using disposable VMs for web browsing.
We require all disks to be encrypted.
We enforce a BIOS boot password, and disable booting from USB.
Provisioning and deployment:
- We use automatic provisioning and deployment of infrastructure and applications. These reduce the need for humans to access production systems directly. For example, our website application deploys itself automatically after any merge into the Git master branch and after all automatic tests have passed. This happens dozens of times per day. In addition, the infrastructure running our website destroys and rebuilds itself automatically once per day, ensuring we’re running the latest patches starting from the OS up.
- We test requirements automatically, including security and privacy requirements (for example, “assert that a system does not try to write to disk, or if it does need to write its internal application log, assert that it does not contain email addresses”). Our hiring and training processes place great emphasis on engineers being able to express requirements as automated tests.
- To guard against the risk of malicious changes to our code, code reviews are required for all changes. For any potentially sensitive repository, git doesn’t allow pushes directly to master. All changes are made in a branch first, and another person must review the pull-request before merging. Read more about our build verification system.
For software that we write ourselves, we design it to be difficult to hack and use extensive testing to convince ourselves that’s true. The methods are specific to the technology stack used in each system, and the sections below will provide more detail.
Some scanning tools are incorporated into continuous integration (CI) processes.
2. Minimizing potential damage
- Security and privacy threat modeling is incorporated into the design phase of any system, as early as possible. Some of our engineering leaders are Microsoft alumni and use the STRIDE method. We ask ourselves, “If this system gets hacked, what’s the worst an attacker can do?” Then we adjust the design to eliminate threats (ideally by not having sensitive data at all) or mitigate them effectively (such as by encrypting or hashing those data).
3. Minimizing the amount of time that a system can remain compromised
Intrusion detection systems.
Automatic reboots—useful for some of our systems that are running read-only from memory (RAM).
Automatic destruction and rebuilding of infrastructure.
Bug bounty program: We invite security researchers to test our systems and receive financial rewards for any problems they find.
Penetration tests: We run tests both by employees and by specialized paid contractors looking for security vulnerabilities.
Automated tests: We design automated tests that assert requirements such as “output from this application may never contain the following types of data,” run with every commit as part of our CI processes. We require all tests to pass before any deployment.
Internal audits: We perform regular audits to confirm compliance with policy (for example: if it’s actually true that a system requires a YubiKey touch to establish an SSH connection).
Security at ExpressVPN: Applications by system
The following section details how security is implemented across each system:
ExpressVPN operates thousands of VPN servers, organized into 160 clusters (server locations) covering 94 countries around the world.
There are people responsible for triaging CVEs and ensuring patch speed.
All deployment and provisioning is automated with Ansible.
Our configuration management system can schedule and execute rolling patch applications and reboots.
SSH hardening: IP whitelist allows SSH via bastion hosts, using limited protocols, and key-based authentication only.
Authentication with YubiKeys that require physical touch for a human to SSH into a system.
Automatic deployments and provisioning, with self-revoking credentials.
Strong security settings on the VPN itself:
Unique secrets per-server.
Weak authentication protocols are disabled.
Strong encryption and hashes.
Clients use strong authentication on servers. They expect both a signature from our CA, as well as a specific common name for a given server. This protects against stolen keys from one server being used to impersonate another server. Also, we can revoke server certificates in less than an hour.
GitHub scanning for vulnerable components.
Static analysis tools running with every commit as part of our CI system.
Code reviews are mandatory for every change.
VPN servers don’t store user data. They don’t have access to, and are physically segregated from, other systems. (This claim was recently validated in a high-profile case with Turkish law enforcement, who gained access to a server physically but were unable to find user data.)
All services and operations run under the least privileged model.
User authentication credentials are strongly hashed.
We write automated tests to assert requirements such as “strongSwan must not log client IP addresses.” These are run as part of our CI system on every commit.
ExpressVPN builds apps for Windows, Mac, Android, iOS, Linux, and routers, as well as browser extensions. Each of these must provide quick access to our VPN server network while also preventing privacy leaks that might be unique to its native platform.
VPN leak-proofing: All VPN applications must deal with the threat of accidentally “leaking” the user’s traffic outside of the tunnel, or giving attackers methods to discover the user’s real IP address. ExpressVPN puts significant effort into proving to ourselves that we do not leak. We have developed an open-source suite of tools to generate and detect such leaks. We believe these tools have been helpful in raising the quality bar for the entire VPN industry. For details, please read about the ExpressVPN leak testing tools.
For apps that have components running as admin (typically needed to control the firewall), there is the threat of local privilege escalation. ExpressVPN apps protect against that class of threats with strong authentication, file system protection, and exposing only a limited set of safe actions to non-administrative users.
There are tampering and spoofing threats when the apps call server-side APIs to authenticate or discover the set of available VPN infrastructure. We mitigate these threats by:
Pinning the CA and common name for all API calls.
For OpenVPN, the client apps authenticate each specific VPN server based on a private CA signature and common name. Stolen keys from one server cannot be used to impersonate another server. We can revoke keys within an hour.
When security issues are discovered, we promptly release fixes and issue updated apps to our customers. We communicate important security information and updates on our blog and through messages within ExpressVPN apps.
In addition to the VPN itself, ExpressVPN apps also connect to ExpressVPN API servers. These servers deliver functionality like logging in, discovering available VPN locations, and contacting customer support.
API servers are rebuilt automatically several times per week, which achieves:
Fast patching of platform components.
Limiting the potential length of time that an attacker can have persistence.
On the latest versions of our apps, user-provided credentials are encrypted asymmetrically by the ExpressVPN apps, then passed through by API servers to downstream systems. The internet-facing API servers do not have access to those credentials, thus reducing the potential damage if those servers were compromised.
Deployment and provisioning are fully automated.
Backend CRM and billing
These are the systems that maintain our database of customers (including their email addresses and account status), handle payments, and allow support staff to interact with these data.
Backend systems are firewalled off and use IP whitelists with reverse proxies, some of which:
Require browser-based certificate client authentication.
Decode incoming HTTPS requests from upstream systems, perform validation, then pass them downstream. This removes a class of attack vectors in platform components such as the webserver application.
SSH via bastion boxes.
Various backend systems communicating with each other:
Queue messages are encrypted and signed with asymmetric keys.
Synchronous API calls require HTTPS, pinned to a private CA.
ExpressVPN maintains an official website with an internet privacy and security blog, a support section, and customer account management pages.
We automatically destroy servers and rebuild them clean from the OS up with latest patches on a daily basis, as described in this blog post.
By design, our website servers do not have direct access to user databases. They interact with backend systems via APIs that expose minimal functionality. For example, there is no feature to “give me a list of all users.” Specific actions such as “show me information about this specific user needed to render the user’s website dashboard” require the user’s password or session-cookie. Thus, if attackers gained root access to a website server, they would only be able to attack data for users who are actively logged in to the ExpressVPN website via that server at that moment.
Our internet-facing web application is relatively lightweight. Very little business logic lives in this application, thus reducing its attack profile. It mainly interacts via backend systems using APIs and queues.
Queue-messages exchanged between frontend and backend are encrypted and signed using asymmetric keys.
Synchronous APIs to reach backend systems require HTTPS with a private CA and use reverse proxies and IP whitelists.
All web requests are served via AWS Cloudfront acting as a reverse proxy, reducing a class of platform-related attack vectors.
Deployment and provisioning are fully automated.
All internet-facing systems are scanned by detectify.com daily.
Excellent customer support from real people is one of ExpressVPN’s greatest assets. Obviously, putting real people in contact with our users must be handled with extreme care. Every aspect of hiring, training, and managing our support staff is handled directly by ExpressVPN. We view customer support as a core competency, both to deliver top-quality service and to be able to sufficiently protect our customer data.
Each customer-support agent has two environments, both fully virtualized and accessed via thin clients, and each boot goes into a clean image:
Secure. Can access only sensitive systems, no general web browsing.
Unsecure. Can do general web browsing, except to sensitive systems. This is where agents would respond to phishing attempts. This environment is always assumed to be hacked and untrusted.
The web application used by customer support to administer customer accounts (e.g., issue payment refunds) has an IP whitelist in the load balancer, and its reverse-proxies require certificate-based client authentication of browsers. Only the agent’s secure environment can access this system.
Our backend application keeps an audit trail of all actions performed by agents.
Access to the physical support office is monitored with CCTV and controlled with biometric authentication.
“Hacking the employees” is a common vector into an organization. Protecting daily-use workstations, especially those that read email and do general web-browsing, is extremely difficult. We try to make attackers’ work as hard as possible, but also design our systems and workflows such that a compromised workstation cannot do much damage. We operate on the principle that we can generally trust the people on our team, but not their laptops.
Physical and logical network isolation of various types of workstations and servers (for example, separating the marketing team from our VPN leak testing lab).
Virtualized desktops, often split into secure/unsecure environments per person.
Dedicated hardware for sensitive access.
A detailed IT policy with maximally automatic enforcement:
BIOS boot password.
Disable booting from USB.
Multi-factor authentication, often with YubiKeys requiring physical touch.
No exporting of personally identifiable information (e.g., customer email addresses) away from production systems.
Password managers to store credentials.
Preference for disposable VMs for web browsing.
PGP email for sensitive communication.
Signals of Trustworthy VPNs: an industry initiative
When users choose a VPN, they are trusting the provider with their online privacy and security. Whether they are ExpressVPN users or not, we believe that everyone should have the information and guidance needed to evaluate whom they can trust.
That’s why we’ve worked with the Center for Democracy and Technology—an independent non-profit organization that champions online civil liberties and human rights around the world—to develop a list of questions that VPN services should be able to answer to signal their trustworthiness.
We’ve published those questions here, along with our responses, and we welcome all VPN providers to join us in answering these questions and raising standards for the entire industry.