Reverse engineering private APIs

We would all prefer to have access to a public API but sometimes, services you would love to add automation to only offer their website to use.

There is no public API, now what?

First let’s state that not all mobile or web applications can be (easily) reversed engineered, some companies invest a lot of money to secure and protect their API, user data. I’m not a hacker, but sometimes being resourceful let you create scripts and automations that can make you save tons of time. Note that private API, as in not documented, are usually not intended to be used by others than developers internal to the service or product, so don’t count on it to be stable, don’t expect to receive a notice if something is going to break, don’t base your entire business on something that can disappear any day.

In some cases, web browser inspectors comes handy. With the surge nowadays of websites developed using React, or more generally client side web applications, the network inspector of your browser often shows all the requests the website is doing in the background in response to your actions on the interface. Coupled with a HTTP proxy like Charles to inspect traffic going through your phone, you can scan pretty much all endpoints a service is using and try to mimic this behavior in scripts.

What’s the catch?

There is often only one problem: authentication. Most web services nowadays are using JWT session tokens, passed in the headers of each authenticated request, and even though you could re-execute a request you copied from your network inspector / HTTP proxy using the same URL, parameters and headers, this session token will eventually expire after an hour (a day at best depending on how their authentication is configured) and you will have to manually extract your access token again and again.

To be honest, I’m often able to reverse engineer everything but the authentication, as a user, I’m fine with that 🙂

Refreshing tokens

This is the best case scenario: if you can sniff and extract the refresh token from the original authentication request, and reverse engineer the token retrieval request, then you can generate a fresh access token every time your script run.

Let the website refresh tokens for you

Chances are the website is refreshing its own access token periodically as well as when launching it, this provides a seamless experience without having to log in every time. Obviously websites trying to secure sensitive data like banks are not doing this, but a lot of other consumer websites and tools are.

In that case, it’s pretty common for these web applications to store access tokens and / or refresh tokens in their local storage. You can see the content of the local storage using the “application” tab of the web inspector (Google Chrome), sometimes this information can also be stored in cookies.

Final note

Don’t be evil.

Remember that you can be in trouble for doing that, only use these techniques to automate something you have the right to do.

Localize your app early

And when I say early, I mean immediately, as soon as you create your project. Trust me, just do it, take the few minutes it takes to setup localization and do it from the first word that will appear on your interface.

Regardless of your app supporting several languages at launch, doing so will not consume additional time, and these few moments will literally save you hours later on, when you actually have to translate the app.

Consistent keys

When adding a new localized key, I always group it with other keys of the same screen and use the same naming convention: snake case + name of the screen followed by a dot and followed by a description of what the key is, eg:

// EventList
"event_list.title" = "Events";
"event_list.new" = "Create Event";

There are many systems out there, find your own and stick to it.

Take shortcuts

Especially if you don’t have to, using NSLocalizedString(_:comment:) everywhere and in the proper way can be cumbersome. So unless your team and process requires it (and in most apps it will certainly not be required), you can take a small shortcut that will greatly help: build an extension to wrap the function and make the comment parameter optional.

//  String+i18n.swift

import Foundation

extension String {
    func i18n(comment: String = "") -> String {
        return NSLocalizedString(self, comment: comment)
    }
}

It’s makes it really to use by calling the i18n() method on the key.

class EventListViewController: UIViewController {

    override func viewDidLoad() {
        super.viewDidLoad()

        // i18n
        title = "event_list.title".i18n()
    }
}

Automate

You are now supporting several languages, nice, it’s time to automate! Please don’t open the string file in each language to add your keys, this is the best way to make mistakes.

Instead, for small to medium projects, you can easily create a shared document on Google Spreadsheet with a “key” column and a column for each supported language, and write a small script to download the sheet as a CSV file and generate one .string file per language.

Good habits makes your life easier

What if I’m doing a really quick and dirty test and don’t want to take the few seconds to write the localizable for it? I understand, and I do it sometimes, in this case, just make sure to:

  1. take note that this piece of code will need localization later
  2. be consistent so that you can retrieve all the places that need localization in one go.

I usually add a // TODO: i18n before or after the code using hard-coded text, it’s easy, and because I always use the same text in the comment, all the occurrences will show up in one project search.

Please use MARK pre-processor directive

In Xcode, the // MARK: and // MARK: - pre-processor directive allows you to create separations in your Swift source files. This directive does not change the behavior of your code, it’s just here to help organizing it.

When, why and how should I use them?

You should use marks to separate the code of your source file into sections. These sections should represent logic units and/or group of declarations (outlets / functions) that are going with each other.

By grouping your code this way, you will often come to realize something doesn’t belong to any category: in this case, it probably belongs elsewhere than in this class, this is a good signal that you could refactor this function or piece of code into another file / class.

Additionally if you end up having groups named “utils” or “utilities”, it’s again a good signal a refactor is needed.

Finally in Xcode 11, the mini-map will highlight these marks, using them will therefore become more important than ever.

Difference between // MARK: and // MARK: –

The difference is subtle but exists, with the added - at the end of // MARK:, Xcode symbol explorer adds a separator before the mark, so when clicking on your symbols in this bar:

it will look like this:

instead of this:

I personally only use the // MARK: - syntax.

Create a snippet

Creating a Xcode snippet allows you to add marks faster and ensure you are using a consistent format thanks to auto-completion.

To create a snippet in Xcode:

  1. from the status bar, click on “Editor” then “Create Code Snippet”
  2. give a name to your snippet (eg: MARK)
  3. Choose a “Completion Shortcut” keyword, I use lowercase “mark” so that writing the beginning of the word triggers the auto-completion
  4. Choose “All” for Completion Scopes
  5. Enter // MARK: - <#category#> in the text box

To use the snippet, start tapping “mark” in your code editor and let the auto-completion do the work.

Dependency Injection in Swift with the help of Sourcery

Dependency Injection is a really simple concept allowing among other things to decouple the different components of your application’s code (and it’s not just about getting rid of these nasty shared instances!).

There is already a ton of resources online that talks about it. There are also different implementations for Swift projects, but not of them were quite fitting to my needs as they are often:
– done at compile time, meaning that your app can crush at runtime if you mis-configured something and you could notice it too late when your app is already released in production
– not type safe, passing optionals all around
– asking for ton of boilerplate
– not that easy to use when it comes to testing
– not very opinionated on what belong to a centralized dependency injection container and what should just be passed around

In a decently sized project I’m working on, transitioning from all the Singleton design pattern used all over the project with shared instances representing most of my controllers, managers and services to something where dependencies are properly injected asks for a big deal of time and re-architecting.

The following is an experimental solution, even though it works in a small project, it’s really easy to generate cyclic dependencies and have your project crash at run-time — do not use in production.

Among all possible dependency injections (initializers, properties, methods, ambient, etc.), I believe passing down dependencies during components initialization is the cleanest way as it allows compile-time enforcement and remove the need for optionals and/or force unwrapping. Sadly, passing down dependencies from where there are likely being initialized (often the root of the application, like the AppDelegate) throughout the app isn’t really something you would consider clean. For example, if a component six levels down the navigation needs a particular dependency to function, you would have to pass it through the five first view controllers for it to be available during the sixth component initialization, this isn’t really elegant as most of these view controllers probably won’t need it and it’s simply tightening up the whole dependency graph for nothing.

With the rise of the Coordinator design pattern, it become easier to centrally manage dependencies. Coordinators is the one place responsible for creating view models and view controllers and can therefore pass on the dependencies in initializers. This is convenient and already cleaner, but let’s imagine this view that requires a lot of dependencies:

class SomeViewModel {
  // MARK: - Initializers
  init(dependencyA: DependencyA, dependencyB: DependencyB, dependencyC: dependencyC, ..., dependencyX: DependencyX) {
    self.dependencyA = dependencyA
    // ...
  }

  // MARK: - Dependencies
  let dependencyA: DependencyA
  let dependencyB: DependencyB
  let dependencyC: DependencyC
  ...
  let dependencyX: DependencyX
}

Now let’s look at the component initializing this view model:

class SomeCoordinator {
  let dependencyContainer: DependencyContainer // For our own sake, let's at least imagine dependencies are centralized somewhere

  func showSomeViewController() {
    let viewModel = SomeViewModel(dependencyA: dependencyContainer.dependencyA,
                                  dependencyB: dependencyContainer.dependencyB,
                                  dependencyD: dependencyContainer.dependencyD,
                                  ...
                                  dependencyX: dependencyContainer.dependencyX)
    let viewController = SomeViewController(viewModel: viewModel)
    navigationController.push(viewController: viewController, animated: true)
  }
}

By now, you must certainly see where I’m going: there is a ton of boilerplate to write just to pass down these dependencies. It’s super annoying to write, and it’s taking a huge amount of space in your code making it not as clear and readable as it should be.

But what solutions do we have? Let’s take it bit by bit.

Surely we could fix the initializer signature from getting too many parameters by grouping the dependencies into some kind of data structure:

class SomeViewModel {
  // MARK: - Initializers
  init(dependencies: SomeViewModel.Dependencies) {
    self.dependencies = dependencies
  }

  // MARK: - Dependencies
  let dependencies: SomeViewModelDependencies

  struct Dependencies {
    let dependencyA: DependencyA
    let dependencyB: DependencyB
    let dependencyC: dependencyC
    ...
    let dependencyX: DependencyX
  }
}

We would still need to initialize the dependencies struct before passing it to the view model, and it introduces one more annoyance in not having the dependencies accessible directly. Instead, the methods insides of view model would have to refer to dependencies.dependencyA to use it.

So, we are left with boilerplate issue. It’s now time for some code generation and Sourcery seemed to be a good tool to use.

I like to keep things isolated, so I will create a new folder at the root of my project named AutoGenerated. This folder will contain three sub-folders:
Generated/ where we will add files generated by Sourcery to our project
Templates/ where we will group our stencil templates
Protocols/ where we will define some types to be detected by Sourcery

Let’s then add the AutoDependencies.swift file in the Protocols/ folder with its content being as simple as:

import Foundation

protocol AutoDependencies {}

In the Templates/ folder, we will then add the AutoDependencies.stencil template, it’s content relatively easy to understand:

{% for type in types.implementing.AutoDependencies %}
// MARK: {{type.name}}

extension {{type.name}} {
{% for dependencies in type.containedTypes.Dependencies %}
{% for variable in dependencies.variables %}
    var {{variable.name}}: {{variable.typeName}} {
        return dependencies.{{variable.name}}
    }
{% endfor %}
{% endfor %}
}

extension DependencyContainer {
    func generate{{type.name}}Dependencies() -> {{type.name}}.Dependencies {
        return {{type.name}}.Dependencies({% for dependencies in type.containedTypes.Dependencies %}{% for variable in dependencies.variables %}{{variable.name}}: {{variable.name}}{% if not forloop.last %}, {% endif %}{% endfor %}{% endfor %})
    }
}

{% endfor %}

For each type implementing the empty AutoDependencies protocol, two things will be done:
– an extension on the type will be created to make and a simple get variable will be made so that you can use dependencyA directly
– an extension on the DependencyContainer will be created to easily create the dependencies struct

Now, let’s configure our project for it generate the code we need at build time. To do that, we simply need to add new Run Script in our target Build Phases (replace XXX with the proper path in your project):

sourcery --sources XXX/\
         --templates XXX/AutoGenerated/Templates/\
         --output XXX/AutoGenerated/Generated/

(note that at this stage, I’m using Sourcery installed on my machine with brew install sourcery)

We now need our class to implement the AutoDependencies protocol, it stays exactly the same as before otherwise:

class SomeViewModel: AutoDependencies {
  // MARK: - Initializers
  init(dependencies: SomeViewModel.Dependencies) {
    self.dependencies = dependencies
  }

  // MARK: - Dependencies
  let dependencies: SomeViewModelDependencies

  struct Dependencies {
    let dependencyA: DependencyA
    let dependencyB: DependencyB
    let dependencyC: dependencyC
    ...
    let dependencyX: DependencyX
  }
}

On save, a AutoDependencies.generated.swift is automatically created by Sourcery. This file looks like:

// MARK: SomeViewModel

extension SomeViewModel {
  var dependencyA: DependencyA {
    return dependencies.dependencyA
  }
  var dependencyB: DependencyB {
    return dependencies.dependencyB
  }
  var dependencyC: DependencyC {
    return dependencies.dependencyC
  }
  ...
  var dependencyX: DependencyX {
    return dependencies.dependencyX
  }
}

extension DependencyContainer {
  func generateSomeViewModelDependencies() -> SomeViewModel.Dependencies {
   return SomeViewModel.Dependencies(dependencyA: dependencyA, dependencyB: dependencyB, dependencyC: dependencyC, ..., dependencyX: dependencyX)
  }
}

We need to add it to the Xcode project by right-clicking on our Generated/ folder and “Add Files to …”.

In our coordinator, we can now simplify:

class SomeCoordinator {
  let dependencyContainer: DependencyContainer // For our own sake, let's at least imagine dependencies are centralized somewhere

  func showSomeViewController() {
    let dependencies = dependencyContainer.generateSomeViewModelDependencies()
    let viewModel = SomeViewModel(dependencies: dependencies)
    let viewController = SomeViewController(viewModel: viewModel)
    navigationController.push(viewController: viewController, animated: true)
  }
}

Want to see what’s in the DependencyContainer?

class DependencyContainer {
  lazy var dependencyA: DependencyA = DependencyA()
  lazy var dependencyB: DependencyB = DependencyB(dependencies: generateDependencyBDependencies()) // Assuming DependencyB is also an AutoDependencies
  lazy var dependencyC: DependencyC = DependencyC()
  ...
  lazy var dependencyX: DependencyX = DependencyX()
}

It’s a really dumb file lazy instantiating my dependencies as they need be. (Obviously, this kind of dependency container is too dumb to manage circular dependencies and is not a proper dependency graph, I let that to you to solve this problem!)

What about testing? The dependency container implementation being a bunch of lazy var, you can easily swap one with a mock:

class SomeViewControllerTests: XCTestCase {

    func testExample() {
        // Given
        let container = DependencyContainer()
        container.dependencyA = MockDependencyA()
        let dependencies = container.generateSomeViewModelDependencies()
        let viewModel = SomeViewModel(dependencies: dependencies)

        // When
        ...

        // Then
        ...
    }

class MockDependencyA: DependencyA {
    override func someMethod() -> Bool {
        return true // Let's say I want this method to always return true in my tests
    }
}

So in the end, what do we have? I will say:
– a functional type safe dependency injection enforced at build time that will probably work in most project
– some low(er) amount of boilerplate: you simply have to declare you type to comply to the AutoDependencies protocol, have a let dependencies: Dependencies and a Dependencies struct with all the desire dependencies
– Sourcery runs automatically when you build, so Xcode might show errors until you actually press Build
– adding a dependency to a type is done in only one place (the Dependencies struct), you simply Build again and you are good to use it in your code
– testable code

One caveat this system is build on-top is that no all dependencies need to be injected this way. For instance, a view controller is still responsible for passing dependencies to subview that will not own necessarily own it, for instance using initializer, property or method injection.

Don’t Be A Free User

An article I really liked to reference disappeared from it’s original website, so here is a copy of its content from the Web Archive.

“Someone builds a cool, free product, it gets popular, and that popularity attracts a buyer. The new owner shuts the product down and the founders issue a glowing press release about how excited they are about synergies going forward. They are never heard from again.

Whether or not this is done in good faith, in practice this kind of ‘exit event’ is a pump-and-dump scheme. The very popularity that attracts a buyer also makes the project financially unsustainable. The owners cash out, the acquirer gets some good engineers, and the users get screwed.

To avoid this problem, avoid mom-and-pop projects that don’t take your money! You might call this the anti-free-software movement.

If every additional user is putting money in the developers’ pockets, then you’re less likely to see the site disappear overnight. If every new user is costing the developers money, and the site is really taking off, then get ready to read about those synergies.

To illustrate, I have prepared this handy chart:

Free Paid
Stagnant losing money making money
Growing losing more money making more money
Exploding losing lots of money making lots of money

What if a little site you love doesn’t have a business model? Yell at the developers! Explain that you are tired of good projects folding and are willing to pay cash American dollar to prevent that from happening. It doesn’t take prohibitive per-user revenue to put a project in the black. It just requires a number greater than zero.

I love free software and could not have built my site without it. But free web services are not like free software. If your free software project suddenly gets popular, you gain resources: testers, developers and people willing to pitch in. If your free website takes off, you lose resources. Your time is spent firefighting and your money all goes to the nice people at Linode.

So stop getting caught off guard when your favorite project sells out! “They were getting so popular, why did they have to shut it down?” Because it’s hard to resist a big payday when you are rapidly heading into debt. And because it’s culturally acceptable to leave your user base high and dry if you get a good offer, citing self-inflicted financial hardship.

Like a service? Make them charge you or show you ads. If they won’t do it, clone them and do it yourself. Soon you’ll be the only game in town!”

— maciej from Pinboard on December 06, 2011