Blog

Implementing a custom UI for a web app that has no API

By  
Matti Tahvonen
Matti Tahvonen
·
On Jan 14, 2025 6:19:55 PM
·

It has happened before, and it happened again. I found myself dissatisfied with software I had to use multiple times a week. As a professional UI developer, I am particularly sensitive to bad UX. When a perfectly functional phone reservation service for my home island’s ferry was replaced with a mediocre web app, I couldn’t resist. I had to create a custom Vaadin UI—one that I actually enjoyed using (even more than the old phone service).

It took me one Monday, but I’ve already gained that time (at least partially) back. Now, reserving a ferry trip is as simple as opening an app and clicking a button—a stark contrast to the error-prone, time-consuming, and frustrating process it replaced. Every time I use the app, it makes me smile 😁 —partly because of its quirky name, FoolFerries, and partly because I know it’s saving time for others, too.

This is a screenshot of the custom web UI for the ferry trip booking app. Now, most often, I open the app from my iPhone and hit the blue Book button. Previously, I had to log in, re-fill all the route and passenger information through cumbersome menus, and verify my selections on another page.

When I explained my project to a colleague (a skilled software developer), their initial reaction was disbelief: How is that even possible? After all, the ferry service didn’t provide an API to build against. This article explores three strategies for tackling a problem like this—creating a custom UI for an app with no API. I’ll also dive into how I built FoolFerries, highlighting some cool features of JDK’s HttpClient that I learned to use along the way.

Three Approaches to Building a Custom UI Without an API

For a Java developer, I can quickly come up with three approaches with different pros and cons. In all of these, we’ll somehow need to impersonate an actual user and the browser, just with different abstractions.

Option 1: Programmatically control a web browser

The most robust and stable approach is to control an actual web browser from your app. With a server-centric stack like Vaadin, it’s straightforward to utilize end-to-end testing frameworks like Selenium or Playwright on the server (or via the server). These frameworks can guide a browser to a specific address, simulate user actions, and extract relevant data from the rendered DOM.

The main drawback is scalability. Every user essentially requires a browser session running on the server, which remains open for the duration of their session. For a small user base—like a handful of people using my ferry reservation app—this could be a viable solution. If I were building FoolFerries for profit, I might have started here and optimized later.

Option 2: Simulate a browser with jsoup

jsoup, a popular JVM library for HTML parsing, can also act as a simple programmable browser. It allows you to fetch HTML, construct a “DOM,” fill out forms, and submit them back to the server—all in pure Java. jsoup handles cookies and headers, making it a suitable tool for interacting with “traditional” web apps.

The downside? It doesn’t execute JavaScript. Modern web apps, even those without public APIs, often rely on JavaScript and AJAX calls. When I started with this approach, I quickly hit a wall. The ferry reservation app used XHRs to fetch some data, which made this approach impractical for my needs.

Option 3: Simulate a browser using an HTTP client

This approach works at the HTTP level, simulating both the user and the browser. I guess some hard-core heroes could do this with java.net.URL, but the Java ecosystem provides numerous more advanced libraries for making HTTP requests (and to consume raw REST services). There are several options already within Spring libraries. There is also a fairly new HttpClient available directly in the JDK that I chose to explore and learn while making this app. It’s kind of a modern (and built-in) replacement for Apache HttpClient.

HttpClient supports things like persistent connections, cookie handling, and HTTP/2 out of the box, but some assembly is naturally required to extract required data and to compile a suitable request payload. If you go down this route, as I did with my FoolFerries app, you'll need to understand how the web works. Using browser developer tools (DOM and network tabs) to reverse-engineer the web app logic is often necessary. Although this approach needs most programming, it is the most lightweight approach for the server (in terms of CPU and memory) as e.g., parsing the full DOM structure is not necessarily needed.

Example using JDKs HttpClient

For FoolFerries, I started by creating a session-scoped Spring Bean to manage a per-user HTTP client. The app uses cookie-based sessions, HTTP/2, and redirects, so I configured HttpClient accordingly:

@SessionScope 
@Component 
public class Session { 
  private final HttpClient client;

  public Session() {
    this.client = HttpClient.newBuilder()
            .cookieHandler(new CookieManager())
            .version(HttpClient.Version.HTTP_2)
            .followRedirects(HttpClient.Redirect.ALWAYS)
            .build();
  }
}

Handling login with Nonces

The original ferry app, based on some e-commerce plugin for WordPress, required a nonce (a one-time token) for login. This meant that I couldn’t simply take the username and password and send it to the server. Instead, I needed to:

  1. Fetch the login page HTML.
  2. Extract the nonce (and HttpClient acquiring a new session cookie)
  3. Submit the credentials along with the nonce.

Here’s how I handled it:

HttpResponse<String> response = client.send(HttpRequest.newBuilder()
        .uri(URI.create("https://booking.finferries.fi/my-account/"))
        .GET()
        .build(), HttpResponse.BodyHandlers.ofString());

String loginformBody = response.body();

String nonce = extractNonce(loginformBody); // helper to regexp nonce

HttpResponse<String> loginResponse = client.send(HttpRequest.newBuilder()
        .uri(URI.create("https://booking.finferries.fi/my-account/"))
        .header("Content-Type", "application/x-www-form-urlencoded")
        .POST(HttpRequest.BodyPublishers.ofString("username=" + username + "&password=" +
         password + "&login=Login&woocommerce-login-nonce=" + nonce))
        .build(), HttpResponse.BodyHandlers.ofString());

if (loginResponse.statusCode() > 399) {
    throw new RuntimeException("Login failed with status code: " + loginResponse.statusCode());
}

After this step, I have an “authenticated session” open to the original web application.

Fetching and submitting data

I reverse-engineered the web app’s AJAX calls and built a BookingService class to handle these interactions. For JSON deserialization, I used Jackson manually (with actual REST clients like JAX RS Client or Spring’s RestClient, you get this part usually for free):

HttpResponse<String> response = client.send(HttpRequest.newBuilder()
       .uri(URI.create("https://booking.finferries.fi/wp-json/wp/v2/ffr_vehicle_type?_fields=id%2Cname&lang=fi&per_page=100"))
       .GET()
       .build(), HttpResponse.BodyHandlers.ofString());

String body = response.body();

List<VehicleType> vehicleTypes = new ObjectMapper().readValue(body, new TypeReference<>() {});

The app automatically selects intelligent default options like the vehicle type, recent route, and departure time, making the booking process nearly effortless. If you, for example, previously traveled to an island with a car, it guesses that next time you’ll be reserving a return trip with the same vehicle. I call that AI 🤪

Submitting multipart forms

The most challenging part was submitting the reservations. The original app uses multipart requests for some forms, which the new HttpClient doesn’t support out of the box. To build the request body, I used the tooling from the good old Apache HttpClient library for this task, even though I was still posting the data with the JDKs client:

HttpEntity entity = MultipartEntityBuilder.create()
        .setCharset(StandardCharsets.US_ASCII)
        .setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
        .addTextBody("key1", "value1")
        .addTextBody("key2", "value2"))
        .build();

Header contentType = entity.getContentType();

HttpResponse<String> response = client.send(HttpRequest.newBuilder()
        .uri(URI.create("https://booking.finferries.fi/checkout-fi/"))
        .header(contentType.getName(), contentType.getValue())
        .POST(HttpRequest.BodyPublishers.ofInputStream(() -> {

            try {
                return entity.getContent();
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        }))
        .build(), HttpResponse.BodyHandlers.ofString());

Later, I discovered the Methanol library, which is an add-on for the JDK's HttpClient. I’ll definitely try that next time I craft the multipart request bodies.

The response to the multipart request above actually presents end users with another HTML form. This form includes a set of parameters (mostly the same as before) as hidden input fields, along with another nonce and a checkbox that must be ticked to finalize the reservation.

For this final step, I opted to use the jsoup library to parse the returned HTML instead of pursuing further optimizations. Since jsoup provides an API to modify form fields (such as ticking the checkbox) and iterate over the field values, crafting the final "yes, I'm sure" multipart request was a straightforward task:

FormElement formElement = Jsoup.parse(bodyHtml).forms().get(0);

// simulate user checking the terms checkbox

formElement.selectXpath("//input[@name='terms']").attr("checked", "checked");

var builder = MultipartEntityBuilder.create()
        .setCharset(StandardCharsets.US_ASCII)
        .setMode(HttpMultipartMode.BROWSER_COMPATIBLE);

formElement.formData().forEach(kv -> {
    builder.addTextBody(kv.key(), kv.value());
});
HttpEntity entity1 = builder.build();

Conclusion

The FoolFerries app transformed a cumbersome process into a single-click operation. Leveraging HttpClient allowed me to build a seamless UI, even without an official API. For Java developers, especially those using Vaadin, this project demonstrates how to harness the full power of the JDK to improve user experience.

Check out the full source code on GitHub. Happy coding!

ui
Matti Tahvonen
Matti Tahvonen
Matti Tahvonen has a long history in Vaadin R&D: developing the core framework from the dark ages of pure JS client side to the GWT era and creating number of official and unofficial Vaadin add-ons. His current responsibility is to keep you up to date with latest and greatest Vaadin related technologies. You can follow him on Twitter – @MattiTahvonen
Other posts by Matti Tahvonen