Mastering HTTP Requests with WebClient in Spring WebFlux

Introduction

In the world of web development, making HTTP requests is a fundamental operation. In this comprehensive guide, we’ll explore how to make HTTP requests with WebClient in Spring WebFlux, including how to set custom http headers, proxy setting, rate limit and retry, error handling, and sse(server sent event) requests, making post requests with multipart/form-data.

1. Understanding WebClient and Spring WebFlux

Before diving into the practical aspects of making HTTP requests with WebClient, let’s briefly understand what WebClient and Spring WebFlux are.

WebClient: WebClient is a non-blocking, reactive client for making HTTP requests in a Spring WebFlux application. It provides a more efficient and scalable way to interact with external resources compared to traditional synchronous approaches.

Spring WebFlux: Spring WebFlux is a part of the Spring Framework that supports reactive programming. It allows developers to build asynchronous and non-blocking applications, making it ideal for handling a high volume of concurrent requests.

2. Getting Started with WebClient

To use WebClient, we need to add the Spring WebFlux and WebClient dependencies to our project. We can do this by including the following dependencies in your pom.xml or build.gradle:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
</dependency>

3. Building Custom Webclient

Building a custom webclient object, including custom http headers, proxy settings as socks5 is quite easy, here is an example:

        try {

            HttpClient httpClient = HttpClient.create().proxy(proxy -> proxy.type(ProxyProvider.Proxy.SOCKS5).address(new InetSocketAddress("192.168.1.8", 1081)));
            httpClient.warmup().block();
            Consumer<HttpHeaders> httpHeaders = httpHeader -> {
                httpHeader.add("Content-Type", "application/json");
                httpHeader.add("Authorization", "Bearer " + apiKey);
            };
            return  WebClient.builder().defaultHeaders(httpHeaders).clientConnector(new ReactorClientHttpConnector(httpClient)).baseUrl(API_ENDPOINT).build();
        } catch (Exception e) {
            logger.error(e.getMessage());
            return null;
        }

4. Making GET Requests

Making GET requests with WebClient is straightforward. Here’s a basic example:

import org.springframework.web.reactive.function.client.WebClient;

WebClient webClient = WebClient.create("https://api.example.com");

webClient
    .get()
    .uri("/resource/{id}", 123)
    .retrieve()
    .bodyToMono(String.class)
    .subscribe(response -> {
        // Handle the response
    });

We can also add rate limit and retry to a webclient object with transformDeferred() method

webClient
    .get()
    .uri("/resource/{id}", 123)
    .retrieve()
    .bodyToMono(String.class)
    .publishOn(this.myScheduler)
    .transformDeferred(RateLimiterOperator.of(this.RateLimiter))
    .transformDeferred(RetryOperator.of(this.retry));

5. Post Requests for SSE

    public Flux<ServerSentEvent<String>> getForSSE(MyRequestEntity requestEntity) {
        try{


            WebClient client = this.buildWebClient();

            ParameterizedTypeReference<ServerSentEvent<String>> type = new ParameterizedTypeReference<ServerSentEvent<String>>() {};
            Map<String, Object> params = new HashMap<>();
            params.put("stream", true);

            return client.post()
                    .uri(uriBuilder ->
                            uriBuilder.path(Constant.API_ENDPOINT).build()
                    ).body(Mono.just(params), Map.class)
                    .accept(MediaType.TEXT_EVENT_STREAM)
                    .retrieve()
                    .bodyToFlux(type)
        } catch (Exception e) {
            e.printStackTrace();
            return Flux.empty();
        }

6. Making Post Requests

For making POST requests, you can use the .post() method along with the request body:

webClient
    .post()
    .uri("/create")
    .bodyValue(requestData)
    .retrieve()
    .bodyToMono(Response.class)
    .subscribe(response -> {
        // Handle the response
    });

Making post request with multipart/form-data, send local file data to remote server, let’s take openai whisper api as an example:

        HttpClient httpClient = HttpClient.create();
        Consumer<HttpHeaders> httpHeaders = httpHeader -> {
            httpHeader.add("Content-Type", "multipart/form-data");
            httpHeader.add("Authorization", "Bearer " + apiKey);
        };
        WebClient client = WebClient.builder().defaultHeaders(httpHeaders).clientConnector(new ReactorClientHttpConnector(httpClient)).baseUrl(getOpenAIAPIEndpoint()).build();
        StringBuilder resultBuilder = new StringBuilder();
        for(String filePath: files) {
            MultipartBodyBuilder builder = new MultipartBodyBuilder();
            builder.part("file", new FileSystemResource(filePath));
            builder.part("model", "whisper-1");
            if (lang != null) {
                builder.part("language", lang);
            }
            ParameterizedTypeReference<R> type = new ParameterizedTypeReference<R>() {};

            try {
                Path filep = Paths.get(filePath);
                logger.info("transcribing {}, size {}MB", filePath, Files.size(filep) / 1024 / 1024);
                R result = client
                        .post()
                        .uri(OPEN_AI_TRANSCRIPT_ENDPOINT)
                        .contentType(MediaType.MULTIPART_FORM_DATA)
                        .body(BodyInserters.fromMultipartData(builder.build()))
                        .retrieve()
                        .bodyToMono(type)
                        .block(Duration.ofMinutes(10));

                if (result != null) {
                    if (result.get("text") != null) {
                        resultBuilder.append(result.get("text"));
                        Files.delete(filep);
                    } else {
                        logger.info("Error for transcribing file {}", filePath);
                    }
                } else {
                    logger.info("Timeout for transcribing file {}", filePath);
                }
            } catch (Exception e) {
                logger.error(e.getMessage());
            }
        }

7. Handling Responses

WebClient allows you to handle responses in various ways, including mapping them to specific classes, extracting headers, and processing the body. For instance:

webClient
    .get()
    .uri("/data")
    .retrieve()
    .toEntity(Data.class)
    .subscribe(responseEntity -> {
        Data data = responseEntity.getBody();
        HttpHeaders headers = responseEntity.getHeaders();
        // Handle the data and headers
    });

8. Error Handling

Error handling is a crucial part of HTTP requests. You can use the .onErrorResume() method to handle errors gracefully:

webClient
    .get()
    .uri("/resource")
    .retrieve()
    .bodyToMono(String.class)
    .onErrorResume(error -> {
        // Handle the error
        return Mono.empty();
    })
    .subscribe(response -> {
        // Handle the response or absence of it
    });

handle HttpStatus.4xx or HttpStatus.5xx error:

        try{
            WebClient client;
            Map<String, Object> params = new HashMap<>();

            return client.post()
                    .uri(uriBuilder ->
                            uriBuilder.path(Constant.OPEN_AI_EMBEDDING_ENDPOINT).build()
                    ).body(Mono.just(params), Map.class)
                    .accept(MediaType.APPLICATION_JSON)
                    .retrieve().onStatus(HttpStatus::is4xxClientError, response -> {
                        logger.error("error, {}", response.statusCode());
                        return Mono.error(new HttpClientErrorException(response.statusCode()));
                    })
                    .onStatus(HttpStatus::is5xxServerError, response -> {
                        logger.error("error, {}", response.statusCode());
                        return Mono.error(new HttpServerErrorException(response.statusCode()));
                    })
                    .bodyToMono(String.class)
        } catch (Exception e) {
            e.printStackTrace();
            return Mono.empty();
        }

9. Summary

Mastering HTTP requests with WebClient in Spring WebFlux is a valuable skill for any developer. In this blog post, we explored the basics of WebClient, making GET and POST requests with custom http headers, parameters, handling responses and errors, also described how to add rate limit and retry to webclient.