In this note I will create a URL shortening service using Spring Restful, Spring Boot, Liquibase, and PostgreSQL database. In a similar way of some shortening services, such as goo.gl, tinyurl.com, and bit.ly, purpose of the shortened URL may be more convenience for website and provide detailed information on clicks a link receives.
1. URL Shortener Service.
The service will receive a request of any client side that contains a original URL, and then process it by applying a hashing algorithm in order to convert the ID stored in database to a shortened URL as the figure below:
When someone clicks on the shortened URL, the service invoked will convert this URL to the initial ID to query the original URL from database.
2. Implementation.
2.1 Project structure.
I used the same design I did on the multi-tier application architecture with Micro-service note to create the URL shortener service.
You can use Maven to create your own new project or download fullcode from GitHub. Install PostgreSQL 9.x and pgAdmin tool to easily manage database and common IDEs such as Eclipse and IntelliJ IDEA recommended.
2.2 A Shortening Algorithm.
In theory background, Bijection function f indicates that ‘ A original URL is mapped to exactly one key, and a key is mapped to exactly one original URL’, which guarantees a 1-to-1 mapping. For example, given an original URL like “https://guides.github.com/activities/hello-world/“, generate a unique shortened URL “tdkhiem.com/P7Cu“, actually unique key is P7Cu.
I used base62 that contains 62 characters to hashing the URL. It takes 2^63 – 1 possible urls
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789
Convert the ID to a shortened URL:
- Used base62 that’s [a-zA-Z0-9].
- Generate a unique ID with a base of 10 ( Create a SEQUENCE to generate next sequence unique ID in PostgreSQL).
- Convert the unique ID to base 62 as the shortened URL.
We also resolve the shortened URL to the initial ID by doing a reverse lookup in our alphabet, and then find the ID in database. A brief explanation of Marcel Jackwerth can be found here for detailed reference.
src/main/java/com/mycompany/app/utils/Base62.java
package com.mycompany.app.utils;
public class Base62 {
public static final String ALPHABET = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
public static final int BASE62 = ALPHABET.length();
/**
* Returns the base 62 string of a long.
* @param a long.
* @return the base 62 string of a long
*/
public static String toBase62(long value) {
final StringBuilder sb = new StringBuilder(1);
do {
sb.insert(0, ALPHABET.charAt((int) (value % BASE62)));
value /= BASE62;
} while (value > 0);
return sb.toString();
}
/**
* Returns the base 62 value of a string.
* @param a string.
* @return the base 62 value of a string.
*/
public static long toBase10(String str) {
return toBase10(new StringBuilder(str).reverse().toString().toCharArray());
}
private static long toBase10(char[] chars) {
long result = 0;
for (int i = chars.length - 1; i >= 0; i--) {
result += toBase10(ALPHABET.indexOf(chars[i]), i);
}
return result;
}
private static long toBase10(long result, int pow) {
return result * (int) Math.pow(BASE62, pow);
}
}
2.3 Create database with liquibase.
First, we create a database in PostgreSQL with name like ‘url-shortener’ by pgAdmin tool or execute SQL script:
-- DROP DATABASE "url-shortener"; CREATE DATABASE "url-shortener"
An example of our urls table:
To make sure that auto-generated ID is unique key and begin with any value, a sequence created in PostgreSQL looks like below:
-- DROP SEQUENCE public.seq_unique_id; CREATE SEQUENCE public.seq_unique_id INCREMENT 1 MINVALUE 1 MAXVALUE 9223372036854775807 START 10000000 CACHE 1; ALTER TABLE public.seq_unique_id OWNER TO postgres;
However, we do not go with any specific database because of development changes, so taking advantages of Liquibase – database-independent library for tracking, managing and applying database schema changes is a good solution. The following implementation to create the url table and the sequence:
src/main/resources/config/liquibase/db-changelog.xml
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-2.0.xsd">
<changeSet id="1" author="khiem.truong">
<createTable tableName ="url">
<column name="id" type="BIGINT" >
<constraints nullable="false" primaryKey="true"/>
</column>
<column name="original_url" type="varchar(1000)"/>
<column name="created_on" type="timestamp"/>
</createTable>
<createSequence incrementBy="1"
sequenceName="seq_unique_id"
startValue="10000000" />
<!-- For insert a new record testing purpose only -->
<loadData tableName="url"
encoding="UTF-8"
file="config/liquibase/urls.csv"
separator=";">
</loadData>
</changeSet>
</databaseChangeLog>
2.4 Implement Service with Spring Restful API, and JPA/Hibernate.
Create URL entity
There are two ways to create an unique sequence ID. First, we use auto-generate ID that already supported by JPA like:
@GeneratedValue(strategy= GenerationType.IDENTITY)
And then add autoIncrement=”true” to db-changelog.xml:
<column name="id" type="BIGINT" autoIncrement="true">
<constraints nullable="false" primaryKey="true"/>
</column>
But, this way will create the sequence with START value (the first insert record is always 1). So I come a next way that creates my own sequence with any START value as well as don’t use any strategy generated ID value of JPA. Actually URLShortener entity as below:
src/main/java/com/mycompany/app/domain/URLShortener.java
package com.mycompany.app.domain;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.Table;
import org.hibernate.annotations.Type;
import org.joda.time.DateTime;
import com.mycompany.app.utils.DomainConstants;
@Entity
@Table(name="url")
public class URLShortener implements DomainConstants{
@Id
@Column(name = "id")
private Long id;
@Column(name = "created_on", updatable = false)
@Type(type = COMMON_DATE_TYPE)
private DateTime createdOn = new DateTime();
@Column(name = "original_url")
private String originalURL;
//Generate Getter and Setter
URL Repository
Using JPA repository of Spring Data JPA minimizes the amount of code needed for implementing repositories, but still provides CRUD capabilities to any domain object, write custom queries, and offers simple abstractions for performing common tasks like sorting an pagination.
We implemented three methods in URLRepository interface that extends JpaRepository. These methods help us find a exiting unique ID, a next sequence unique ID, and a original URL (prevent no duplicated original URL).
src/main/java/com/mycompany/app/repository/URLRepository.java
package com.mycompany.app.repository;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import java.util.Optional;
import com.mycompany.app.domain.URLShortener;
public interface URLRepository extends JpaRepository<URLShortener, Long>{
@Query(value = "from URLShortener u where u.id = :id")
Optional<URLShortener> findById(
@Param(value = "id") Long id);
@Query(value = "from URLShortener u where u.originalURL = :originalURL")
Optional<URLShortener> findByOriginalURL(
@Param(value = "originalURL") String originalURL);
@Query(nativeQuery = true, value = "SELECT nextval('seq_unique_id')")
Long getIdWithNextUniqueId() ;
}
URL Service
All logic business of URL service implemented here. In particular, the first method saveUrl saves an original URL to database and return a shortened URL. The second method getURL enables us reverse the original URL from the shortened URL. In addition, domain default injected via setter injection with domain.shortener property.
src/main/java/com/mycompany/app/repository/URLRepository.java
import java.util.Optional;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import com.mycompany.app.api.dto.URLShortenerDTO;
import com.mycompany.app.domain.URLShortener;
import com.mycompany.app.repository.URLRepository;
import com.mycompany.app.utils.Base62;
@Service
public class URLServicesImpl implements URLService {
@Autowired
private URLRepository urlRepository;
private final String domain;
@Autowired
public URLServicesImpl(@Value("${domain.shortener}") String domain) {
this.domain = domain;
}
/**
* Reverse the original URL from the shortened URL
*/
public URLShortenerDTO getURL(String shortenURL) {
URLShortenerDTO dto = new URLShortenerDTO();
if (validateURL(shortenURL)) {
//Remove domain to shortened URL if possible.
String str = shortenURL.replace(this.domain +"/", "");
// Resolve a shortened URL to the initial ID.
long id = Base62.toBase10(str);
// Now find your database-record with the ID found
Optional<URLShortener> urlShortener = urlRepository.findById(id);
if(urlShortener.isPresent()) {
// Mapped domain to DTO
URLShortener url = urlShortener.get();
dto.setId(url.getId().toString());
dto.setShortenedURL(shortenURL);
dto.setOriginalURL(url.getOriginalURL());
dto.setCreatedOn(url.getCreatedOn().toString());
}
}
return dto;
}
/**
* Save an original URL to database and then
* generate a shortened URL.
*/
public URLShortenerDTO saveUrl(String originalURL) {
URLShortener url = new URLShortener();
if (validateURL(originalURL)) {
originalURL = sanitizeURL(originalURL);
// Quickly check the originalURL is already entered in our system.
Optional<URLShortener> exitURL = urlRepository.findByOriginalURL(originalURL);
if(exitURL.isPresent()) {
// Retrieved from the system.
url = exitURL.get();
} else {
// Otherwise, save a new original URL
url.setId(urlRepository.getIdWithNextUniqueId());
url.setOriginalURL(originalURL);
url = urlRepository.save(url);
}
}
//TODO Should handle the url didn't save successfully and return null.
return generateURLShorterner(url);
}
/**
* Generate a shortened URL.
*/
private URLShortenerDTO generateURLShorterner(URLShortener url) {
// Mapped domain to DTO
URLShortenerDTO dto = new URLShortenerDTO();
dto.setId(url.getId().toString());
dto.setOriginalURL(url.getOriginalURL());
dto.setCreatedOn(url.getCreatedOn().toString());
// Generate shortenedURL via base62 encode.
String shortenedURL = this.domain +"/" + Base62.toBase62(url.getId().intValue());
dto.setShortenedURL(shortenedURL);
return dto;
}
/**
* Validate URL not implemented, but should be implemented to
* check whether the given URL is valid or not
*/
private boolean validateURL(String url) {
return true;
}
/**
* This method should take into account various issues with a valid url
* e.g. www.google.com,www.google.com/, http://www.google.com,
* http://www.google.com/
* all the above URL should point to same shortened URL
* There could be several other cases like these.
*/
private String sanitizeURL(String url) {
if (url.substring(0, 7).equals("http://"))
url = url.substring(7);
if (url.substring(0, 8).equals("https://"))
url = url.substring(8);
if (url.charAt(url.length() - 1) == '/')
url = url.substring(0, url.length() - 1);
return url;
}
}
URL APIs
Spring RESTful web service is a feature of Spring Framework. The service will handle POST request for api/url/shorten with a originalURL parameter in the query string. The POST request should return a 200 OK response with JSON in body that represents a URLShortenerDTO. It should look something like this:
{
"id": "10000000",
"originalURL": "www.liquibase.org/documentation/index.html",
"createdOn": "2017-06-22T12:08:57.952+07:00",
"shortenedURL": "http://tdkhiem.com/P7Cu"
}
Similarly, the service also handle GET request for api/url/reverse with shortenedURL parameter in the query string.
Noted that Spring uses the Jackson JSON library to automatically marshal instances of type URLShortenerDTO into JSON. Moreover, if Jackson can’t deserialize JSON payload with `Optional` fields in JDK 8, or you want to customize joda.time format. I already resolved it in Jackson Configuration.
package com.mycompany.app.api.dto;
public class URLShortenerDTO {
private String id;
private String originalURL;
private String createdOn;
private String shortenedURL;
// Generate Getter and Setter.
}
src/main/java/com/mycompany/app/rest/URLResourceImpl.java
package com.mycompany.app.rest;
import static org.springframework.http.MediaType.APPLICATION_JSON_VALUE;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.ResponseStatus;
import org.springframework.web.bind.annotation.RestController;
import com.mycompany.app.api.URLResource;
import com.mycompany.app.api.dto.URLShortenerDTO;
import com.mycompany.app.service.URLService;
@RestController
@RequestMapping("/api/url")
public class URLResourceImpl implements URLResource {
@Autowired
private URLService urlService;
@RequestMapping(value = "/shorten",
method = RequestMethod.POST,
produces = APPLICATION_JSON_VALUE)
@ResponseStatus(HttpStatus.OK)
@ResponseBody
public URLShortenerDTO saveURL(@RequestParam(value = "originalURL") String originalURL) {
return urlService.saveUrl(originalURL);
}
@RequestMapping(value = "/reverse", method = RequestMethod.GET)
@ResponseStatus(HttpStatus.OK)
@ResponseBody
public URLShortenerDTO getURL(@RequestParam(value = "shortenedURL") String shortenedURL) {
return urlService.getURL(shortenedURL);
}
}
@RestController annotation used from Spring 4.x. The service controller simply populates and return a URLShortenerDTO object, and then the object will be written directly to HTTP response as JSON. The opposite was true for traditional MVC controller basing on view technology to perform server-side rendering of the greeting data to HTML.
2.5 Make the application executable with Spring Boot.
Spring Boot does some magic to help us automatically manage dependency and configuration. For example, if we add springboot-starter-web dependency by default it will pull all the commonly used libraries while developing Spring MVC applications such as spring-webmvc, jackson-json, validation-api and tomcat. After that, it will configure the commonly registered beans like DispatcherServlet, ResourceHandlers, MessageSource etc beans with sensible defaults.
Our Spring Boot Application look like:
src/main/java/com/mycompany/app/Application.java
package com.mycompany.app;
import ch.qos.logback.classic.LoggerContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import java.io.IOException;
@Configuration
@ComponentScan
@EnableAutoConfiguration
@EnableJpaRepositories
public class Application {
@Autowired
private Environment env;
private static final Logger LOGGER = LoggerFactory.getLogger(Application.class);
@PostConstruct
public void initApplication() throws IOException {
LOGGER.info("Running with Spring profile(s) : {}", env.getActiveProfiles());
}
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
@PreDestroy
public void shutdownLogback(){
LOGGER.info("Shutting down Logback");
LoggerContext lCtx = (LoggerContext) LoggerFactory.getILoggerFactory();
lCtx.stop();
}
}
2.6 Test the service.
I used Postman Rest Client tool to capture POST/GET request. Sending a POST request to Tomcat Server with port 8081 (http) as below:
POST http://localhost:8081/api/url/shorten?
originalURL=http://www.liquibase.org/documentation/index.html
And JSON response on Postman screenshot look like:
Likely, we will look up the original URL from the shortened URL by sending a GET request to server.
GET http://localhost:8081/api/url/reverse?shortenedURL=http://tdkhiem.com/P7Cu
JSON response:
{
"id": "10000000",
"originalURL": "www.liquibase.org/documentation/index.html",
"createdOn": "2017-06-22T12:08:57.952+07:00",
"shortenedURL": "http://tdkhiem.com/P7Cu"
}
3. Summary
A simple URL Shortener has done with Spring Restful, Liquibase, and PostgreSQL, which just provides basic ideas, and particular implementation for future references. I acknowledge contributors from my list of reference.
References:
- How to code a url shortener, http://stackoverflow.com/questions/742013/how-to-code-a-url-shortener
- Java Code – Url Shortener, https://gist.github.com/rakeshsingh/64918583972dd5a08012
- Class to encode a string into base 62, https://gist.github.com/jdcrensh/4670128
- Creating a URL Shortener with NodeJs, Express, and MongoDB, https://coligo.io/create-url-shortener-with-node-express-mongo/
- Spring Data JPA Tutorial Part Nine: Conclusions, https://www.petrikainulainen.net/programming/spring-framework/spring-data-jpa-tutorial-part-nine-conclusions/
- Introduction to Spring Data JPA, http://www.theserverside.com/discussions/thread/80065.html
- Benefits and usage of spring data JPA , http://javaenterpriseworld.blogspot.com/2014/02/benefits-and-usage-of-spring-data-jpa.html
- Why Spring Boot, https://dzone.com/articles/why-springboot
Known high severity security vulnerability detected in com.fasterxml.jackson.core:jackson-databind 2.8.11.1.