Web APIs: basics every developer needs to know

An API is an interface that makes it easy for one application to ‘consume’ capabilities or data from another application. By defining stable, simplified entry points to application logic and data, APIs enable developers to easily access and reuse application logic built by other developers. It allows for the clear separation between interface and implementation. A well designed API allows its user to rely only on the published public interface, abstracts implementation details. This enables API developer to evolve the system independently of the client and augurs well for the development of highly scalable systems. In the case of ‘web APIs’, that logic and data is exposed over the network.

Now, Web API is the hot currency in the digital world. Organisations like Google, Amazon, Facebook, Salesforce etc., are essentially selling their services via APIs. So:

• APIs can be among a company’s greatest assets
• Customers invest heavily on APIs: buying, learning , writing clients.
• API is public contract, therefore its developers need to honour it.
• Cost to stop, change using an API can be prohibitive.
• Successful public APIs capture customers.

Every API in the world follows some sort of paradigm or architectural style like Control Language, Distributed Object, RPC, Resource-based architecture (REST) and query language.

Control Languages provide an economical and efficient way for application programs to control remote process, usually residing in hardware (like firmware of a printer). Hewlett-Packard’s PCL printer language is one such example of control language[1]. Control languages involve sending compact escape sequence codes that are embedded in the data stream between computer and peripheral hardware. These escaped sequence control commands are interpreted by the embedded software and appropriate functionality takes place. Control languages are by its very nature system specific and are not viable for building scalable, general purpose systems.

Remote procedure calls(RPC) allow programs to call procedures located on other machines. When a process on machine A calls a procedure on machine B, the calling process on A is suspended, and execution of the called procedure takes place on B. Information can be transported from the caller to the callee in the parameters and can come back in the procedure result. No message passing at all is visible to the programmer. However, it’s not easy for clients to invoke remote procedure calls. They may establish connections to remote systems through “low-level” protocols like the BSD Socket API. Developers that use these mechanisms must convert the data types defined on the remote computing platform to corresponding types on the local platform and vice versa. This process is called data marshalling. This can be a daunting task because different platforms use different character encoding schemes (e.g., ASCII, EBCDIC, UTF-8, UTF-16, Little and Big Endianness) to represent and store data types. App Developers who work at this level must therefore understand how the remote platform encodes data and how it interprets any byte stream received.

Remoting technologies like CORBA and DCOM have made it much easier to share and use remote procedures. Hewlett-Packard’s Orblite project is one such effort to build CORBA based distributed object communication infrastructure [2]. Orblite infrastructure can be used to communicate between processes running on computer with processes running on hardware devices like digital scanner, printer etc. The pieces involved in the distributed call in Orblite is figure 1. It involves generating common procedure call signature via an Interface Definition Language(IDL) compiler using a contract defined in the IDL language. This process generates Stub in client and Skeleton in server. Both communicating parties must agree on transmittable types before hand and this is usually done using a Common Data Representation (CDR) format. With this setup client and server can be implemented in different technology and hardware stacks. RPC protocol is free to use any transport mechanism like TCP, HTTP, TCP over USB etc

Fig 1. Pieces involved in a CORBA distributed call.

Though CORBA based system is very good improvement over RPC in terms interoperability, there is still a lot tight coupling in terms of IDL and CDR which affects scalability and independent evolution of system. The systems thus developed are also very complex in nature. You can see that in the below figure 2 which traces logical flow of a remote method invocation across all subsystems.

Fig 2. The logical flow of a remote method invocation.

HTTP mitigates many of these issues because it enables clients and servers that run on different computing platforms to easily communicate by leveraging open standards. But the challenge is how can clients use HTTP to execute remote procedures? One approach is to send messages that encapsulate the semantics for procedure invocation. One can use open standards of data representation like XML and JSON to transmit data between client and server. There are many concrete implementations/standards of http based RPC like XML-RPC, JSONRPC and Simple Object Access Protocol (SOAP). Among these SOAP is the most famous. SOAP provides a layer of metadata which describe things such as which fields correspond to which datatypes and what are the allowed methods and so on. SOAP uses XML Schema and a Web Services Description Language (WSDL) for this purpose. This metadata allows clients and server to agree upon the public contract of communication.

For example a SOAP based system for communicating between process running in Desktop and firmware running in a digital scanner will have a WSDL defining operations like – GetScannerCapabilities, CreateScanRequest, CancelScanRequest, GetCurrentScanJobInfo – and values and their correspond datatypes applicable to each operation.

But, it can be noted that number of operations, their semantics and parameters are unique to each system. This poses great deal of problem in integration of disparate systems, as developers have to consider WSDLs of every other system that is to be integrated. Though SOAP allows for Service Oriented Architecture (where domain specific services are exposed via http web services), non uniformity among web services is rather limiting.

For example, consider the SOAP based services to work on Amazon S3 buckets and their individual objects , we can notice an explosion of operations to be considered. Also, though SOAP web services use HTTP protocol as transport mechanism, they use only POST http method. So, we are not taking advantage of idempotence and cacheabilty of GET http method and partial update semantics of PUT method.

Bucket Webservices

• ListAllMyBuckets
• CreateBucket
• DeleteBucket
• ListBucket
• GetBucketAccessControlPolicy
• SetBucketAccessControlPolicy
• GetBucketLoggingStatus
• SetBucketLoggingStatus

Object Webservices

• PutObjectInline
• PutObject
• CopyObject
• GetObject
• GetObjectExtended
• DeleteObject
• GetObjectAccessControlPolicy
• SetObjectAccessControlPolicy

So, next improvement in web APIs is to use a Resource Oriented API called Representational State Transfer (REST). It is an architectural style that is defined by a specific set of constraints. REST calls for layered client/server systems that employ stateless servers, liberal use of caching on the client, intermediaries, and server, a uniform interface. REST views a distributed system as a huge collection of resources that are individually managed by components. Resources may be added or removed by (remote) applications, and likewise can be retrieved or modified. [3]

There are four key characteristics of what are known as RESTful architectures

1. Resources are identified through a single naming scheme
2. All services offer the same interface, consisting of at-most four operations, as shown in Table below
3. Messages sent to or from a service are fully self-described
4. After executing an operation at a service, that component forgets everything about the caller (stateless execution)

In REST based APIs, HTTP is used as a complete application protocol that defines the semantics of service behaviour. It usually involves four HTTP methods with the below semantics:

Usual semantics of REST API is that when you do a POST on a collection of resources(which has a unique URI), a new resource is created in that collection and unique URI is returned to the newly created resource. We can perform a GET on the newly created resource URI to get all its information in some representation. Using PUT on this URI, we can update partial resource(that is, only necessary parts). We can use the DELETE operation on the URI to remove resource from the collection permanently. This application semantics holds good for any resource based services and thus helping clients to integrate disparate systems and also helps us reason about communication across subsystems easily. In REST style, server-side data are made available through representations of data in simple formats. This format is usually JSON or XML but could be anything.

Most of the above mentioned operations on AWS S3 buckets and objects can be easily modelled on only two URIs and four http methods as:

GET, POST, PUT, DELETE /api/buckets ?query_param1=val…
GET, POST, PUT, DELETE /api/buckets/:object_id ?query_param1=val…

A URI can choose support a limited number of http methods and all GET requests are idempotent thus provides for caching in intermediaries and thus greatly improves efficiency.

One of the tenets of RESTFul architecture, that is less widely used is Hypermedia which provides “next available actions” in the response of an API. Roy Fielding in his paradigm defining thesis about REST called this as HATEOAS (Hypermedia as the Engine of Application State). HATEOAS allows us to develop elegant, self discoverable API system. For example, if we develop a HATEOAS API for Multi-functionality Printers, get operation on the URI api/capabilities return printer capabilities like Print, Scan, Fax with link to access these capabilities like /api/capabilities/scan, /api/capabilities/print and /api/capabilities/fax. A GET on /api/capabilities/scan will return links to access scanner capabilities like /api/capabilities/scan/flatbed, /api/capabilities/scan/ auto_doc_feeder and so on.

A HATEOS API [5]

curl http://localhost:8080/spring-security-rest/api/customers

{
  "_embedded": {
    "customerList": [
      {
        "customerId": "10A",
        "customerName": "Jane",
        "companyName": "ABC Company",
        "_links": {
          "self": {
            "href": "http://localhost:8080/spring-security-rest/api/customers/10A"
          },
          "allOrders": {
            "href": "http://localhost:8080/spring-security-rest/api/customers/10A/orders"
          }
        }
      },
      {
        "customerId": "20B",
        "customerName": "Bob",
        "companyName": "XYZ Company",
        "_links": {
          "self": {
            "href": "http://localhost:8080/spring-security-rest/api/customers/20B"
          },
          "allOrders": {
            "href": "http://localhost:8080/spring-security-rest/api/customers/20B/orders"
          }
        }
      },
      {
        "customerId": "30C",
        "customerName": "Tim",
        "companyName": "CKV Company",
        "_links": {
          "self": {
            "href": "http://localhost:8080/spring-security-rest/api/customers/30C"
          }
        }
      }
    ]
  },
  "_links": {
    "self": {
      "href": "http://localhost:8080/spring-security-rest/api/customers"
    }
  }
}

One of the downsides of RESTFul APIs is that client may need to call multiple APIs to get different resources to piece together information needed by the client. When resources are related forming a graph of relations, it becomes extra difficult in RESTFul architecture to express the need to retrieve selective information from the web of relations among resources. A new API style that is gaining currency these days called ‘GraphQL’ mitigates this problem.[6].

GraphQL is basically RPC with a default procedure providing a query language, a little like SQL. You ask for specific resources and specific fields, and it will return that data in the response. GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more. GraphQL reduces the number of HTTP requests to retrieve data from multiple resources.

However, endpoint-based API’s are able to utilise the full capabilities of HTTP protocol to cache response data, but GraphQL dispatches queries through POST requests to a single endpoint. So, advantage of out of the box http caching is lost and API developers are needed to device custom caching mechanism themselves.

There are emerging standards for API documentation like OpenAPI pioneered by Swagger. Swagger allows for API design, API documentation, API development, API testing, API mocking, API governance and API monitoring. We can alternative documentation tools like Sphinx, along with its extensions. API documented using Sphinx looks something like below.

Another issue in maintaining resource based APIs like RESTFul APIs is versioning. We have API URI and at some point it will need to be replaced or have new features added while keeping the older APIs still supporting existing clients. There are many solutions to solve this issue, each having its own merits and demerits. One popular approach is to embed version number in the API URIs like /api/v1. But REST purists frown upon this approach as they see it breaking the fundamental concept of REST – evolvability. A resource is meant to be more like a permalink. This permalink (the URL) should never change. But the practical downside to version approach is that pointing v1 and v2 to different servers can be difficult.

This issue of server setup for different version can be resolved by putting version number in in the hostname(or subdomain) like “https://apiv1. example.com/places”. Another approach to API versioning is to put version info in body or query parameters or custom request header or as part of content negotiation. [7]

Overall, Web APIs are the new digital currency using which digital services are sold. Essentially services provided Facebook, Google, Amazon, Salesforce etc are via APIs. So, organisations take great care in defining, documenting and maintaining their Web APIs. Web APIs are public contracts and hence every software engineering due diligence exercised in developing key software systems should be followed for Web APIs also.

References:

[1] http://www.hp.com/ctg/Manual/bpl13210.pdf
[2] https://www.hpl.hp.com/hpjournal/97feb/feb97a9.pdf
[3] Distributed Systems, Third edition by Maarten van Steen and Andrew S. Tanenbaum. ISBN: 978-90-815406-2-9
[4] https://spring.io/understanding/HATEOAS
[5] https://www.baeldung.com/spring-hateoas-tutorial
[6] https://blog.apisyouwonthate.com/understanding-rpc-rest-andgraphql-2f959aadebe7
[7] Phil Sturgeon. “Build APIs You Won’t Hate”.