16 min read

Fundamentals of backend engineering — HTTP

Hi Guys, If you have followed me for a while, you must know almost all of my articles are based on a real world project and I cover steps to create and deploy that project.

Hi Guys, If you have followed me for a while, you must know almost all of my articles are based on a real world project and I cover steps to create and deploy that project. I have created several Full stack projects and I have been working in the industry for almost 4 years now. There are lot of things I have learnt along the way and I’m trying to use this article to share those experiences with you.

In software engineering, most of the time there are no right or wrongs, what we usually do is we think about the trade offs between different approaches and go with the one that make sense. As the first part of the backend engineering fundamentals, I will discuss about HTTP.

HTTP Requests

Fundamentals of backend engineering — HTTP — figure 1

Let’s take a practical example where you send a letter to a friend who is living in another country. Letter first fetched by the mail man living in your area, then taken to a one distribution centre near your area, from there it will be taken from a different mail man and all these details are abstracted to you. In a similar way when we send a request from our web browser, we don’t see the underlying network implementations, but it goes from client to a server and then respond. And similar to mail delivery, we send our address and recipient address and data with the request. In networking we have different models like TCP/IP and OSI. So only the Application layer which is the top most layer visible to everyday users.

HTTP versions

HTTP was introduced mainly as a protocol to support communication between browsers and web servers. This used TCP as the transport protocol. It has evolved over the period of time from it’s version 0.9 (1991) to 3 (2022 released).

  • 0.9 (First release) — Only allowed getting information from a server. GET method was the only method supported. Released as a plain text protocol.
  • 1 — Introduced Header, Versioning, Status code, Content-type, New methods (POST, HEAD).
  • 1.1 — Introduced Host header, Persistent connections, Continue status, New methods (PUT, PATCH, DELETE, CONNECT, TRACE, and OPTIONS)
  • 2— Request multiplexing, Request prioritization, Automatic compressing, Connection reset, Server push, More importantly this release made HTTP a binary protocol.
  • 3 — Major change is transport protocol used is QUIC (On top of UDP)

Three basic features that make HTTP simple and powerful

  • Connectionless protocol — which means the browser submits the request and the client then disconnects from the server. The client waits for a response after disconnecting.
  • HTTP is media independent and it can transport any data type if both the client and server are able to handle the data content.
  • Stateless — Because HTTP is connectionless. Client and server are aware of each other only during the connected period and once the connection ends, data is not retained by either party.

Analogy of a HTTP request

  • Request line — Its starts with method token followed by the Request URI, protocol version, and ending with (a carriage return by a line feed). Elements are separated by SP characters (Request-Line = Method SP Request-URI SP HTTP-Version CRLF)
  • The resource identified by a request
  • Request header fields

HTTP request Methods

  • GET — The _GET_ method requests a representation of the specified resource. Requests using _GET_ should only retrieve data.
  • HEAD — The _HEAD_ method asks for a response identical to a _GET_ request, but without the response body.
  • POST — The _POST_ method submits an entity to the specified resource, often causing a change in state or side effects on the server.
  • PUT — The _PUT_ method replaces all current representations of the target resource with the request payload.
  • DELETE — The _DELETE_ method deletes the specified resource.
  • CONNECT — The _CONNECT_ method establishes a tunnel to the server identified by the target resource.
  • OPTIONS — The _OPTIONS_ method describes the communication options for the target resource.
  • TRACE — The _TRACE_ method performs a message loop-back test along the path to the target resource.
  • PATCH — The _PATCH_ method applies partial modifications to a resource.

Safe methods

So over the years with new releases, HTTP has introduced sevaral methods. Some of the request methods are known as safe methods. The purpose of these methods are just to retrive information and do not change the state of origin server.

Ex: HEAD, GET, OPTION and TRACE

Idempotent methods

An HTTP method is idempotent if the intended effect on the server of making a single request is the same as the effect of making several identical requests. Simple meaning is repeating the operation multiple times produces the same result as executing it once

Ex: GET, HEAD, PUT, DELETE, OPTIONS and TRACE

HTTP authentication

The general HTTP authentication framework is the base for a number of authentication schemes.

  • Basic — base64-encoded credentials
  • Bearer — bearer tokens to access OAuth 2.0-protected resources
  • Digest
  • HOBA
  • Mutual
  • Negotiate/NTLM
  • VAPID
  • SCRAM
  • AWS4-HMAC-SHA256

HTTP cookies

An HTTP cookie (web cookie, browser cookie) is a small piece of data that a server sends to a user’s web browser. The browser may store the cookie and send it back to the same server with later requests. Typically, an HTTP cookie is used to tell if two requests come from the same browser — keeping a user logged in, for example. It remembers stateful information for the stateless HTTP protocol.

  • Session management
  • Personalisation
  • Tracking

HTTP conditional requests

HTTP has a concept of conditional requests, where the result, and even the success of a request, can be changed by comparing the affected resources with the value of a validator. Such requests can be useful to validate the content of a cache, and sparing a useless control, to verify the integrity of a document, like when resuming a download, or when preventing lost updates when uploading or modifying a document on the server.

HTTP Compression

  • File format compression (loss less and lossy are 2 types)
  • End-to-end compression
  • Hop-by-hop compression

HTTP Caches

The HTTP cache stores a response associated with a request and reuses the stored response for subsequent requests.

  • Private caches — A private cache is a cache tied to a specific client — typically a browser cache. Since the stored response is not shared with other clients, a private cache can store a personalized response for that user.
  • Shared caches — The shared cache is located between the client and the server and can store responses that can be shared among users. And shared caches can be further sub-classified into proxy caches and managed caches.
  • Proxy caches — In addition to the function of access control, some proxies implement caching to reduce traffic out of the network. This is usually not managed by the service developer, so it must be controlled by appropriate HTTP headers and so on.
  • Managed caches — Managed caches are explicitly deployed by service developers to offload the origin server and to deliver content efficiently. Examples include reverse proxies, CDNs, and service workers in combination with the Cache API.

HTTPS

HTTP is not a secured protocol as the request is sent in plain text. So for someone monitoring the session can read what’s in the request.

The SSL (Secure Sockets Layer) protocol was added to HTTP to provide a layer of encryption between browsers and servers. You have to add certificates when creating the HTTPS server.

Content negotiation

In HTTP, content negotiation is the mechanism that is used for serving different representations of a resource to the same URI to help the user agent specify which representation is best suited for the user (for example, which document language, which image format, or which content encoding).

  • Server-driven Negotiation — Best representation is made by an algorithm which is located at the server
  • Agent-driven Negotiation — The user agent performs the selection of the best representation for a response after receiving an initial response from the origin server
  • Transparent Negotiation — It is a combination of both server-driven negotiation and agent-driven negotiation

HTTP response status codes

Information responses

  • 100 — Continue (This interim response indicates that the client should continue the request or ignore the response if the request is already finished.)
  • 101 — Switching Protocols (This code is sent in response to an [_Upgrade_](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade) request header from the client and indicates the protocol the server is switching to.)
  • 102 — Processing (This code indicates that the server has received and is processing the request, but no response is available yet.)
  • 103 — Early hints (This status code is primarily intended to be used with the [_Link_](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Link) header, letting the user agent start preloading resources while the server prepares a response.)

Successful responses

  • 200 — OK (The request succeeded.)
  • 201 — Created (The request succeeded, and a new resource was created as a result. This is typically the response sent after _POST_ requests, or some _PUT_ requests.)
  • 202 — Accepted (The request has been received but not yet acted upon. It is noncommittal, since there is no way in HTTP to later send an asynchronous response indicating the outcome of the request. It is intended for cases where another process or server handles the request, or for batch processing.)
  • 203 — Non Authoritative Information ( This response code means the returned metadata is not exactly the same as is available from the origin server, but is collected from a local or a third-party copy. This is mostly used for mirrors or backups of another resource. Except for that specific case, the _200 OK_ response is preferred to this status.)
  • 204 — No content (There is no content to send for this request, but the headers may be useful. The user agent may update its cached headers for this resource with the new ones.)
  • 205 — Reset content (Tells the user agent to reset the document which sent this request.)
  • 206 — Partial Content (This response code is used when the [_Range_](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Range) header is sent from the client to request only part of a resource.)
  • 207 — Multi status (Conveys information about multiple resources, for situations where multiple status codes might be appropriate.)
  • 208 — Already reported (Used inside a _<dav:propstat>_ response element to avoid repeatedly enumerating the internal members of multiple bindings to the same collection.)
  • 226 — IM used (The server has fulfilled a _GET_ request for the resource, and the response is a representation of the result of one or more instance-manipulations applied to the current instance.)

Redirection messages

  • 300 — Multiple choices (The request has more than one possible response. The user agent or user should choose one of them. (There is no standardized way of choosing one of the responses, but HTML links to the possibilities are recommended so the user can pick.))
  • 301 — Moved Permanently (The URL of the requested resource has been changed permanently. The new URL is given in the response.)
  • 302 — Found (This response code means that the URI of requested resource has been changed temporarily. Further changes in the URI might be made in the future. Therefore, this same URI should be used by the client in future requests.)
  • 303 — See other (The server sent this response to direct the client to get the requested resource at another URI with a GET request.)
  • 304 — Not modified (This is used for caching purposes. It tells the client that the response has not been modified, so the client can continue to use the same cached version of the response.)
  • 307 — Temporary redirect (The server sends this response to direct the client to get the requested resource at another URI with the same method that was used in the prior request. This has the same semantics as the _302 Found_ HTTP response code, with the exception that the user agent must not change the HTTP method used: if a _POST_ was used in the first request, a _POST_ must be used in the second request.)
  • 308 — Permanent redirect (This means that the resource is now permanently located at another URI, specified by the _Location:_ HTTP Response header. This has the same semantics as the _301 Moved Permanently_ HTTP response code, with the exception that the user agent must not change the HTTP method used: if a _POST_ was used in the first request, a _POST_ must be used in the second request.)

Client error responses

  • 400 — Bad request (The server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).)
  • 401 — Unauthorized (Although the HTTP standard specifies “unauthorized”, semantically this response means “unauthenticated”. That is, the client must authenticate itself to get the requested response.)
  • 402 — Payment required (This response code is reserved for future use. The initial aim for creating this code was using it for digital payment systems, however this status code is used very rarely and no standard convention exists.)
  • 403 — Forbidden (The client does not have access rights to the content; that is, it is unauthorized, so the server is refusing to give the requested resource. Unlike _401 Unauthorized_, the client’s identity is known to the server.)
  • 404 — Not found (The server cannot find the requested resource. In the browser, this means the URL is not recognized. In an API, this can also mean that the endpoint is valid but the resource itself does not exist. Servers may also send this response instead of _403 Forbidden_ to hide the existence of a resource from an unauthorized client. This response code is probably the most well known due to its frequent occurrence on the web.)
  • 405 — Method not allowed (The request method is known by the server but is not supported by the target resource. For example, an API may not allow calling _DELETE_ to remove a resource.)
  • 406 — Not Acceptable (This response is sent when the web server, after performing server-driven content negotiation, doesn’t find any content that conforms to the criteria given by the user agent.)
  • 407 — Proxy authentication required (This is similar to _401 Unauthorized_ but authentication is needed to be done by a proxy.)
  • 408 — Request timeout (This response is sent on an idle connection by some servers, even without any previous request by the client. It means that the server would like to shut down this unused connection. This response is used much more since some browsers, like Chrome, Firefox 27+, or IE9, use HTTP pre-connection mechanisms to speed up surfing. Also note that some servers merely shut down the connection without sending this message.)
  • 409 — Conflict (This response is sent when a request conflicts with the current state of the server.)
  • 410 — Gone (This response is sent when the requested content has been permanently deleted from server, with no forwarding address. Clients are expected to remove their caches and links to the resource. The HTTP specification intends this status code to be used for “limited-time, promotional services”. APIs should not feel compelled to indicate resources that have been deleted with this status code.)
  • 411 — Length required (Server rejected the request because the _Content-Length_ header field is not defined and the server requires it.)
  • 412 — Preconditioned failed (The client has indicated preconditions in its headers which the server does not meet.)
  • 413 — Payload too large (Request entity is larger than limits defined by server. The server might close the connection or return an _Retry-After_ header field.)
  • 414 — URI too long (The URI requested by the client is longer than the server is willing to interpret.)
  • 415 — Unsupported media type (The media format of the requested data is not supported by the server, so the server is rejecting the request.)
  • 416 — Range not satisfiable (The range specified by the _Range_ header field in the request cannot be fulfilled. It’s possible that the range is outside the size of the target URI’s data.)
  • 417 — Expectation failed (This response code means the expectation indicated by the _Expect_ request header field cannot be met by the server.)
  • 418 — I’m a teapot (The server refuses the attempt to brew coffee with a teapot.)
  • 421 — Misdirected request (The request was directed at a server that is not able to produce a response. This can be sent by a server that is not configured to produce responses for the combination of scheme and authority that are included in the request URI.)
  • 422 — Unprocessable content (The request was well-formed but was unable to be followed due to semantic errors.)
  • 423 — Locked (The resource that is being accessed is locked.)
  • 424 — Failed depenedancy (The request failed due to failure of a previous request.)
  • 425 — Too Ealry (Indicates that the server is unwilling to risk processing a request that might be replayed.)
  • 426 — Upgrade required (The server refuses to perform the request using the current protocol but might be willing to do so after the client upgrades to a different protocol. The server sends an [_Upgrade_](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade) header in a 426 response to indicate the required protocol(s).)
  • 428 — Precondition required (_The origin server requires the request to be conditional. This response is intended to prevent the ‘lost update’ problem, where a client __GET__s a resource’s state, modifies it and __PUT_s it back to the server, when meanwhile a third party has modified the state on the server, leading to a conflict.)
  • 429 — Too many requests (The user has sent too many requests in a given amount of time (“rate limiting”).)
  • 431 — Request header fields too large (The server is unwilling to process the request because its header fields are too large. The request may be resubmitted after reducing the size of the request header fields.)
  • 451 — Unavailable for legal reasons (The user agent requested a resource that cannot legally be provided, such as a web page censored by a government.)

Server error responses

  • 500 — Internal server error (The server has encountered a situation it does not know how to handle.)
  • 501 — Not implemented (The request method is not supported by the server and cannot be handled. The only methods that servers are required to support (and therefore that must not return this code) are _GET_ and _HEAD_.)
  • 502 — Bad gateway (This error response means that the server, while working as a gateway to get a response needed to handle the request, got an invalid response.)
  • 503 — Service unavailable (The server is not ready to handle the request. Common causes are a server that is down for maintenance or that is overloaded. Note that together with this response, a user-friendly page explaining the problem should be sent. This response should be used for temporary conditions and the _Retry-After_ HTTP header should, if possible, contain the estimated time before the recovery of the service. The webmaster must also take care about the caching-related headers that are sent along with this response, as these temporary condition responses should usually not be cached.)
  • 504 — Gateway timeout (This error response is given when the server is acting as a gateway and cannot get a response in time.)
  • 505 — HTTP version not supported (The HTTP version used in the request is not supported by the server.)
  • 506 — Varaint also negotiates (The server has an internal configuration error: the chosen variant resource is configured to engage in transparent content negotiation itself, and is therefore not a proper end point in the negotiation process.)
  • 507 — Insufficient storage (The method could not be performed on the resource because the server is unable to store the representation needed to successfully complete the request.)
  • 508 — Loop detected (The server detected an infinite loop while processing the request.)
  • 510 — Not extended (Further extensions to the request are required for the server to fulfill it.)
  • 511 — Network authentication required (Indicates that the client needs to authenticate to gain network access.)

This is a long list of responses and you don’t need all to know. Btw please note that most of these informations are from

So this is the end of the HTTP note :P see you with another tutorial.

Happy Coding :D

Also published on Medium.