The HTTP Protocol

Last time we discussed internet protocols pretty broadly, in order to gain an understanding of what's happening when we connect to a server. But as web developers, we're going to spend most of our time working with the HTTP protocol, so let's discuss that this time.

The HTTP protocol follows a client-server model of communication. The client makes a request to the server. The server then returns a response. After the response is returned, the connection is closed and they don't talk until the next request from the client.

Note that the server is software, not hardware - your laptop can be a server just like a big data center computer. Likewise, the client doesn't need to be a web browser.

Let's take a look at how HTTP requests and responses are structured.

The HTTP Request

An HTTP request has 3 parts: the request line, the headers and the body. The body is optional.

The request line looks like GET localhost:3000 HTTP/1.1. This says the HTTP method, the URL and the version of HTTP being used. There are several different HTTP methods, GET and POST are the most common. In fact, whenever you send a form in the browser, it's sent as a GET or POST request!

  • GET retrieves a resource.
  • POST sends data to the server.
  • PUT creates a resource or replaces it.
  • PATCH updates a resource.
  • DELETE deletes a resource.
  • OPTIONS gives information about who's allowed to connect to this resource, and how.

The headers contain metadata about the message. There are many different headers, but one good example is the Content-Type header, which says what type of message is being sent by a POST request. For example, it could be text/plain or application/json.

The HTTP Response

An HTTP response also has 3 parts: the status line, the headers, and the body. Like requests, the body is optional.

The status line of an HTTP response contains a status code, which tells the client what to do with the response. There are different variations, but they're indicated by a range (i.e. 400-429 are all failures, but only 418 tells you the server is a teapot). The ranges are:

  • 100 gives information (used during uploads, for example)
  • 200 means the request succeeded.
  • 301 tells the client to redirect to another page.
  • 400 means the request failed due to a problem with the request.
  • 500 means the server failed in some way.

Some headers are only found in responses, but many of them are the same as in the requests.

Playing with HTTP

This is a simple web server which will respond to different requests in different ways. Try visiting the different routes in your web browser and see what they do!

Notice how we set the status code and headers differently for each request.

const express = require('express')
const app = express()
app.use(express.text())

app.get('/', function (req, res) {
  res.status(200)
  res.send("This is a normal response.")
})

app.get('/301', function(req, res) {
  res.status(301)
  res.setHeader("Location","https://fd93.me")
  res.send()
})

app.get('/400', function(req, res) {
  res.status(400)
  res.setHeader("Content-Type","text/plain")
  res.send("It's not me, it's you.\nBad request.")
})

app.get('/500', function(req, res) {
  res.status(500)
  res.setHeader("Content-Type","text/plain")
  res.send("It's not you, it's me.\nServer error.")
})

app.get('/418', function(req, res) {
  res.sendStatus(418)
})

app.post('/', function (req, res) {
  if (req.body.length == 0) {
      res.status(400)
      res.send("Invalid input.")
  }
  else {
    res.status(200)
    res.setHeader("Content-Type","text/plain")
    res.send(req.body)
  }

})

app.listen(3000)

Intro to cURL

Testing in our browser is often not practical when testing backend services. For example, the service we want to connect to might not provide a webpage, or might be behind several layers of security. Instead we'll often use tools like cURL.

cURL (pronounced 'curl') is a UNIX command line tool which lets you send HTTP requests directly to different IP addresses. It's very useful for testing out backend interfaces and checking that they do what you expect them to.

There are also widely used tools like Insomnia which make testing backend services easy, but for now curl will work fine for us. As it's a CLI tool it also has the advantage that we can build it into our scripts very easily.

Let's try testing some different types of requests with cURL.

Try curl -D - https://fd93.me > fd93.log in your console, then open up fd93.log in your editor of choice. You should see the status line first, then a number of headers, and finally the body. Most of the log will be the body of the request.

Now run the backend app from earlier again. You'll notice that we never used the final method in that file, because it's a POST request.

Try this command and see what it returns:

curl -D - --data "" -H "Content-Type: text/plain" -X POST localhost:3000

Now try sending something as part of your POST body: 

curl -D - --data "EXAMPLE TEXT" -H "Content-Type: text/plain" -X POST localhost:3000

Note that we set the Content-Type header by using the -H flag. This is because cURL will try to use application/x-www-form-urlencoded by default when using the --data flag. This is the same format your browser uses when sending encoded form data.

Knowledge Check

  1. What do the client and the server do in HTTP?
  2. Do the client and server stay connected after the initial response in HTTP?
  3. What do the following status codes mean?
    1. 400
    2. 200
    3. 301
    4. 500
  4. What are the most common HTTP request types?
  5. What's the different between PUT and PATCH?
  6. If we want to return some text in JSON format, which header should we use?

Assignment

Starting from the original server given above, make the following changes and test them with cURL or your browser.

  1. Change the response on the / route to return a JSON object by using the header Content-Type: application/json. The format of the JSON object is up to you.
  2. Now change the response on / to return the HTML string <h1>Hello world</h1> by using Content-Type: text/html.
  3. Change the redirect on /301 to point to a different URL.
  4. Make the /418 route do something interesting by manipulating the Content-Type header and the body directly.
  5. Add new routes for the HTTP methods we haven't yet addressed: PATCH, PUT, DELETE, and OPTIONS. Make each of them return the status code 501 (not implemented).
  6. Take a look at the other HTTP status codes and what they represent. Try using some of them with cURL or your browser and see what they do.

Further Reading