JavaScript Networking: AJAX, Requests and CORS
March 19, 2018
Introduction
In application development, networking is a huge part of the process. More often than not, your application will require loading remote data as either a functional part of your app or as a way to reduce the amount of data you need in your application on the first load.
JavaScript, being a versatile language can run on both a front-end environment (in a web-browser), or in a back-end environment (such as a Node.js server or an Electron application). Because of this, there are different ways you should approach making requests and receiving data.
AJAX and XMLHttpRequest
Since 1999, Ajax (Asynchronous JavaScript & XML) has existed as a means of being able to make requests to remote servers from the client-side (one of the things old Internet Explorer actually managed to get right). The legendary XMLHttpRequest allows us to be able to make requests to remote servers.
The basic XMLHttpRequest (GET)
Here is the XMLHttpRequest in it’s basic form:
function fetchJSONFile(path, callback){
var httpRequest = new XMLHttpRequest();
httpRequest.onreadystatechange = function() {
if (httpRequest.readyState === 4) {
var success = (httpRequest.status === 200);
if (callback) callback(success ? JSON.parse(httpRequest.responseText) : null);
}
};
httpRequest.open('GET', path);
httpRequest.send();
}
fetchJSONFile('http://json-schema.org/example/card.json', function(data) {
if (data) console.log('received', data);
else console.error('request failed');
}); Above we have a simple functional representation of how to fetch a JSON file.
XMLHttpRequest is actually a constructor function, as per the use of the new keyword on line 2. We use this line to tell the Browser that we are about to make a HTTP request, which builds an object instance we can interface with.
On line 3, we bind a callback function to onreadystatechange, which will fire once the state of the HTTP request changes. This will include all changes in state, which is why on line 4 we are checking to see the end result of the request. A value of 4 means that the request has finished executing (allowing us to process the fetched data).
An import thing to note is how the data is encapsulated within the httpRequest object itself. Even though we are using a callback for onreadystatechange, that function is not passing any data into the function call. Instead, we access the httpRequest variable we initialized on line 2.
On line 5, we check the HTTP status returned by the request. In this specific example, we are checking for a status of 200, which means that the request has completed without any special conditions. HTTP codes can vary depending on where we are choosing to request data, and the specific data we request.
On line 6, we check if we have a callback defined. If we do, we then attempt to parse the value depending on whether the success condition on line 5 passed. This is done via a ternary logic, which acts as a short-hand way or assigning values based on a condition.
In this case (on line 6), if the request was a success, we will attempt to parse the JSON value received in the request, and then pass that value as the callback’s main argument. If the request was not a success, we then return null as the callback argument value.
The above example does not account for broken JSON responses (which can happen in the real-world). Keep that in mind for when configuring your own XMLHttpRequests
On line 9, we use the open method to configure the XMLHttpRequest to prepare a HTTP request of GET to the specified path (which we passed as an argument to the function fetchJSONFile).
GETis a subset of HTTP requests for specifically getting a resource from a remote location. There are several request types, including (but not limited to POST, PATCH, PUT, and HEAD).
And then on line 10, we use the send method to tell the XMLHttpRequest to execute and send the request to the remote server.
When we call fetchJSONFile on line 12, we pass both the path and the callback used for fetching the JSON. In this case, on line 13/14 we have set up the callback to log the returned response, or log an error if we are unable to.
The basic XMLHttpRequest (POST)
In the previous example, we are sending a GET request to the remote server, which simply involved fetching a remote JSON dataset and returning it’s value. However, there are likely to be several instances where we must first send JSON data to a remote server.
With a few minor alterations, we are able to convert the previous example into a function for both sending and receiving JSON.
function postJSON(path, postData, callback) {
var payload = JSON.stringify(postData);
var httpRequest = new XMLHttpRequest();
httpRequest.onreadystatechange = function() {
if (httpRequest.readyState === 4) {
var success = (httpRequest.status === 200 || httpRequest.status === 201);
if (callback) callback(success ? JSON.parse(httpRequest.responseText) : null);
}
};
httpRequest.open('POST', path);
httpRequest.setRequestHeader('Content-Type', 'application/json;charset=UTF-8');
httpRequest.send(payload);
}
postJSON('somepostableurl.com', { data: 'hello world' },function(data) {
if (data) console.log('received', data);
else console.error('request failed');
});Instead of fetchJSONFile, we named this function postJSON, as it both sends and expects JSON as a response. On line 2 we attempt to convert the postData argument into a JSON string we can send to the remote server.
Since JSON does not allow complex objects, passing a function within the
postDataparameter will cause the function call to fail, since you can’t convert a function into valid JSON with JSON.stringify.
We modified the open method on line 11 to use POST instead of GET, since we are sending the remote server data (and it will be expecting data from us). On line 12, we added a custom request header for ‘Content-Type’, which tells the remote server that we are sending data in JSON format.
This is important when interfacing with APIs, since the remote server needs to be able to determine the difference between different types of requests without having to guess. This helps reduce the amount of load on servers when sending requests, since now the server knows exactly what form of data to expect and will reject anything that doesn’t match that form.
This can also be used to secure connections, since now request bodies must follow a stricter request schema (JSON or XML) instead of accepting plain raw-text or binary streams.
We also modified the send method to pass a single argument, which is the JSON data we wish to send to the remote server.
On line 14, we can see this code in action. When calling postJSON (with a valid POST url), you can both send and receive JSON data to and from a remote server.
Making it modern (ES2015 way)
Since XMLHttpRequest is the most widely used format for making and sending requests on the client-side, it comes as no surprise that many of the examples you will find today are very outdated. Thankfully, it is more than possible to rewrite a basic XMLHttpRequest to use more of an ES2015 format:
function postJSON(path, postData, expectedResponse = 200) {
return new Promise((resolve, reject) => {
const payload = JSON.stringify(postData);
const httpRequest = new XMLHttpRequest();
httpRequest.onreadystatechange = () => {
if (httpRequest.readyState === 4) {
const success = (httpRequest.status === expectedResponse);
if (success) resolve(JSON.parse(httpRequest.responseText));
else reject(new Error('did not receive expected response'));
}
};
httpRequest.open('POST', path);
xmlhttp.setRequestHeader('Content-Type', 'application/json;charset=UTF-8');
httpRequest.send(payload);
});
}
postJSON('somepostableurl.com', { hello: 'world' })
.then(response => console.log(response))
.catch(error => console.error(error));This fairly closely resembles the original XMLHttpRequest format, save a few noticeable differences. On line 2, we are returning an ES2015 Promise, which is a way of natively handling asynchronous code using special chaining of then and catch. On line 2 as well as line 6, we are using shorthand arrow functions instead of the full function syntax, which is making this code much less verbose than the original variation.
We also defined a custom parameter called expectedResponse, which is responsible to the whether we allow request to resolve or reject. We are using the default parameters feature in ES2015 to state that the expected response should be 200 by default if none is specified.
One thing to not is the syntax resulting from the use of the returned Promise. On line 19 through 21 we are using then to handle when a response has been followed through with, and the catch to handle any internal exceptions or unexpected response from the server.
Fetch-API
In modern browsers (with the exception of Internet Explorer), a new method has been introduced called fetch, which streamlines the way requests can be sent and response received. It was implemented due to the verbosity of the XMLHttpRequest code, which, while reliable, has not aged well at all. to use the Fetch API, we simply do the following:
Basic Fetch GET example (binary response)
const request = new Request('hello-world.jpg');
fetch(request)
.then(response =>
response.blob().then((binary) => {
// do something with the binary received
})
)
.catch(error => console.error(error));The above code is a basic example of how to interface with the modern Fetch API. On line 1, we initialize a new native type called Request, which gives us a convenient way to construct request bodies for sending out multiple requests. The first parameter to the Request constructor is the url of the response we are making the request to, and the second are any special modifiers (which we will show an example shortly).
The Fetch API uses native ES2015 Promises to handle asynchronously making and receiving data. On line 3, we are using the then callback to process the response from the request (in the event that it passed). The response parameter returned is a instance of the Response object, which is a new native helper class that helps manage and streamline the parsing of data received from servers.
On line 4, we are calling the blob method on the response, which signifies that we are attempting to convert the response to a binary blob format (for manually saving or converting into whatever data we need). blob returns a Promise that resolved once the browser has finished parsing the request into the specified format, which is also the same when parsing JSON, rawText, or any other formats. We then have access to this data on line 5 as data we can handle.
The catch on line 7 handles when the request could not be completed, so any internal errors or non-successful requests will go here.
Fetch with POST data (JSON response)
Just like XMLHttpRequests, we are able to send JSON data using the post HTTP method:
const request = new Request('https://somepostablesite.com/postendpoint', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
});
fetch(request)
.then(response =>
response.json().then((jsonObject) => {
// do something with the binary received
})
)
.catch(error => console.error(error));Starting on line 1, we are passing a second parameter into the request object to specify special modifiers that need out request to have. In this specific example, we are specifying that we are using method POST, and that the server should expect a Content-Type of JSON. We then pass the request into the fetch function on line 7, and then on line 9 we are able to parse the given response as JSON using the json method.
On line 8, because the request passed we now have the full JavaScript object sent to use by the remote endpoint we sent the request to.
Server-side Requests
Node.js in JavaScript, while not having access to the Fetch API or XMLHttpRequest, requires a developer to use node modules in order to ensure that data is being sent and received correctly. The most stable way to currently send requests in Node.js is via the request module from npm (which is a package manager that manages Node.js related packages).
You can install it into a Node.js project via npm i -S request
And then access it within a npm project like this:
Sending a request (GET with JSON)
const request = require('request');
request({
uri: 'http://json-schema.org/example/card.json',
method: 'GET',
json: true,
}, (error, response) => {
if (error) return console.error('Request failed', error);
// print response body
console.log(response.body);
});In this example, we are simply using the request module’s callback form to allow us to handle requests on a single layered level.
On line 1 we included the node module request into the file, which allows us to interface with the library in order to create requests. On line 3 we start the request by defining a configuration object.
Uri represents the location of the resource we are trying to fetch, relative to the server being run. For instance, had the uri value been some-file.json instead of a full HTTP path, the request module would have looked on the server’s public file-system for that file relative to the current path. This operates much in the same way as fetching a static asset such as a stylesheet on a web-page, which does not require you to specify the full path.
On line 5, we specify the method of which the request is sent. While GET is the preferred default for most requests, for readability it’s generally a good practice to have the methods or your requests readable and concrete (making it much easier to debug).
On line 6, we are setting a conditional flag json, which tells the module that we are both sending (and likely expecting) a JSON response from the remote API. This will cause the body property on the callbacks’s response to be a full JavaScript object instead of plain-text.
On line 7, we specify the callback that runs after the request has been resolved. If there is something wrong with the request send or the response received from the remote resource, then the error argument will by truthy, set to the value of the Error object related to the problem.
A truthy value is any value to the JavaScript engine (in this case V8) that is true, not null, or has a length greater than 0. An Error object is seen by V8 as a truthy value, which is why on line 8 we are able to simply set a conditional statement to check for the presence of an error. If there is no error, the value passed will be null, which resolves to false.
The value returned in the response argument of the callback is a native Node.js HTTP response object, which contains a parsed JSON body thanks to the Request module. We can then use this data for our server.
Sending a request (POST with JSON)
const request = require('request');
request({
uri: 'http://somejsonendpoint.com/data-to-send',
method: 'POST',
json: { hello: 'world' },
}, (error, response) => {
if (error) return console.error('Request failed', error);
// print response body
console.log(response.body);
});While similar to the request above, there are two main differences to take note of.
The first is that the method value on line 5 is POST instead of GET, which tells the request module that we are likely going to be sending data. This becomes really useful on line 6, since assign the json parameter we used in the previous example to our JSON object, which represents the object we will be sending to the remote server.
On line 9 onwards, assuming the request send and the response received are valid, we should now have access to the remote API’s response on the body property of the response.
Reading files (GET with filesystem streaming)
It is also possible to read and stream remote files using the request module. Here’s a basic example of this in action:
const fs = require('fs');
const request = require('request');
request('http://thecatapi.com/api/images/get?format=src&type=jpg')
.pipe(fs.createWriteStream('./cat.jpg'))
.on('error', e => console.error(e))
.on('close', () => {
// runs after we finished streaming the resource
});This code will stream a random cat photo to ./cat.jpg on the Node.js server relative to the project’s root directory. The request module allows us to stream remote file and resources as raw binary, which the module can interpret, allowing us to pipe (stream) it to wherever we see fit.
One line 1, we included the native fs module, which gives us a way to interact with the local system filesystem, which we end up using on line 5 on the pipe callback. The pipe callback tells the request module that we wish to stream the request to another stream, which in this example is a filesystem location.
When an error occurs, the stream stops and we are notified via the error event handler on line 6. On line 7, we are also able to execute code when the stream has finished.
Cross-Origin Resource Sharing (CORS)
Cross-Origin Resource Sharing, or rather CORS, is a web feature that restricts what hosts (or rather websites/devices) have permissions to request a specified resource. This may include things like images, files, or even entire APIs.
If you have a front-end web application hosted on Domain A, and are requesting data from Domain B, depending on the headers set on the response request (from Domain B) the client might not have access to that given resource.
An example of this in action would be the header Access-Control-Allow-Origin, which specifies which Domains are permitted to make requests to a given domain or resource. An Access-Control-Allow-Origin value of * would mean that all requests to a given resource are permitted, while values like www.google.ca would mean that only google.ca would be allowed to make requests to that specific endpoint or resource.
An example of CORS in action (Using Express + CORS)
const express = require('express');
const cors = require('cors');
// create web application
const app = express();
const PORT = 8080;
const corsOptions = {
origin: 'http://yourdomain.ca',
optionsSuccessStatus: 200,
};
app
.use(cors(corsOptions))
.get('/item/:id', (req, res, next) =>
res.json({ msg: 'This is CORS-enabled for only yourdomain.ca.' }))
.listen(PORT, () => console.log(`listening on port ${PORT}`));In the above example, we are including libraries for express, which is a popular Node.js web framework, and CORS, which is a middleware for creating and parsing CORS related endpoints.
On line 8, we define the CORS configuration we would like to use for our entire web app. In this example, we are only allowing the domain http://yourdomain.ca from making requests to the web application. This means that any requests from other domains will be blocked by default.
The actual web application setup begins on line 13, where on line 14 we tell express to use our CORS configuration on all endpoints within our application. Following on line 15 and 16, we are adding a testable GET endpoint to our application.
On line 17, we finally start our application on the specified port, which in this case is port 8080 on our development system. Any subsequent requests will require the requesting domain to be http://yourdomain.ca in order for a response to be sent to the client.
Why CORS?
CORS is a good feature to have — when configured, it can prevent other sites from trying to load or access your private api or content. Some websites are known to mock other websites in order to do malicious things like steal banking information. By implementing CORS, you are preventing third-party sources from using your content or API for their own purposes.
Conclusion
Making requests is a fundamental part of how web frameworks are built, since sending and receiving data is needed in order to allow dynamic content and web applications to function. We went over ways of sending requests in a web browser, as well as within a Node.js environment.