ScienceDirect API in C#
by Cyrus Gomes
These recipe examples use the Elsevier ScienceDirect Article (Full-Text) API. Code was tested and sample data downloaded from the ScienceDirect API on February 2024 via http://api.elsevier.com and https://www.sciencedirect.com/.
You will need to register for an API key from the Elsevier Developer portal in order to use the ScienceDirect API. This tutorial content is intended to help facillitate academic research. Before continuing or reusing any of this code, please be aware of Elsevier’s API policies and appropiate use-cases, as for example, Elsevier has detailed policies regarding text and data mining of Elsevier full-text content. If you have copyright or other related text and data mining questions, please contact The University of Alabama Libraries.
ScienceDirect APIs Specification: https://dev.elsevier.com/sd_api_spec.html
Elsevier How to Guide: Text Mining: https://dev.elsevier.com/tecdoc_text_mining.html
Setup#
First, install the CURL and jq packages by typing the following command in the terminal:
!sudo apt install curl jq
Then, we set a directory where we want the Science_Direct directory for our projects to be created:
!mkdir Science_Direct
Finally, we change the directory to the folder we created:
%cd Science_Direct
Create a variable for API Key#
Save your API key to a separate text file, then create a variable for your key. Avoid displaying your API key in your terminal (to prevent accidental sharing).
# Create the key file
!touch "apiKey.txt"
We use the following command to access the key as Jupyter does not allow variable sharing for bash scripts.
# Read the key from the file
!apiKey=$(cat "apiKey.txt")
Identifier Note#
We will use DOIs as the article identifiers. See our Crossref and Scopus API tutorials for workflows on how to create lists of DOIs and identfiers for specific searches and journals. The Elsevier ScienceDirect Article (Full-Text) API also accepts other identifiers like Scopus IDs and PubMed IDs (see API specification documents linked above).
Create an executable for API calls#
First, we can initialize a folder for the all the project files and change to that directory:
!mkdir api_results
%cd api_results
We utilize the %%file command to create the following makefile which will compile our program and create an executable.
%%file makefile
# Set the variable CC to gcc, which is used to build the program
CC=gcc
# Enable debugging information and enable all compiler warnings
CFLAGS=-g -Wall
# Set the bin variable as the name of the binary file we are creating
BIN=sDirect_data
# Create the binary file with the name we put
all: $(BIN)
# Map any file ending in .c to a binary executable.
# "$<" represents the .c file and "$@" represents the target binary executable
%: %.c
# Compile the .c file using the gcc compiler with the CFLAGS and links
# resulting binary with the CURL library
$(CC) $(CFLAGS) $< -o $@ -lcurl
# Clean target which removes specific files
clean:
# Remove the binary file and an ".dSYM" (debug symbols for debugging) directories
# the RM command used -r to remove directories and -f to force delete
$(RM) -rf $(BIN) *.dSYM
Writing makefile
This command is used again to create our .c file which contains the code for the program
%%file sDirect_data.c
#include <curl/curl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/*CURL program that retrieves Science Direct data from
https://api.elsevier.com/content/article/doi/ */
int main (int argc, char* argv[]) {
// If arguments are invalid then return
if (argc < 2) {
printf("Error. Please try again correctly. (./sDirect_data -doi [doi] -key [key])\n");
return -1;
}
// Initialize the CURL HTTP connection
CURL *curl = curl_easy_init();
// Bits of the URL that are joined together later
char api[] = "https://api.elsevier.com/content/article/doi/";
char url[1000];
char label1[] = "?APIKey=";
char label2[] = "&httpAccept=text/xml";
char doi[] = "10.1016/j.tetlet.2017.07.080";
// Check if CURL initialization is a success or not
if (!curl) {
fprintf(stderr, "init failed\n");
return EXIT_FAILURE;
}
/* Here are different ways of calling the program in the
command line and integrating doi and parameter fields.*/
// Has the -doi flag: /sDirect_data -doi
if ((argc==2) && (strcmp(argv[1],"-doi")==0)) {
// Combine the API and default DOI to produce a functioning URL
sprintf(url, "%s%s", api, doi);
}
// Has the -doi flag and field: ./sDirect_data -doi [doi]
else if ((argc==3) && (strcmp(argv[1],"-doi")==0)) {
// Combine the API and custom DOI
sprintf(url, "%s%s", api, argv[2]);
}
// Has the -doi and -key flags and the key field: ./sDirect_data -doi -key [key]
else if ((argc==4) && (strcmp(argv[2],"-key")==0) && (strcmp(argv[1],"-doi")==0)) {
// Combine the API, default DOI, and key to produce a functioning URL
sprintf(url, "%s%s%s%s%s", api, doi, label1, argv[3], label2);
}
// Has the -key and -doi flags and the key and doi field: ./sDirect_data -key [key] -doi [doi]
else if ((argc==5) && (strcmp(argv[1],"-key")==0) && (strcmp(argv[3],"-doi")==0)) {
// Combine the API, custom DOI, and key to produce the URL
sprintf(url, "%s%s%s%s%s", api, argv[4], label1, argv[2], label2);
}
// Has the -doi and -key flags and the doi and key field: ./sDirect_data -doi [doi] -key [key]
else if ((argc==5) && (strcmp(argv[3],"-key")==0)) {
//combines the API, custom DOI, and key to produce the URL
sprintf(url, "%s%s%s%s%s", api, argv[2], label1, argv[4], label2);
}
// If the arguments are invalid then return
else {
printf("./sDirect_data -doi [doi] -key [key]\n");
curl_easy_cleanup(curl);
return 0;
}
// Set the URL to which the HTTP request will be sent to
// First parameter is for the initialized curl HTTP request, second for the option to be set, and third for the value to be set
curl_easy_setopt(curl, CURLOPT_URL, url);
// If result is not retrieved then output error
CURLcode result = curl_easy_perform(curl);
// If result is not retrieved then output error
if (result != CURLE_OK) {
fprintf(stderr, "download problem: %s\n", curl_easy_strerror(result));
}
// Deallocate memory for the CURL connection
curl_easy_cleanup(curl);
return EXIT_SUCCESS;
}
Overwriting sDirect_data.c
!make
gcc -g -Wall sDirect_data.c -o sDirect_data -lcurl
1. Retrieve full-text XML of an article#
This example downloads an XML file with the article full-text by calling the API. The DOI used in this example comes from a Tetrahedron Letters article:
%%bash
# Store the key in the key variable
key=$(cat apiKey.txt)
# -key [key] can also be used to input the key to program
# ./sDirect_data -doi "$doi" -key "$key"
# Call the program using a doi and assign it to a variable
fulltext1=$(./sDirect_data -doi "10.1016/j.tetlet.2017.07.080")
# Save the output to fulltext1.xml
echo "$fulltext1" > fulltext1.xml
2. Retrieve plain text of an article#
This example downloads a text file with the article full-text by calling the API. The DOI used in this example comes from a Tetrahedron Letters article:
%%bash
# Store the key in the key variable
key=$(cat apiKey.txt)
# -key [key] can also be used to input the key to program
# ./sDirect_data -doi "$doi" -key "$key"
# Call the program using a doi and assign it to a variable
fulltext2=$(./sDirect_data -doi "10.1016/j.tetlet.2022.153680")
# Save the output to fulltext2.txt
echo "$fulltext2" > fulltext2.txt
3. Retrieve full-text in a loop#
This example retrieves the full-text for a list of articles given their DOIs. This example downloads the articles as plain text, and the examples are Tetrahedron Letters articles.
%%bash
# List of 5 DOIs for testing
dois=('10.1016/j.tetlet.2018.10.031',
'10.1016/j.tetlet.2018.10.033',
'10.1016/j.tetlet.2018.10.034',
'10.1016/j.tetlet.2018.10.038',
'10.1016/j.tetlet.2018.10.041')
# Store the key in the key variable
key=$(cat apiKey.txt)
# Call the program using a DOI and assign it to a variable
for doi in "${dois[@]}"; do
# Can't save files with a '/' character on Linux
filename=$(echo "$doi" | tr '/' '_')
# Concatenate "_plain_text.txt" to the filename
filename="${filename}_plain_text.txt"
# -key [key] can also be used to input the key to program
# ./sDirect_data -doi "$doi" -key "$key"
# Call the program using a DOI and assign it to a variable
article=$(./sDirect_data -doi "$doi")
# Save the output to a .txt file
echo "$article" > "$filename.txt"
done