USA Spending API in C#
by Cyrus Hartwin Gomes
The USA Spending Website: https://www.usaspending.gov/
The USA Spending API: https://api.usaspending.gov/
These recipe examples were tested on December 12, 2023
Setup#
First, install the CURL package by typing the following command in the terminal:
!sudo apt install curl
Then, install the jq package by typing the following command in the terminal:
!sudo apt install jq
Now we create a directory for our projects to be created:
!mkdir USA_Spending
Finally, we will change the directory to the one we created:
%cd USA_Spending
1. Get agency names and toptier codes#
To obtain data from the API, it is useful to first build a dictionary containing agency names and toptier codes, the latter of which will be used to access subagency data.
First, we can initialize a folder for the current project that we are working on. And then change to that directory:
!mkdir APIdata
%cd APIdata
We utilize the %%file command to create the following makefile which will compile our program and create an executable.
%%file makefile
# Set the variable CC to gcc, which is used to build the program
CC=gcc
# Enable debugging information and enable all compiler warnings
CFLAGS=-g -Wall
# Set the bin variable as the name of the binary file we are creating
BIN=api_data
# Create the binary file with the name we put
all: $(BIN)
# Map any file ending in .c to a binary executable.
# "$<" represents the .c file and "$@" represents the target binary executable
%: %.c
# Compile the .c file using the gcc compiler with the CFLAGS and links
# resulting binary with the CURL library
$(CC) $(CFLAGS) $< -o $@ -lcurl
# Clean target which removes specific files
clean:
# Remove the binary file and an ".dSYM" (debug symbols for debugging) directories
# the RM command used -r to remove directories and -f to force delete
$(RM) -rf $(BIN) *.dSYM
The command is used again to create our .c file which contains the code for the program
%%file api_data.c
#include <curl/curl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/* CURL program that retrieves api results from usaspendig with the given id.
Custom property fields can be added */
int main (int argc, char* argv[]) {
// If arguments are invalid just return
if (argc < 2) {
printf("Error. Please try again correctly. (./api_data -id [id])\n");
return -1;
}
// Initialize the CURL HTTP connection
CURL *curl = curl_easy_init();
// Bits of the url that are joined together later
char api[] = "https://api.usaspending.gov";
char url[1000];
char default_id[] = "/api/v2/references/toptier_agencies/";
// Check if CURL initialization is a success or not
if (!curl) {
fprintf(stderr, "init failed\n");
return EXIT_FAILURE;
}
// When the command ./census_data -id is used
if ((argc==2) && (strcmp(argv[1],"-id")==0)) {
// Combine the api and the default id
sprintf(url, "%s%s", api, default_id);
}
// ./census_data -id [id]
else if ((argc==3) && (strcmp(argv[1],"-id")==0)) {
// Combine the api and the custom id
sprintf(url, "%s%s", api, argv[2]);
}
// If the arguments are invalid just return
else {
printf("./api_data -id [id]\n");
curl_easy_cleanup(curl);
return 0;
}
// Set the url to which the HTTP request will be sent to
// first parameter is for the initialized curl HTTP request, second for the option to be set, and third for the value to be set
curl_easy_setopt(curl, CURLOPT_URL, url);
// If result is not retrieved then output error
CURLcode result = curl_easy_perform(curl);
// If result is not retrieved then output error
if (result != CURLE_OK) {
fprintf(stderr, "download problem: %s\n", curl_easy_strerror(result));
}
// Deallocate memory for the CURL connection
curl_easy_cleanup(curl);
return EXIT_SUCCESS;
}
Writing api_data.c
The command is used again to create our .c file which contains the code for the program:
!make
!./api_data -id | jq '.["results"][0]'
{
"agency_id": 1146,
"toptier_code": "310",
"abbreviation": "USAB",
"agency_name": "Access Board",
"congressional_justification_url": "https://www.access-board.gov/cj",
"active_fy": "2023",
"active_fq": "4",
"outlay_amount": 9232761.09,
"obligated_amount": 8863660.56,
"budget_authority_amount": 11366458.51,
"current_total_budget_authority_amount": 11889863708699.81,
"percentage_of_total_budget_authority": 9.559788731373906e-07,
"agency_slug": "access-board"
}
# Display total number of agencies in the data
!./api_data -id | jq '.["results"] | length'
108
Now we can create a dictionary containing the agency names as keys and the toptier codes as the data.
%%bash
# Get the raw data from the api
raw_data=$(./api_data -id | jq '.["results"]')
# Modify retrieved json data to retrieve agency name and top tier code
agencies_and_top_codes=$(echo "$raw_data" | jq -r '.[0:] | map({(.["agency_name"]): .["toptier_code"]}) | add')
# Print the modified data
echo "${agencies_and_top_codes}" | tee output.txt | head -n 5
{
"Access Board": "310",
"Administrative Conference of the U.S.": "302",
"Advisory Council on Historic Preservation": "306",
"African Development Foundation": "166",
We can also print the toptier code for a particular agency. This will be useful when building URLs to view other data from the API.
# Can be changed to any department
!cat output.txt | jq -r '.["Department of Transportation"]'
069
2. Retrieving Data from Subagencies#
The toptier_codes dictionary we created above contains every agency name in the API. For this example, we’ll look at the total obligations of each subagency of the Department of Defense.
%%bash
# Retrieve the toptier_code
# Can be set to the desired department
toptier_code=$(cat output.txt | jq -r '.["Department of Defense"]')
# Create complete url for retrieving API data
url="/api/v2/agency/${toptier_code}/sub_agency/"
# Get the raw data from the API
raw_data=$(./api_data -id "${url}"| jq '.["results"]')
# Modify retrieved JSON data to retrieve agency name and top tier code
subagencies=$(echo "$raw_data" | jq -r '.[0:] | map({(.["name"]): .["total_obligations"]}) | add')
# Sort the subagencies in alphabetical order
subagencies=$(echo "$subagencies" | jq 'to_entries | sort_by(.key) | from_entries')
# Print the modified data
echo "${subagencies}" | tee subagencies.json
{
"Defense Advanced Research Projects Agency": 36296392,
"Defense Health Agency": 15718671.85,
"Defense Information Systems Agency": 1598100.49,
"Defense Threat Reduction Agency": 1958308.27,
"Department of the Air Force": 178875720.31,
"Department of the Army": 503264075.56,
"Department of the Navy": 250867292.31,
"Missile Defense Agency": 42996.26,
"Uniformed Services University of the Health Sciences": 33887146.44,
"Washington Headquarters Services": 58951661
}
3. Accessing Fiscal Data Per Year#
We can use the API to examine the annual budget of an agency from 2017 onward.
%%bash
# Retrieve the toptier_code
# Can be set to the desired department
toptier_code=$(cat output.txt | jq -r '.["Department of Health and Human Services"]')
# Create complete url for retrieving api data
url="/api/v2/agency/${toptier_code}/budgetary_resources/"
# Get the raw data from the api
raw_data=$(./api_data -id "${url}"| jq '.["agency_data_by_year"]')
# Modify retrieved json data to retrieve agency name and top tier code
# All the years before 2023 are used
# Start from index 2 as the year 2024 and 2023 are skipped
# since fiscal years cannot be used as key, it is converted to strings
budgetary_resources_data=$(echo "$raw_data" | jq -r '.[2:] | map({(.["fiscal_year"] | tostring): .["agency_total_obligated"]}) | add')
# Sort the subagencies in alphabetical order
budgetary_resources_data=$(echo "$budgetary_resources_data" | jq 'to_entries | sort_by(.key) | from_entries')
# Print the modified data
echo "${budgetary_resources_data}" | tee budgetary_resources_data.txt
{
"2017": 1646989531123.68,
"2018": 1679128003253.74,
"2019": 1814270463757.37,
"2020": 2198882208891.79,
"2021": 2355524286884.46,
"2022": 2452969781323.39
}
4. Breaking Down Budget Categories#
We can use the API to view the breakdown the spending of a particular agency.
%%bash
# Retrieve the toptier_code
# Can be set to the desired department
toptier_code=$(cat output.txt | jq -r '.["Department of the Interior"]')
# Create complete url for retrieving api data
url="/api/v2/agency/${toptier_code}/obligations_by_award_category/"
# Get the raw data from the api
obligations_by_category_data=$(./api_data -id "${url}"| jq '.')
# Output the data to .txt file
echo "${obligations_by_category_data}" > obligations_by_category_data.txt
# Print the modified data
echo "${obligations_by_category_data}" | jq '.["results"]'
[
{
"category": "contracts",
"aggregated_amount": 593973658.84
},
{
"category": "direct_payments",
"aggregated_amount": 505458682.16
},
{
"category": "grants",
"aggregated_amount": 309093580.28
},
{
"category": "idvs",
"aggregated_amount": 1033114.79
},
{
"category": "loans",
"aggregated_amount": 0
},
{
"category": "other",
"aggregated_amount": 416415.66
}
]
It may be useful to limit our results to those with aggregated amounts greater than 0:
%%bash
# Reformat the data where category : aggregated_amount and aggregated amount > 0
formatted_data=$(cat obligations_by_category_data.txt | jq -r '.results | map(select(.aggregated_amount > 0) | {(.category): .aggregated_amount}) | add')
echo "${formatted_data}"
{
"contracts": 593973658.84,
"direct_payments": 505458682.16,
"grants": 309093580.28,
"idvs": 1033114.79,
"other": 416415.66
}