rOpenGov R packages for open government data analytics

usdoj: For Accessing U.S. Department of Justice (DOJ) Open Data

Sat, 01 Apr 2023 00:00:00 +0000

usdoj, a package for fetching data from the United States (US) Department of Justice (DOJ) API, was released as part of the rOpenGov project. usdoj provides easy access to US DOJ press releases, blog entries, and speeches. Optional parameters allow users to specify the number of results starting from the earliest or latest entries, and whether these results contain keywords. Data is cleaned for analysis and returned in a data frame.

US DOJ press releases, blog posts, and speeches are an official media through which the United States government publicizes front line information about law, enforcement, and crime that may be of interest to members of the public, researchers and analysts, and members of other government branches. They include coverage for divisions such as the Federal Bureau of Investigation (FBI), the Offices of the United States Attorneys (USAO), the National Security Division (NSD), the Civil Division, the Tax Division, the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF), the Drug Enforcement Administration (DEA), and more. New media are published on a regular basis

usdoj makes this media accessible in an analysis-ready format through three functions that search for and return relevant results: doj_press_releases(), doj_blog_posts(), and doj_speeches(). Data is cleaned and structured before it is returned as a data frame with fields for the body text, date, title, url, the name of the corresponding division, to name just a few.

library(usmap)
library(lubridate)
library(tidyverse)
library(usdoj)

# press_releases <- doj_press_releases(n_results = 100000, search_direction = "DESC")
# write_csv(press_releases, "press_releases_doj_intro.csv")
press_releases <- read_csv("press_releases_doj_intro.csv")

state <- statepop$full
count <- list()

for(state_name in state) {
  count <- append(count, sum(str_count(press_releases$name, state_name))) }

df <- data.frame(state = unlist(state), count = unlist(count))

earliest_date <- ymd(min(press_releases$date))
earliest_date <- paste0(month(earliest_date, label = TRUE), " ", day(earliest_date), ", ", year(earliest_date))
  
latest_date <- ymd(max(press_releases$date))
latest_date <- paste0(month(latest_date, label = TRUE), " ", day(latest_date), ", ", year(latest_date))

plot_usmap(data = df, 
           values = "count", 
           color = "#4682b4") + 
  scale_fill_continuous(low = "white", 
                        high = "#4682b4", 
                        name = "n", 
                        label = scales::comma) + 
  theme(legend.position = "right") +
  labs(title = "US DOJ Press Releases Involving the FBI Corresponding to State", 
       subtitle = paste0("Raw Count From ", earliest_date, " to ", latest_date), 
       caption = "This plot was generated using data from usdoj. It visualizes the raw count of press releases that are tagged 
       as involving both the FBI and a state's office of the United States Attorney.")

Demonstration: Text Mining United States Department of Justice Open Data

The data returned by usdoj is in a format that can easily undergo additional processing for analysis. The purpose of this section is to show one way of doing this while walking through the steps for performing a TF-IDF (term frequency-inverse document frequency) analysis in order to see which words are characteristic to certain divisions, and not others.

Installing and Loading Libraries

usdoj can be installed from CRAN (using install.packages("usdoj")) or from rOpenGov’s r-universe. For this tutorial we will also use the tidyverse and tidytext libraries.

library(usdoj)
library(usmap)
library(tidyverse)
library(tidytext)
library(lubridate)

We will start by collecting US DOJ press releases with the corresponding function, doj_press_releases(). By default, the most recently published records are returned. Passing search_direction = "ASC" to the function will instead return data starting at the earliest published records. usdoj automatically flattens nested fields. The resulting data frame is easily text mined.

press_releases <- doj_press_releases(n_results = 700)

We will also save the date range present in the data for use in our visualization (later on).

earliest_date <- ymd(min(press_releases$date))
earliest_date <- paste0(month(earliest_date, label = TRUE), " ", day(earliest_date), ", ", year(earliest_date))
  
latest_date <- ymd(max(press_releases$date))
latest_date <- paste0(month(latest_date, label = TRUE), " ", day(latest_date), ", ", year(latest_date))

write_csv(press_releases, "intro_pt_2.csv")
press_releases <- read_csv("intro_pt_2.csv", show_col_types = FALSE)

A single field may contain multiple values. For example, the field “name” contains the (sometimes multiple) US DOJ divisions related to a press release, as shown by lines 7 and 9. A single press release may relate to USAOs across multiple states or may implicate multiple offices.

head(press_releases$name, 10)

[1] "Office of the Attorney General"            
[2] "Civil Rights Division"                     
[3] "Civil Division"                            
[4] "Criminal Division"                         
[5] "Environment and Natural Resources Division"
[6] "Office of the Deputy Attorney General"     
[7] "Environment and Natural Resources Division"
[8] "Tax Division"                              
[9] "Criminal Division"                         
[10] "Tax Division"

In this demonstration we will process the text body, transforming the dense blocks of natural language text into a structure that is more easily quantifiable.

tail(press_releases$body, 2)

For this demonstration, we will just compare the words relating to United States Attorney Offices (USAOs) across different states. We will do this by removing mentions of the other divisions from the “name” field and filtering for just press releases that contain USAO as a division.

state_names <- paste(statepop$full, collapse = "|USAO - ")

press_releases$name <-  str_extract(press_releases$name, paste0("USAO - ", state_names))

usao_press_releases <- press_releases %>%
  filter(str_detect(name, "USAO"))

The following code tokenizes the body text, a process through which dense paragraphs are separated into one-word-per-row.

tokenized_press_releases <- usao_press_releases %>%
  select(body, name) %>%
  unnest_tokens(word, body)

For this demonstration we will remove digits because they occur frequently in the data set and, for our purposes, they don’t reveal much meaningful information.

cleaned_tokenized_press_releases <- tokenized_press_releases %>%
  slice(which(!str_detect(word, "[[:digit:]]")))

In preparation of performing a TF-IDF analysis, we will count the number of times a word appears in each unique “name” grouping. In other words, if the same word appears in “Civil Division” and “Antitrust Division,” then the count will be “one” for each division (as opposed to “two,” reflecting the overall count). To remove typos and other such errors, we will also remove words that have been stated less than 5 times.

counted_tokenized_press_releases <- cleaned_tokenized_press_releases %>%
  count(name, word) %>%
  filter(n > 5)

head(counted_tokenized_press_releases)

We will now gather the overall word count per “name” grouping and use bind_tf_idf() to see which words are characteristic of one grouping and not the others.

total_words_per_group <- counted_tokenized_press_releases %>% 
  group_by(name) %>% 
  summarize(total = sum(n)) %>%
  ungroup()

counts_and_totals <- left_join(counted_tokenized_press_releases, total_words_per_group)

usao_press_releases_tf_idf <- counts_and_totals %>%
  bind_tf_idf(word, name, n)

We can now visualize which words are characteristic of one “name” grouping and not another. In the following code we take the top 10 words per name grouping and plot them based on their TF-IDF scores.

top_usao_press_releases <- usao_press_releases_tf_idf %>%
  group_by(name) %>%
  arrange(desc(tf_idf)) %>%
  slice(1:10) %>%
  ungroup() 

ggplot(top_usao_press_releases,
       aes(tf_idf, reorder_within(word, tf_idf, name),
           fill = name)) +
  labs(title = "TF-IDF Scores By USAO Grouping",
       subtitle = paste0("US DOJ Press Releases From ", earliest_date, " to ", latest_date)) + 
  geom_col(show.legend = FALSE) +
  facet_wrap(~name, ncol = 2, scales = "free") +
  scale_y_reordered() + 
  labs(x = "tf-idf", y = NULL)

Regions package for Eurostat sub-national statistics

Wed, 16 Jun 2021 00:00:00 +0000

The new version of our rOpenGov R package regions was released today on CRAN. This package is one of the engines of our experimental open data-as-service Green Deal Data Observatory, Economy Data Observatory, and Digital Music Observatory prototypes, which aim to place open data packages into open-source applications.

In international comparison the use of nationally aggregated indicators often have many disadvantages: they inhibit very different levels of homogeneity, and data is often very limited in number of observations for a cross-sectional analysis. When comparing European countries, a few missing cases can limit the cross-section of countries to around 20 cases which disallows the use of many analytical methods.

Working with sub-national statistics has many advantages: the similarity of the aggregation level and high number of observations can allow more precise control of model parameters and errors, and the number of observations grows from 20 to 200-300.

Figure 1: The change from national to sub-national level comes with a huge data processing price: internal administrative boundaries, their names, codes codes change very frequently.

Yet the change from national to sub-national level comes with a huge data processing price. While national boundaries are relatively stable, with only a handful of changes in each recent decade, the change of national boundaries requires a more-or-less global consensus. But states are free to change their internal administrative boundaries, and they do it with high frequency. This means that the names, identification codes and boundary definitions of sub-national regions change very frequently. Joining data from different sources and different years can therefore be very difficult.

Figure 2: Our regions R package helps the data processing, validation and imputation of sub-national, regional datasets and their coding.

There are numerous advantages in switching from a national level of the analysis to a sub-national level. This, however, requires more effort in data processing, validation and imputation. Our regions package aims to help in this process.

You can review the problem, and the code that created the two map comparisons, in the Mapping Regional Data, Mapping Metadata Problems vignette article of the package. A more detailed problem description can be found in the Working With Regional, Sub-National Statistical Products vignette.

This package is an offspring of the eurostat package on rOpenGov. It started as a tool to validate and re-code regional Eurostat statistics, but it aims to be a general solution for all sub-national statistics. It will be developed parallel with other rOpenGov packages.

Get the Package

You can install the development version from GitHub with:

devtools::install_github("rOpenGov/regions")

or the released version from CRAN:

install.packages("regions")

You can review the complete package documentation on regions.dataobservaotry.eu. If you find any problems with the code, please raise an issue on Github. Pull requests are welcome if you agree with the Contributor Code of Conduct

If you use regions in your work, please cite the package.

Economic and environmental impact analysis with iotables

Fri, 04 Jun 2021 00:00:00 +0000

We have released a new version of iotables as part of the rOpenGov project. The package, as the name suggests, works with European symmetric input-output tables (SIOTs). SIOTs are among the most complex governmental statistical products. They show how each country’s 64 agricultural, industrial, service, and sometimes household sectors relate to each other. They are estimated from various components of the GDP, tax collection, at least every five years.

SIOTs offer great value to policy-makers and analysts to make more than educated guesses on how a million euros, pounds or Czech korunas spent on a certain sector will impact other sectors of the economy, employment or GDP. What happens when a bank starts to give new loans and advertise them? How is an increase in economic activity going to affect the amount of wages paid and and where will consumers most likely spend their wages? As the national economies begin to reopen after COVID-19 pandemic lockdowns, we can utilize SIOTs to calculate direct and indirect employment effects or value added effects of government grant programs to sectors such as cultural and creative industries or actors such as venues for performing arts, movie theaters, bars and restaurants.

Making such calculations requires a bit of matrix algebra and a solid understanding of input-output economics, direct, indirect effects, and multipliers. Economists, grant designers, and policy makers have those skills, but until now, such calculations were either made in cumbersome Excel sheets, or proprietary software, as the key to these calculations is to keep vectors and matrices, which have at least one dimension of 64, perfectly aligned. We made this process reproducible with iotables and eurostat on rOpenGov

Accessing and tidying the data programmatically

The iotables package is in a way an extension to the eurostat R package, which provides programmatic access to the Eurostat data warehouse. The reason for releasing a new package is that working with SIOTs requires plenty of meticulous data wrangling based on various metadata sources, apart from only accessing the data itself. When working with matrix equations, the bar is higher than with tidy data. Not only must your rows and columns match, but their ordering must strictly conform to the quadrants of the matrix system, including the connecting trade or tax matrices.

When you download a country’s SIOT table, you receive a long form data frame, a very-very long one, which contains the matrix values and their labels like this:

## Table naio_10_cp1700 cached at /var/folders/nb/sxk6cbzd5455n3_rhxnw2xnw0000gn/T//Rtmp7lAZZa/eurostat/naio_10_cp1700_date_code_FF.rds

# we save it for further reference here 
saveRDS(naio_10_cp1700, "not_included/naio_10_cp1700_date_code_FF.rds")

# should you need to retrieve the large tempfiles, they are in 
dir (file.path(tempdir(), "eurostat"))

dplyr::slice_head(naio_10_cp1700, n = 5)

## # A tibble: 5 x 7
##   unit    stk_flow induse  prod_na geo       time        values
##   <chr>   <chr>    <chr>   <chr>   <chr>     <date>       <dbl>
## 1 MIO_EUR DOM      CPA_A01 B1G     EA19      2019-01-01 141873.
## 2 MIO_EUR DOM      CPA_A01 B1G     EU27_2020 2019-01-01 174976.
## 3 MIO_EUR DOM      CPA_A01 B1G     EU28      2019-01-01 187814.
## 4 MIO_EUR DOM      CPA_A01 B2A3G   EA19      2019-01-01      0 
## 5 MIO_EUR DOM      CPA_A01 B2A3G   EU27_2020 2019-01-01      0

The metadata reads like this: the units are in millions of euros, we are analyzing domestic flows, and the national account items B1-B2 for the industry A01. The information of a 64x64 matrix (the SIOT) and its connecting matrices, such as taxes, or employment, or $CO_{2}$ emissions, must be placed exactly in one correct ordering of columns and rows. Every single data wrangling error will usually lead to an error (the matrix equation has no solution), or, what is worse, in a very difficult to trace algebraic error. Our package not only labels this data meaningfully, but creates very tidy data frames that contain each necessary matrix of vector with a key column.

iotables package contains the vocabularies (abbreviations and human readable labels) of three statistical vocabularies: the so called COICOP product codes, the NACE industry codes, and the vocabulary of the ESA2010 definition of national accounts (which is the government equivalent of corporate accounting).

Our package currently solves all equations for direct, indirect effects, multipliers and inter-industry linkages. Backward linkages show what happens with the suppliers of an industry, such as catering or advertising in the case of music festivals, if the festivals reopen. The forward linkages show how much extra demand this creates for connecting services that treat festivals as a ‘supplier’, such as cultural tourism.

Example

Let’s take Slovakia’s employment data as an example and match it with the latest structural information on from the Symmetric input-output table at basic prices (product by product) Eurostat product.

## Downloading employment data from the Eurostat database.

## Table lfsq_egan22d cached at /var/folders/nb/sxk6cbzd5455n3_rhxnw2xnw0000gn/T//Rtmp7lAZZa/eurostat/lfsq_egan22d_date_code_FF.rds

A quick look at the Eurostat website already shows that there is a lot of work ahead to make the data look like an actual symmetric input-output table.

iotable’s iotable_get() function downloads the data and does basic labelling and preprocessing on the raw Eurostat files. Because of the size of the unfiltered dataset on Eurostat, the following code may take several minutes to run.

sk_io <-  iotable_get ( labelled_io_data = NULL, 
                        source = "naio_10_cp1700", geo = "SK", 
                        year = 2015, unit = "MIO_EUR", 
                        stk_flow = "TOTAL",
                        labelling = "iotables" )

## Reading cache file /var/folders/nb/sxk6cbzd5455n3_rhxnw2xnw0000gn/T//Rtmp7lAZZa/eurostat/naio_10_cp1700_date_code_FF.rds

## Table  naio_10_cp1700  read from cache file:  /var/folders/nb/sxk6cbzd5455n3_rhxnw2xnw0000gn/T//Rtmp7lAZZa/eurostat/naio_10_cp1700_date_code_FF.rds

## Saving 808 input-output tables into the temporary directory
## /var/folders/nb/sxk6cbzd5455n3_rhxnw2xnw0000gn/T//Rtmp7lAZZa

## Saved the raw data of this table type in temporary directory /var/folders/nb/sxk6cbzd5455n3_rhxnw2xnw0000gn/T//Rtmp7lAZZa/naio_10_cp1700.rds.

The input_coefficient_matrix_create() creates an input coefficient matrix, which is used for most of the analytical functions.

$a_{ij}$ = $X_{ij}$ / $x_j$

It checks that the columns are in correct order and additionally it fills up 0 values with 0.000001 to avoid division with zero.

input_coeff_matrix_sk <- input_coefficient_matrix_create(
  data_table = sk_io
)

## Columns and rows of real_estate_imputed_a, extraterriorial_organizations are all zeros and will be removed.

Then you can create the Leontieff-inverse, which contains all the structural information about the relationships of 64x64 sectors of the chosen country (in this case, Slovakia) ready for the main equations of input-output economics:

I_sk <- leontieff_inverse_create(input_coeff_matrix_sk)

And extract the primary inputs:

primary_inputs_sk <- coefficient_matrix_create(
  data_table = sk_io, 
  total = 'output', 
  return = 'primary_inputs')

## Columns and rows of real_estate_imputed_a, extraterriorial_organizations are all zeros and will be removed.

Now let’s try to figure out what happens when the government tries to stimulate the economy in three sectors: agriculture, car manufacturing, and R&D with 1 billion euros. Direct effects measure the initial, direct impact of the change in demand and supply for a product. When production goes up, it will create demand in all supply industries (backward linkages) and create opportunities in the industries that use the product themselves (forward linkages).

direct_effects_create( primary_inputs_sk, I_sk ) %>%
  select ( all_of(c("iotables_row", "agriculture",
                    "motor_vechicles", "research_development"))) %>%
  filter (.data$iotables_row %in% c("gva_effect", "wages_salaries_effect", 
                                    "imports_effect", "output_effect"))

##            iotables_row agriculture motor_vechicles research_development
## 1        imports_effect   1.3684350       2.3028203            0.9764921
## 2 wages_salaries_effect   0.2713804       0.3183523            0.3828014
## 3            gva_effect   0.9669621       0.9790771            0.9669467
## 4         output_effect   2.2876287       3.9840251            2.2579634

Car manufacturing requires a large number of imported components, so increased demand for cars will also create growth in importing activities. Increase in R&D activity will mostly affect local wages because research is job-intensive. As we can see, the effect on imports, wages, gross value added (which will end up in the GDP) and output changes are very different in these three sectors.

This is not the total effect, because some of the increased production will translate into income, which in turn will be used to create further demand in all parts of the domestic economy. The total effect is characterized by multipliers.

The multipliers can be solved with the following function:

multipliers_sk <- input_multipliers_create( 
  primary_inputs_sk %>%
    filter (.data$iotables_row == "gva"), I_sk )

And select a few industries:

set.seed(12)
multipliers_sk %>% 
  tidyr::pivot_longer ( -all_of("iotables_row"), 
                        names_to = "industry", 
                        values_to = "GVA_multiplier") %>%
  select (-all_of("iotables_row")) %>%
  arrange( -.data$GVA_multiplier) %>%
  dplyr::sample_n(8)

## # A tibble: 8 x 2
##   industry               GVA_multiplier
##   <chr>                           <dbl>
## 1 motor_vechicles                  7.81
## 2 wood_products                    2.27
## 3 mineral_products                 2.83
## 4 human_health                     1.53
## 5 post_courier                     2.23
## 6 sewage                           1.82
## 7 basic_metals                     4.16
## 8 real_estate_services_b           1.48

Package vignettes

The Germany 1990 provides an introduction of input-output economics and re-creates the examples of the Eurostat Manual of Supply, Use and Input-Output Tables, by Jörg Beutel (Eurostat Manual).

The United Kingdom Input-Output Analytical Tables Daniel Antal, based on the work edited by Richard Wild is a use case on how to correctly import data from outside Eurostat (i.e. not with eurostat::get_eurostat()) and join it properly to a SIOT. We also used this example to create unit tests of our functions from a published, official government statistical release.

Finally, Working With Eurostat Data is a detailed use case of working with all the current functionalities of the package by comparing two economies, Czechia and Slovakia and guides you through a lot more examples than this short blogpost.

Our package was originally developed to calculate GVA and employment effects for the Slovak music industry, and similar calculations for the Hungarian film tax shelter. We can now programmatically create reproducible multipliers for all European economies in the Digital Music Observatory, and create further indicators for economic policy making in the Economy Data Observatory.

Environmental Impact Analysis

Our package allows the calculation of various economic policy scenarios, such as changing the VAT on meat or effects of re-opening music festivals on aggregate demand, GDP, tax revenues, or employment. But what about $CO_{2}$, methane and other greenhouse gas effects of the reopening festivals, or increasing meat prices?

Technically our package can already calculate such effects, but to do so, you have to carefully match further statistical vocabulary items used by the European Environmental Agency about air pollutants and greenhouse gases.

The last released version of iotables is Importing and Manipulating Symmetric Input-Output Tables (Version 0.4.4). Zenodo. https://doi.org/10.5281/zenodo.4897472, and we are already working on a new major release. In that release, we are planning to build in the necessary vocabulary into the metadata functions to increase the functionality of the package, and create new indicators for our Green Deal Data Observatory. This experimental data observatory is creating new, high quality statistical indicators from open governmental and open science data sources that has not seen daylight yet.

rOpenGov and the EU Datathon Challenges

rOpenGov is a community of open governmental data and statistics developers with many packages that make programmatic access and work with open data possible in the R language. Reprex is a Dutch-startup that teamed up with rOpenGov and other open collaboration partners to create a technologically and financially feasible service to exploit reproducible research products for the wider business, scientific and evidence-based policy design community. Open data is a legal concept - it means that you have the rigth to reuse the data, but often the reuse requires significant programming and statistical know-how. We entered into the annual EU Datathon competition in all three challenges with our applications to not only provide open-source software, but daily updated, validated, documented, high-quality statistical indicators as open data in an open database. Our iotables package is one of our many open-source building blocks to make open data more accessible to all.

Visualizing City of Helsinki procurements with geofi-package

Thu, 01 Apr 2021 00:00:00 +0000

City of Helsinki public procurements have been available as open data since 2014. High quality data like this is obviously of great interest to many and [several interesting applications and visualizations] have been made available.

With some additional open data from Finnish Patent and Registration Office and data wrangling, the location of the city supplier companies could be made visible. Geofi-package, just released in CRAN, provides excellent tools for this sort of task, alongside dplyr-package’s lightning-fast join and mutate operations.

The Patent and Registration Office data could be accessed by making API calls with the unique Business ID (“Y-tunnus”) of each company. A limitation was that information was available only for limited companies, cooperatives and similar entities, leaving out public institutions, third sector (independent sector) actors and sole proprietor type enterprises.

Hadley Wickham’s httr-package vignette Best practices for API packages provided a good starting point for building our own custom function “prh_api”, which made it possible to access company information with relative ease. In practice the task was not only smooth sailing as the API had a limit of 300 calls per minute, to be shared between all API users. Downloading information took approximately 4 seconds for one company (15 calls per minute), which added up to a significant amount of hours when the dataset had over 30,000 unique Business IDs.

Downloading and processing the data

Rows with invalid BIDs can be removed with hetu-package’s bid_ctrl-function:

library(hetu)
library(dplyr)
helsingin_ostot <- read.csv("http://openspending.hel.ninja/files/ostot/helsingin-ostot-all.csv")
helsingin_ostot$valid_ytunnus <- bid_ctrl(helsingin_ostot$toimittaja_ytunnus)
helsingin_ostot2 <- helsingin_ostot[which(helsingin_ostot$valid_ytunnus),]

At this point I produced a vector or unique business ID’s from the dataset, so that same information would not be downloaded more than once, and use this dataset to download data from Patent and Registration Office API. However, as the process is so time consuming, I will not reproduce the process. Below is an example with just one Business ID number “0494571-4”:

# A vector containing unique Business IDs
unique_ytunnus <- unique(helsingin_ostot2$toimittaja_ytunnus)

# Example: Getting information for one Business ID and filtering the data

yrityksen_tiedot <- jsonlite::fromJSON("http://avoindata.prh.fi/tr/v1/0494571-4", simplifyVector = TRUE)
poimitut_tiedot <- NULL
poimitut_tiedot$businessId <- yrityksen_tiedot$results$businessId
poimitut_tiedot$street <- yrityksen_tiedot$results$addresses[[1]]$street[1]
poimitut_tiedot$city <- yrityksen_tiedot$results$addresses[[1]]$city[1]
poimitut_tiedot$postCode <- yrityksen_tiedot$results$addresses[[1]]$postCode[1]

poimitut_tiedot <- as.data.frame(poimitut_tiedot)

helsingin_ostot3 <- left_join(x = helsingin_ostot2, y = poimitut_tiedot, by = c("toimittaja_ytunnus" = "businessId"))

With more than one Business ID, the code above can be made into its own function and used with lapply function.

If the company information could be downloaded from the API, the information most likely contained the zip code, address and city of the company. If these are missing it was most likely due to the API call producing error 404. Below is visualized the number of missing zip codes:

# Prepared dataset that has above operations
load("~/helsingin_ostot3.RData")
library(ggplot2)
ggplot(helsingin_ostot3, aes(fill=is.na(postCode), x=year)) +
  geom_bar(position="stack", stat="count") +
  labs(x = "Year", y = "Rows", fill = "Missing \nzip code")

The closer we are to present day, the smaller the proportion of missing data becomes.

Top-20 municipalities with most procurements

Company’s zip code is a good starting point to determine where purchased services, items and materials come from. The data could be visualized with zip code areas, but that would produce a hard to read map with too many details. Municipality level visualization will be adequate for our purposes.

While zip code areas and municipality borders do not always align perfectly, the zip code area can be assigned to the municipality which has the majority of buildings in the zip code area (Tilastokeskus 2020). Keen readers may have noticed that the data from API already had city and even street level data, but as city names can be in Finnish or in Swedish, it is simpler to look up municipality names by using an unambiguous zip code value.

library(geofi)
library(dplyr)
zipcodes <- geofi::get_zipcodes(year = 2021)

# Transform sf-object to a regular data frame
zipcodes <- as.data.frame(zipcodes) %>% 
  select(kuntanro, posti_alue)
helsingin_ostot4 <- dplyr::left_join(x = helsingin_ostot3, y = zipcodes, by=c("postCode" = "posti_alue"))

municipalities <- geofi::get_municipalities(year = 2021)
municipalities <- municipalities %>% 
  select(kunta, kunta_name)

helsingin_ostot4 <- dplyr::right_join(x = municipalities, y = helsingin_ostot4, by=c("kunta" = "kuntanro"))

# Group procurements by municipality
helsingin_ostot5 <- helsingin_ostot4 %>% 
  group_by(kunta_name) %>% 
  summarise(kunta_summa = sum(as.numeric(summa), na.rm = FALSE)) 

# Print top 20 municipalities
slice_max(helsingin_ostot5, order_by = kunta_summa, n = 20)

## Simple feature collection with 20 features and 2 fields (with 1 geometry empty)
## geometry type:  MULTIPOLYGON
## dimension:      XY
## bbox:           xmin: 215353.4 ymin: 6640920 xmax: 588843.4 ymax: 7298654
## projected CRS:  ETRS89 / TM35FIN(E,N)
## # A tibble: 20 x 3
##    kunta_name  kunta_summa                                                  geom
##    <chr>             <dbl>                                    <MULTIPOLYGON [m]>
##  1 <NA>       12164973424.                                                 EMPTY
##  2 Helsinki    5214226687. (((402737.7 6680700, 402069.8 6680535, 400326.8 6678…
##  3 Espoo        741586021. (((375773.7 6691597, 377355.9 6680366, 379983.8 6681…
##  4 Vantaa       629136624. (((392811.8 6694857, 399192.7 6692524, 396012.8 6689…
##  5 Kuopio       470253261. (((581015.6 7009317, 585462.3 7007121, 588843.4 7002…
##  6 Kouvola      149758496. (((511075 6780902, 512323 6772856, 508132 6769735, 5…
##  7 Tuusula      129772884. (((397271.6 6711736, 391885.8 6710060, 392411.8 6702…
##  8 Vaasa         90367794. (((259700 7001591, 253892.6 6990776, 242914.5 700120…
##  9 Turku         61425478. (((251038.1 6731422, 245370.1 6713651, 244865.9 6708…
## 10 Tampere       60124288. (((346626.3 6854536, 347270.1 6836030, 334112.1 6814…
## 11 Hyvinkää      51569708. (((396836.4 6726577, 392888.2 6717250, 385516.6 6715…
## 12 Kerava        49833397. (((399192.7 6692524, 392811.8 6694857, 392727.4 6700…
## 13 Raasepori     44527631. (((299881.1 6640940, 297472.1 6640920, 296412.5 6642…
## 14 Oulu          42513844. (((418101.2 7220618, 417351.5 7219858, 415092.4 7219…
## 15 Lahti         42463258. (((448838.6 6774406, 453144.3 6766188, 452204.9 6761…
## 16 Nurmijärvi    22549476. (((385516.6 6715109, 388809.1 6711136, 382590.6 6697…
## 17 Kemi          21518412. (((396561 7287772, 392601.9 7283067, 392361.4 728345…
## 18 Raisio        20195233. (((239031.6 6717088, 236093.7 6712495, 230992.2 6711…
## 19 Porvoo        18877028. (((441843.9 6673817, 440194.5 6673207, 436276.1 6673…
## 20 Padasjoki     18830945. (((422133.8 6800321, 420573.8 6797563, 415159 679860…

As expected, the largest sums for procurements were from Helsinki itself and the neighbouring cities of Espoo and Vantaa. Somewhat surprisingly, Kuopio wedges ahead of Kouvola and Tuusula, which are located geographically closer to the Helsinki metropolitan area.

However, the largest amount is credited to the NA group, with 12 billion euros over 8 years. This probably includes for the most part procurements from third sector entities and public sector organizations, highlighting their large role in the Finnish economy.

Choropleth and flow map of the top 20 municipalities

library(sf)
library(dplyr)

# Remove NA group
helsingin_ostot6 <- helsingin_ostot5 %>% 
  filter(kunta_name %in% setdiff(helsingin_ostot5$kunta_name, c(NA)))

kunnat_top20_summat <- slice_max(helsingin_ostot6, order_by = kunta_summa, n = 20)

# Highlighting the top 20 with red borders
ggplot() +
  geom_sf(data = helsingin_ostot6, aes(fill = kunta_summa), color = alpha("white", 1/3)) +
  labs(fill = "Helsingin ostot, €") +
  scale_fill_gradient2(n.breaks = 6, trans = "log10") +
  geom_sf(data = kunnat_top20_summat, col="red", size=1)

Geofi-package has the option to draw municipality central localities as POINT-geometries. With small modification these can be turned into LINESTRINGs, which have a starting point at the municipality and end point in Helsinki, that can be thought of as flow markers. The example below is very rudimentary, but at their best flow maps can be very beautiful and convey information in fresh and elegant ways.

keskukset <- geofi::municipality_central_localities

# Turn ALL CAPS municipality names to Capital Case with custom capwords-function found in base R
capwords <- function(s, strict = FALSE) {
    cap <- function(s) paste(toupper(substring(s, 1, 1)),
                  {s <- substring(s, 2); if(strict) tolower(s) else s},
                             sep = "", collapse = " " )
    sapply(strsplit(s, split = " "), cap, USE.NAMES = !is.null(names(s)))
}

keskukset$teksti <- capwords(keskukset$teksti, strict = TRUE)

keskukset <- left_join(keskukset, as.data.frame(helsingin_ostot6)[,1:2], by = c("teksti" = "kunta_name"))

# Count the distance between municipalities and Helsinki for later use
keskukset$distance_to_hel <- NULL
keskukset$distance_to_hel <- st_distance(keskukset$geom, y=keskukset$geom[210,])
keskukset$distance_to_hel <- as.integer(keskukset$distance_to_hel / 1000)

# Make linestrings
keskukset_linestring <- st_cast(st_union(keskukset$geom[1,], keskukset$geom[210,], by_feature=TRUE),"LINESTRING")
for (i in 1:nrow(keskukset)) {
  keskukset_linestring[i] <- st_cast(st_union(keskukset$geom[i,], keskukset$geom[210,], by_feature=TRUE),"LINESTRING") 
}

keskukset_helsinkiin <- keskukset

keskukset_helsinkiin$geom <- keskukset_linestring

keskukset_helsinkiin <- keskukset_helsinkiin[which(keskukset_helsinkiin$teksti %in% kunnat_top20_summat$kunta_name),]

# Line thickness: 0 for Helsinki, Espoo and Vantaa, and then 4,3,2,2,1...
ggplot() +
  geom_sf(data = helsingin_ostot6, aes(fill = kunta_summa), color = alpha("white", 1/3)) +
  labs(fill = "Helsingin ostot, €") +
  scale_fill_gradient2(n.breaks = 6, trans = "log10") +
  geom_sf(data = arrange(keskukset_helsinkiin, desc(kunta_summa)), col=alpha("red", 1/2), size=c(0,0,0,4,3,2,2,rep(1, 13)))

For the above example to work, it is important to keep the desired data object in the class “sf” so that ggplot2 can find geom column without trouble.

Effect of distance and number of companies in a municipality

Finally, I will illustrate how the number of companies in a municipality and municipality’s distance from Helsinki affect how much city of Helsinki buys from there.

library(sf)

# Get number of companies in each municipality from Statfin
# /PXWeb/api/v1/fi/StatFin/yri/alyr/statfin_alyr_pxt_11dc.px
library(pxweb)
library(fuzzyjoin)

pxweb_query_list <- 
  list("Vuosi"=c("2019"),
       "Kunta"=c("*"),
       "Tiedot"=c("Tplukumaara2"))

# Download data 
px_data <- 
  pxweb_get(url = "https://pxnet2.stat.fi/PXWeb/api/v1/fi/StatFin/yri/alyr/statfin_alyr_pxt_11dc.px",
            query = pxweb_query_list)

# Convert to data.frame 
px_data_frame <- as.data.frame(px_data, column.name.type = "text", variable.value.type = "text")

yritykset <- left_join(x = px_data_frame, y = as.data.frame(helsingin_ostot5), by=c("Kunta"="kunta_name"))

# Remove "KOKO SUOMI", "Tuntematon" (Unknown) ja municipalities that had no procurements from Helsinki
yritykset <- yritykset[which(!is.na(yritykset$kunta_summa)),]
# Remove geom-column
yritykset <- as.data.frame(yritykset)
yritykset <- yritykset[,-5]

keskukset <- geofi::municipality_central_localities

keskukset <- left_join(keskukset, as.data.frame(helsingin_ostot6)[,1:2], by = c("teksti" = "kunta_name"))

keskukset$distance_to_hel <- NULL
keskukset$distance_to_hel <- st_distance(keskukset$geom, y=keskukset$geom[210,])
keskukset$distance_to_hel <- as.integer(keskukset$distance_to_hel / 1000)

keskukset <- keskukset %>% 
  dplyr::select(teksti, distance_to_hel, geom)
  
# Fuzzyjoin-package removes the need for custom functions 
yritykset <- fuzzyjoin::regex_left_join(x = yritykset, y = keskukset, by=c("Kunta" = "teksti"), ignore_case = TRUE)

# Remove outlier, Helsinki
yritykset <- yritykset[-which(yritykset$Kunta == "Helsinki"),]

# Draw scatter-plots with smoothened curves
par(mfrow=c(1,2))
scatter.smooth(x=yritykset$`Yritysten toimipaikat (lkm)`, y=log10(yritykset$kunta_summa), span = 1/5)
scatter.smooth(x=yritykset$distance_to_hel, y=log10(yritykset$kunta_summa))

# Compare two different regression models
fit1 <- lm(log10(kunta_summa) ~ `Yritysten toimipaikat (lkm)`, data=yritykset)
fit2 <- lm(log10(kunta_summa) ~ `Yritysten toimipaikat (lkm)` + distance_to_hel, data=yritykset)

# If needed, draw regression plots
# abline(lm(log10(kunta_summa) ~ `Yritysten toimipaikat (lkm)`, data=yritykset))
# abline(lm(log10(kunta_summa) ~ `Yritysten toimipaikat (lkm)` + distance_to_hel, data=yritykset))

summary(fit1)

## 
## Call:
## lm(formula = log10(kunta_summa) ~ `Yritysten toimipaikat (lkm)`, 
##     data = yritykset)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.2431 -0.7991  0.0834  0.9631  2.4039 
## 
## Coefficients:
##                                 Estimate Std. Error t value            Pr(>|t|)
## (Intercept)                   4.75557723 0.07845464   60.62 <0.0000000000000002
## `Yritysten toimipaikat (lkm)` 0.00039387 0.00003411   11.55 <0.0000000000000002
##                                  
## (Intercept)                   ***
## `Yritysten toimipaikat (lkm)` ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.159 on 294 degrees of freedom
## Multiple R-squared:  0.312,  Adjusted R-squared:  0.3097 
## F-statistic: 133.3 on 1 and 294 DF,  p-value: < 0.00000000000000022

summary(fit2)

## 
## Call:
## lm(formula = log10(kunta_summa) ~ `Yritysten toimipaikat (lkm)` + 
##     distance_to_hel, data = yritykset)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.3314 -0.7943  0.1029  0.9196  2.7113 
## 
## Coefficients:
##                                  Estimate  Std. Error t value
## (Intercept)                    5.24186205  0.13519748  38.772
## `Yritysten toimipaikat (lkm)`  0.00036832  0.00003366  10.943
## distance_to_hel               -0.00158104  0.00036166  -4.372
##                                           Pr(>|t|)    
## (Intercept)                   < 0.0000000000000002 ***
## `Yritysten toimipaikat (lkm)` < 0.0000000000000002 ***
## distance_to_hel                          0.0000172 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.126 on 292 degrees of freedom
##   (1 observation deleted due to missingness)
## Multiple R-squared:  0.3543, Adjusted R-squared:  0.3499 
## F-statistic: 80.13 on 2 and 292 DF,  p-value: < 0.00000000000000022

We notice that the number of companies in a municipality and distance from Helsinki are significantly correlated with how successful companies from these municipalities are in selling goods and services to Helsinki. There are, however, some interesting outliers in smaller municipalities that punch above their weight in Helsinki’s procurements. The dataset provides an excellent starting point in identifying these companies and, perhaps, learning from their example.

Conclusion

Many of the largest companies in Finland have their headquarters in the capital region (cf. Manninen & Tölli 2019), which may explain why Helsinki, Espoo and Vantaa are so well represented in Helsinki’s procurements. It might be interesting to compare in the future whether regional capitals such as Turku and Tampere also buy majority of their goods and services from the capital region or if they have their own local ecosystems.

Idealized conditions of perfect competition (no barriers to entry or exist, perfect information, zero transaction costs etc.) do not exist even within a relatively homogeneous national framework, let alone within a heterogeneous single market area such as the EU. For different industry advocacy groups, government organizations and companies support for greater access to EU single market offers great potential and active policy measures aim to lower those barriers to entry to foster competitiveness. Perhaps there is still work left undone in opening up access to local markets such as Helsinki.

Hetu-package for handling of Finnish personal identity codes

Thu, 29 Oct 2020 00:00:00 +0000

General information

Hetu-package for R is meant for algorithmic handling of Finnish personal identity numbers (PINs). The package is especially useful for those who wish to extract information from or validate a large number of PINs at a time.

The toolset for analyzing Finnish PINs was initially developed as a part of sorvi-package, but was later made into a separate package. The development of the hetu-package reached an important milestone in Fall 2020 when it was published in CRAN.

The development of hetu-package is closely related to sweidnumbr, a similar package meant for analyzing Swedish personal identity numbers (PINs) and organizational identity numbers (OINs). Hetu-package shares similar function names with sweidnumbr, when applicable.¹

Finnish personal identification code, hetu

Personal identification code (or: national identification number, national identity number, personal identification number or PIN) is meant to be a unique identifier for individuals. Finnish personal identification number (henkilötunnus, hetu for short) consists of date (DDMMYY), century marker (-, + or A), personal number (NNN) and checkmark (C). Males have an odd personal number and females an even personal number.²

Personal identity codes are widely used in public and private sectors alike. They are not confidential or secret information, but like every personal information, handling hetu-codes requires consent from the individual or a valid reason.

Algorithmic handling of hetu-pins

Analyzing and extracting information from Finnish personal identity numbers is rather straightforward even with a naked eye. Hetu-package naturally excels in handling large number of PINs, which would be cumbersome otherwise.

Hetu-package has functions to extract the following information:

hetu_date / pin_date: date of birth
hetu_sex / pin_sex: sex, Male or Female
hetu_age / pin_age: age in years, months or days (at the time of the query or at a desired date)
hetu_ctrl / pin_ctrl: validity check for the PIN, TRUE or FALSE

Use of hetu-package

Installing the package in R from CRAN:

install.packages("hetu")

Loading the package and setting a few imaginary PINs for testing:

library(hetu)
example_pins <- c("010101-0101", "111111-111C")

Hetu-function is the backbone of the package and majority of the information that can be extracted is available as a simple data frame:

knitr::kable(hetu(example_pins))

hetu	sex	p.num	checksum	date	day	month	year	century	valid.pin
010101-0101	Female	010	1	1901-01-01	1	1	1901	-	TRUE
111111-111C	Male	111	C	1911-11-11	11	11	1911	-	TRUE

There are several alternatives in extracting specific information about a group of PINs, for example date of birth. If the output of the hetu-function is saved as an object, all columns can be normally subsetted. For the convenience of the end user, the information in the data frame can also be extracted by using extract-parameter in the hetu-function or by using one of the specialized functions:

# Extracting sex
hetu(example_pins, extract = "sex")

## [1] "Female" "Male"

hetu_sex(example_pins)

## [1] "Female" "Male"

# Extracting date of birth
hetu(example_pins, extract = "date")

## [1] "1901-01-01" "1911-11-11"

hetu_date(example_pins)

## [1] "1901-01-01" "1911-11-11"

# Extracting information on validity
hetu(example_pins, extract = "valid.pin")

## [1] TRUE TRUE

hetu_ctrl(example_pins)

## [1] TRUE TRUE

# Information that can be extracted only with extract-parameter
hetu(example_pins, extract = "p.num")

## [1] "010" "111"

In contrast to other information, extracting age works only with a specialized function. In this example we will also introduce the ability to generate random PINs with rhetu-function:

example_pins2 <- rhetu(5, start = "1950-01-01", end = "1995-05-07")
# Age in years
hetu_age(example_pins2)

## The age in years has been calculated at 2021-01-31.

## [1] 33 69 62 31 43

# Age in months
hetu_age(example_pins2, timespan = "months")

## The age in months has been calculated at 2021-01-31.

## [1] 403 839 752 383 521

# Age in 2011
hetu_age(example_pins2, date = "2011-01-01")

## The age in years has been calculated at 2011-01-01.

## [1] 23 59 52 21 33

# Visualization: boxplot grouped by sex
example_pins3 <- rhetu(20, start = "1950-01-01", end = "1995-05-07", p.male = 0.5)
boxplot(hetu_age(example_pins3)~hetu_sex(example_pins3), xlab = "", ylab = "Age in years", col=c("cyan", "magenta"))

## The age in years has been calculated at 2021-01-31.

In some cases diagnostics information for invalid PINs might be useful:

hetu_diagnostic("321399-000G")

##           hetu is.temp valid.p.num valid.checksum correct.checksum valid.date
## 21 321399-000G   FALSE       FALSE          FALSE            FALSE      FALSE
##    valid.day valid.month valid.year valid.length valid.century
## 21     FALSE       FALSE       TRUE         TRUE          TRUE

# Print only certain columns
hetu_diagnostic("321399-000G", extract = c("valid.p.num", "valid.length"))

##           hetu valid.p.num valid.length
## 21 321399-000G       FALSE         TRUE

Business Identity Numbers (Y-tunnus, BID)

As in sweidnumbr, hetu-package has two functions that can be used with Finnish Business Identity Numbers (y-tunnus). Finnish business identity numbers have the form 1234567-8, where the last number is a checknumber.³ The following functions are available:

bid_ctrl(bid): checks the valiity of the BID, TRUE or FALSE
rbid(n): generates n BIDs

example_bids <- rbid(2)
example_bids

## [1] "7128741-6" "1963928-5"

bid_ctrl(example_bids)

## [1] TRUE TRUE

No additional information can be extracted from BIDs.

References

More information about sweidnumbr can be found e.g. from this blogpost: Magnusson, Mans & Bulow, Erik. 2015. R made personal (at least for swedes)!. URL: https://ropengov.org/2015/08/r-made-personal-at-least-for-swedes/↩︎
Digital and Population Data Services Agency (Digi- ja väestötietovirasto). The personal identity code. URL: https://dvv.fi/en/personal-identity-code ↩︎
Finnish Patent and Registration Office. The Business Information System (BIS). URL: https://www.prh.fi/en/kaupparekisteri/rekisterointipalvelut/ytj.html ↩︎

geofi R-package for accessing Statistics Finland spatial data

Tue, 11 Feb 2020 10:53:45 +0000

We have slowly started developing a follow-up package with ropengov-posse for gisfin-package named geofi. Package provides access to few sources of Finnish open geospatial data from R. We are focusing in administrative regions at the moment and our primary source of data is Statistics Finland and their wfs-api. You can use functions in geofi fecth data such as municipality borders, postal code areas sekä population grids.

geofi is not published in CRAN yet and you cant install it using install.packages(). But you can install it directly from Github with remotes::install_github("ropengov/geofi") and try out the following examples. For quick access try our Shiny app at: https://muuankarski.shinyapps.io/geofi_selain/.

Municipalility borders

library(geofi)
library(ggplot2)

municipalities <- get_municipalities(year = 2020, scale = 4500)
ggplot(municipalities) + 
  geom_sf(aes(fill = as.integer(kunta))) +
  scale_fill_viridis_c()

Postal code areas

zipcodes <- get_zipcodes(year = 2020) 
ggplot(zipcodes) + 
  geom_sf(aes(fill = as.integer(posti_alue)), color = alpha("white", 1/3)) +
  scale_fill_viridis_c()

Population grids

pop_grid <- get_population_grid(year = 2018, resolution = 5)
ggplot(pop_grid) + 
  geom_sf(aes(fill = objectid), color = alpha("white", 1/3)) +
  scale_fill_viridis_c()

Regions (maakunnant), health care districts (sairaanhoitopiirit) and many more regional breakdowns are based on municipality divide. get_municipalities()-function returns data containing these attribute variables (year 2020), that you can use to aggregate from municipality level upwards.

library(dplyr)
municipalities <- get_municipalities(year = 2019, scale = 4500)
regions <- municipalities %>% 
  group_by(maakunta_name_fi) %>% summarise()
ggplot(regions) + 
  geom_sf(aes(fill = maakunta_name_fi)) +
  scale_fill_viridis_d()

You can join geofi datas with other attribute, too. Below is an example on how to get (non-spatial statistical) data from Statistics Finland and create a map on population at municipality level.

library(tidyr)
library(pxweb)
library(janitor)

municipalities17 <- get_municipalities(year = 2017)

# pull municipality data from Statistics Finland
pxweb_query_list <-
  list("Alue 2019"=c("*"),
       "Tiedot"=c("*"),
       "Vuosi"=c("2017"))
px_data <-
  pxweb_get(url = "http://pxnet2.stat.fi/PXWeb/api/v1/fi/Kuntien_avainluvut/2019/kuntien_avainluvut_2019_aikasarja.px",
            query = pxweb_query_list)
# Convert to data.frame
tk_data <- as.data.frame(px_data, column.name.type = "text", variable.value.type = "text")
tk_data2 <- tk_data %>%
  rename(name = `Alue 2019`) %>%
  mutate(name = as.character(name),
         # Paste Tiedot and Vuosi
         Tiedot = paste(Tiedot, Vuosi)) %>%
  select(-Vuosi) %>%
  spread(Tiedot, `Kuntien avainluvut`) %>%
  as_tibble()
tk_data3 <- janitor::clean_names(tk_data2)

# Join with Statistics Finland attribute data
dat <- left_join(municipalities17, tk_data3)

ggplot(dat) + 
  geom_sf(aes(fill = vakiluku_2017), color = alpha("white", 1/3)) +
  scale_fill_viridis_c(trans = "sqrt")

Take a look at the Github-site and join us!

Cost of academic publishing in Finland 2010-2017

Wed, 05 Dec 2018 00:00:00 +0000

Subscription costs to scientific publishers in Finland 2010-2017

This post provides a brief overview of the subscription prices paid by Finnish research institutions to academic publishers in 2010-2017.

Finland is possibly the only country that has systematically released subscription prices that research libraries pay to academic publishers as open data. The data is available for all major research institutions in Finland. Recently, an updated data set for 2010–2017 was made openly available at avointiede.fi. In addition, full text agreements with many publishers have been made available. The subscription price data was initially provided by Finnish Ministry of Education and Culture, and its Open Science and Research Initiative funded 2014–2017, after a successful Freedom of Information request by the Finnish Open Science community, as summarized elsewhere. This post updates our earlier analysis. For source code, see main.R.

Overall subscription costs 2010-2017

Based on the data collected by the Ministry of Education, Finland paid in total 198.7 million EUR subscription and other fees on scientific publishing in 2010-2017. The average annual costs for in Finland were 25 MEUR.

Data for the top-10 publishers in the UK 2010-2014 is available in Lawson, Meghreblian & Brook, 2017 (Table 1). During this period the UK paid altogether 4319 MEUR (rough estimate based on the exchange rate June 12, 2016) for the top-10 publishers. Finland paid 62 MEUR for the same top-10 publishers in 2010-2014. This is 17.1% of the UK expenditure per capita. It could be that the data is not directly comparable but this will require further investigation.

Information for Finland is available by agreement type, organization type, and subscription category.

Costs by publisher

Overall, the Finnish data covers 376 unique publishers. The figure indicates the total subscription fees paid to the top publishers 2010-2017. One third of the total costs go to Elsevier, which has been often criticized for its huge profit margins. The costs are given per bundle, so we cannot compare average journal prices among individual publishers based on this data.

The total costs paid to scientific publishers by Finland have increased roughly 10% per year in 2010-2017 (annual increase is indicated in the left figure). The top-10 publishers correspond to 75% of the overall costs (right figure).

## Warning: The `i` argument of ``[`()` can't be a matrix as of tibble 3.0.0.
## Convert to a vector.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_warnings()` to see where this warning was generated.

Let us compare the relative increase in publisher costs. The costs are normalized to 1 in 2010, and the top 10 publishers with the highest cost increase in 2010-2017 are shown. The 275 publishers that did not have declared costs in 2010 or 2017 are excluded.

Costs by organization

The Finnish data collection includes 82 organizations. The universities (‘yliopisto’) are responsible of 78.9% of all costs (left figure); University of Helsinki had the highest total costs in 2010-2017 (37.9 MEUR; top institutions shown in the right figure).

Finally, let us compare the relative share of costs per institution. The top organizations with the highest total costs are shown.

Presenting at useR2017 in Brussels

Wed, 05 Jul 2017 00:00:00 +0000

Annual useR-conference is organised this year in Brussels, Belgium. rOpenGov will be presenting in two occasion on Thursday 6th. Leo Lahti will give an general overview of rOpenGov in his lightning talk rOpenGov - Community project for open government data at 17.50 in room 3.02. At 13.30 in Plenary room Markus Kainu will speak about Community-based learning and knowledge sharing - Teaching R withing organisation using edu -package.

eurostat package published in R Journal

Fri, 14 Apr 2017 00:00:00 +0000

A formal publication on the eurostat CRAN R package has now appeared on-line in R Journal.

Further details on installation and use are provided at the package homepage, tutorial and cheat sheet. Blog posts provide further and more advanced examples on the package use.

We are also collecting information about publications using the eurostat R package, suggestions are welcome.

Scientific journal subscription costs in Finland 2010-2015

Fri, 10 Jun 2016 00:00:00 +0000

Detailed information on journal subscription costs paid to individual publishers by the Finnish research institutions has been released by the Finnish Ministry of Education and Culture, and its Open Science and Research Initiative funded 2014–2017 (Academic Publishing Costs in Finland 2010–2015).

With this, Finland becomes to our knowledge the first country where annual subscription fees for all individual publishers and all major research institutions have been made available, spanning the years 2010-2015. Similar information has been previously released for some, but not all publishers and research institutions in the UK and US; and related activities are ongoing in several countries (see the recent blog post by Stuart Lawson).

Access to literature is fundamental to academic research but this has been challenged with the rapidly increasing prices of academic journals, which university libraries find increasingly difficult to fund. According to the data from the US Association of Research Libraries the academic journal subscription charges increased 4x faster than inflation in 1986-2007. The relative variation in prices is also considerable among the publishers (Bergstrom et al. PNAS 2014). Limited access to detailed pricing information and agreement details are likely to result in suboptimal contracts (Cockerill, 2006; Shieber, 2009). Improved access to subscription costs can hence be expected to lead to better deals and lower costs for the universities. It can also facilitate transition to the Open Access (OA) publishing model.

Motivated by all this, and following Tim Gowers successful FOI request on the subscription costs for scientific journals in UK, we made a similar request in Finland together with the Open Knowledge Finland association and other Open Science advocates. After the Finnish universities turned down my FOI request in summer 2014, we appealed in court, which decided the case positive for us in August 2015. For an English summary of this process, see a separate post in the MostlyPhysics blog.

Here I provide a brief preliminary analysis of the data on journal subscription fees that was collected and released by the Ministry of Education Open Science Initiative in Finland. I have abbreviated some terms as detailed in the source code of this analysis is maintained in Github.

Overall subscription costs 2010-2015

Finland paid in total 131.1 million EUR subscription and other fees on scientific publishing in 2010-2015. The overall breakup of the costs is available as a separate table. The average annual costs for in Finland were 22 MEUR in 2010-2015; this is one third of the annual subscription costs in Austria (70 MEUR; Bauer et al., 2015), and two thirds of the annual expenditure (31 MEUR) in New Zealand. Data for the top-10 publishers in the UK 2010-2014 is available in Lawson, Meghreblian & Brook, 2015 (Table 1). During this period the UK paid altogether 4319 MEUR (rough estimate based on the exchange rate June 12, 2016) for the top-10 publishers. Finland paid 61 MEUR for the same top-10 publishers in the same period, which is roughly 17% of the UK expenditure per capita (unexpectedly low?). The costs in the other countries seem unexpectedly high compared to Finland, I wonder what is the explanation. Either Finland can negotiate considerably better deals, or, perhaps more likely, the figures from the different countries are not directly comparable but this will require further investigation.

The data set covers all Finnish universities, major public institutions, and a number of special libraries, information services and other smaller institutions. Open access article processing charges (APCs) are not included in this data collection as far as I can see, although it would be interesting in its own right. It was recently reported by Naukkarinen (2016) that in 2014 18% of the articles in Finnish universities were published as open access. It was also estimated that publishing all articles as Open Access would have cost 17 MEUR, whereas the subscription fees in 2014 were 22 MEUR. This suggests that the transition to the Open Access model might be a good idea. Overall, there are roughly 35 000 peer-reviewed academic journals globally (Ware & Mabe, 2015); less than a third of these (11 000) are open access (DOAJ, 2016). A common complaint regarding the (golden) OA model is the relatively high cost of the article processing charges, typically paid from primary research funding. With this funding model, money going OA is frequently seen as being taken away from other research activities, such as experiments or research personnell. In contrast, publication costs in the conventional subscription model are funded through university libraries with secret discloure agreements with the publishers and hence largely masked from the research community. In order to assess the true costs of scientific publishing and in order to facilitate shift to OA, the costs of the subscription model must be made more transparent and money used for conventional subscription must be diverted to funding the costs of OA publishing.

The full data set details the subscription fees also by agreement type, organization type, and subscription category.

Costs by publisher

The Finnish data covers 244 individual publishers (see annual costs by publisher). The figure indicates the total subscription fees paid to the top publishers 2010-2015. Over one third of the total costs go to Elsevier, which has been often criticized for its huge profit margins. The costs are given per bundle, so we cannot compare individual publishers on a per article or per citation basis based on this data set. It was recently estimated elsewhere, however, that Elsevier’s prices per citation are roughly 3x higher than with non-profit publishers; Emerald, Sage, and Taylor & Francis had roughly 10x higher prices (Bergstrom et al. PNAS 2014).

The total costs paid to scientific publishers by Finland have increased roughly 10% per year in 2010-2015 (annual increase is indicated in the left figure). The top-10 publishers correspond to 77% of the overall costs (right figure). See a separate table for full annual costs by publisher (the top-10 publishers shown below).

Publisher (costs in MEUR)	2010	2011	2012	2013	2014	2015	Total	%
Total	17.30	19.01	21.33	22.13	24.31	27.03	131.10	100.00
Elsevier	6.41	6.84	7.38	7.65	8.10	8.58	44.96	34.30
Wiley	1.66	2.06	2.29	2.43	2.57	2.65	13.67	10.43
Ebsco	0.81	1.76	1.97	2.09	2.00	1.70	10.33	7.88
Springer	1.30	1.36	1.41	1.45	1.48	1.50	8.49	6.48
ProQuest	0.75	0.83	0.97	0.98	1.01	1.24	5.77	4.40
Thomson Reuters	0.49	0.54	0.67	0.73	0.77	0.94	4.13	3.15
SAGE Publications	0.50	0.55	0.65	0.66	0.71	0.86	3.92	2.99
American Chemical Society (ACS)	0.51	0.53	0.59	0.62	0.63	0.78	3.66	2.79
Nature Publishing Group	0.38	0.42	0.58	0.64	0.67	0.62	3.30	2.52

Let us compare the relative increase in publisher costs. The costs are normalized to 1 in 2010, and the top 10 publishers with the highest cost increase in 2010-2015 are shown. The 120 publishers that did not have declared costs in 2010 or 2015 (see here) are excluded.

Costs by organization

The Finnish data collection includes 63 organizations (see annual costs by organization). The universities (‘yliopisto’) are responsible of 79% of all costs (left figure); University of Helsinki had the highest total costs in 2010-2015 (24.4 MEUR; top institutions shown in the right figure). The Table indicates the annual costs for the top organizations.

Organization (costs in MEUR)	2010	2011	2012	2013	2014	2015	Total	%
Total	17.30	19.01	21.33	22.13	24.31	27.03	131.10	100.00
University of Helsinki	3.32	3.52	3.91	4.12	4.46	5.05	24.39	18.60
Aalto University	2.25	2.40	2.67	2.71	2.88	3.07	15.98	12.19
University of Turku	1.50	1.60	1.76	1.79	2.04	2.35	11.04	8.42
University of Oulu	1.49	1.57	1.74	1.83	2.03	2.17	10.84	8.26
University of Eastern Finland	0.90	1.35	1.43	1.53	1.69	1.82	8.73	6.66
University of Jyväskylä	1.07	1.14	1.27	1.31	1.57	1.73	8.08	6.16
University of Tampere	0.91	1.00	1.09	1.14	1.28	1.41	6.82	5.20
Tampere University of Technology	0.90	1.05	1.17	1.19	1.21	1.30	6.82	5.20
Åbo Akademi University	0.75	0.78	0.79	0.78	0.89	0.97	4.96	3.78

Finally, let us compare the relative increase in costs across the institutions. The costs are normalized to 1 in 2010, and the top 10 organizations with the highest cost increase in 2010-2015 are shown. The 12 organizations that did not have declared costs in 2010 or 2015 (see here) are excluded.

R made personal (at least for swedes)!

Thu, 20 Aug 2015 00:00:00 +0000

Background

– Who are you? asked Mr Doe.

– I’m a Hindu! Namrata from India replied.

– I’m a statistician! said Günther from Germany.

People of different nationalities tend to identify themselves using different characteristics. In India, your identity might rely on your religion, while in other countries your profession might take its place. In Sweden, you might identify yourself with your almost-world-known (!?) personal identification number (“pin”). This 10 digit number is given to you almost immediately after birth and it often stays with you until your very last breath. The number is similar to a “social security number” but it has a much broader use and it is considered public. It is used in public registers (for education, work, tax payment, healthcare, car ownership etc) and it often serves as a membership number or customer id within companies and member unions. It is also essential for example in the public health and quality registers maintained in Sweden (and other Scandinavian countries) and used for reaserch.

Motivation

Naturally, the “pin” is used extensively to distinguish individuals in data sets analysed by R. The number also helps to match data from different sources and it can bring some demographic background data into the bargain, such as birth date (age), sex and geographic origin (depending on your birth year).

Up until now however, with the lack of a consistent R convention to handle “pins”, the number might be treated as either a 10 or 12 digit numeric (with or without century prefix), a character (with hyphen or a ‘+’-sign to distinguish birth date from suffix numbers) or as a factor variable. But the pin is not a number (to add, subtract or logarithm pins is just nonsense) and it contains more information than captured by the individual characters in a string. Luckily, the new R package sweidnumbr (released on CRAN) is here for rescue!

Example

Let’s look at some data (all pins are fake; they have a valid syntax but do not identify any real individuals):

library(sweidnumbr)

## sweidnumbr: R tools to handle swedish identity numbers.
## https://github.com/rOpenGov/sweidnumbr

knitr::kable(tail(fake_pins,10))

	pin	name
53	19471130-3022	TWIST, LIS
54	19440311-1131	NOBLESSE, RAGNAR JOHN
55	20000805-0523	NILSSON, CHOK
56	19240622-2286	CADBURY, LOVISA
57	19020517-1798	PLOPP, AUGUST
58	20050111-1123	MINT, MARIA ADA
59	19370215-1590	NILSSON, BARRY
60	19970430-3023	BERG, ANTO
61	20031010-1023	CENTER, PALL
62	20010218-1823	CACAO, EDA

So far, pin is just a standard character vector but let’s change that to benefit from all of sweidnumbr’s features:

pin <- as.pin(fake_pins$pin)

## Assumption: Pin of format YYMMDDNNNC is assumed to be less than 100 years old

str(pin)

##  'AsIs' chr [1:62] "191212121212" "201212121212" "191212121212" ...

We can now also investigate some demographic characteristics almost on the fly (note that pins contained geographical information only up to 1989):

par(mfrow = c(1,2))
hist(pin_age(pin), 20, col = "lightgreen", main = "Age distribution")

## The age has been calculated at 2021-01-29.

pie(table(pin_sex(pin)), main = "Sex distribution")

pin_birthplace(pin[1:8])

## [1] Stockholms län              Born after 31 december 1989
## [3] Stockholms län              Born after 31 december 1989
## [5] Born after 31 december 1989 Stockholm stad             
## [7] Stockholms län              Born after 31 december 1989
## 28 Levels: Stockholm stad Stockholms län Uppsala län ... Born after 31 december 1989

Formats

as.pin can recognize pins in several different formats such as:

 as.pin(c("191212121212", "1212121212", "121212-1212", "121212+1212"))

## Assumption: Pin of format YYMMDDNNNC is assumed to be less than 100 years old

## [1] "191212121212" "201212121212" "201212121212" "191212121212"
## Personal identity number(s)

It also checks that the numbers follow the correct pin syntax:

as.pin("181212121212") # Pins were introduced in 1946 and only for people not deceased before that

## Warning in as.pin.character("181212121212"): Erroneous pin(s) (set to NA).

## [1] NA
## Personal identity number(s)

pin_ctrl("191212121211") # The last digit is a control number that is checked against preceeding digits

## [1] FALSE

luhn_algo("191212121211") # The correct control number can be calculated by the Luhn algorithm

## 'multiplier' set to: c(0, 0, 2, 1, 2, 1, 2, 1, 2, 1, 2, 0)

## [1] 2

Organisational numbers

Not only individual has their personal identification number, so do companies and NGO:s. These features are covered by the oin group of functions in the package. Feel free to try them out …

Other countries

An analogous conversion function is availale for the Finnish social security numbers in the sorvi package.

Keep in touch!

… and feel free to suggest enhancements and report bugs to https://github.com/rOpenGov/sweidnumbr/issues

Digital humanities with R

Fri, 12 Jun 2015 00:00:00 +0000

Digital humanities is one of the fields where reproducible research is now becoming increasingly popular. We got an opportunity to highlight the latest advancement in this field in Paris Digital Humanities event, June 12 at Deutsches Historische Institut Paris.

The ‘digital humanities lab’ is an interactive experiment where the audience actively participates in designing the analysis of massive library catalogues. In particular, we focused on publishing activity in the early modern period by mining the British Library ESTC catalogue 1470-1800. To facilitate this interactive session, we prepared reproducible presentation slides with Rmarkdown. To be completed in the workshop, carrying out data analysis on-the-fly together with the audience. To reproduce the preliminary slides (we will complete them during the workshop!), clone the rOpenGov slide repository and run the following commands in R:

library(rmarkdown)
render("slides/20150612-Paris/20150611-Paris.Rmd")

Unfortunately the ESTC data itself is not public, so we could not share it. The slides and source code are fully reproducible, however, so you can modify the template for your own purposes and check out our summaries of the ESTC data collection at our estc site. You may also like to read the related excellent blog post of Douglas Duhaime on the same data set.

We used a combination of RStudio and the estc and bibliographica R packages that are designed for bibliographic data analysis. Combined with the vast analytical capabilities of the R statistical ecosystem, these custom tools for digital humanities provide a rapid development toolkit for reproducible research of historical document collections.

A hierarchical model of Finnish apartment prices

Thu, 11 Jun 2015 00:00:00 +0000

Probabilistic programming approach for regional trends in apartment prices

Basing on open data from Statistics Finland, we at Reaktor modelled Finnish apartment prices and their trends on zip-code level, in the years 2005–2014. Estimates from the model are available as an interactive visualization.

Why do (statistical) modeling?

The original price data consists of local (geometric) mean sales prices per year. The number of sales is available as well. If there are less than six sales, the mean price is censored.

Partly missing data and noise from low number of transactions make it hard to evaluate local price levels, let alone their changes, except on the most urban areas.

Yearly numbers of transactions for a few random zip codes are depicted on the left below. Censored slots are with red. On the right, all year-zip slots are ordered on the x-axis by their available number of sales data. 17.5% of slots are censored, and about half of the mean prices are either missing or based on less than 30 transactions.

Mean price of 6–30 sales is not a reliable estimate of the local mean, and deriving trends from so few sales is not going to be successful. (Still, it is repeatedly tried: There have been several top and bottom lists of apartment prices and their development published in the Finnish media lately. The media estimates are based on this raw data.)

A statistical model has a concept of a price level behind individual sales. It can then make a distinction between systematic variation of the underlying price level over time and place, and random variation that is not explainable within the model. This is in contrast to looking at raw data without a model; then all variation is taken at face value.

When the model is estimated, it produces the underlying price level as its output. Of course, because the model cannot explain all variation in data, the price level estimates will also have a random component: Instead of a fixed value, we get a probability distribution. Means, trends, confidence intervals, etc., can be computed from these posterior distributions. Provided the model is sensible, these estimates of underlying price trends are more informative than the raw data. Uncertainty of the estimates also reveals when data is not enough to draw any conclusions.

Some properties of zip code areas, like population density, will correlate strongly with apartment prices. Such properties, if known and included in the model, allow us to generalize over zip codes: estimates of price level become available even on places where the data is sparse or there is no data at all. Of course, uncertainty will then be higher, and the model will tell us that.

Adjacency of the areas, or their hierarchy, can be used in the similar way to allow generalization between close-by areas. Once topography or hierarchy is parameterized to the model, it can see correlations between areas that are close to each other, or within the same larger area in the hierarchy. It will then generalize over areas, which helps especially when an area has few observations by itself.

Note that as with demogrpahic covariates, no strict assumptions about geographic dependency are coded into the model. Rather, including these auxiliary parts allows the model to use dependency where it exists.

Below, the map on the left shows raw mean prices over the whole period 2005–2014. White areas are without any available data. Map on the right shows the (mean) price level estimated from a model.

Below, yearly mean prices and estimates of the underlying price level are depicted for some zip codes at Espoo, part of the capital area of Finland. Shading around the lines indicate uncertainty of the estimates. Even within this relatively urban region, estimates from some areas are quite noisy: 02150 or Otaniemi, 02240 or Friisilä, 02330 or Kattilalaakso, etc. Some areas have no sales at all. (But they may not have apartments either. The model does not know whether apartments exist.)

The model can be used for forecasting, but future prices or trends will have large uncertainty, even larger than indicated by the model. The current model has quadratic shape for the temporal dependency. It was chosen to fit the data of the last decade and to give an idea of past price development that is easy to summarise. There is no reason why future changes in economy and policy would follow the same pattern. Althought relative development of areas is more accurately predicted than absolute price levels or trends, the model is at its best at describing past development of apartment prices, especially their spatial differences. There is no guarantee future will follow the same pattern.

The model, data, and environment are described in more detail below.

Environment and data

We used R for almost all data manipulation, modeling and visualizations. Model itself was estimated with Stan. Source code for the project, except for the web site, is available in our GitHub repo.

The libraries pxweb and gisfin make it easy to get apartment prices and other data from the public API’s. The libraries are developed in the Louhos and Ropengov projects. Zip code areas are from Duukkis. Scripts used for downloading data and manipulating it are available in our repo.

Map

Zip code polygons are available at least through the Paavo API and from Duukkis. Finland’s archipelago is extensive and complex, so zip codes extend well onto the Baltic sea. Paavo offers two versions of the polygons, with or without the sea area. The zip code areas with sea can be intersected with the sea shore line, giving us a quite beautiful Finnish zip code map. The problem is its size: over 20MB in the GeoJSON format. We ended up using the polygons from Duukkis: They are a good compromise between size and accuracy.

The set of zip codes varied a little bit from source to source, so some zip codes areas may be missing from the visualizations. A few small areas have no population, or the information about population is missing. These polygons are without a price estimate and appear as grey on the maps.

Note that the model produces price and trend estimates even for areas with no apartments: just the relative position of the zip code and the local population density are enough for computing the estimate.

Model

Of the past sales, the model has yearly (geometric) average per location, and the associated number of sales, if these are not censored ($n<6$). The latter scales the variance of the mean as an estimator of the population mean. Population mean is here the hypothetical mean of all potential apartment sales on the areas. Of course all apartments are not sold at the same rate, so mean is biased towards prices of the apartments that are sold more often.

Sparseness of the data is a problem especially for estimates of temporal price changes, and also for comparison of areas. Predictive covariates for the zip code areas are therefore valuable. A quite extensive set of demographic variables is available in the Paavo data, but of these the model so far has only the population density included. It is probably the most predictive of the covariates, although not necessarily causal from the economics point of view.

Spatial structure is included as a zip code prefix hierarchy. For example 02940 is within the Uusima district (0), city of Espoo (02), and northern Espoo (029). The hierarchy allows the model to see similarity within these and other equivalent nested areas. Real spatial continuity in the form of a Markov field or a latent gaussian field would be an alternative, but it would be much harder to estimate with the chosen tools, and may not be better on modelling administrative areas that are nested, after all.

Temporal change of prices is as interesting as their overall level. The model could have a separate price level for each year, but continuity over time would then be lost, and there would be no predictions. Also the relationships between price trends and covariates (population density) and between trends and spatial or hierarchical structure would be hard to define, for there would be no unique trend. These reasons and simplicity favor a simple temporal parameterization, as the current quadratic model. Combining hierarchy and covariates with a more flexible temporal model, like a gaussian process, is an interesting research question.

In total, there are three parameters on the zip code level affecting log-scale prices: price level, its trend, and change of trend. On the next geographic hierarchy level three other parameters appear: the influences of (logarithmic) population density on price, trend and its change. These six parameters, three plus their interactions with population density, appear also on upper hierarchy levels. On each hierarchy level, the model has multinormal priors for the three or six parameters, and hyperpriors for the variance and covariance of the multinormal distribution. The covariances bind different parameters together, so that for example price level helps the estimation of price trend, or the influence of population density.

In summary, the lowest level of the model for the log prices is

$$ \log h_{it} = \beta_{i1} + \beta_{i2} t + \beta_{i3} t^2 + \beta_{i’4} d_i + \beta_{i’5} d_i t + \beta_{i’6} d_i t^2 , $$

$$ \log y_{it} \sim \textrm{t}\ \left(\log h_{it}, \sqrt{\sigma^2_y + \frac{\sigma^2_w}{n_{it}}}, \nu\right) , $$

where $i$ refers to the zip code area, $t$ is time, $\beta$ are coefficients specific to the zip code $i$, $i’$ is the first prefix hierarchy level of the zip code (population density parameters are constant within each $i’$-area), $t()$ is the t-distribution, $\sigma_y$ is standard deviation of the underlying (log) price levels over years, $\sigma_w$ standar deviation of the prices within the measurement unit (year $\times$ zip), and $\nu$ the degrees of freedom of the residual t-distribution. Note that the linear model is for log-scale prices. The complete model is best described by the source code. Estimate for $\nu$ is around 6.5, that is, residuals are with a bit heavier tails than normal. From the covariance parameters (Omega in the source) one sees that price level and trend correlate at the lowest level ($r$=0,28), as do trend change and price level ($r$=0,43). So price differences between areas have been growing during the last ten years, probably due to urbanisation, a global trend.

Plotting area-wise prices and its changes against population density, one sees the expected correlation: remote areas are loosing in the sense of price, trend and trend change.

The model has been written and estimated with the probabilistic programming language Stan (http://mc-stan.org/). Stan produces a Monte Carlo estimation algorithm from a generative model description.

Possible improvements

The model could have more demographic covariates. Dealers have reminded us about the predictivity of the sales volume. Obviously, the number of sales is not in a predictive role in the current model.

Apartment vary by their size, age, etc., but this heterogeneity is not taken into account. Trends are biased towards apartments that are sold more often. Some public data is available where sales are separated by size and age of the condos, but these data are even more sparse than the aggregate data used here.

On the Finnish equivalent of this blog, Herra Huu suggested whitened parameterization, which may help with estimation.

Reproducible posters with R, LaTeX, tikz and Sweave

Sun, 07 Jun 2015 00:00:00 +0000

rOpenGov is all about reproducible research, so preparing a reproducible poster became our mission after having the chance to present at International Conference on Computational Social Science (ICCSS 2015) in Helsinki, June 8-11, 2015 (poster 36 in Monday session 15:30-17:00).

We blended R code and output with the standard poster contents. The poster sources download data with our eurostat R package and automatically generates the final figures and the overall layout.

Whereas several ready-made layouts, such as baposter, beamerposter, tikzposter, latextemplates.com and other options were available and could be useful for fast poster design, they also limit the available options as the graphical elements are laid out as tightly defined text boxes. This is not suitable for all purposes, and mixing ready-made styles with free design is potentially confusing. I also bumped into some problems in incorporating R code with some of these templates.

Therefore we ended up using the plain LaTeX/tikz combination which allows reproducible design of arbitrary poster layouts and schematic figures, as well as automated numbering of figures and references. The R/Sweave allows incorporation of R code and output (figures, tables, text). The a0poster style provided appropriate font sizes and other LaTeX utilities for posters. If you are a frequent LaTeX user, we warmly recommend familiarizing with tikz. For further details, see the poster sources.

To reproduce the A0 poster (PDF), clone the rOpenGov poster repository and run the following commands in R:

library(knitr)
knit2pdf("poster/2015-ICCSS/poster.Rnw")

Finished poster

Finnish Meteorological Institute open data added to rOpenGov

Tue, 30 Sep 2014 00:00:00 +0000

An R package fmi for the Finnish Meteorological Institute open data API has been released. The package provides an access from R to many data sets including weather conditions, climate scenarios, sea related observations and sun radiation in Finland. Installation instructions, examples and other details are provided in the tutorial. The package is in beta and test experiences and contributions are welcome! For the contact details, see the package home page. The package has been jointly developed with rwfs package, which provides a generic access to WFS interfaces in R.

Note Oct 10, 2020: the package is now fmi2

Berlin Open Knowledge Festival July 15-17, 2014

Fri, 18 Jul 2014 00:00:00 +0000

rOpenGov developers met at the Open Knowledge Festival in Berlin, July 15-17. We discussed the development of representation formats for social/political science data sets. The psData package for panel series data is now under active development, and further plans include similar structures for election and other data sources. Standardized representation formats will allow the development and application of analysis algorithms beyond particular applications. Robert Gentleman from Bioconductor has said: ‘If everybody puts their - - data into the same kind of box, it doesn’t matter how the data came about, but that box is the same and can be used by analytic tools. Really, I think it’s data structures that drive interoperability.’

Overall, the OKFest 2014 was a great success with an interesting program and good athmosphere. Thanks for all organizers and participants!

NIPS/MLOSS workshop on Machine Learning Open Source Software

Tue, 10 Dec 2013 00:00:00 +0000

The rOpenGov project was today highlighted at the Neural Information Processing Systems (NIPS) conference, one of the main forums for machine learning and scientific computation. We gave a presentation at the Machine Learning Open Source Software (MLOSS) workshop.

A number of other interesting projects were present, including iPython, Factorie, scikit-learn, and many others. For a full list, see the MLOSS site.

While the rOpenGov ecosystem is based on R, we actively seek connections with related initiatives in other languages, such as Python or Julia. The iPython is particularly interesting in this regard with its fluent tools to mix code from various languages with automated document generation tools that support LaTeX and markdown. An added R support for this project would extend the applicability of rOpenGov toolkits beyond the R community. Looking forward!

R Ecosystem for Open Government Data and Computational Social Science

Mon, 09 Dec 2013 00:00:00 +0000

Hello world !

This is to announce the rOpenGov project and encourage new contributions to promote community-driven development of R tools for open government data and computational social science.

A community of independent package developers around open goverment data analytics is now emerging at rOpenGov. Independent projects dedicated to US, Russia, Finland, Poland, Austria, and OpenStreetMap have already joined in; more are coming and we will keep you posted through this blog.

The rapidly emerging governmental and other open data streams provide novel opportunities for social sciences, data journalism, and citizen participation across the globe while computational tools to utilize these resources are lacking. A community-driven software ecosystem provides a scalable solution and a potential to revolutionize the field, taking advantage of the lessons learned in Bioconductor, rOpenSci, and related projects. The umbrella site focusing on open government data R tools will give added visibility and recognition for independent package developers, an opportunity to attract contributors, and a forum for exchanging ideas and information.

The project is in beta, and more information will be added soon. You can follow us on rOpenGov blog, Twitter, Google+, and IRC (ropengov@Freenode). If you are working on related projects, don’t hesitate to get in touch!

Community

Mon, 01 Jan 0001 00:00:00 +0000

Developers

The network is coordinated Leo Lahti, Pyry Kantanen and Markus Kainu. It all started from the Finnish Louhos blog back in 2010.

We are grateful to all contributors! For a full list, see R-Universe and the individual projects pages in github.

The rOpenGov blog is part of R bloggers.

Contact

You can follow us in Twitter, reach us in Gitter or via email: ropengov - at - googlegroups.com

In administrative matters, contact the coordinators or other developers as relevant.

How to contribute?

rOpenGov is a community of independent R package developers. If you are working on related projects, don’t hesitate to get in touch!

Adding new packages to rOpenGov:

Open a new repository or transfer existing repository under the rOpenGov Github-organization (contact the coordinators to obtain necessary permissions).
Add the package under R-Universe

Contribute to the existing rOpenGov packages:

Open Issue or Pull Request to the package of your choice in GitHubissa

Possible ways to contribute:

Use the R-packages in your projects.
Contribute to development. We will gladly accept code contributions, bug reports or other suggestions for improvements, or new R packages that fit the scope. We are also welcoming tutorials and guidelines.
Write a blog post in your own blog or send your writing to gain visibility through the rOpenGov blog.
Provide feedback on packages, blog posts, website, or other issues.

Projects

Mon, 01 Jan 0001 00:00:00 +0000

The rOpenGov R package development versions are maintained in GitHub. For contributions, check out our Community-page to get in touch or simply just make a pull request in GitHub.

Package	Description	Homepage	Forks	Issues		Updated